Towards Generative AI for Course Content Production: Expert Reflections

AI’s influence is growing globally, with policy strategies often overlooking its educational implications. Various studies highlight AI’s integration in schools and its potential to enhance learning through analytics and intelligent tools (Baker & Inventado, 2014; Chen et al., 2023; Fischer et al., 2020; Niemi, 2021). One of the promising applications of generative AI in education is its potential to assist educators in generating course materials. By synthesising and rephrasing existing content, generative AI can save time and effort for educators, allowing them to focus on more complex aspects of course design and pedagogy (Alshater, 2022; Terwiesch, 2023). Thus, governments invest in developing AI tools to benefit from this technology in content production. For example, the UK government invested up to £2 million in AI tools for Oak National Academy.¹ This online learning platform offers a suite of practical AI tools designed for and freely available to teachers, which will help develop new, free resources powered by AI for teachers, such as lesson planners and classroom quizzes.

There is increasing support for utilising generative AI in content production, ranging from creating instructional content to providing automated assessment feedback and supporting academic services (Pelletier et al., 2022). As stated in Bektik et al.’s (2024) literature review, generative AI can aid the content production process by creating prompts for open-ended questions that align with the learning goals; generating quality rubrics that clearly and concisely explain exactly what students need to accomplish; creating prompts for formative assessment activities that provide ongoing feedback to inform teaching and learning; helping in creating lesson plans for specific courses, developing customised resources and learning activities; carrying out assessment and evaluation and supporting the writing process of research; suggesting ideas for graphical content; creating ways to make content more engaging with interactive exercises and simulations; creating analogies to turn abstract concepts into concrete ones; generating reflective questions based on course readings or video scripts; generating student learning tasks and feedback based on existing course content; generating engaging video scripts based on existing written content, including suggesting on-screen graphics; and generating scenarios or case studies based on existing content (Herft, 2023; Liu et al., 2023; McInnes, 2023; Michel-Villarreal et al., 2023; Rahman & Watanobe, 2023). We have reviewed practical examples currently employed for such tasks and further examined examples of existing practices, which are discussed next.

Various companies and platforms have developed AI-powered tools since ChatGPT was launched in late 2022 to aid in curriculum generation, enhancing the course creation process for educators. We have reviewed these tools and their functionalities, including examining each tool’s features, capabilities and specific contributions to educational content creation and course design. Table 1 summarises some notable developments in this field. The table encapsulates the functionalities and distinctive aspects of each AI tool or platform, illustrating the diverse ways AI is being integrated into educational technology to support teaching and learning processes.

These developments suggest that course designers could potentially create highly engaging course content simply by providing a generative AI engine with a few sophisticated text prompts that are aligned with student learning outcomes. For many course developers and subject matter experts, course creation is among the most time-consuming tasks they undertake. Overall, AI-based content-generation tools can speed up the course development process. However, since these AI-powered tools listed above are not open source (apart from EduWeaver), their operational mechanisms remain unclear, as does whether this speed-up process sacrifices the content quality. Consequently, there is a need to evaluate the capabilities of generative AI in producing course content, which is the focus of this paper.

To obtain a list of tasks associated with curriculum development, we reviewed available internal University documentation for the development of the curriculum. This included details of the business planning process, the decision-making process for curriculum development, academic governance, advice and guidance on various elements, including assessment, learning outcomes and review.

We found that curriculum development first appears within the business planning process. However, this is perhaps better viewed as a management rather than a curriculum development activity. Therefore, we have included a range of core activities that follow from the moment someone first begins to describe the actual aims and content of a course. Many will be familiar with writing educational aims and learning outcomes aligned to benchmarked statements and other standards. Most will also have considered course outlines and assessment strategies. Some people will be familiar with the use of personas to help focus on the needs of the intended learners. In some institutions where materials are intended for large numbers of students or for distance learning, there may be a house style and the need for different levels of editing. This editing might also include checking and standardising XML tags to ensure they are ready for a virtual learning environment (VLE). All these and more are set out in Figure 1. The details in the figure are not intended to be exhaustive but to indicate the range of activities that we might expect to see in curriculum development.

Figure 1.

The potential for generative AI to support common curriculum tasks. (The inner circle is an alphabetical list of common curriculum tasks and the outer ring contains an example of how each might be supported by generative AI).

Figure 1 shows several of the main tasks associated with developing a curriculum and how generative AI might contribute to completing them. The list of tasks in the inner ring is a collation of items taken from our institutional references relating to curriculum development. The outer, italicised ring gives an example of how we might expect generative AI to support the completion of the task for each item. The items are arranged in alphabetical order. The cycle represents the iterative nature of the process of developing a curriculum.

Each of the tasks outlined above will result in tangible outcomes, and course authors, students and other stakeholders will have expectations regarding the quality associated with the produced materials. These expectations are often only partly made explicit in training materials and guidance for course authors but more often are part of the implicit knowledge of the course production team, who, over the years, have learned how to write high-quality course materials. The importance of this implicit knowledge is further amplified by research that indicates that, as of now, it is not clear how course quality is best conceptualised (Schiering et al., 2023). In this research, we rely on the expert knowledge of staff to judge the quality of the generated content. We included multiple experts to cover a wide range of perspectives on course quality. A further reason to work with experts is that there are currently no standards or benchmarks that allow us to judge the quality of the content for the specific tasks outlined above. The research on the various dimensions that potentially influence course quality (for example, see Lowenthal & Hodges, 2015; MacDonald & Thompson, 2005) looks at the quality of a course as a whole but not at quality dimensions for the individual tasks of course production, which was required for this research.

In summary, the literature review showed that educational researchers and education companies see the potential of using generative AI to support curriculum production tasks and our curriculum development tasks mapping showed that many tasks could potentially benefit from generative AI. The literature review also revealed that the research around generative AI and course content production is currently in an exploratory phase and more research is needed to understand the quality of the rapidly generated course content. In this context, this qualitative research aims to assess whether generative AI can support teachers in creating course content. This research is novel, timely and of high importance as the introduction of generative AI in the curriculum production process could significantly impact the education sector. Our research question is:

One of the core tasks of a course production team is preparing various course outlines. These outline the structure and content of the course. The actual writing of course content often follows this initial step, drafting content and refining it iteratively for each of the outline’s points.

We tasked the AI with producing an outline of content suggestions based on a subject-specific big idea and draft content for one of the suggestions. The course team provides the central theme and the AI generates a course outline based on that idea. Ullmann et al. (2024) presented a prompt that suggested big ideas for a specific course topic. These big ideas are meant to hook students and create interest in the topic. The prompt presented there can inform this prompt by using one of the generated big ideas.

We developed the following prompting sequence that consisted of two prompts to create a course outline and a follow-up prompt, which created course content based on a pre-defined structure for one of the outline’s points. The first prompt set up the interaction and asked the user to specify a big idea for the course content:

‘You are assisting an apprenticeship teacher in creating course content for a single week of an online apprenticeship course.

The AI composed a completion consisting of three major sections. The first section asked the user to provide the week’s big idea or central theme. In the second section, the AI showed an exemplary course outline for 7 days of the week, starting with an introduction, several content sections and, towards the end of the week, a group activity and a reflection and question-and-answer section. In the third section, the AI drew up a section exemplifying how it would ask the user for feedback.

The follow-up prompt provided the big idea for the week: ‘The big idea that my students should work on this week is Ethics and bias in data’. This idea was one of the big ideas generated by another prompt reported by Ullmann et al. (2024).

The completion followed the structure that the AI proposed in the previous completion. It restated the big idea: ‘What are the ethical considerations and potential biases in data collection, analysis and usage, and how can we address them?’ This followed seven sections for each day of the week. For example, it proposed for Day one the following:

‘Day 1: Introduction to Ethics and Bias in Data

Similarly, it proposed for the other days of the week. Day two was about ethical data collection, day three was about the identification and mitigation of bias, day four was about ethical decision-making, day five was about data governance and policy, day six proposed a workshop on ethical problem-solving and day seven was about reflecting on the week and a question-and- answer session.

The final prompt asked the AI to produce course content about one of the bullet points of the previously generated outline. The prompt also specified a special structure asking to include a surprising example, sections about explaining the topic and a section aimed at aiding the learners to reflect on the topic:

‘Please create course content about the topic: Defining data ethics and understanding its importance. Start with a surprising example demonstrating the importance of the topic. Then, explain the topic in three paragraphs. Then, summarise the content in one paragraph. Finally, help the student to critically reflect on the topic by creating three reflective questions’.

The AI created a completion that consisted of three requested parts. As for the example, it proposed the following:

‘Imagine a scenario where a seemingly harmless fitness tracking app collects data on users’ physical activities, locations and health metrics. This data is then sold to insurance companies without users’ explicit consent, leading to increased premiums for those identified as high-risk based on the collected data. The users are unaware of this breach of trust and the ethical implications it carries, highlighting the critical need for ethical standards in data handling’.

It then explained in three paragraphs what data ethics is about, its importance and the consequences of ignoring data ethics. This is followed by a summary and three reflective questions.

As outlined in the methods section, the prompts and completions were sent to five experts who commented on them individually. Afterwards, the experts met to discuss their comments in a structured one-hour-long meeting.

Regarding outlining generating prompts 1 and 2, the experts found that the AI’s proposed outline can help the course team draft a first outline, which they then need to adapt further and contextualise considering the whole course. They also thought that the outline might help to provide another perspective on the course outline, which may help adapt an existing outline. The experts found an implicit pedagogical structure embedded in the response of the AI, which can help the learners understand the materials. The proposed sequence of contents started with an introduction to the topic. This was followed by several content-specific sections, which also build from the more concrete issues, such as the ethics of data collection, to more abstract topics, such as data governance. After the content-laden sections, a group and reflective exercise were suggested to be useful in strengthening the learning of the contents. This structure can help to guide the learner, starting from introductory concepts and moving them to more advanced concepts and strengthens the learning with activities and exercises at the end of the course block.

Regarding content-generating prompt three, the experts found that the generated text provided a first draft of texts which then needed further adaption and change. The generated example, the three content paragraphs and the summary all contain valid points to make, but as they stand, would need further development. This is tied to another observation, the created text lacks citations. This is likely a limitation of the prompt compared to that of generative AI, as these systems can produce text, including citations. Furthermore, the generated text would need to be checked for inaccuracies. For example, the AI proposed the following sentence when defining data ethics (example was shortened): ‘Data ethics is a branch of ethics (…). It encompasses a wide array of ethical issues, including privacy, accuracy and consent, particularly (…)’. Here, it was not clear why accuracy was mentioned along with privacy and consent. Instead of accuracy, a better point to mention could have been bias and fairness.

The next investigated task was the production of instructions for an assessment.

Writing instructions for assessments is a core part of the course production team. Students benefit from clear instructions. Here, we envisioned the AI producing draft instructions for an assessment. The course production team specifies the topic of the assessment and reviews and edits the outputs generated by the AI.

We used two prompts. The first prompt set up the interaction with the course production team. It asked the course writing team to provide information about the course and assessment topic. It also specified that the AI should generate assessment instructions. The final prompt that we created was:

‘You are assisting an apprenticeship teacher in creating instructions and grading criteria for an assessment in a distance learning context.

This prompt combined several steps into one. The idea was to let the AI generate a sequence of responses, which would guide the user through the steps.

The AI created a detailed completion of the prompt. It started by outlining the purpose of the interaction: ‘I am looking forward to assisting you with the creation of the assessment instructions and grading criteria for your course’. It then detailed information that the user was supposed to provide: ‘What is the subject matter of the course?’ and ‘Could you give a detailed description of the topic that the assessment will cover?’. In addition to the information that was explicitly specified in the prompt, the AI added five additional user input requests, such as ‘What are the learning objectives that this assessment aims to evaluate?’, ‘Are there any specific guidelines or requirements that the students must follow (e.g., word count, citation style, presentation length)?’ and others. The experts found that the additional information requested by the AI would have been useful for specifying the assessment guidance. Still, they also noted that it would have substantially increased the amount of time to retrieve and prepare the requested information.

The AI has also already drafted an example of what the assessment instructions could potentially look like. It starts with a greeting message to the students: ‘Dear Students, As part of your [Course Name], you are required to complete an assessment that will evaluate your understanding of [Assessment Topic]’. This is followed by a template proposing six sections for the assignment instructions, including sections such as ‘Assignment Type: [e.g., Research Essay/Project/Presentation]’, ‘Topic Description: [Provide a detailed description of the topic and what aspects need to be covered]’, submission deadline, submission method and others.

As a response to the completion, we chose the following second prompt providing a course topic, subject of the assignment and learning outcomes:

‘The course topic is about Ethics and Bias in data. The subject of the assignment is Ethical considerations when collecting data. The learning outcomes are (a) the development and presentation of justification, rationale and argument; (b) your ability to communicate complex ideas in an appropriate and focussed manner; and (c) your ability to select and critically appraise appropriate resources on a given topic’.

The AI created a completion that followed the template AI had proposed earlier, consisting of an introduction, assignment type, topic description, learning objectives, requirements, submission deadline and submission method. The AI chose an essay as an assignment type. The generated introduction introduces the chosen topic with the following words:

‘Dear Students,

After this introduction, it offers a short description of the topic:

‘You are to write an essay that discusses the ethical considerations involved in the collection of data. Your essay should cover key ethical principles, potential biases that may arise in data collection and strategies to mitigate these issues. You should also include real-world examples to support your analysis’.

This is followed by the three learning objectives outlined in the prompt, some requirements such as word length, a section to add information about the deadline and submission method.

Overall, the experts noted that the generated text presented a good start for drafting the assignment instructions. The assignment creators would need to think about adding more details and depth. A different prompt might potentially open up more ideas for assessments, for example, by asking it to provide multiple options for the module team to consider.

The experts commented on the additional information that the AI asked the user to add, which was not explicitly mentioned in the prompt. They were all seen as good suggestions, such as learning objectives or guidelines. However, it was also noted that providing all the AI-requested information also adds to the user’s workload.

The experts noted that the AI also created an exemplary template for the prompt response without explicitly specifying this in the prompts. The experts found that it could be useful for the course writer to see such templates to structure the assessment instructions. This can serve as the starting point for creating a template that fits the requirements of the particular assessment instructions, making use of one of the capabilities of generative AI, which is to write content according to a template structure.

One of the challenges for anyone creating a curriculum within higher education is to ensure alignment with external benchmarks. An aspect of this challenge is creating appropriate learning outcomes and objectives that both fit with external requirements and the internal criteria set down by the institution. Within UK higher education, one available qualification is the Level 7 Research Scientist Apprenticeship (https://www.instituteforapprenticeships.org/apprenticeship-standards/research-scientist-v1-0). This is a master’s level degree. Within the specification is a set of 23 statements of intended learning objectives, often referred to as knowledge, skills, and behaviors (KSBs) because they are arranged within three categories: knowledge, skills and behaviours. If a university creates a programme to support this degree, it has to demonstrate that it delivers these KSBs. This creates an immediate problem for universities which have developed an internal model specifying a different number of learning outcomes or categories. Like many universities, ours keeps numbers relatively low, aiming for around 12 and has four categories: knowledge and understanding (KU), cognitive skills (CS), key skills (KS) practical and professional skills (PS).

Given this context, we tasked the generative AI with aligning the learning outcome categories of the university with the 23 externally specified apprenticeship learning outcomes. We envisioned that the output of this process could be reviewed by the curriculum team and used to identify the degree of alignment between internal and external learning outcomes. We developed a series of prompts towards the solution. We chose a conversational approach in which the activity was broken into steps in which each step could be checked and revised before progressing to the next. The first step was to provide the generative AI with the 23 KSB statements. Hence, the initial prompt was:

‘The following is a list of KSBs, these are descriptors I would like to ask you to perform some actions on. Each KSB comprises a paragraph of text. It begins with an identifier in the form’, ‘Kx:’, ‘Sx:’, or ‘Bx:’. Where ‘x’ is an integer. The descriptor follows the colon. The list is: ‘The generative AI responded with’:

The prompt included a list of all KSBs copied from the apprenticeship standards website. For brevity, the whole list is not shown here.

The generative AI responded:

We asked the generative AI:

In our institution, we use four categories of learning outcomes: KU; CS); KS; practical and PS. Please produce the most suitable statements that could be used within these four categories to cover all the concepts, skills and ideas set out in the text of the KSBs. Please use the abbreviations of the learning outcome categories, minimise the number of statements and write in the form ‘At the end of your study you will…’ but omitting these words.

The generative AI responded:

‘Certainly! I’ll create concise statements for each of the four learning outcome categories based on the concepts, skills and ideas set out in the KSBs.

In the remainder of the response, generative AI outlined 11 more learning outcomes, resulting in 3 within each category (KU1, KU2, KU3, CS1, CS2, etc.).

Our final prompt was:

‘Produce a table with the KSBs as the first column, these 12 learning outcome statements (identified by their codes) as the header row and a 1 in each cell to identify where each KSB links to each of the learning outcomes’.

Generative AI produced the following table (we added the highlighting to rows with no matches):

The experts found that generative AI can support developing learning outcomes in the required internal format and style and in alignment with a defined set of external requirements.

They also found that the tool readily produces a tabular representation of a mapping between two lists of statements. They noted that not all the KSB items were found to match. We will consider this below.

The original list of KSBs had three categories and generative AI successfully arranged the generated learning outcomes into the four internal categories as requested. The allocations made are also appropriate, with each learning outcome fitting well into the allocated category and with no other category offering a better home. These learning outcomes are written to an appropriate level, matching that of the KSBs and with none looking out of place.

Considering the quality of the mapping, the experts found that generative AI successfully produced a table in the way requested. To compare the automatically generated mapping with an expert mapping, one of the experts created a mapping by hand. On comparing both mappings, the expert found that all the KSBs were covered by the internal learning outcomes, although the text of four of the learning outcomes could be improved with small changes. Those that could be improved include the three cells where generative AI did not identify a link. Overall, the manual mapping agreed with the large language model (LLM) on 86% of the cells for the three KSBs. A small number, four, of the links indicated by the LLM were completely incorrect.

Overall, the experts found that generative AI can provide a strong starting point for developing a set of compliant learning outcomes. Through very simple prompts, the AI generated a workable list of learning outcomes in minutes rather than the morning or afternoon it might otherwise be expected to take. This provided a good foundation text for further development. The mapping produced by the LLM was also done in a couple of minutes and is helpful as a point of comparison for one’s own expert mapping. The points of disagreement provide opportunities to consider the rationale of our choices and would ultimately lead to improvements in the wording.