Analysis of influencing factors of SPOC course teaching effect using structural equation modelling

M. Scriven, in 1967, distinguished between ‘summative evaluation’ and ‘formative evaluation’ for the first time in his book ‘The Methodology of Evaluation’. The concept of summative and formative evaluation was first introduced in. The educator B. S. Bloom introduced this assessment into the field of teaching and learning. He divided it into diagnostic, formative and summative evaluation, which Bloom saw as an integral part of teaching and learning [10]. Based upon their extensive research on teaching practice, Black and William found that objective and comprehensive classroom formative evaluation can significantly improve students’ academic performance in the short term. This is especially true for students who are otherwise underachieving [11]. Scholars have largely focussed on the application of big data technology in the field of educational assessment in terms of evaluating online teaching effectiveness. For example, Mou [12] constructed a personalised learning evaluation model based on electronic schoolbags. Based on big data analytics, Zhen-Chao et al. [13] developed a digital learning developmental evaluation system. The traditional flipped classroom teaching model was refined by Xiao et al. [14] into 10 sessions, and ‘learning assessment’ was used throughout the entire process. Concerning teaching models, Tan and Gu [15] examined the composition of learning effectiveness assessment indices for the hybrid model of open education. According to Bai [16], ‘formative assessment refers to the evaluation of students’ learning progress and refers to the evaluation of teaching and learning in the teaching process’. Formative assessment focuses on the improvement of the learning process, emphasises the timely feedback of assessment information and aims to improve students’ learning effectiveness and teachers’ teaching through regular assessment. As a result of their research, Luo and Hu [17] concluded that combining formative and summative assessment and moderately increasing the importance of formative assessment in the assessment system helps learners to pay attention to their own interests, emotions and skills when learning English.

From the above analysis, it is evident that the current scope of research on learning assessment methods is relatively extensive. Scholars have proposed learning behaviour analysis and prediction models based on big data, but they have not assessed the learning effects of the blended teaching model, particularly the SPOC model, and empirical research on learning assessment methods for online and offline blended teaching models is still insufficient. Xiao et al. [14], for instance, proposed a multifaceted evaluation system that combined pre-class and in-class assessments, but further application and evaluation are required for the flipped classroom teaching effect. Tan and Gu [15] proposed the effect assessment index of the blended teaching model but only validated this set of effect assessment index through a questionnaire survey.

The objective of this paper is to reconstruct a more ideal learning effectiveness evaluation model for evaluating students’ learning abilities and application skills through the evaluation of three exogenous latent variables using structural equation modelling: the teacher's teaching method, students’ learning ability and the classroom environment.

Before constructing the SPOC evaluation model, a study of the SPOC teaching mode has been conducted using the course ‘Aviation Broadcasting’ from the Zhengzhou Institute of Aviation Industry Management in China as an example. The nature, teaching format, teaching objectives and methodologies, as well as the teaching process of SPOC, were examined, as well as factors affecting the evaluation of SPOC with full-time undergraduate university students in the classes of 2019, 2020 and 2021. ‘Aviation Broadcasting’ is a course oriented to cultivate students’ comprehensive ability of aviation service and management. Through the study of this course, students can adapt to the context and control the scene according to the cabin environment, maintain cabin safety and improve cabin service quality. As a result of sorting out the factors affecting course evaluation from the available data, three dimensions affecting SPOC evaluation, namely, student learning ability ζ 1, teacher design ζ 2 and classroom learning atmosphere ζ 3, were selected as latent variables to construct a structural equation model (SEM) for SPOC evaluation (Figure 1).

Fig. 1

The following hypothesis is also proposed.

A SEM may replace regression analysis, analysis of variance, correspondence analysis, factor analysis and cluster analysis to clarify the roles of individual indicators on the total and their interactions with one another [18]. In structural equation modelling, variables that cannot be measured directly and accurately are called latent variables and are measured indirectly through external indicators [19]. Using the research in the education sector and the operability and orientation of evaluation indicators, this study selects 19 observed variables to form a model and forms an index system of SPOC course teaching effectiveness influences, as shown in Table 1.

As described in Table 1, the observed variables are the measurement errors or residuals, which are represented by the residual term e. Based on the hypothesis proposed above, AMOS software was used to plot the SEM of the factors influencing the teaching effectiveness of SPOCs, as shown in Figure 2.

Fig. 2

Fig. 3

This study refers to the teaching questionnaire of the relevant SPOC and develops a questionnaire in conjunction with the relevant situation of the course ‘Aviation Broadcasting’ at the Zhengzhou Institute of Aviation Industry Management, China, and distributes it via an online survey during the course cycle. In the first part of the questionnaire, the students are asked basic information, such as their age, gender and grade level; the second part is the main part of the questionnaire, and it covers three aspects, including students’ learning ability, teachers’ teaching designs and the classroom learning environment. The questions in the questionnaire were based on a 10-point Likert scale [20] where 1 represents ‘very low’ or ‘very poor’ and 10 represents ‘very high’ or ‘very good’ and students were asked to respond according to their thoughts. Students of the class of 2018, 103 students of the class of 2019, and 107 students of the class of 2020 at the Zhengzhou Institute of Aviation Industry Management in China were surveyed. A total of 358 questionnaires were distributed, 302 valid questionnaires were returned and the effective return rate was 84.4%.

Generally, data reliability is defined as the degree of consistency or stability of the measurement results, and a high degree of reliability indicates greater confidence in the measured results. SPSS 17.0 software was used to analyse the model data for internal consistency reliability and to proofread the abnormal data. The reliability of internal consistency is usually determined by the Cronbach's α coefficient value [17], where coefficients of 0.9–0.8 are considered excellent, 0.8–0.7 are considered good and 0.7–0.65 are considered acceptable. Table 2 provides the values of the coefficients of confidence for each latent variable in terms of the internal consistency of the questionnaire data.

In Table 2, it can be seen that the α values of all four latent variables are >0.7 and that the reliability coefficient for the overall study is >0.9, which means that the questionnaire design is reasonable, the quality of the study data is high and the study data have a high intrinsic reliability that can be used for further analysis.

To ensure the validity of the questionnaire, the validity of this study was analysed using an exploratory factor analysis to assess the validity within and among factors. The KMO and Bartlett tests were conducted on the question items to examine the suitability of the study sample for factor analysis. The results are shown in Table 3.

The Kaiser KMO measures are as follows: ≥ 0.9 means very suitable, 0.8 means suitable, 0.7 means average, 0.6 means not very suitable and ≤ 0.5 means extremely unsuitable. The KMO value approaches 1 when the sum of squared simple correlation coefficients among all variables is greater than the sum of squared partial correlation coefficients. The closer the KMO value is to 1, the stronger the correlation between variables and the more suitable the original variables are for factor analysis. According to Table 3, the KMO value of this study is 0.858, the Bartlett's sphericity test value approximates the chi-square 1337.333, the degree of freedom is 69 and the P value is 0.000 < 0.001, indicating that the data of this questionnaire are suitable for factor analysis.

Model test analysis was performed using AMOS, and the study used the cardinality ratio of freedom (x^2/df), goodness-of-fit index (GFI), adjusted goodness-of-fit index (AGFI), Bentler comparative fit index (CFI), root mean square error of approximation (RMSEA), non-canonical fit index (TLI) and value-added fitness index (IFI). Table 4 shows the results of the tests, which were compiled.

The cardinality ratio of freedom value of 1.098 is less than the standard value of 3, and the model fit is excellent; meanwhile, the RMSEA value 0.019 is <0.05, and the GFI, NFI, RFI, IFI and TLI values are all >0.9, indicating that the model is well constructed and the results are reliable.

The results of the hypothesis testing for the SEM are presented in Table 5, where *** indicates that P < 0.001 indicates a very significant effect of the hypothesis. 0.001 < P < 0.01 means that the hypothesis has a more significant effect, and 0.01 < P < 0.05 means that the hypothesis has a generally significant effect. It can be seen in the table that all of the hypotheses of this study have passed the hypothesis test.

Based on the standardised path coefficient for Hypothesis 1, which is 0.412 and P < 0.001, students’ learning ability is extremely significant in determining the effectiveness of SPOCs. It is the most influential factor. Thus, the stronger the learning ability of a student, the more likely they will be able to take the initiative to pre-study before class, summarise and review after class and at the same time find resources, think creatively, and be able to extend knowledge and work in a team, along with gaining more knowledge from taking the SPOC. Comparatively to the teacher's teaching design and the classroom learning environment, students’ commitment to the SPOC can enhance the course's effectiveness.

The standardised path coefficient for Hypothesis 2 was 0.401 and P < 0.001. This suggests that the instructional design of the instructor was highly significant in influencing the quality of the SPOC. Teachers’ planning and organisation of online and offline teaching in SPOC, the course scheduling, as well as their teaching styles, methods and styles, all have a significant impact on the teaching effectiveness of the course. In addition, the number of assignments, number and difficulty of questions, the timeliness with which assignments are corrected, as well as the fairness and openness of the rules for grading students’ grades all have an impact on the effectiveness of SPOCs.

According to the standard path coefficient of Hypothesis 3, which was 0.177 and P = 0.012, classroom learning atmosphere has a generally significant effect on the teaching effectiveness of SPOCs, but the effect is relatively modest compared with other latent variables. In terms of SPOC teaching effectiveness, the influence of classroom climate (interaction between teachers and students, communication between students and teachers in online and offline classes and whether students actively participate in group activities and take the initiative to undertake tasks) can be compensated by other active learning behaviours of students and improvements to classroom instructional design.

Based on the standardised path coefficient for Hypothesis 4, P < 0.001, student learning ability has a highly significant impact on instructional design. The current findings indicate that teachers’ classroom instructional design is generally constrained by students’ learning abilities and learning bases. Students’ better learning foundation can stimulate teachers’ enthusiasm for teaching design, and teachers can improve teaching design according to the depth of students’ knowledge. When students have strong learning ability and teamwork ability, the completion of teaching design is high, which can also stimulate teachers’ enthusiasm for teaching design and form a virtuous circle.

Accordingly, the standardised path coefficient for Hypothesis 5 is 0.221 and P = 0.043, indicating that the classroom learning environment has a generally significant impact on students’ learning ability. As a result, the more teacher-student interaction and communication in the classroom, the better the students’ learning abilities will be. Students actively participate in group activities and take the initiative to undertake tasks, which can enable students to transform knowledge into practice and deepen their understanding of knowledge. The higher the frequency of teacher-student interaction, the higher the student's attention to the classroom. Interesting interactive methods can also arouse students’ willingness to learn and improve their learning ability.

It was determined that the standardised path coefficient for Hypothesis 6 was 0.372 and P = 0.009, indicating that the teacher's instructional design was more significant in influencing the classroom learning environment. The teacher's teaching design in terms of teaching style, style, etc. impacts students’ motivation and enjoyment of learning in the classroom. Teaching methods that are receptive to students, moderately difficult and large assignments and open, fair and equitable grading methods can all influence the learning environment in the classroom.

The SEM was tested and calculated using AMOS software to obtain the standardised output of the model, i.e., the observed variable path coefficients, as shown in Figure 4.

Fig. 4

As can be seen from Figure 4, the most important influence on the teaching effect of SPOC is students’ learning ability, where the most important observed variable on students’ learning ability is students’ active thinking ability, with a path coefficient of α4 = 0.79; the second most important influence is students’ problem-solving ability, with a path coefficient of α5 = 0.77; the third most important influence is students’ ability to find resources, with a path coefficient of α3 = 0.69. The other observed variables in order of importance were students’ ability to summarise and review after class, ability to work in teams, ability to extend their eyes and ability to preview before class, with a path coefficient of α2 = 0.55 > α7 = 0.45 > α6 = 0.44 > α1 = 0.41.

The most influential observed variable on teachers’ instructional design was teachers’ teaching methods with a path coefficient of α11 = 0.89; the next most important influence was grade evaluation with a path coefficient of α14 = 0.87; the third most important influence was teaching style with a path coefficient of α10 = 0.83. The other observed variables in order of importance were course scheduling, classroom planning and organisation, homework volume and assignment design, and the path coefficients were α9 = 0.71 > α8 = 0.55 > α13 = 0.54 > α12 = 0.48.

The most influential observed variable on classroom learning climate was classroom communication with a path coefficient of α16 = 0.61; the next most important influence was task taking with a path coefficient of α17 = 0.52; the third most important influence was teacher-student interaction with a path coefficient of α16 = 0.48.

In addition to ζ 1, ζ 2 and ζ 3, course satisfaction and course learning goal completion evaluation also influence the teaching effectiveness of SPOCs with the path coefficient α18 = 0.50 > α19 = 0.38, which means that the influence of course satisfaction is stronger than that of course learning goal completion evaluation.

As opposed to the teacher's teaching design or classroom learning environment, the students’ internal motivation is more important. It is essential that students pay attention to the learning of professional courses, increase their interest in the courses and learn to think independently and build a knowledge system on their own. During cooperative tasks in the course, students should be guided by the idea of taking action and contributing to the team's success.

The teaching team actively explores flexible and effective teaching methods and integrates heuristic teaching, case studies, simulations and experiential learning. It also focuses on the interaction between teaching and learning in the classroom. By improving teaching methods, students’ interest in learning will be stimulated and their ability to innovate will be enhanced. Implement all-round guidance for students’ learning processes by categorising and organising three time dimensions: before, during and after class. In line with the teaching content, relevant learning tasks are set to clarify key points of knowledge and clarify key questions. Students complete online teaching resources, such as courseware, videos, audios and cases independently, following the prompts. Practice questions are added to the video playback to stimulate participation and enhance learning enthusiasm. The learning effects of students are evaluated through online tests after stage learning. Students provide feedback on problems they cannot solve through the learning platform, and teachers focus on key and difficult content offline based on the feedback students provide.

During the SPOPC, the advantages of online and offline integration should be fully tapped to develop multiple applications. This should take into consideration the needs of students with distinct personalities, preferences and levels of learning. Moreover, this should also combine the emphasis on cultivating theoretical research with the emphasis on cultivating time-based abilities and be involved throughout the entire course to achieve a balanced level of cultivation of comprehensive literacy and applied knowledge and abilities.

Evaluation is established as follows: develop a diversified learning evaluation system, explore a diversified assessment and evaluation model that integrates online and offline courses, integrate formative and summative evaluations, encourage independent learning, process learning and experiential learning and assess overall evaluation results of the courses based on both process and summative evaluation.

Evaluation of the course process includes online assignments from the course resource platform, classroom interaction, check-in, audio and video task points, chapter quizzes, group tasks, chapter learning times, discussion participation and scores for offline learning behaviour. By the end of the course, the summative assessment will include a theoretical knowledge question bank test and a simulated scenario-based practical training project assessment. This final assessment will test the students’ theoretical mastery and their practical application ability, measure the effectiveness of the teaching activities and comprehensively reflect the students’ learning ability and the level of the SPOC, ensuring that the learning objectives and requirements are met and the students learn what they need to know.

The course is an important carrier for students’ training. In some cases, students’ attitudes and learning behaviours can objectively and truly reflect that a professional course is meeting their professional ability development needs to some extent. The college student's learning process is more independent and autonomous in comparison to that of secondary school students. In the teaching process, the teacher is the object of learning activities, while the students are the subjects of those activities.

Based upon traditional education, most students are more introverted in terms of interactive communication and they are not adept at asking questions during the learning process. There will be poor communication between students, teachers and classmates, and the effectiveness of classroom instruction will be considerably reduced. As a way to improve the learning effects of students’ courses, students should take advantage of the opportunity to communicate with others, whether through online or offline classes. Additionally, they are expected to participate in classroom discussions and research projects, create sparks of wisdom through the collision of ideas and keep a positive classroom atmosphere. By taking advantage of the opportunity to communicate with others, they will be able to break free from traditional thinking, actively participate in classroom discussions and research, create sparks of wisdom through the collision of ideas, foster an active classroom environment and cultivate a positive attitude towards learning so that they may become better learners.