Measuring usage versus preferences for online study materials among business-majored undergraduates

One line of inquiry focuses on learners’ preferences of study materials. Preference refers to how students, in their own opinion, value certain types of learning resource over the others. Research on students’ preferences of study materials are thus comparative by nature: typical studies explore at least two types of learning resources and how differently they appeal to learners and support learning processes.

The majority of studies in this topic in the context of online learning are descriptive as they capture students’ self-reported perceptions through questionnaires and/or interviews and generalise into characteristics of preference for that particular population under investigation. For example, Frydrychová et al. (2014), in analysing survey results for an online learning development project in the Czech Republic, report that as social sciences students rely fully on digital materials for their mastery of e-subjects, the majority prefer online lectures with illustrated PowerPoint slides the most, followed by printed texts, then online texts with graphics and hyperlinks to external resources. Video sequences and animations are less welcomed, but still roughly 30% of the surveyed students feel the need to include these materials in the course. The authors point to the benefits of multimedia learning over traditional text-based learning in terms of facilitating information processing with more sensory experiences to explain the preference for online lectures and interactive texts, as well as the flexibility of access and the stronger sense of choice for students: they can pick materials to work online (streaming) or offline (printing) as they wish. Another study in a different discipline presents contradictory results but with rather similar explanations. Montayre and Sparks (2018) examine nursing students’ preference of learning resources in their online bioscience course, through surveys, and conclude that the students are much more interested in video materials than in any kind of text. The students themselves cite their so-called sensory learning styles (e.g. visual and auditory learners) and accessibility as reasons for their preferences. It is noted that they clearly favour online, live-streaming videos on the Internet over recorded materials in CDs/DVDs, which underlines the need for ‘wider’ access and more interaction during a learning episode to improve both engagement and effectiveness of learning. For a more holistic descriptive inquiry, Kobayashi (2017) assesses college students’ preferences over 15 learning media based on perceived usefulness. In her findings, among the study materials, lecture notes and assignments got the best reviews (from more than 80% of participants); online multimedia slide presentations, live lectures, online videos and online collaboration tools (e.g. Google Docs) are also considered useful (50%–65%), while discussion boards and CDs/DVDs are the lowest rated materials (only 20%–25%). Similar to the special remark by Montayre and Sparks (2018), Kobayashi takes note of the students’ preference for live, online audio-visual materials over their offline counterparts, deeming the online resources as much more convenient and expansive for self-study. She also comments that the unpopularity of discussion boards, as asynchronous media to promote student interactions, may indicate that students attending online courses are more inclined to individual, autonomous learning when given a wealth of study materials. The high opinions for lecture notes and assignments somehow resemble the preference for PowerPoint slides and printed texts in Frydrychová et al.’s study (2014), which implies that the basic method of content delivery through text documents remains essential even in a media-rich environment. Overall, the reviewed articles show that self-reported preferences among students vary greatly, yet there is a clear need for diversified, multimedia learning resources in online education in view of accommodating students’ motivation.

Several studies attempt to go beyond the descriptive purpose: they compare the perceptions of value and interest for two or three types of learning resources and link such comparisons to the learning outcomes. For instance, York et al. (2009) set up a small-scaled quasi-experimental research to identify medical students’ preference of content delivery methods and compare their respective mastery of the content in a single-mode online learning environment (i.e. without any face-to-face components during a course). Participants are randomly divided into two groups who study the same course with two distinct types of study materials: group A with non-printable interactive slides, which contain animated graphics, hyperlinks to external resources and auditory feedback, and group B with printable PDF texts with select graphics and hyperlinks. Members of both groups complete a survey of preference before engaging in their course; the result shows that they prefer to have access to both types of materials, and if they have to choose one type, more students favour the interactive option than the text option. In terms of academic performance, however, there is no significant difference between the two groups in their test scores, which means – in line with the argument of Clark (1994) – the delivery methods do not impact the mastery of the subject matter. This finding demonstrates the general consensus that online learning students prefer a variety of study materials and may be attracted towards a particular type of instruction but can still manage to reach the learning targets when given less favourable but equally informative materials. In other words, accommodating students’ preferences in course design can lead to higher levels of satisfaction and/or motivation, but not necessarily better effectiveness of learning. Similar conclusions are made by Ge (2019) in his work on mismatch between preferred and assigned study materials among language e-learners. The author compares results of six groups of students following six combinations of media preferences (auditory vs. visual) and media assignments (audio-only vs. on-screen text-only vs. audio and on-screen text combined) as they participate in a fully online course of English proficiency. The final analysis indicates that the two completely mismatched groups (i.e. students who preferred visual materials but were given audio-only resources, and vice versa) did not perform significantly worse than the other groups, especially the completely matched ones. This means the warning that failure to tailor the virtual environment to students’ individual needs can cause failure of online learning is not substantiated. It also puts a question mark on the reverse assumption of positive correspondence among academic preference, participation and productivity in the pursuit of autonomous learning in online education. Furthermore, Ge’s work (2019) reiterates the benefits of multimedia learning as the findings suggest that students who study a combination of audio and visual materials perform better, regardless of their original singular preference. In short, correlational studies involving students’ perceptions of study materials do not confirm any clear association between self-reported preference and actual achievement, but they support the diversification of learning resources as a general approach that can improve the engagement and effectiveness of online education.

Another line of research covers learners’ usage of study materials. These studies are recent and contingent to LMS data analysis as they unobtrusively track learners’ online behaviour through log files of given systems. Compared to self-reported questionnaires and qualitative interviews with students, LMS access data are a more reliable source of empirical evidence to reflect the reality of resource use by the students on a virtual platform. Many recent studies take advantage of learning analytics recorded by LMS to investigate how students access different study materials and how patterns of access can relate to individual characteristics and/or learning outcomes.

To describe students’ usage of study materials, researchers rely on the tracking frequency of access (i.e. number of views/clicks onto a content item) and/or duration of access (i.e. amount of time spent on a content item). A typical and simple study using the first method of estimating resource use is proposed by Atherton et al. (2017), in which the raw frequency of access per student is used as the proxy for his/her engagement with the online study materials (lecture notes, lecture slides, tutorial questions and tutorial answers). The team sorts the students’ assessment results in an increasing order, then plots such results against their respective frequencies of access and identifies similar positive trends between the two elements among both full-time and part-time participants of the given openaccess course. Another example with a more specific focus to distinguish the levels of usage among different study materials is the work by Chan et al. (2021). In this research, they document medical undergraduates’ access to five types of learning resources, namely, problem-based documents, videos, quizzes, discussion forums and other unclassified materials, and report that students use the documents and the videos the most, while the forums the least. As the data were collected in 4 years with four cohorts of students, the authors also discover how the frequencies of access to different materials changed over time for each cohort, for which they explain as corresponding to the students’ progress through their bachelor’s programme.

Li and Tsai (2017) identify patterns of access to study materials with the second method of measurement and then examine how the patterns may impact students’ academic performances in a blended course (i.e. students engage in both face-to-face and online learning activities to achieve the course outcomes). Particularly, for the online components, they recorded the time (in seconds) each student spent on each of the following study materials: video lectures, lecture slides, shared assignments and posted messages on the discussion board. After standardising the data, they employ cluster analysis to divide the students into three groups viewing behaviours: ‘consistent users’, who actively consult all materials, especially the interactive ones; ‘slide-intensive users’, who spend more time on lecture slides than on other types of materials; and the ‘less-use users’, who spend the least time on all materials. To examine the impact of usage of study materials on learning achievements, the authors compare the scores of the three clusters of students. Statistical analyses reveal that the consistent and slide-intensive users perform significantly better than the less-use users in all assessment forms, and the consistent users even outperform the slide-intensive users in formative assessment.

Several lines of evidence-based research take account of both frequency and duration of access in describing patterns of usage of learning resources. For example, Lust et al. (2013) examine students’ regulation of tool use throughout a course period by tracking both numbers of views/clicks and time spent on different study materials, classified as functional learning ‘tools’ (e.g. Web lectures as basic information tools, discussion boards as cognitive communication tools, quizzes and exercises as cognitive knowledge modelling tools). With such rich data, the authors were able to describe the general trends of access during the course, notably students access Web lectures (basic information tools) most often but spend the least time there, as opposed to quizzes and exercises (knowledge modelling tools). Also, the students seem to regulate the intensity and diversity of their tool use based on the organisation of the course. It is noted that the course employs a blended learning environment, which requires the students to attend physically and/or digitally to different learning activities, so this could be a factor to help students’ tooluse regulation in the online LMS platform. The study then employs cluster analysis to delineate specific diverging patterns of online usage among the students and then covariance analysis to link those behavioural patterns to learning performances. Interesting findings include a large number of tool-use patterns (12), only a few of which (2) can be considered effective regulation and aligned with the changing requirements during the course, and the significant impact of tool-use regulation on course achievements, as students of certain patterns (e.g. intensive users towards the end of the course) perform much better than their peers (e.g. non-users and limited users). Another exemplary work is by Lento (2018) who processes the rich data of usage differently. He also collects both the number and duration of views by students on seven types of study materials, divided into assessment-based materials (weekly practice tests), self-study ‘dynamic’ audio-visual materials (demonstration problems, course videos and online tutorials) and self-study ‘static’ visual materials (lecture notes, in-class solutions and case solutions) for the online component of a blended course. In describing students’ usage throughout the course period, Lento only relies on aggregate numbers to visualise the fluctuations of access, similar to how Lust et al. (2013) describes general trends of usage. The finding still reveals certain patterns of tool-use regulation, though not specific for any further grouping of students: access to self-study materials drastically peaks before exam dates and freefalls right after, which signifies cramming habits, while access to assessment-based materials is less volatile, less concentrated around exam dates and decreased towards the end of the course, which signifies the effects of requiring students to use them on a more regular basis. In terms of relating usage and performance, this study employs regression analysis, with final examination scores as the independent variable and accesses to different materials as dependent variables. The results show that the use of assessment-based materials is more positively correlated to learning achievement than the use of self-study materials, and within the selfstudy materials, the use of dynamic resources is more positively correlated to the achievement than the use of static resources.

Overall, many studies on students’ usage of online study materials have agreed on (1) the strong association between patterns of usage and learning performances, mostly in a positive direction between the intensity of access and better evaluative results, and (2) students’ active changes in their online learning behaviour in response to the course organisation and different academic requirements. Methodologically, the usage of study materials is often measured through the frequency and/or duration of access to content items, which is/are automatically tracked by the LMS. In addition, there are currently two designs of analysis to explore the relationship between academic usage and performance: The first one is directly through correlation and regression analysis, processing total and/or average access to study materials against test scores, which will yield a global picture of how the entire body of students behaves and performs academically, and the second is to run cluster analysis to categorise students based on their patterns of usage and then compare test scores of those clusters of students, which arguably provide more in-depth and nuanced descriptions of both the usage and the impact of usage on performance.

An intriguing observation from the scattered literature of engaging learners and learning resources online is that there is not yet an inquiry that connects students’ reported preferences with their actual usage of study materials with a view to educational effectiveness. In other words, the three elements of learners’ preference, behaviour and performance in an online learning environment have never been linked in the same case of investigation. As briefly reviewed earlier, preference and performance have been paired to look for potential influences and so have behaviour and performance, with the two approaches providing mixed results. Threading the three of them together for a better-informed conclusion thus remains a gap to be addressed. Furthermore, given that both students’ feedback of preference and LMS-recorded behaviours play important roles in course management and material development, it is desirable to study both in relation to each other so as to arrive at the best practice(s) of educational data analysis for improving the instructional design.

The present study thus sought to fill the identified gap of research. Specifically, it examined the interplay between students’ preference for and usage of online study materials, on the one hand, and possible implication(s) for academic performances, on the other hand, through two research questions:

The first question links preference with the usage of study materials, so its answers will identify possible correspondences and/or discrepancies between students’ behaviours and opinions regarding their own experience and the effectiveness of online learning. This can help clarify any misconception from course instructors and designers about what students do, how they organise learning in the virtual settings and how they react to different learning resources, as well as any ambiguous perception by students themselves of how they should improve their learning efficiency. The second question builds on the result of the first question, connecting the figured correspondences and/or discrepancies with students’ performances in examination to testify how effectively students have understood their academic needs and thus actively made use of the online course materials to support their learning. Overall, the research can provide insights for (re)considering current approaches to course design and redesign, thereby ensuring that learning is enhanced (rather than impeded) by technology.

Participants of this study were 203 undergraduates from various business-related majors in a Faculty of Economics and Business at a large research-intensive university in Belgium. They all enrolled in a Dutch-taught course on business information systems (BIS) in the spring term of 2020. As the present study took place within the scope of a bigger research project by the faculty, the researcher did not directly engage with those students to collect their data but received a part of relevant and readily pseudonymised data to process for the proposed research purpose. As a result, concrete demographic information such as name, age, gender or previous academic records of the students was not known and taken into consideration for analysis. The data gathering was approved by the in-charge personnel of the faculty’s project and the ethical committee of the university.

The present study specifically looked into the second half of the given course, where the students had to study 100% online. The BIS course in Spring 2020 was originally planned as a 15-week blended course, combining on-campus face-to-face classes with off-campus online learning components. There are six modules of learning, which covered a total of 23 classes (both on- and off-campus) from February to May 2021. Due to impacts of the COVID-19 pandemic that forced all educational activities at the university to move online since mid-March, the first three modules (Modules 1, 2 and 3) were carried out as planned in the blended format, while the last three modules (Modules 4, 5 and 6) were taught entirely through the LMS and other digital platforms. There are two reasons for the focus on the second half of the course. Firstly, as required, learning was delivered exclusively online from March onwards, students’ online learning behaviour that was automatically and exhaustively recorded in the LMS becomes a well-informed reflection of students’ learning activities in that same period. Secondly, during this period, with three modules of content, four types of online learning materials were employed in varying degrees: Web lectures, massive open online course (MOOC)/ knowledge clips, discussion boards and online quizzes. This variety means more possibilities of comparison in usage, preference and effectiveness of certain materials both within a module and across different modules.

It should be noted that given the course was originally designed as a blended course, the transition into a fully online course in the second half was rather smooth as most of the mentioned online materials had been produced and made available before the pandemic pause. To be more specific, except for the Web lectures, which must be recorded with the prepared slides/notes to replace face-to-face lectures, the MOOC/knowledge clips, discussion boards and quizzes were all functional on the LMS since the beginning of the course. In other words, with online learning being an integral part of the blended course by design, even though the learning in the second half of the course can arguably be viewed as emergency remote learning, the instructional design was originally intentional to facilitate learning activities on digital platforms (both LMS-based and with external resources) and later boosted during the pandemic pause to best support the drastic change in the academic context and interpersonal interactions.

In summary, there were three kinds of data: (1) survey responses about the preference of study materials, (2) activity logs on the LMS to record the usage of the given study materials and (3) assessment results on Toledo to reflect learning performances during the course. After the collection phase, data were processed and synthesised into a master Excel file with the basic organisation per student (example in Table 2). The statistical analysis was then run by SPSS software.

The first research question concerns the possible relationship between students’ preference and usage of online study materials. For this question, a k-means clustering analysis was conducted with the preference data and the usage data to categorise students into groups (clusters) of distinct preference and usage patterns. Descriptions of each cluster would reveal specific correspondences and/or discrepancies between preference and usage among the students. The choice of cluster analysis was based on insights from the literature review and considerations of how complex and constrained the data sources were (i.e., the survey of preference, as an instrument, could not offer straightforward comparisons, and there was a forced reliance on the frequency of access after processing Toledo activity logs).

The second question stipulates that the link between the preference and usage of online study materials can impact students’ learning performances. To answer it as a follow-up for the first question, group comparisons were conducted. Students’ cluster membership (as specified from the previous cluster analysis) served as an independent variable and all their performance indicators as dependent variables. After checking the assumptions of normal distribution and homogeneity of variance for the dependent variables, a multivariate analysis of variance (MANOVA) was carried out.

Descriptive statistics of students’ self-reported preferences (ranked order of favour) and actual usage (frequency of access) of online study materials are represented in Tables 3 and 4. Overall, regarding preferences, clearly, the students prefer studying online to attending traditional face-to-face lectures, and they have quite diverse opinions on which combinations of online study materials appeal to them most. All combinations that include Web lectures generally rank higher than those with knowledge clips, which indicates the students’ preference for Web lectures as the main source of knowledge transfer over knowledge clips. However, this is apparently not aligned with the numbers of actual usage: even though there are fewer clips than Web lectures, on average, students access the clips slightly more frequently than the lectures. The statistics show some discrepancies between the preference and usage of practice quizzes and discussion boards as well, but considering the different nature of access of these two materials, it is difficult to make descriptive comparisons.

To classify students into groups of preference/usage patterns, a k-means cluster analysis was performed, and three clusters were identified. A quick run of analysis of variance with cluster membership as an independent variable and all indicators for preference and usage as dependent variables revealed that the three clusters differ in all indicators, except for one (preference for the combination of Web lectures and quizzes, F(2, 200) = 0.516, p =.598). Figures 1 and 2 compare the three clusters in terms of preference and usage of study materials.

Figure 1:

Figure 2:

Cluster 1 (n = 45) has students who indicated preference for Web lectures and quizzes (as their highest-ranked materials involve either or both) but actually used them less than other materials. They do not seem to regard discussion boards highly (as their lowest-ranked materials involve this type of materials) and actually used them less than other materials. Compared to other clusters, the students in Cluster 1 also accessed all materials most sparingly, as seen in their overall below-average usage in z-score comparisons. Cluster 2 (n = 77) consists of students who showed clear preference for Web lectures as the three combinations involving Web lectures received markedly high scores (3–4 out of 5). This is reflected to an extent in their usage as they accessed all the separate materials in a high and roughly similar frequency. They had very low opinions on knowledge clips, especially the combination of knowledge clips and quizzes but still used the clips quite actively. Compared to other clusters, the students in Cluster 2 accessed all materials most frequently, as seen in their above-average usage in z-score comparisons. Cluster 3 (n = 81) includes students who generally preferred online multimedia learning and used all the materials in the same frequency, except for the discussion boards. They reported a strong favour for any combinations of online study materials and a distinct dislike for any options involving face-to-face lectures. Compared to other clusters, the students in Cluster 3 accessed all materials moderately, as seen in their average usage in z-score comparisons, with the exception of the below-average usage of the discussion board.

For confirmation, a post hoc test for the analysis of variance on the identified clusters was conducted. Tests on assumptions of normal distribution and homogeneity of variance for the dependent variables revealed that they were not satisfied, so a Games–Howell test was performed. Table 5 provides descriptive statistics of the clusters and the statistically significant results of Games–Howell multiple pairwise comparisons among the clusters.

The post hoc test results confirm that the students in Cluster 1 were particularly not fond of discussion boards among their self-reported preferences and they used all the materials much less than the other clusters. The students in Cluster 3 indeed have very clear preferences for combinations of online study materials against traditional lectures, and they accessed them all, except for the discussion boards. The students in Cluster 2 indeed have clear preferences for combinations involving Web lectures, yet they used all the given materials and apparently the only cluster who consulted the discussion boards actively.

In sum, the cluster analysis provides a clearer understanding of the preference–usage patterns among the students. Cluster 1, the smallest cluster with about a fourth of the sample, includes generally less active users who reportedly prefer materials more suitable for self-study such as Web lectures and quizzes. Cluster 2 includes very active users who reportedly prefer multimedia online learning, specifically with Web lectures. Cluster 3, the largest cluster, includes fairly active users who reportedly prefer all kinds of multimedia online learning.

A MANOVA test was conducted with cluster membership as the independent variable and all performance indicators as dependent variables. The performance indicators include both module-specific scores and total scores for both take-home assignments and the final examination that the students had to do. A few scores were missing (final examination scores for one student from Cluster 1 and five students from Cluster 3). Tests of normal distribution and homogeneity of variance showed several violations. Specifically, the data on assignment scores for all clusters show a ceiling effect (i.e. a lot of students have high scores on the assignments), while final exam scores generally conform to normal distribution. However, the F-test in (M)ANOVA is still robust against non-normality if it is caused by skewness (Blanca et al., 2017), so this should not pose a major concern. Levene’s tests showed that homogeneity of variances among the three clusters was not satisfied for data on final exam scores for Module 5 (e.g. based on mean: F(2, 194) = 4.908, p = 0.08), while it was satisfied for all other performance indicators. This means that a separate ANOVA was needed to run the data on final exam scores for Module 5, while the rest of the performance indicators could enter the MANOVA. If there are possible significant differences among the clusters per performance indicator, different post hoc tests will be administered for the indicators based on their homogeneity or heterogeneity of variances.

Results of the analysis, however, revealed no significant differences in the assessment scores among the three clusters (see Table 6). Post hoc tests were thus not needed. This means that despite their diverging preference–usage patterns in online learning, the students’ performance was not impacted.