The development of student feedback literacy through peer feedback in the online learning environment

Defined as scoring lists that guide students and facilitate their learning (Cheng & Chan, 2019), rubrics offer instructors the opportunity to introduce the criteria to students to consider in their assignments (Panadero & Romero, 2014). Rubrics allow students to be aware of what is expected of them, to recognise the applicable criteria and to make sense of the feedback given to them (Prins, de Kleijn, & van Tartwijk, 2017). In this respect, it is recommended that instructors should integrate rubrics into their formative and summative evaluations in higher education (Iglesias Pérez, Vidal-Puga, & Pino Juste, 2020). This study was carried out by using the rubric proposed by Göktaş (2016) for academic writing in instructional technologies. After permission was duly obtained from its author, the rubric was adapted to the study by reviewing the relevant literature (Razı, 2015). Since the research context of this study does not exactly match Göktaş's study, modifications were made by taking expert opinion to fit the context of the present study. The scope of the rubric was reduced to the scope of academic writing in the context of literature review on instructional technologies and its use in education. The tool was evaluated by two experts based on the following criteria: (i) coverage of academic writing for instructional technologies and essential acquisitions for its use in education; (ii) adequacy of the number of items; (iii) the intelligibility of the items; (iv) overlaps between the items; (v) descriptive descriptions of items reflect that item; (vi) the nature and number of degrees of the items; and (vii) usefulness of the tool.

The rubric contained 18 items rated on a three-option scale. For example, the respondents were expected to evaluate the statement ‘The effect of the instructional technology on the educational environment has been discussed from different points of view.’ by choosing the option closest to their actual thoughts: needs improving (0), acceptable (1) or good (3). Also, the tool was expanded by adding an open-ended question (‘What are your suggestions for improving the research report you have reviewed?’). The purpose of this question was to receive peer feedback messages from undergraduate students who evaluated assignments as a group. In addition, the rubric featured a section to include demographic information about the assessing group.

In this study, online questionnaires consisting of open-ended questions were used to learn the students’ opinions. The questions in this tool were drafted after the literature review and refined through expert opinion. Both at the beginning and end of the process, certain questions were addressed including demographic query and the following questions: (i) How eager are you to participate in the process planned for your preparing the assignment? (ii) What do you think about receiving peer feedback on your assignments? and (iii) Do you have any peer feedback experience? Upon the completion of the feedback process, another questionnaire of open-ended items was filled out by the students online reflecting on the process. The items in the form were based on the model by Carless and Boud (2018) and perfected by taking expert opinion. The questions in this round were as following: (i) How do you find receiving or giving feedback in the online learning environment? (ii) Has there been any change in your perspective on feedback at the end of this process? If yes, how? (iii) Has there been any change in your ability to consider the feedback you receive and decide where and how to use it, at the end of this process? If yes, how? (iv) How did you feel when you received supporting and critical feedback? (v) Has there been any change in your feelings about receiving feedback? If yes, how? (vi) How has the supporting and critical feedback you received during the process affected your effort to improve your assignment? Please explain. (vii) Have you acquired any useful insights or gains for your future education life? Please specify. (viii) What do you think about receiving or giving peer feedback in a similar process in your future education life?

In this study, the mixed research method was used; therefore, the collected data were analysed using quantitative and qualitative analysis methods, as appropriate. The reliability of the rubric was examined by looking at the agreement between the assessors. The consistency between the independent assessors’ scores for draft 1 and draft 3 according to the rubric was calculated with the kappa statistic. Draft 1 is the first assignment that the groups prepare without any peer feedback. Draft 2 is its revised version based on peer feedback with the rubric. Draft 3 is the final version of the revised assignments based on peer feedback provided on draft 2 in synchronous discussions. The kappa statistic is frequently used in determining the reliability between raters, which is proposed by Cohen (Bıkmaz Bilgen & Doğan, 2017). It is based on the correction of chance alignment. Pre- and post-test scores were analysed separately by SPSS 21. Agreement was found to be 0.705 for the pretest and 0.710 for the post-test. These values were interpreted by considering the levels of agreement suggested by Landis and Koch (1977), and the strength of the agreement was found to be significant in the range of 0.61–0.80.

In the study, the difference between draft 1 and draft 3 was evaluated. The obtained data were analysed through descriptive analysis. Descriptive statistics is the best statistical technique to summarise the obtained data and interpret them in connection with the research questions (Özsoy, 2010). With descriptive statistics, a large number of data can be organised, expressed in numbers and converted into information through tabulation (Gürsakal, 2012). Another statistical technique, the Wilcoxon signed-rank test, is applied to see whether scores of the subjects vary significantly under two different conditions (Foster, 2001). It is the non-parametric equivalent of the dependent t-test, and the data must be obtained through a scale with a minimum interval, far from normal distribution (Karagöz, 2010). Content analysis was performed on the peer feedback messages written in response to the following item in the rubric: ‘What are your suggestions for improving the research report you have reviewed?’ In order to analyse the content of the peer feedback messages collected, the coding scheme of Cheng, Liang, and Tsai (2015) was used. This scheme is composed of four dimensions: affective, cognitive and metacognitive dimension, and irrelevant comments. The affective dimension consists of ‘supporting’ and ‘opposing comments’. Peer feedback messages implying praise or supporting ideas (e.g., ‘Congratulations to our friends on their good work!’ or ‘Well done!’) were classified as the supportive feedback, whereas statements lacking any compliment or support (e.g., ‘Thank you for the efforts.’) were not treated so. On the other hand, peer feedback messages simply expressing negative feelings (e.g., ‘This assignment lacks the features of a proper research report.’ or ‘We think this assignment is lacking in many ways.’) were put in the opposing comments category. The cognitive dimension consists of three categories: direct correction, personal opinion and guidance. Messages focusing on the accuracy of the assignment (e.g., ‘The bibliography is not written according to the spelling rules.’ or ‘The table of articles should not be in the form of article tag.’) were classified as direct correction. Peer feedback messages expressing general opinions were analysed under the personal opinion category. Also, comments emphasising the shortcomings of draft assignments (e.g., ‘We think that the same topic is referred to so often.’ or ‘The advantages of the topic are mentioned, but they are not given under a separate heading and disadvantages are not mentioned at all. There is no conclusion or a part for recommendations at all.’) were considered under the personal opinion category. Lastly, feedback providing clear guidance to revise work (e.g., ‘It would be better to make explanations under the tables in the method heading.’ or ‘The basic concepts of instructional technology could have been explained more clearly.’) were addressed in relation with guidance.

The metacognitive dimension consists of two categories: evaluating and reflecting. Peer feedback messages evaluating or verifying the knowledge, skills or chosen strategies in the reports (e.g., ‘If the assignment had been based on the given template, it would have been evaluated more positively.’) were classified as the evaluating category, while comments containing critical messages to be extensively thought or reflected by peers (e.g., ‘By looking at the other assignments, the shortcomings of this work can easily be seen.’) were addressed as the reflecting category. Lastly, messages showing no relevance to affective, cognitive or metacognitive feedback categories were all classified as irrelevant comments. All of the feedback messages in the study were coded separately by the assessors against the dimensions and sub-dimensions in the aforementioned scheme. Then, consistency of their coding was checked. Reviewing and coding were repeated until agreement was achieved between the independent assessors to classify each message under the appropriate size and category.

The data obtained through the questionnaires were analysed by content analysis. During this analysis, datasets were read repeatedly to expose themes and codes. Analysis results are presented in tables with a detailed list of themes, codes and frequencies (Boyatzis, 1998; Yıldırım & Şimşek, 2013). Reliability was provided by two different assessors who extracted the codes and calculated the frequencies. For consistency in coding, assessors read datasets repeatedly and then extracting codes and frequencies individually. Then, they had a meeting and checked the consistency of the codes. They reached a consensus by discussing the data, when necessary. The codes with which consensus and disagreement were reached were clarified. Then, the frequencies were re-examined for the disputed codes, and they were analysed carefully until consensus was reached. Reliability was calculated by using the formula suggested by Miles and Huberman (1994) over the codes with consensus and disagreement. The confidence rate was found to be 0.81.

Research highlights peer feedback activities that allow supporting the development of student feedback literacy. In this study, a two-stage process consisting of providing written peer feedback with rubrics and participating in simultaneous online discussions were carried out. Yu and Liu (2021) thought that peer discussions on feedback can bring a deeper understanding of the assessment criteria, generation of alternative ideas and build-up of students’ capacity to expand their insight, ultimately leading to developed feedback literacy. In this study, the participation rate of the groups in the written activities based on the rubric feedback was equal to or above 70% (n = 14). As few as 20% (n = 4) of the groups provided peer feedback on nine assignments. A smaller percentage of the groups, 10% (n = 2), evaluated even less products than the minimum requirement. Winstone, Mathlin, and Nash (2019) drew attention to the importance of proactive participation of students through feedback in order to get the most out of feedback. As for the online discussions, It was created a platform to review all the assignments produced here. However, during the discussions applied to both the presenters and the feedback providers, the students did not seem motivated. This was especially true for the assessing groups as they expressed their opinions in writing, despite the availability of audio and video features in the online sessions.

The literature shows that feedback in written, audio and video forms can be used as complementary to each other. While Ene and Upton (2018) emphasised that chatting on written feedback can produce beneficial results, Wood (2022) reported that peer feedback presented in video form supports written feedback. According to Er, Dimitriadis, and Gašević (2020), dialogue over feedback can make it easier for students to make sense of the input they are given. Online peer discussions were conducted to elaborate on the written peer feedback as a part of the present study, but it proved unsuccessful. In Zheng et al. (2018), in his study examining the effect of simultaneous discussions on written peer feedback on writing performance, it is seen that the participants who will provide peer feedback in simultaneous discussions are selected and appointed to draft in a certain number within the framework of certain criteria. Wood (2022) pointed out similar results in his research with a small study group. In this study, all students were given the task of providing peer feedback in simultaneous discussions. The lack of a more systematic planning by assigning certain students to certain assignments may be the reason for the current situation. On the other hand, Zhan (2019) believed that if interpersonal relationships between participants are compromised due to feedback discussions, this can be a source of concern for participants. This means that feedback loops cannot be closed without comfortable interaction between buyers and providers.

In this study, the content of feedback messages provided by peers was analysed. The results suggested that the feedback given in this study predominantly fall under the cognitive dimension. In addition, feedback focusing on the accuracy of the product, expressing general opinions and providing clear guidance for reviewing the product seems to have been preferred by peers. About 42% (n = 214) of the classified feedback messages provided clear guidance on how the work can be improved. While around 17% (n = 86) of the feedback messages expressed personal opinions, approximately 12.5% (n = 64) of those were classified as direct correction. Likewise, Foo (2021) claimed that the cognitive dimension makes up the most common type of feedback provided by students. Nearly 27% (n = 138) of the peer feedback was found to be in the affective dimension, supporting comments particularly. The groups that provided this kind of feedback underlined the admirable aspect(s) of the works and added their praise or supportive opinions.

These findings together imply that the students went beyond learning about the course content, specific tasks and relevant criteria, and they were involved in more complex processes such as explanation, justification, comparison and problem-solving in the process of providing feedback. Feedback-literate students develop capacities to make sound academic judgements about the work of their own and others (Carless & Boud, 2018). According to Han and Xu (2020), peer feedback can expand students’ self-regulation and self-assessment skills by engaging them in assessment and evaluation. In this study, 66% (n = 35) of the participants at the end of the process reported enhanced abilities or skills of making sense of the received feedback and using it effectively, 24.5% (n = 13) of the participants noted no change or improvement in aforementioned skills, another 34% (n = 18) reported progress in being open to different perspectives throughout the implementation, a group of students corresponding to 28% (n = 15) of the study participants stated that their collaborative working skills improved, 7.5% (n = 4) developed skills of being objective, 5.5% (n = 3) developed empathy skills and 5.5% (n = 3) reported better critical thinking ability at the end of the process.

Carless and Boud (2018) recommended analysing exemplars as one of the well-established learning activities for their proposed model. They suggested using more than one exemplar to stress that a high-quality product can manifest in a variety of ways. Emphasis is placed on the crucial role of making students share and discuss their academic judgements through dialogue on exemplars or online interaction. In this study, a model or sample assignment was shared with the students and analysed based on the rubric by the instructor to show how to assess strong and weak aspects of the work during a 60-min online class. The importance of scaffolding and training to be provided by the teacher before creating and receiving feedback is obvious (Min, 2006; Zong et al., 2020). In the current study, content analyses of the peer feedback messages revealed that the students provided feedback in the affective and cognitive dimensions, but the rate of metacognitive feedback was as low as about 1%.

In the model proposed by Carless and Boud (2018), it is an important characteristic of feedback-literate students to maintain their emotional balance in peer feedback uptake. The students listed a variety of feelings, such as happy, good, proud, motivated and neutral, in response to the feedback received from peers. There were students who felt in one of these ways as a reaction to either supporting or critical feedback. Overall, 28% (n = 15) of the students felt bad, while 37% (n = 20) stated that they did not feel good, and 35% (n = 19) felt neutral in the face of critical feedback. Researchers have pointed out that feedback can arouse a range of emotional responses that can affect confidence and motivation and that students may not necessarily cope with emotional aspects in the feedback process (To, 2016; Winstone, Nash, Parker, & Rowntree, 2017; Paterson, Paterson, Jackson, & Work, 2020). Moreover, Kennette and Chapman (2021) have emphasised that not all feedback can be positive and that students should learn how to make their works better and cope with failure besides other things.

When the implementation was completed, 47% (n = 25) of the participants implied a change of feelings as a result of receiving peer feedback. Of these, 24% (n = 6) detailed such change with additional information. In this scope, some students stressed that they gained the ability to manage their emotions, and some others referred to increased flexibility or maturity. Cheng et al. (2015) handled students’ cognitive development and maturity as a variable involved in the relationship between peer feedback and writing performance. According to researchers, students who have become mature enough are more inclined to anticipate thought-provoking or reflective comments, rather than unhelpful, comments that contain approval, praise or negative affective feedback about the assignment.

Feedback-literate students process and act on the feedback data they receive (Molloy et al., 2020). To make sense of the received information, students need to be actively busy and use it to inform further work, thereby closing a feedback loop (Carless & Boud, 2018). In the present study, the students stated that their effort was altered positively as a result of the feedback they received. A large percentage of the participants, 94% (n = 50), said that the feedback increased their effort or motivation for bettering their assignment. These qualitative findings were verified with quantitative support. Statistical difference between the scores the students got from the first and the last draft of their assignment proved the significant variation between the first and last copies.

This study researched students’ feedback literacy development through peer feedback in an online learning environment. First of all, the students’ views were explored about appreciation of feedback. Before the 14-week implementation, 56% (n = 30) of the students declared a positive approach towards peer feedback, but this figure increased to 88% (n = 47) after the implementation. The students thought that the peer feedback was useful for improving their assignment and supporting their personal development or learning. Nonetheless, 9% (n = 5) of the students were not sure about appreciating peer feedback, and another 2% (n = 1) reported a negative opinion. Yu and Liu (2021) claimed that there are individual differences in the way students form their feedback perception. Han and Xu (2020) have discovered various effects of instructor feedback following peer feedback on individual students depending on learner factors such as language ability, beliefs and motivation. Tian and Zhou (2020) have discovered the individual and contextual factor shaping student uptake of feedback coming from different sources. The results obtained here are consistent with the results of past research. In this study, 2% (n = 1) of the students regarded peer feedback and the overall process negatively. The respondent found the feedback given by their friends unnecessary or useless. However, it must be remembered that the other students in the same group had a positive opinion of the same material. Another 9% (n = 5) of the students were unsure about uptake of peer feedback. Among those, 2% (n = 1) attributed their own judgement to the existence of unnecessarily critical or useless feedback. The hesitant respondents justified their attitude with the technology literacy level, intra-group communication problems, the excessive length of the process and likelihood of adverse social communication or interaction among the students.

In total, there were 20 groups and 20 draft assignments. The students were expected to revise their drafts in the light of the peer feedback and upload the revised versions to the system by the specified date. A total of 18 assignments were handed in, which means that two groups did not submit the final draft. The remaining 18 assignments were evaluated by the researchers. There was no change or revision between the first and last drafts of the two assignments, yet they were included in the statistical analysis. As a result, it was understood that the students’ performances were significantly affected at the end of the 14-week process built on peer feedback.

It is considered essential to create and receive peer feedback for student feedback literacy. In this study, only one model or sample assignment was used. No instructional activity was carried out, other than showing the sample to the students and analysing it against the rubric criteria and finally answering the students’ questions during one of the 1-h online classes. No preliminary training was provided to the students on how to create peer feedback in different dimensions and categories, and how to interpret or receive the feedback provided.

The students were informed that they would get their midterm marks based on participation in the 14-week instructional activity along with the quality of their final work. The activities or steps implied by participation in the process were explained to them. Participation in online discussions is one of these steps or activities. However, no planning was made regarding the path and minimum requirement of participation for those who took part as feedback providers in these discussions. In other words, this part of the process was flexible. Only the process for peer feedback to be provided asynchronously with rubrics was planned in detail and explained to the students.

In an online learning environment, peer feedback can be a way to encourage the development of student feedback literacy. This study focuses on the effect of peer feedback with consecutive rubrics and online discussions in the online learning environment. The purpose of the online discussions on peer feedback provided by rubrics was to discuss the feedback in depth and to generate alternative ideas. However, those sessions did not prove as efficient as expected. It is recommended to elaborate the path to be followed in online discussions, to explain the roles to be undertaken by the participants and to assign participation and interaction as part of the overall grade for the course. Besides, it may be useful to appoint feedback providers to assignments in online discussions so that smaller discussion groups can be formed.

In this study on generating and receiving peer feedback, a sample assignment was shared with the students and evaluated with a rubric during an online lesson. During this activity, the students’ questions were answered by the teacher. Carless and Boud (2018) pointed out that exemplars can also help maintain student balance regarding standards by eliminating some unwanted surprises that may arise from unexpected teacher judgements. Therefore, an opportunity can be provided for students to discuss samples before moving on to the task of creating and receiving feedback. The number of exemplars can be increased. Moreover, training on creating and receiving feedback in the affective, cognitive and metacognitive dimensions and affiliated categories can be added to the model. Foo (2021) drew attention to the importance of providing feedback that encourages higher order thinking of students.