Digitizing Dunhuang Cultural Heritage: A User Evaluation of Mogao Cave Panorama Digital Library
Article Category: Research Paper
Publicado en línea: 22 ago 2017
Páginas: 49 - 67
Recibido: 10 may 2017
Aceptado: 25 jun 2017
DOI: https://doi.org/10.1515/jdis-2017-0014
Palabras clave
© 2017 Xiao Hu, Eric M. Y. Ho & Chen Qiao
This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 License.
Cultural heritage conveys evidence of culture and society, and is a precious resource for education, research, and appreciation. However, non-moveable heritage sites such as historic townscapes and landscapes, archaeological sites and remains, and standing buildings and monuments are subject to geo-location constraints and erosions caused by natural forces (Herbert, 1995). In recent years, the interest has been growing in applying digital technology to documenting and preserving cultural heritage (Garoufallou, Siatri, & Balatsoukas, 2008). Digital libraries (DLs) have played an important role in digitalizing and preserving cultural heritage materials, providing users with access to digital surrogates of these precious materials from anywhere at any time. DLs have thus contributed to the solutions to both heritage preservation and geo-location independent access. This, in turn, makes DLs agents for promoting local history and cultural heritage to a wider audience (Tait et al., 2016).
Dunhuang is a small town in Northwest China. It is well-known worldwide for the nearby Mogao Caves (also called Mogao Grottoes) which is a world heritage site listed by the United Nations Educational, Scientific and Cultural Organization (UNESCO). The Mogao Grottoes “consist of 492 caves containing over 45,000 m2 of murals, 2,415 painted sculptures, and more than 4,000 Apsaras statues” (Lu & Pan, 2011, p.2) that reflect the cultural interchange within the oriental areas via the Silk Road from the 4th to the 14th century (Lu & Pan, 2011). It has great historical, scientific, and artistic values for the study of ancient Chinese, the central Asian history, Buddhism, and ancient art (Lu & Pan, 2011). Aiming for preserving and promoting the precious Dunhuang heritage using digital tools, the Digital Dunhuang project has been undertaken by Dunhuang Research Academy with the support of the Andrew W. Mellon Foundation (Wang et al., 2015; Zhou, 2011). The Mogao Cave Panorama DL is one of the first end-user facing products in the Digital Dunhuang project that makes digital images of Dunhuang heritage accessible online. It is thus critical to understand how it serves the end-users and how it can be further improved from the users’ perspectives. The findings will not only benefit the Panorama DL, but also all end-user oriented digital presentations of Dunhuang heritage, such as interactive media presentations or 3D immersive exhibitions. The methods used in this study also provide a reference for future evaluations of digital platforms of cultural heritage in Dunhuang and beyond.The Dunhuang Mogao Cave Panorama DL is available on the Dunhuang Research Academy website
Homepage of the Mogao Cave Panorama DL© of Dunhuang Research Academy.
This study focuses on evaluating the Mogao Cave Panorama Digital Library from the end-user point of view. Four groups of end-users were recruited, based on their professions, to conduct usability tests with nine information seeking tasks designed specifically for the Dunhuang heritage. Findings of this study will help reveal how users interact with the Panorama DL. User friendly and problematic areas will be identified, based on which suggestions can be made for designers and developers in Digital Dunhuang projects to improve this DL and other digital presentations of Dunhuang heritage. In addition, as a case study on user evaluation of cultural heritage DLs, this work showcases the methodology, from task design, participant sampling to quantitative and qualitative data analysis, which can be adopted and/or adapted by future studies in related topics.
Digitization of cultural heritage treasures has attracted remarkable attention in recent years (Hampson et al., 2012). A number of DLs and online repositories of cultural heritage sites have been established worldwide, with the help of computational technologies. HistoricPlacesLA
The Eternal Egypt
Without any doubt, all these digital technologies in the aforementioned DLs can be made use by the Digital Dunhuang project. In fact, an immersive exhibition of Cave 220 using virtual reality technique and 360-degree panoramic projection has been held in China and overseas (Kenderdine, 2013).
Notwithstanding their irreplaceable advantages in preserving culture heritage, DLs also need careful designs and evaluation to ensure their usefulness and values. There are a number of studies on evaluating the usability of cultural heritage DLs and exploring ways of improvement. For example, Garoufallou, Siatri, and Balatsoukas (2008) evaluated the usability of VeriaGrid, an online virtual map DL focusing on the provision of information about cultural heritage in Veria, Greece. Their findings indicated that, although novel features such as recommended points of interests were added into the system, users still encountered difficulties in locating texts describing the heritage sites in the DL. Suggestions were then proposed to address this issue accordingly. Likewise, Jeng (2008) investigated the New Jersey Digital Highway, a DL built for demonstrating the history and culture of the state of New Jersey. Their findings identified issues in navigation and layout of the DL. Barak et al. (2009) examined the usability of MOSAICA, a Web 2.0-based DL for the preservation and presentation of cultural heritage in Europe. They applied online questionnaires and semi-structured interviews to obtain evidence of the usefulness, ease of use, and attractiveness of the DL. The authors reported positive views toward MOSAICA’s usage.
Smith and Rowley (2012) analyzed websites of nine public libraries in the Greater Manchester area in UK for the Web presence of local heritage, on the aspects of usability, content, aesthetics, and Web 2.0 services. Together with interviews with local study librarians, their findings suggest that, among other things, researchers and practitioners should continue developing knowledge of users’ expectations and information seeking behaviors, which can inform decision making on preservation practice, strategy, and policies (Smith & Rowley, 2012). In their framework of “information representation design,” Abbas et al. (2016) also argued that, in the spirit of user-centered design, understanding of users’ information behaviors and needs is crucial for designing information systems to support users’ information seeking tasks.
The Dunhuang Mogao Cave Panorama DL is the first panorama image collection of the Dunhuang heritage available online for public access from anywhere at any time. It is also one of the first of its kind for stone cave temples and murals. It was released to the public in 2015 and has yet to be systematically evaluated by end users. This study aims to fill this gap by exploring its performances with regard to usability and user experience. Specifically, we aim to answer the following research questions (RQ):
To what extent does the Mogao Cave Panorama DL facilitate information seeking tasks in different levels? How does the Mogao Cave Panorama DL perform in each criterion of the user evaluation?
Usability tests with a think aloud protocol were conducted in this study, followed by post-test interviews. A set of information-seeking tasks were designed for the participants to complete with the Panorama DL during the usability tests.
As different users may have different purposes when using the DL, four groups of representative end-users were identified for this study. The first two groups are users with professional purposes, such as artists or designers who seek materials to stimulate their creation (group 1), and school teachers who look for examples or evidence for the courses they teach (group 2). The other two groups are users with non-professional purposes, such as the general public who seeks background information of the Mogao Caves for entertainment or for trip preparations (group 3), and students who want to learn new knowledge about this splendid cultural heritage (group 4). Participants in this study were then recruited according to these considerations, through the authors’ professional and personal networks.
Fourteen participants in total joined this study. There were six males and eight females, aged between 18 and 29 with an average age of 23. The occupations of the participants diverged, ranging from college students (6 of them), administrative clerks (4), school teachers (2) to multimedia professionals (2). Half of the participants used the Internet for 6–10 years, while the others used over 10 years. All participants had used the panoramic function in Google Map.
We selected the evaluation criteria of the usability tests with considerations of 1) the usability evaluation model adapted from Jeng (2005), and 2) the importance and definition of the website’s interactivity (Liu, 2003). The framework of evaluation criteria is shown in Figure 2, which includes the aspects of effectiveness, efficiency, satisfaction, and interactivity (Jeng, 2005).
Framework of usability evaluation criteria, adapted from (Jeng, 2005).
The aspect of
As users need to perform relevant tasks in the usability tests, we designed nine information seeking tasks in four categories of different cognitive complexity levels (Spool, 1999), with considerations of the contexts of Mogao Caves as well as panorama representations of cultural heritage, including colors, shapes, structures, and themes (Shiaw, Jacob, & Crane, 2004).
Tasks 1 to 3 belong to
Task 1: Does the collection include any caves that are central-pillared (which means there is pillar in the middle of the cave)?
Task 2: Please locate one item that is constructed in the Northern Wei, Tang, and Yuan Dynasty respectively.
Task 3: In the collection you are working with, how many caves can you find displaying in the Digital Exhibition Center?
Tasks 4 and 5 are in the category of
Task 4: Please try to compare the central statue in Cave 275 and the first statue on the right in Cave 322, and see if the right hand of the first statue (the central one in Cave 275) is damaged.
Task 5: Among the caves mentioned in task 3, list two which are decorated with the Thousand- Buddha motif (shown in Figure 3):
Example of Thousand-Buddha motif© of Dunhuang Research Academy.
Tasks 6 to 8 are
Task 6: You would like to know more about the artistic style (e.g. color and shape) of those Flying Apsaras (an example shown in Figure 4.) from the Mogao cave arts. Name one cave from which you can find such information?
Example of Flying Apsaras© of Dunhuang Research Academy.
Task 7: Other than Buddha stories and Chinese mythology, you would like to know more about how people lived in ancient China (e.g. their economic, military, political status, and activities) from the wall paintings and statues. Which item(s) will you go for?
Task 8: Which of the following theme(s) best describe the paintings on the roof of Cave 249?
A). War; B). Chinese mythology; C). Hunting; D). Music performing.
Finally, task 9 is the
Task 9: Please identify and describe in general if there is/are any difference(s) between the artistic styles of the cave arts in the Tang and Yuan Dynasty?
In the usability testing, efficiency of the DL was measured by time (in seconds) spent in completing the information seeking tasks (i.e. task completion time).
Effectiveness of the DL in supporting information seeking was measured by the proportion of tasks that were successfully completed A task was considered to be successfully completed if the participant was able to finish it without committing any fatal errors such as giving obviously wrong answers or giving up the task.
During the usability test, each participant conducted all the nine tasks. They were instructed to follow the think aloud protocol. The test sessions were audio recorded and then transcribed into text afterwards. All the participants were voluntarily recruited and no monetary payments were provided.
After the usability tests, the participants were asked to answer a set of post-test questions in an interview. The participants were asked to rate the DL in the aforementioned criteria of usability. The interview also included open-ended questions for soliciting detailed explanations behind the ratings. The actual questions are presented in Table 1. All the rating questions were on a five-point Likert scale, ranging from difficult/unclear/disagree (point 1) to easy/clear/agree (point 5). The interviews were audio recorded and then transcribed into text for analysis.
Interview questions on evaluation criteria.Criterion Question Ease of use Was the DL easy to use? Why or why not? Organization of information How do you think about the organization of information on the DL? Why? Labeling Do you think the categories of the DL are clearly labeled? Why or why not? Visual appearance Do you think the DL is visually attractive? Why or why not? Content What do you think about the accuracy of the information on the DL? Why? Error correction Can you recover from mistakes easily? Please give an example. Active control When exploring the site, do you have a lot of control over what you can do? Two-way communication Do you think the DL allows two-way communication between you and the DL? Synchronicity Do you think the DL is fast in responding to your requests? Please give an example.
To measure the effectiveness and efficiency of the DL, we counted the number of participants who were not able to successfully complete a task and the average time used by the participants for completing each task. The results are shown in Table 2. Tasks in the
Task completion statistics.Task 1 Task 2 Task 3 Task 4 Task 5 Task 6 Task 7 Task 8 Task 9 Avg. Avg. time (seconds) 157.2 21.6 99.6 28.2 118.6 108.5 80.2 86.3 110.9 90.0 #. Failures 1 0 4 0 4 2 4 7 0 2.44
To compare the effectiveness of the DL in supporting different user groups, we calculated the task success rate of each participant which is defined as the number of successful tasks divided by the total number of tasks (i.e. 9). The results are shown in Table 3, which seem to suggest that the DL worked very well for multimedia professionals, followed by teachers and college students. To further understand the difference across users, we also compared task success rates between genders, age groups, experience with the Internet, and interests in cultural heritage. An obvious difference was found between genders, with males outperforming females by more than 10%. Longer history of Internet use seems to have helped a little too. These observations seem to suggest that experience and competency on IT skills might benefit the use of this DL. However, further work with a larger sample of users in various groups is needed for a systematic cross group comparison.
Comparison of task success rate among participants.Dimension Groups Success rate Occupation Admin clerks 75.03% College students 83.35% Secondary school teachers 83.35% Gender Female 77.80% Age 25–30 81.96% Experience with the Internet 6–10 years 80.97% Interests in cultural heritage Somewhat uninterested 81.50%
The rating scores on perceived satisfaction and interactivity in post-task interview are aggregated across participants and presented in Table 4. The results indicate that the DL offers pleasant experience by providing aesthetic appearance (median = 4), guidance for correcting errors (median = 4), and active and instant controls (median = 4); whereas it is quite weak in providing two-way communication with users (median = 2).
Participant ratings on perceived satisfaction and interactivity.Dimension Rating (five-point Likert scale) Median Mean Ease of use 3 3.2 Organization of information 3 3.4 Labeling 3 3.4 Visual appearance 4 3.6 Content 3 3.4 Error correction 4 3.9 Active control 4 3.8 Two-way communication 2 1.9 Synchronicity 4 4.0
Correlation analyses among task completion time, success rate, user satisfaction, and interactivity reveal that satisfaction and interactivity are significantly correlated (Spearman’s rank-order correlation coefficient
In order to explore the factors affecting usability of the DL in more depth, we analyzed the participants’ think-aloud and the post-task interview transcripts. Major findings are summarized as follows.
Evidence shows that sometimes responses from the DL were slow, resulting in participants spending longer time in completing the information seeking tasks. For example, in task 8 which required participants to identify the themes of the roof painting of a given cave, one participant had the experience:
One reason for this glitch might be the conditions of users’ hardware or network. Because the visual materials in the DL are of high resolution, it may require a fast network speed for transmission and a fast client computer for rendering the visual elements. The fact that users’ network conditions and accessing devices may vary should be taken into consideration in DL design, and multiple choices of image quality (i.e. file size) can be provided.
Statistics have shown that higher-level tasks (those in
These comments suggest that the layout and arrangement of the digital images could have room for improvement, and more descriptions and labels are desirable, in both facilitating users’ information seeking processes, but also improving user experience in navigation.
As navigation in a panorama DL heavily relies on its view control, whether the control function is easy to use can affect users’ experience with the DL to a large extent. There appears to be several issues of view control in users’ think aloud discourses. The problem of adjusting the position and rotation speed of the view control was mentioned quite often:
Another participant who also complained about the view control claimed that the contradictive direction controls of the DL to his preferred ways made him sometimes confused and move in the wrong way.
Other problems include insufficient instructions on the view control, no “back” buttons for returning to previous pages, and inconsistency of the interface.
There were different views among participants regarding the DL’s way of organizing information. Some participants appreciated the organization structure ordered by cave numbers and dynasties:
Others, however, argued that the organization of information on this DL could be improved by arranging the exhibits in a mini navigation map which could mark the caves and the surroundings in a way aligned with the geographic configurations in the real-world. In this way, all the exhibits could be linked together, allowing a more intuitive and straightforward navigation:
From the participants’ think-aloud discourses and comments on the DL’s performance on labeling, it is found that most of the labeling issues were related to the mini-map available in the scene, “
Besides, participants complained that many of the cave arts presented in the DL were unlabeled or labeled unclearly. As a result, users with limited relevant knowledge were not able to understand the meanings of the arts. For example, one participant commented on the labeling system in the DL:
The application of panoramic photos for reconstructing the caves in virtual space makes the DL visually appealing to the participants. The visual appearances of the DL were regarded as highly detailed, of superior quality, high resolution and realism. Some participants (e.g. No. 03 and No. 06) found the sunlight effects on the main page “
However, the visual appearance of the homepage was criticized by participants, for a large part of the page was covered with thumbnail images representing the exhibits. Besides, the image effects of the homepage were not consistent with other pages, reducing the consistency of the DL and visual attractiveness.
Most participants agreed that the content on the DL was authoritative and accurate. Nonetheless, some participants commented that, as they had never been to Dunhuang, and they had very little knowledge of the caves to tell if the virtual caves are in fact the same as the real ones.
In general, the DL performed well on this criterion. One issue that might cause unexpected errors was the default option of opening new tab when an item was clicked. While some participants liked this design for easy return to previous pages, others complained that having too many tabs might result in accidental closings of some tabs.
Participants agreed that the DL provided good human-computer interaction experiences in viewing the exhibits. However, they complained that most interactions were only one-directional:
Most participants mentioned that it would be desirable to have more two-way communication between users and the DL, as well as between users.
The results of quantitative data analysis help answer research questions Q1 (To what extent does the Mogao Cave Panorama DL facilitate information seeking tasks in different levels?). The success rates on information seeking tasks of high complexity levels were lower than those on tasks of low complexity levels. This may imply that the DL lent limited support to relatively complex information seeking tasks. One possible way to improve could be to provide more navigation support and meaningful labels to facilitate users’ exploration with the DL. In addition, more textual descriptions about the background, themes, and interpretations of the caves and the arts (i.e. murals, statues) can be added to help users learn new knowledge during the search and interaction process.
The qualitative data analysis helps answer research question Q2 (How does the Mogao Cave Panorama DL perform in each criterion of the user evaluation?). Although the DL performed generally well in the usability tests, issues were identified in most evaluation criteria, suggesting room for future improvements. For example, system responses were sometimes slow (efficiency); it was not easy to adjust mouse positions and rotation speeds (ease of use); the hotspots in the minimap lack semantic labels (labeling); and bi-directional communication in the system was weak (interactivity).
Suggestions can be made to address these issues: 1) options can be provided to make the image quality adjustable according to users’ hardware and network conditions, so that timely responses from the DL can be ensured regardless of different accessing conditions; 2) to improve ease of use, the navigation system may provide users with ways to adjust mouse movement direction and speed; 3) navigation can be facilitated by a mini-map that situates the caves into their surroundings; 4) the original mini-map with hotspots can incorporate labels such as titles or cave numbers, for easy reference by the users; 5) considering some users may have little background knowledge about Dunhuang, the DL can add introductions to cultural heritage materials, which will also make the DL more attractive and interesting; 6) to help users recognize the authority and accuracy of this DL, information on the development of the DL could be added, such as the image creation process (e.g. technologies used in preserving and representing the caves), and background of the parties who created the images and those who developed the DL itself; and 7) interactivity can be enriched, by possibly creating a virtual character that can guide and support users in navigation. Alternatively, more communication channels (e.g. chatting forum, mailing lists, social tagging, and Weibo account) can be established to enhance library-to-user and user-to-user discussions, as well as timely updates with the latest news about the DL.
DLs provide a strategic solution for culture heritage preservation and access. The usability and user experience of DLs are critical for the success of this strategy. In this paper, we presented a user evaluation of the Mogao Cave Panorama DL, aiming to identify issues that end-users may encounter when using such DLs. Usability tests were conducted based on a framework of evaluation criteria and nine information seeking tasks designed for the Dunhuang cultural heritage. The results help identify the strengths of the panorama DL in visual appearance and error correction, as well as areas for improvement, particularly in user control and two-way communication. From the results, we proposed suggestions for further improving this DL. It is our hope that, after the improvement, the DL can better convey the educational, research, and artistic values of the Dunhuang Mogao Grottoes, and at the same time offering a multipurpose and user-customized access experience.
The evaluation framework, the designed information seeking tasks, and the interview questions can be adopted or adapted in evaluating other DLs and digital platforms of Dunhuang heritage. Very recently, the Digital Dunhuang project released the e-dunhuang