Sentiment analysis of reviews on cappadocia: The land of beautiful horses in the eyes of tourists
, et | 31 janv. 2024
Article Category: Research Article
Publié en ligne: 31 janv. 2024
Pages: 188 - 197
Reçu: 06 sept. 2023
Accepté: 28 août 2023
DOI: https://doi.org/10.2478/ejthr-2023-0015
Mots clés
© 2023 Özge Barış-Tüzemen et al., published by Sciendo
This work is licensed under the Creative Commons Attribution 4.0 International License.
Social media has an important role as a source of information for travellers around the world. Social networks have led to widespread changes in “business-to-business”, “business-to-customer” and “customer-to-customer” communications. The Internet has evolved from a broadcast medium to a participatory platform that allows people to themselves be the “media” for collaboration and information sharing (Leung et al., 2013). According to a 2021 Statista report, there were 859 million reviews on TripAdvisor as of 2019 (Statista, 2021b). In July 2019 alone, 224 million people visited the TripAdvisor website. The website Booking.com, another popular and well-known travel platform, was visited 697 million times in the same period (Condor Ferries, 2021). In addition, Twitter had an average of 330 million active users per month in 2019 (Statista, 2021a).
These statistics clearly show that the dominance of social media in our lives has increased. With this increase, it is seen that corporate communication has become democratized (Kietzmann et al., 2011). The ability to share information through social media has created significant changes in the bargaining power of consumers and decreased information asymmetries. Stories and experiences shared by people across many online platforms (blogs, microblogs, Twitter, Facebook, TripAdvisor, Booking.com, YouTube, etc.) have made social media into a megatrend, significantly affecting the tourism sector (Leung et al., 2013).
The tourism industry, whose stakeholders are more than in many other industries, is the largest job-creating sector on the planet. Developing technology and the internet play a significant role for tourism organizations and destinations in increasing their market share. Advances in search engines and social networks have increased the number of people who plan and experience their travels around the world. These developments have also affected the efficiency of tourism organizations and their business models, as well as the way consumers communicate with these organizations. There have been many new players in the tourism sector since these developments, and the importance of tourism for a growing number of national and regional economies has been increasingly recognized (UNWTO, 2001; Buhalis & Law, 2008). In short, the internet and social networks have significantly changed the distribution channels of tourism-related information, as well as travel planning and consumption habits (Xiang & Gretzel, 2010).
Tourism is both a service and a knowledge-intensive industry. Therefore, it is essential to understand how changes in technologies and consumer behaviour are affecting the distribution and availability of travel-related information. Technology and social media are needed to create an avant-garde tourism experience. Information and communication technologies are powerful tools for reshaping tourism as they create totally new products, communication networks, business models, industry structures, and types of companies. Therefore, they are an important resource for travel companies seeking an enhanced competitive advantage and maximum profit in the global market (Sheldon, 2006).
There is extensive literature on social media analytics that combines web scraping, machine learning applications with recent software, and statistical techniques to collect, clean, analyse, and understand large amounts of data. With these methods, opinions and beliefs about products can be examined for commercial purposes, such as tracking and identifying trending topics and popular sentiment (Xiang et al., 2017). In particular, with social media playing an increasingly in both travellers’ decisions and tourism activities and management, studies are increasingly focusing on the effects of user ratings and comments on tourism destinations and hotel management practices (see Vermeulen and Seegers, 2009; Ye et al., 2009; Xiang et al., 2008; Schmunk et al., 2014; Schuckert et al., 2015; Fang et al., 2016; Geetha et al., 2017; Cheng and Jin, 2019; Kyaw and Wai, 2019; Nohh et al., 2019; Büyükeke et al., 2020; Dina, 2020; Paolanti et al, 2021; Özen, 2021).
Turkey is in the Mediterranean climate zone, and its clean beaches and coves, historical sites, and natural beauty all endow it with great potential for tourism. Turkey has hosted many civilizations throughout history and offers a wide range of options to visitors of all preferences and tastes. Each one of its seven regions has its own unique natural sites. Therefore, the development of the tourism sector could play an important role in solving socio-economic problems in Turkey, such as unemployment, inflation, and migration (Bulut, 2018). Turkey, which welcomed 51.2 million tourists in 2019, is ranked 6th in the world in terms of number of arrivals. However, the country ranks 13th in the world in terms of tourism revenues, which stood at 29.8 billion dollars in 2019 (Republic of Turkey Ministry of Culture and Tourism, 2020).
The Cappadocia region, which encompasses the Nevsehir, Aksaray, Nigde, Kayseri, and Kirsehir provinces, is one of Turkey’s most important tourism destinations. The Rocky Cappadocia region, which is a narrower area, consists of Uchisar, Urgup, Avanos, Goreme, Derinkuyu, Kaymakli, Ihlara, and their surroundings. It is an important tourism destination of both natural and historical interest and has been listed as a UNESCO World Heritage Site since 1985. Natural and cultural tourism is popular in the region, encompassing experiences such as thermal baths, ballooning, horse-back riding, and pottery, just to name a few (Ahiler Development Agency Plans for Future, 2015; Republic of Turkey Ministry of Culture and Tourism, 2021). Common activities include hiking and horseback riding in Ihlara; visiting the Red and Rose Valleys, or the Göreme or Zelve open-air museums; and taking a hot air balloon tour over the region.
Analysing social media data in the tourism industry will allow both local businesses and policymakers to understand the decision-making behaviours of visitors and to be aware of the opportunities and pitfalls in the industry. In this context, we investigated the social media data of the Rocky Cappadocia region through sentiment analysis and various machine-learning methods.
In the next part of the study, we briefly introduce social media and sentiment analysis methods. Later, we explain sentiment classification and machine learning algorithms. In the fourth section, we discuss the data set and findings, and finally, we present the conclusion and evaluations.
When using social media, people interact with each other and share information, opinions, and many other things freely. At the same time, companies and organizations may reach customers more easily and enjoy advantages such as offering them personalised options. According to Appel et al. (2019), social media sites should be seen as “digital places” where people manage important parts of their lives, rather than just platforms that offer digital media and technology services. When evaluated from this perspective, social media will mean “less about specific technologies or platforms and more about what people do in these environments” (Appel et al., 2019, p. 80).
On the other hand, reviews made by social media users cannot be treated as the single source of truth. False information can spread very quickly on social media. Thus, users need to pay attention to this kind of information and always double-check the information they find on social media.
Sentiment analysis is a method for analysing emotions by evaluating posts on internet platforms. Making inferences from the related platforms on the feelings and opinions of people provides an understanding of human behaviour, with implications for efficiency in areas such as international public influence, business decisions, and policy development (Alamoodi et al., 2020). Sentiment analysis has many other applications in education, health, and tourism management, among many other areas (Balahadia et al., 2016; Alayba et al., 2017; Ye et al., 2009; Saura et al., 2019). It is fast and effective for making large-scale inferences and decisions at a low cost (Choi et al., 2017). For these reasons, it is an essential tool for understanding social trends.
To analyse the general public opinion about the Cappadocia region, we used sentiment analysis, an effective and frequently-used method in big-data analytics. In sentiment analysis, the main idea in a text is classified by applying natural-language processing and text analytics. Sentiment analysis aims to understand the attitude of the author by detecting the emotional polarity of a text and classifying it as positive, negative, or neutral (Luo et al., 2013, pp. 53–54). For this, a dictionary-based emotion score is determined for each word in the text using flexible and open-source programming languages such as R, Python, and related packages. Later, this score, determined on the basis of the selected words, is calculated for the whole text. As a result, the entire text can be classified as positive, negative, or neutral (Luo et al., 2013).
Sentiment analysis studies are generally performed with three different methods: the lexicon-based method, the machine learning method and the hybrid approach. The lexicon-based approach “detects the sentiment based on a sentiment lexicon, which includes a collection of known and pre-compiled sentiment terms. It is divided into two, namely dictionary-based approach and corpus-based approaches, using statistical or semantic methods to find sentiment polarity” (Medhat et al., 2014, p. 1098). Machine learning approaches apply machine learning algorithms by taking advantage of syntactic and/or linguistic features, including sentiment lexicons (Maynard & Funk, 2011). The hybrid approach combines both machine learning and a lexicon-based method with manually written linguistic rules. The classifiers in this approach are used gradually. Thus, if one method cannot classify a document, the algorithm passes to the next method until the document is classified or all methods are tried (Prabowo & Thelwall, 2009; Manda, 2019). In this study, opinions of tourists about the Cappadocia region, which is one of the most popular tourist destinations in Turkey, is examined using machine learning algorithms.
Machine learning techniques for sentiment classification have gained attention because of the possibility of modelling many features while also capturing the context, adapting easily to changing inputs, and measuring the degree of uncertainty with which a classification is made (Boiy & Moens, 2008). There are two basic kinds of machine learning algorithms: supervised and unsupervised learning. “A supervised learning algorithm takes a known set of input data (training set) and known responses to data (output) and trains a model to generate reasonable predictions for response to new input data.” (MathWorks, 2016, p.2). Unsupervised learning involves pattern recognition without the involvement of a target feature. In other words, all variables are used as features (Alloghani et al., 2019). Three different supervised machine learning algorithms are employed in the study. These classification algorithms are Naïve Bayes (NB), Support Vector Machines (SVM), and Random Forest (RF).
Naïve Bayes, which is the simplest example of a probabilistic classifier, is the probability (P (C | d)) that a
“Support Vector Machine is a supervised machine algorithm that uses statistical learning theories. It works as separating the classes in the dataset with an optimal hyperplane that maximizes the margin between classes” (Yu et al., 2012, p.232). In other words, SVM maximizes the distance between support vectors belonging to different classes. “The interesting property of SVM is that it is an approximate implementation to the structure risk minimization (SRM) in statistical learning theory, rather than the empirical risk minimization method from which the classification function is derived by minimizing mean square error (MSE)” (Song et al., 2002, p.440). As a powerful machine learning algorithm, SVM works well in text classification, as it has good generalization ability in a high-dimensional feature space (Joachims, 1998; Wei et al., 2012; Nohh et al., 2019).
Random forests are an important bagging modification that create a collection of de-correlated trees and then averages them. Random forests are used to improve bagging variance reduction by scaling down the correlation among trees without changing the variance too much. This is done by randomly selecting input variables during the tree growing process. “On many problems, the performance of random forests is very similar to boosting, and they are simpler to train and tune” (Hastie et al., 2009, pp. 587–588). “It gives good results in data sets that contain categorical variables with a large number of variables and class labels, have missing data or exhibit an uneven distribution” (Aydın, 2018, p.172).
In the current study, the success of classification algorithms is decided by confusion matrix, mean absolute error (MAE) and root mean square error (RMSE) criteria. The confusion matrix gives the actual and predicted classification data computed from a classification system. The performance of these kinds of systems is often determined using the data in the matrix. A two-class classification problem confusion matrix is shown below:
Accuracy is the statistic that determines the success of the machine learning model. As seen in Equation (1), the accuracy rate is the ratio of data classified as correct to all data:
The Confusion Matrix for Two-Class Classification Problem
| “Positive” | TP | FP |
| “Negative” | FN | TN |
“Note: TP is True Positive, FP is False Positive, FN is False Negative, and TN is True Negative.” Source: Visa et al. (2011)
In this study, we collected social media reviews on Göreme National Park and Göreme Open Air Museum in the Rocky Cappadocia region from the TripAdvisor website. The collection of reviews, also known as data scraping, was done using the Python programming language. A total number of 4,770 English-language reviews were collected from April 2011 to January 2021. In order to analyse the reviews, first we converted the text into numbers. To do that, we used the “term frequency-inverse document frequency” (TF-IDF) method, which supposes that a document is just a stack of words. Therefore, the relative importance of any word in the document can be calculated, that is, vectorized, by considering the frequency of the word in the document and its popularity in the compilation (Kim et al., 2019, p.17). We retained only the most common 2,000 words in the dataset, as less common words do not play a crucial role in classification. In addition, a word needed to be in at least five reviews, and at most in 70% of all the reviews. The rationale behind choosing 70% as a threshold was that words that occur in more than 70% of documents are very common and unlikely to play any role in classifying sentiment. Finally, we created a term document matrix using the CountVectorizer module.
We applied data cleaning before creating the term document matrix. After that, we converted all characters in the reviews to lowercase and removed punctuation and numeric characters. Then, we removed the words that do not significantly affect the meaning of the sentence using the StopWords application and lemmatized the text to map the various forms of the words to the root form using WordNet Lemmatizer. Two classifications, positive and negative, were made according to the ratings of the reviews. We labelled reviews that have ratings of 1, 2 and 3 as negative, and 4 and 5 as positive. 70% of the data set was used as the training set, and the remaining part was used for the test set.
In this part of the study, we make conclusions about the machine learning algorithm results and interpretations regarding the sentiment polarity of tourists who visited Göreme Open Air Museum and Göreme National Park. In this context, first, the distribution of reviews by rating is shown in Figure 1. According to Figure 1, the tourist reviews of Cappadocia generally show satisfaction with the experience that they have had. In the rating distribution of the reviews, the comments with ratings of 4 and 5 constitute approximately 90% of the total reviews, while the remaining ratings compose 10%.
Figure 1:
Distribution of Reviews According to Rating
Source: Authors’ computation.
Figure 2 shows the length of the reviews according to the rating. 5-star reviews have the lowest median size, while the highest median sizes are for 1 and 2-star reviews. In addition, considering the minimum number of characters, it can be observed that 1- and 2-star comments are longer than those given other rating values. The greater length of the negative reviews is included in the figure.
Figure 2:
Review Length by Rating
Source: Authors’ computation.
In Table 2, some of the negative and positive reviews of the Cappadocia Region are given. As can be seen from the table, positive reviews are generally shorter than negative ones. On the other hand, staff is the aspect of the location most frequently complained about in negative reviews. This indicates that the businesses in the region should take steps to increase the quality of their service by taking into account these reviews.
Reviews and Rating
| “DISSAPOINTED WITH THE STAFF – VERY RUDE, RUDE STAFF.” | “My son had to use the toilet as soon as we entered. So we went back out to the toilets that were at the entrance in the shop. They wouldnt let us back in, they wanted us to buy new tickets. They even called the ploice on us.” | 1 |
| “A total scam” | “We paid £200 expecting a private driver with a English speaking guide instead we got a miserable driver that didn’t even communicate with us and didn’t even have the curtesy to even get out of the car…no guide and we were left to our own devices!!!completely ruined our whole…” | 1 |
| “Wonderfull but local authorıtıes dont take care of thıs museum,” | “I was shocked when I saw such a pıece of neglected hıstory and no care from yhe local authorıtıes that made me almost call the police. People takıng peaces of the caves and takıng home can you beleıve ıt?Apart of ıt certainly ıt a place to vısit.” | 1 |
| “Disappointed with the visit” | “We visited Zelve Open Air Museum the day before, which is why the Goreme Open Air Museum was very disappointing for us. By comparison, GoremeOAM was 3x more expensive than Zelve, most of the sites inside Goreme OAM was closed, too many people were around…” | 1 |
| “Beautiful, Very beautiful place.” | “I went there 2 months ago. It was great trip. My guide told me history about the place. It is really great” | 5 |
| “Amazing” | “One of the best place in Cappadocia, let me write correctly, It is one of the best place in World.” | 5 |
| “Stunning,” | “Absolute stunning frescoes in the dark church… Well worth the extra 8 lira, A must see in Cappadocia and an easy walk from the towns center” | 5 |
| “Amazing,” | “Göreme Open Air Museum is a magnificent open air park, ideal for a guided tour. Many caves are great for exploring. It is good to see ‘churches’ in some of these caves.” | 5 |
Source: TripAdvisor
The most frequently mentioned bigrams all appear in the reviews in Figure 3. In fact, the most common bigrams in the text are “open air,” “national park,” and “goreme air.” However, since the reviews of Göreme Open Air Museum and Göreme National Park are being examined within the scope of the study, these words, which are likely to be seen in every review, have been removed from the chart. Apart from this, one of the most important buildings of Göreme Open Air Museum, the Dark Church, is the most mentioned bigram in the reviews. Rocks composed of lava and ash are also frequently mentioned in the reviews.
Figure 3:
Top 20 Bigrams in Reviews
Source: Authors’ computation.
Finally, Figure 4 shows the frequency of the 20 most common trigrams in the reviews. According to the chart, tourists like the Cappadocia Region and it is seen that the Dark Church is mentioned a lot in trigrams as well. The popularity of the hot air balloon ride, one of the important activities in the region, is also reflected in the reviews, and the local people living in caves were also the focus of tourists’ interest.
Figure 4:
Top 20 Trigrams in Reviews
Source: Authors’ computation.
Performance evaluation of the classification algorithms is examined using the micro-average values of F1-Score, Recall, and Precision statistics, and the outcome is presented in Table 3. While the macro-average treats all classes equally, the micro-average supports larger classes. Thus, in cases where there are class imbalances, the micro-average value is preferred (Sokolova & Lapalme, 2009; Nohh et al., 2019). sssss
Performance Evaluation
| “RF” | 90.91 | 0.9091 | 0.9091 | 0.9091 | 0.7764 | 0.18169 |
| “SVM” | 90.98 | 0.9098 | 0.9098 | 0.9098 | 0.7749 | 0.1802 |
| “NB” | 91.05 | 0.9105 | 0.9105 | 0.9105 | 0.7734 | 0.1788 |
Source: Authors’ estimation.
Table 3 also shows the RMSE and MAE results. The degree of success achieved by each of the three algorithms is very similar. However, Naïve Bayes algorithm performed the classification process with the highest accuracy rate. At the same time, the lowest RMSE and MAE values belonged to the Naïve Bayes algorithm. Based on these results, we found the Naïve Bayes algorithm to be preferable for classifying the reviews.
In this study, we analysed sentiment about the Cappadocia region, which is one of Turkey’s most important tourism destinations. We employed various machine learning algorithms and visualization methods for this purpose. We collected 4,770 English reviews for Göreme Open Air Museum and Göreme National Park in the Cappadocia region. For classification purposes, we carried out Random Forest, Naïve Bayes, and Support Vector Machine supervised machine learning methods. We used accuracy, RMSE and MAE criteria for performance evaluation. Although the classification algorithms performed very similarly, we determined that the Naïve Bayes algorithm was the most successful among them.
Most of the reviews (90%) indicated that their writers had positive feelings about the region. Tourism revenues in both the regional and the national economy are one of the most important components of Turkey’s GDP. Cappadocia, with its fairy chimneys, valleys, underground cities, churches, inns, and cultural structures, carries the traces of a civilization thousands of years old. It can be seen from the chart in the findings section, churches are mentioned in most of the reviews, meaning religious tourism is popular. The Dark Church, in particular, was at the centre of tourists’ attention. Strategic plans should be made to attract more visitors from various parts of the world by preserving the spirit and potential of the Cappadocia Region. The number and focus of negative reviews show that it is easy to eliminate the negative factors with some small improvements in customer service. The success of the tourism sector will eventually contribute to the income level of the locals as well.
Language is a powerful mechanism that enables people to express themselves. Studies on sentiment analysis, also known as opinion mining, are one of the most interesting and rapidly-growing research areas of natural language processing and text mining. A wide range of sentiment analyses can be made for every field through social media reviews, which are a crucial artifact of the information age. Social media allows individuals and organizations to reach larger audiences at much less cost than surveys, interviews, and similar methods. In addition, on social platforms, people share their feelings and thoughts in line with their own wishes. This causes these opinions to be more candid than others gathered by more conventional methods. It is thought that sentiment analysis on the tourism sector using social media reviews will help countries and businesses make better strategic decisions for the future. In addition, it will prevent the information-asymmetry problem in the global competitiveness, and thus enhance awareness of opportunities and threats in the market.
This study reveals the overall opinion of the visitors about Cappadocia region and thus serves as a mirror for service providers in the region. To be able to improve the quality perceptions of the region, locals, service providers and decision makers should take these reviews into account and keep increasing service quality accordingly. A significant limitation of this study, however, is that only a certain part of the Cappadocia Region is investigated. Thus, as a suggestion for further investigations, expanding the research to the whole region, or to other tourist destinations, may further reveal and contribute to the tourism potential of Turkey.