<abstract xmlns="http://www.w3.org/1999/xhtml"><p>Combined with the actual situation, this paper explores how to develop a crawler method based on the specific framework for the complete interface of steam manufacturers and stores, which should be able to automatically and efficiently crawl the data of specific targets, analyze the dynamic pages, and complete the data cleaning, downloading, saving and other operations, explore the methods of general data analysis, and Analyze the downloaded data, extract useful information from it, analyze and summarize the specific crawler method and data analysis method through practical application.</p></abstract>

Combined with the actual situation, this paper explores how to develop a crawler method based on the specific framework for the complete interface of steam manufacturers and stores, which should be able to automatically and efficiently crawl the data of specific targets, analyze the dynamic pages, and complete the data cleaning, downloading, saving and other operations, explore the methods of general data analysis, and Analyze the downloaded data, extract useful information from it, analyze and summarize the specific crawler method and data analysis method through practical application.


<div xmlns="http://www.w3.org/1999/xhtml"><sec id="j_ijanmc-2020-018_s_001"><label>I.</label><div>INTRODUCTION</div><p>The 21st century is a book written by information. With the rapid development of information technology, today’s society has become a huge information polymer, and there are various kinds of data in this huge polymer. Data is a kind of embodiment of information. In this era of information explosion, how to efficiently find the data we want from all kinds of miscellaneous data and extract them from the network in batches has become a key problem. However, sometimes the unprocessed data itself may be confusing for people. How to process the huge and complex data obtained through what kind of technical means, and finally become an intuitive number, or trend, and become the information that people can obtain intuitively is also a very important topic to be studied in this data age.</p></sec><sec id="j_ijanmc-2020-018_s_002"><label>II.</label><div>STATISTICAL INVESTIGATION ON THE PREFERENCE SALES VOLUME</div><p>In this project, the American Steam online game platform mall is selected as the research object of the crawler. By setting a specific game company as a search keyword in steam’s online mall, the data of all works of the company in steam mall are crawled, and the useful information is extracted by analyzing the basic data of each manufacturer’s preference for game production type, series sales volume, and praise In addition, the game manufacturers are comprehensively scored and evaluated.</p></sec><sec id="j_ijanmc-2020-018_s_003"><label>III.</label><div>RELEVANT TECHNOLOGY AND FRAMEWORK</div><p>This project will use the scrapy framework based on Python language to crawl steam website. Python as a language has the advantages of lightweight, simplicity, wide range of application and so on. At present, various crawler frameworks and application libraries based on Python have been very mature, among which the crawler framework is very popular in the application of general web crawlers. Its first version was released in 2008, and now it is quite mature as a crawler framework. The basicprinciple of the scrapy framework is shown in Figure <a ref-type="fig" href="#j_ijanmc-2020-018_fig_001">1</a>.</p><figure id="j_ijanmc-2020-018_fig_001" position="float" fig-type="figure"><h2>Figure 1.</h2><figCaption><p>Basic principles of Scrapy frame</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_ijanmc-2020-018_fig_001.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/6471f5c4215d2f6c89db6d45/j_ijanmc-2020-018_fig_001.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240705T072256Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240705%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=9c3289079137b56f865033e7b22aa236f5095dfc483ad4508aab2f473645ccce" class="mw-100"/></figure></sec><sec id="j_ijanmc-2020-018_s_004"><label>IV.</label><div>DESIGN OF CRAWLER</div><sec id="j_ijanmc-2020-018_s_004_s_001"><label>A.</label><div>General design idea</div><p>The process of crawler itself is actually to simulate the user’s operation on the browser with a program. First of all, the starting point and range of crawling need to be specified. As the target of crawling is for manufacturers and their works, the interface of manufacturers is taken as the starting point. For example, the page of paradox, a manufacturer, first analyzes the entire manufacturer’s page, and finds that the page links and information of all games or game related DLC downloads of the manufacturer are stored in the recommendation div framework of each sub recommendation of recommendations rows, as shown in Figure <a ref-type="fig" href="#j_ijanmc-2020-018_fig_002">2</a></p><figure id="j_ijanmc-2020-018_fig_002" position="float" fig-type="figure"><h2>Figure 2.</h2><figCaption><p>Investigation of HTML page structure of steam manufacturers by using viewers</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_ijanmc-2020-018_fig_002.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/6471f5c4215d2f6c89db6d45/j_ijanmc-2020-018_fig_002.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240705T072256Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240705%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=d934144d8519819c484a3d32414cac53723cd2504d774db02c37b16930d97081" class="mw-100"/></figure></sec><sec id="j_ijanmc-2020-018_s_004_s_002"><label>B.</label><div>Design and implementation of reptile functions</div><p>The crawler architecture is composed of items, spiders, piplings and middleware. Among them, items are mainly used to define the items to be crawled, spiders are responsible for defining the whole process of crawling, what means to crawl, pipes are responsible for the basic operations such as data cleaning and saving, middleware can be responsible for the bridge service of scratch and other plug-ins or architectures.</p><p>First, the items to be crawled are defined in the items file. Finally, these items may be submitted to the analysis part for data analysis. The specific design and implementation code is:</p><p>import scrapy</p><p>class SteamDevItem(scrapy.Item):</p><p>  # define the fields for your item here like:</p><p>  # name = scrapy.Field()</p><p>  qry_nam = scrapy.Field()</p><p>  if_dev = scrapy.Field()</p><p>  pub_sum = scrapy.Field()</p><p>  pub_gam_sum = scrapy.Field()</p><p>  pub_dlc_sum = scrapy.Field()</p><p>  dev_nam = scrapy.Field()</p><p>  pub_nam = scrapy.Field()</p><p>  gam_title = scrapy.Field()</p><p>  res_date = scrapy.Field()</p><p>  gam_type = scrapy.Field()</p><p>  gam_tag = scrapy.Field()</p><p>  if_muti = scrapy.Field()</p><p>  gam_score = scrapy.Field()</p><p>  gam_score_sum = scrapy.Field()</p><p>  gam_score_ratio = scrapy.Field()</p><p>pass</p></sec><sec id="j_ijanmc-2020-018_s_004_s_003"><label>C.</label><div>Spider design</div><p>The design of spider is the key point of this project. Whether the initial dynamic page connection or the last static page information crawling mode will be defined in this file. In this project, spider will be named steam, and some key implementation codes will be pasted here, with running results and some notes attached. First, introduce start_ the design method of dynamic page crawling of selenium in requests method:</p><p>chrome_opt = webdriver.ChromeOptions()</p><p>   prefs = {</p><p>“profile.managed_default_content_settings.images”: 2,</p><p>   ’permissions.default.stylesheet’: 2</p><p>  }</p><p>   chrome_opt.add_experimental_option(“prefs”, prefs)</p><p>browser    =</p><p>webdriver.Chrome(options=chrome_opt)</p><p>   browser.get(“https://store.steampowered.com/” + Qry_sta + “/” + Qry_Target)</p><p>   bs  =  BeautifulSoup(browser.page_source, ‘html.parser’)  #Beautiful Soup</p><p>The specific store connections of each product exist in the a anchor label of each entry, and these connections are read to the defined links using the loop_ In the list list, crawling of the list is completed, but sometimes the text and picture in the entry may contain a tag, and they all point to the same page. If direct application may cause repeated crawling, a loop is used here, and if not in statement is used to de duplicate the list.</p><p>After using the print statement to verify the function of the module, the verification results are shown in Figure <a ref-type="fig" href="#j_ijanmc-2020-018_fig_003">3</a>.</p><figure id="j_ijanmc-2020-018_fig_003" position="float" fig-type="figure"><h2>Figure 3.</h2><figCaption><p>List of URLs obtained by selenium and beautiful soup</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_ijanmc-2020-018_fig_003.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/6471f5c4215d2f6c89db6d45/j_ijanmc-2020-018_fig_003.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240705T072256Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240705%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=9d10fabce97ef19d6f1cd2a0f8fcdd751a9daf4b7adb20403898992d7a9e9e20" class="mw-100"/></figure></sec><sec id="j_ijanmc-2020-018_s_004_s_004"><label>D.</label><div>Start directional climbing</div><p>After designing and debugging the spider, run the CMD command window of the system, open the root directory of the crawler file, and input the crawler stream-o SteamDev.csv, crawl the target website. Input - O SteamDev.csv The purpose is to let the crawler save the last crawled data in the form of CSV table. The saved data appears in the project root. See Figure <a ref-type="fig" href="#j_ijanmc-2020-018_fig_004">4</a> for the climbing process.</p><figure id="j_ijanmc-2020-018_fig_004" position="float" fig-type="figure"><h2>Figure 4.</h2><figCaption><p>Executing the start request method selenium pop-up browser to crawl the dynamic page</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_ijanmc-2020-018_fig_004.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/6471f5c4215d2f6c89db6d45/j_ijanmc-2020-018_fig_004.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240705T072256Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240705%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=d7b73e6eb775f7c27f31c94a43080bd21066d3a98181a0a76a1685860cd40916" class="mw-100"/></figure></sec></sec><sec id="j_ijanmc-2020-018_s_005"><label>V.</label><div>DATA ANALYSIS</div><p>Next, we will perform basic visual operations on the crawled data in the form of operation tables. In the crawler project, we crawled for the Paradox Interactive publisher. The crawled data is presented in the form of CSV tables, as shown in Figure <a ref-type="fig" href="#j_ijanmc-2020-018_fig_005">5</a>.</p><figure id="j_ijanmc-2020-018_fig_005" position="float" fig-type="figure"><h2>Figure 5.</h2><figCaption><p>Crawled data list</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_ijanmc-2020-018_fig_005.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/6471f5c4215d2f6c89db6d45/j_ijanmc-2020-018_fig_005.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240705T072256Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240705%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=f03d9d2c6279a074406511ec95f112b5ec3c43e0bff52f274d03b259f1c27f84" class="mw-100"/></figure><figure id="j_ijanmc-2020-018_fig_006" position="float" fig-type="figure"><h2>Figure 6.</h2><figCaption><p>Output the publisher platform follower ranking chart</p></figCaption><img xmlns:xlink="http://www.w3.org/1999/xlink" xlink:href="graphic/j_ijanmc-2020-018_fig_006.jpg" src="https://sciendo-parsed.s3.eu-central-1.amazonaws.com/6471f5c4215d2f6c89db6d45/j_ijanmc-2020-018_fig_006.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&amp;X-Amz-Date=20240705T072256Z&amp;X-Amz-SignedHeaders=host&amp;X-Amz-Expires=18000&amp;X-Amz-Credential=AKIA6AP2G7AKP25APDM2%2F20240705%2Feu-central-1%2Fs3%2Faws4_request&amp;X-Amz-Signature=ea19ccca277c17950a72fa48eceb7ca82e93eca58ccd0f49bff3650114800b7a" class="mw-100"/></figure><p>Through the use of spreadsheets and further collation of the crawled data, the following data are obtained: the publisher has published 396 works in steam platform, of which the majority of DLC has published 334 DLC, most of the games published are single player games, and each game published in its mall has an average of 6800 reviews, of which the proportion of favorable reviews is about 76.4 8%, see the chart below for detailed visual analysis.</p></sec><sec id="j_ijanmc-2020-018_s_006"><label>VI.</label><div>CONCLUSION</div><p>Through demonstration and part of practice, this paper explores the process of data crawling and basic data analysis of dynamic pages by combining the general Python’s story framework with selenium + beautiful soup through crawling the steam online game mall website.</p><p>The crawler has good scalability. For example, if you want to compare the crawling data of multiple game manufacturers, you can write a query manufacturer list to get the product URL list from the dynamic web page of the manufacturer list first. In terms of anti-crawler, selenium itself has a very good anti crawler ability. If you want to further anti crawler, you can also expand multiple cookies, and even establish a proxy IP pool.</p></sec></div>

Application Research of Crawler and Data Analysis Based on Python

International Journal of Advanced Network, Monitoring and Controls

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

{"article-title":"Application Research of Crawler and Data Analysis Based on Python"}

Combined with the actual situation, this paper explores how to develop a crawler method based on the specific framework for the complete interface of steam...

Application Research of Crawler and Data Analysis Based on Python

Published Online: Jul 13, 2020

Page range: 64 - 70

DOI: https://doi.org/10.21307/ijanmc-2020-018

Keywords
Python, Scrapy, Selenium, BeautifulSoup

© 2020 Wu Hejing et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Application Research of Crawler and Data Analysis Based on Python

Published Online: Jul 13, 2020

Page range: 64 - 70

DOI: https://doi.org/10.21307/ijanmc-2020-018

KeywordsPython, Scrapy, Selenium, BeautifulSoup

© 2020 Wu Hejing et al., published by Sciendo

This work is licensed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.

Keywords
Python, Scrapy, Selenium, BeautifulSoup