Application Research of Crawler and Data Analysis Based on Python

The process of crawler itself is actually to simulate the user’s operation on the browser with a program. First of all, the starting point and range of crawling need to be specified. As the target of crawling is for manufacturers and their works, the interface of manufacturers is taken as the starting point. For example, the page of paradox, a manufacturer, first analyzes the entire manufacturer’s page, and finds that the page links and information of all games or game related DLC downloads of the manufacturer are stored in the recommendation div framework of each sub recommendation of recommendations rows, as shown in Figure 2

Figure 2.

The crawler architecture is composed of items, spiders, piplings and middleware. Among them, items are mainly used to define the items to be crawled, spiders are responsible for defining the whole process of crawling, what means to crawl, pipes are responsible for the basic operations such as data cleaning and saving, middleware can be responsible for the bridge service of scratch and other plug-ins or architectures.

First, the items to be crawled are defined in the items file. Finally, these items may be submitted to the analysis part for data analysis. The specific design and implementation code is:

import scrapy

class SteamDevItem(scrapy.Item):

  # define the fields for your item here like:

  # name = scrapy.Field()

  qry_nam = scrapy.Field()

  if_dev = scrapy.Field()

  pub_sum = scrapy.Field()

  pub_gam_sum = scrapy.Field()

  pub_dlc_sum = scrapy.Field()

  dev_nam = scrapy.Field()

  pub_nam = scrapy.Field()

  gam_title = scrapy.Field()

  res_date = scrapy.Field()

  gam_type = scrapy.Field()

  gam_tag = scrapy.Field()

  if_muti = scrapy.Field()

  gam_score = scrapy.Field()

  gam_score_sum = scrapy.Field()

  gam_score_ratio = scrapy.Field()

pass

The design of spider is the key point of this project. Whether the initial dynamic page connection or the last static page information crawling mode will be defined in this file. In this project, spider will be named steam, and some key implementation codes will be pasted here, with running results and some notes attached. First, introduce start_ the design method of dynamic page crawling of selenium in requests method:

chrome_opt = webdriver.ChromeOptions()

   prefs = {

“profile.managed_default_content_settings.images”: 2,

   ’permissions.default.stylesheet’: 2

  }

   chrome_opt.add_experimental_option(“prefs”, prefs)

browser    =

webdriver.Chrome(options=chrome_opt)

   browser.get(“https://store.steampowered.com/” + Qry_sta + “/” + Qry_Target)

   bs  =  BeautifulSoup(browser.page_source, ‘html.parser’)  #Beautiful Soup

The specific store connections of each product exist in the a anchor label of each entry, and these connections are read to the defined links using the loop_ In the list list, crawling of the list is completed, but sometimes the text and picture in the entry may contain a tag, and they all point to the same page. If direct application may cause repeated crawling, a loop is used here, and if not in statement is used to de duplicate the list.

After using the print statement to verify the function of the module, the verification results are shown in Figure 3.

Figure 3.

After designing and debugging the spider, run the CMD command window of the system, open the root directory of the crawler file, and input the crawler stream-o SteamDev.csv, crawl the target website. Input - O SteamDev.csv The purpose is to let the crawler save the last crawled data in the form of CSV table. The saved data appears in the project root. See Figure 4 for the climbing process.

Figure 4.