1. bookVolume 38 (2022): Issue 1 (March 2022)
    Special Issue on Price Indices in Official Statistics
Journal Details
License
Format
Journal
eISSN
2001-7367
First Published
01 Oct 2013
Publication timeframe
4 times per year
Languages
English
access type Open Access

Estimating Weights for Web-Scraped Data in Consumer Price Indices

Published Online: 29 Mar 2022
Volume & Issue: Volume 38 (2022) - Issue 1 (March 2022)<br/>Special Issue on Price Indices in Official Statistics
Page range: 5 - 21
Received: 01 May 2020
Accepted: 01 Apr 2021
Journal Details
License
Format
Journal
eISSN
2001-7367
First Published
01 Oct 2013
Publication timeframe
4 times per year
Languages
English
Abstract

In recent years, there has been much interest among national statistical agencies in using web-scraped data in consumer price indices, potentially supplementing or replacing manually collected price quotes. Yet one challenge that has received very little attention to date is the estimation of expenditure weights in the absence of quantity information, which would enable the construction of weighted item-level price indices. In this article we propose the novel approach of predicting sales quantities from their ranks (for example, when products are sorted ‘by popularity’ on consumer websites) via appropriate statistical distributions. Using historical transactional data supplied by a UK retailer for two consumer items, we assessed the out-of-sample accuracy of the Pareto, log-normal and truncated log-normal distributions, finding that the last of these resulted in an index series that most closely approximated an expenditure-weighted benchmark. Our results demonstrate the value of supplementing web-scraped price quotes with a simple set of retailer-supplied summary statistics relating to quantities, allowing statistical agencies to realise the benefits of freely available internet data whilst placing minimal burden on retailers. However, further research would need to be undertaken before the approach could be implemented in the compilation of official price indices.

Keywords

Auer, J., and I. Boettcher. 2017. “From Price Collection to Price Data Analytics: How New Large Data Sources Require Price Statisticians to Re-think their Index Compilation Procedures. Experiences from Web-Scraped and Scanner Data.” In International Working Group on Price Statistics: Fifteenth Meeting of the Ottawa Group, May 10–12, 2017. Eltville am Rhein: Ottawa Group. Available at: https://www.ottawa-group.org/Ottawa/ottawagroup.nsf/4a256353001af3ed4b2562bb00121564/1ab31c25-da944ff5ca25822c00757f87/$FILE/From%20price%20collection%20to%20price%20-data%20analytics%20-Josef%20Auer,%20Ingolf%20Boettcher%20-Paper.pdf (accessed February 2020). Search in Google Scholar

Cavallo, A. 2017. “Are Online and Offline Prices Similar? Evidence from Large Multi-Channel Retailers.” American Economic Review 207(1): 283–303. DOI: https://doi.org/10.1257/aer.20160542.10.1257/aer.20160542 Search in Google Scholar

Cavallo, A., and R. Rigobon. 2016. “The Billion Prices Project: Using Online Prices for Measurement and Research.” Journal of Economic Perspectives 30(2): 151–178. DOI: https://doi.org/10.1257/jep.30.2.151.10.1257/jep.30.2.151 Search in Google Scholar

Chevalier, J., and A. Goolsbee. 2003. “Measuring Prices and Price Competition Online: Amazon.com and BarnesandNoble.com.” Quantitative Marketing and Economics 1(2): 203–222. DOI: https://doi.org/10.1023/A:1024634613982.10.1023/A:1024634613982 Search in Google Scholar

Hisano, R,. and T. Mizuno. 2010. “Sales Distribution of Consumer Electronics.” Physica A: Statistical Mechanics and its Applications 390(2): 309–318. DOI: https://doi.org/10.1016/j.physa.2010.09.033.10.1016/j.physa.2010.09.033 Search in Google Scholar

Hull, I., M. Lof and M. Tibblin. 2017. “Price Information Collected Online and Short-Term Inflation Forecasts.” In IFC – Bank Indonesia Satellite Seminar on Big Data: International Statistical Institute Regional Statistics Conference, March 20–24, 2017. Bali: International Statistical Institute. Available at: https://www.bis.org/ifc/publ/ifcb44e.pdf (accessed February 2020). Search in Google Scholar

ILO, International Labour Office. 2020. Consumer Price Index Manual: Concepts and Methods, Draft January 2020. Geneva: International Labour Office. Available at: https://www.imf.org/~/media/Files/Data/CPI/cpi-manual-concepts-and-methods.ashx?la=en (accessed May 2020). Search in Google Scholar

Nygaard, R. 2015. “The Use of Online Prices in the Norwegian Consumer Price Index.” In International Working Group on Price Statistics: Fourteenth Meeting of the Ottawa Group, May 20–22, 2015. Urayasu City: Ottawa Group. Available at: https://www.ottawagroup.org/Ottawa/ottawagroup.nsf/4a256353001af3ed4b2562bb00121564/d012f001b8a1cf6cca257eed008074c9/$FILE/Ragnhild%20Nygaard%20(Statistics%20Norway-%20The%20use%20of%20online%20prices%20in%20the%20Norwegian%20Consumer%20Price%20Index.pdf (accessed February 2020). Search in Google Scholar

ONS, Office for National Statistics. 2019a. Retail Sales, Great Britain: April 2019. Newport: Office for National Statistics. Available at: https://www.ons.gov.uk/businessindustryandtrade/retailindustry/bulletins/retailsales/april2019 (accessed December 2020). Search in Google Scholar

ONS, Office for National Statistics. 2019b. Introducing Alternative Data Sources into Consumer Price Statistics. Newport: Office for National Statistics. Available at: https://www.ons.gov.uk/economy/inflationandpriceindices/articles/introducingalternativedatasourcesintoconsumerpricestatistics/2019-05-09 (accessed May 2020). Search in Google Scholar

ONS, Office for National Statistics. 2020a. Consumer Price Inflation Basket of Goods and Services: 2020. Newport: Office for National Statistics. Available at: https://www.ons.gov.uk/releases/consumerpriceinflationbasketofgoodsandservices2020 (accessed May 2020). Search in Google Scholar

ONS, Office for National Statistics. 2020b. Retail Sales, Great Britain: April 2020. Newport: Office for National Statistics. Available at: https://www.ons.gov.uk/businessindustryandtrade/retailindustry/bulletins/retailsales/april2020 (accessed December 2020). Search in Google Scholar

ONS, Office for National Statistics. 2020c. Consumer Prices Development Plan: Updated March 2020. Newport: Office for National Statistics. Available at: https://www.ons.gov.uk/economy/inflationandpriceindices/articles/consumerpricesdevelopmentplan/updatedmarch2020 (accessed May 2020). Search in Google Scholar

Polidoro, F., R. Giannini, R.L. Conte, S. Mosca and F. Rossetti. 2015. “Web Scraping Techniques to Collect Data on Consumer Electronics and Airfares for Italian HICP Compilation.” Statistical Journal of the IAOS 31(2): 165–176. DOI: https://doi.org/10.3233/sji-150901.10.3233/sji-150901 Search in Google Scholar

Stanley, M.H.R., S.V. Buldyrev, S. Havlin, R.N. Mantegna, M.A. Salinger and H.E. Stanley. 1995. “Zipf Plots and the Size Distribution of Firms.” Economics Letters 49(4): 453–457. DOI: https://doi.org/10.1016/0165-1765(95)00696-D.10.1016/0165-1765(95)00696-D Search in Google Scholar

Ten Bosch, O., and R. Griffioen. 2016. “On the Use of Internet Data for the Dutch CPI.” In Meeting of the Group of Experts on Consumer Price Indices: United Nations Economic Commission for Europe, May 2–4, 2016. Geneva: United Nations. Available at: https://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.22/2016/Session_2_Netherlands_on_the_use_of_internet_data_for_the_Dutch_CPI.pdf (accessed February 2020). Search in Google Scholar

Touzani, S., and R.V. Buskirk. 2015. “Estimating Sales and Sales Market Share from Sales Rank Data for Consumer Appliances.” Physica A: Statistical Mechanics and its Applications 451(1): 266–276. DOI: https://doi.org/10.1016/j.physa.2016.01.030.10.1016/j.physa.2016.01.030 Search in Google Scholar

Van Loon, K., and D. Roels. 2018. “Integrating Big Data in the Belgian CPI.” In Meeting of the Group of Experts on Consumer Price Indices: United Nations Economic Commission for Europe, May 7–9, 2018. Geneva: United Nations. Available at: https://www.unece.org/fileadmin/DAM/stats/documents/ece/ces/ge.22/2018/Belgium.pdf (accessed February 2020). Search in Google Scholar

Willenborg, L. 2017. Elementary Price Indices for Internet Data. The Hague: Statistics Netherlands. Available at: https://www.cbs.nl/en-gb/background/2017/25/elementary-price-indexes-for-internet-data (accessed December 2020). Search in Google Scholar

Recommended articles from Trend MD

Plan your remote conference with Sciendo