<abstract xmlns="http://www.w3.org/1999/xhtml"><p>One of the research fields significantly affected by the emergence of “big data” is computational linguistics. A prominent example of a large dataset targeting this domain is the collection of Google Books Ngrams, made freely available, for several languages, in July 2009. There are two problems with Google Books Ngrams; the textual format (compressed with Deflate) in which they are distributed is highly inefficient; we are not aware of any tool facilitating search over those data, apart from the Google viewer, which, as a Web tool, has seriously limited use. In this paper we present a simple preprocessing scheme for Google Books Ngrams, enabling also search for an arbitrary n-gram (i.e., its associated statistics) in average time below 0.2 ms. The obtained compression ratio, with Deflate (zip) left as the backend coder, is over 3 times higher than in the original distribution.</p></abstract>

One of the research fields significantly affected by the emergence of “big data” is computational linguistics. A prominent example of a large dataset targeting this domain is the collection of Google Books Ngrams, made freely available, for several languages, in July 2009. There are two problems with Google Books Ngrams; the textual format (compressed with Deflate) in which they are distributed is highly inefficient; we are not aware of any tool facilitating search over those data, apart from the Google viewer, which, as a Web tool, has seriously limited use. In this paper we present a simple preprocessing scheme for Google Books Ngrams, enabling also search for an arbitrary n-gram (i.e., its associated statistics) in average time below 0.2 ms. The obtained compression ratio, with Deflate (zip) left as the backend coder, is over 3 times higher than in the original distribution.

Google Books Ngrams Recompressed and Searchable

Lodz University of Technology, Institute of Applied Computer Science, al. Politechniki 11, 90-924 Łódź, Poland

University of Szczecin, Institute of Information Technology in Management, Mickiewicza 64, 71-101 Szczecin, Poland

Foundations of Computing and Decision Sciences

{"article-title":"Google Books Ngrams Recompressed and Searchable"}

One of the research fields significantly affected by the emergence of “big data” is computational linguistics. A prominent example of a large dataset targeting...

Google Books Ngrams Recompressed and Searchable

Data publikacji: 22 gru 2012

Zakres stron: 271 - 281

DOI: https://doi.org/10.2478/v10209-011-0015-8

This content is open access.