Google Web Crawling

The web has grown in 10 years from 26 million unique URLs to 1 trillion unique URLs. This makes it a very challenging task for Google to crawl all the pages. The amazing part is that they do this a couple of times a day to keep their search engine uptodate. See the blog posting for more details

http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html

Comments

Popular posts from this blog

Spring Framework

Never use a String

JSON Web Token