Google Web Crawling
The web has grown in 10 years from 26 million unique URLs to 1 trillion unique URLs. This makes it a very challenging task for Google to crawl all the pages. The amazing part is that they do this a couple of times a day to keep their search engine uptodate. See the blog posting for more details
http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html
http://googleblog.blogspot.com/2008/07/we-knew-web-was-big.html
Comments