Thursday, March 03, 2011

What Is A Googlebot?

Seems I do have an admirer, but it's not a real person:

What is Googlebot?

Googlebot is Google's web crawling bot (sometimes also called a "spider"). Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.

We use a huge set of computers to fetch (or "crawl") billions of pages on the web. Googlebot uses an algorithmic process: computer programs determine which sites to crawl, how often, and how many pages to fetch from each site.

For most sites, Googlebot shouldn't access your site more than once every few seconds on average. However, due to network delays, it's possible that the rate will appear to be slightly higher over short periods. In general, Googlebot should download only one copy of each page at a time. If you see that Googlebot is downloading a page multiple times, it's probably because the crawler was stopped and restarted.

Googlebot was designed to be distributed on several machines to improve performance and scale as the web grows. Also, to cut down on bandwidth usage, we run many crawlers on machines located near the sites they're indexing in the network. Therefore, your logs may show visits from several machines at google.com, all with the user-agent Googlebot. Our goal is to crawl as many pages from your site as we can on each visit without overwhelming your server's bandwidth

Basically, that means that my blog posts are being archived by Google, which is probably why when Carrie Cann or Carrie's Bar & Grill are Googled, my blog and many of my posts show up in the search. So, in a way, it's a good thing, as it makes my blog more accessible to those interested in my stories, and as you can see, it went into overdrive this past week!




1 comment:

Carrie said...

While this "thing" shows up on my Sitemeter, it has zero registered visits on my Stat Counter. I'm beginning to think this is some kind of google bot (well, the ISP IS googlebot, LOL). I'm working on trying to figure out why Google is botting me.