When I think of Googlebot I prefer to think of a cute little robot speeding off wearing a hat. He’s on a quest to find information and create a knowledgeable index. The truth is disappointing though, it’s just a computer program developed by Google that scoots around adding pages to its index. Let’s take a look at what it does.

Google Bot? Crawler? Spider? What is it?

Everyone calls it something different but you will never catch me calling it a spider…ever. All of these terms mean the same thing. It’s a computer program that crawls web pages by using links. It finds updated and new content, reads it, and suggests what should be added to the Google index and what shouldn’t.

This index is essentially Google’s brain, which is where all of the knowledge of the internet sits. Google uses a lot of computers to send their crawlers out to scour every nook and cranny of the internet to find out what is on them and how to best handle what they find. Every search engine has its own crawler, but Googlebot is the one used by Google.

How Does Googlebot Work?

Googlebot uses databases and sitemaps that were previously created during its crawls to decide where it should go next. After a crawler finds new links on a site, it adds them to the list of pages to be visited next.

When it finds links or even broken links, it will make an instant note so the index can be updated in the system. Based on what is found, the program will automatically determine how often it will crawl pages. To ensure that Googlebot is able to index your website correctly you need to check its crawlability. As long as your site is available, the crawlers should come around often.

There are different robots that have different purposes. For example, the Adsense and AdsBot check ad quality while the Mobile Apps android checks Android Apps. We won’t go into detail on what they are called, just know there are multiple crawlers and each one has a purpose.

How Googlebot Visits Your Site

If you want to find out how often Googlebot comes to your site and what it's doing while it’s there, you can check your log files or open the Crawl section of the Google Search console. If you have the capability or are willing to learn how to get into the advanced stuff, you can optimize the crawl performance of your site, you can check out tools like Screaming Frog and Kibana. 

The IP addresses of Googlebots are not released by Google. The IP addresses change frequently but you can do a reverse IP lookup. It is possible for spammers to spoof a user-agent name but they cannot fake an IP address. Google has actually listed a way for you to verify the validity of their Googlebot visits though.