When I think of Googlebot I prefer to think of a cute little robot speeding off wearing a hat. He’s on a quest to find information and create a knowledgeable index. The truth is disappointing though, it’s just a computer program developed by Google that scoots around adding pages to its index. Let’s take a look at what it does.
Google Bot? Crawler? Spider? What is it?
Everyone calls it something different but you will never
catch me calling it a spider…ever. All of these terms mean the same thing. It’s
a computer program that crawls web pages by using links. It finds updated and
new content, reads it, and suggests what should be added to the Google index
and what shouldn’t.
This index is essentially Google’s brain, which is where all
of the knowledge of the internet sits. Google uses a lot of computers to send
their crawlers out to scour every nook and cranny of the internet to find out
what is on them and how to best handle what they find. Every search engine has
its own crawler, but Googlebot is the one used by Google.
How Does Googlebot Work?
Googlebot uses databases and sitemaps that were previously
created during its crawls to decide where it should go next. After a crawler
finds new links on a site, it adds them to the list of pages to be visited
next.
When it finds links or even broken links, it will make an
instant note so the index can be updated in the system. Based on what is found,
the program will automatically determine how often it will crawl pages. To
ensure that Googlebot is able to index your website correctly you need to check
its crawlability. As long as your site is available, the crawlers should come
around often.
There are different robots that have different purposes. For
example, the Adsense and AdsBot check ad quality while the Mobile Apps android
checks Android Apps. We won’t go into detail on what they are called, just know
there are multiple crawlers and each one has a purpose.
How Googlebot Visits Your Site
If you want to find out how often Googlebot comes to your
site and what it's doing while it’s there, you can check your log files or open
the Crawl section of the Google Search console. If you have the capability or are
willing to learn how to get into the advanced stuff, you can optimize the crawl
performance of your site, you can check out tools like Screaming Frog and
Kibana.
The IP addresses of Googlebots are not released by Google. The IP addresses change frequently but you can do a reverse IP lookup. It is possible for spammers to spoof a user-agent name but they cannot fake an IP address. Google has actually listed a way for you to verify the validity of their Googlebot visits though.
0 comments:
Post a Comment