Wednesday, June 29, 2022

Published June 29, 2022 by with 0 comment

SEO Basics - What is Crawlability?


If you want your website to rank in search engines, it is important that you build a flawless SEO plan and create relevant, helpful content for your readers. 

Understanding basic SEO is important to ensure that you outrank your competition and stay at the top of the search results. One important aspect of SEO is crawlability.

What is the Crawler? 

Search engines, like Google, consist of what is known as a crawler, an index, and an algorithm. Crawlers follow links. Google’s crawler for example is known as Googlebot. Its job is to find your website, render it, read it, and save the content in its index. 

The crawler follows the links on the internet. Other names you may see the crawler referred to are the robot, a bot, or a spider (Which is something I will never call it). Its job is to go around the internet 24 hours a day. Once the crawler comes to a website, it saves the HTML version of the site to a massive database referred to as the index. 

This index is updated every time the crawler comes around your website and finds a revised or new version of it. The frequency that the crawler comes around your website will depend on how important it thinks your site is and how frequently you make changes. Basically, how you interact with your website property will determine how often it comes around.

What is Crawlability?

Crawlability is a term used to determine the possibilities Google does have to crawl your website. You can even block crawlers on your site. There are a few ways that you can block crawlers from coming to your website or a specific page on your website. Keep in mind that doing this will be directly telling Google not to come to your page which will mean it won’t show up in search results. 

There are a few things that will prevent Google from crawling or indexing your website. These include: 

Your robots.txt file blocks the crawler. Google will not come to your website, or the page specifically listed.

Before crawling your website, the Google crawler looks at the HTTP header of your page. This header contains a status code that crawlers rely on. If the status code has been deleted or doesn’t exist, Google will not crawl your website. 

If the robots meta tag on a given page blocks the engine from indexing it, the crawler will still navigate through it, but will not add the page to its index. 



Post a Comment