I’m pretty sure that you are here to read about the facts and reasons why your site is not indexed by Google. There are many other people like you who are facing these issues while indexing their site. After a thorough research on this topic, we are here with the list of valid reasons and answers to each and every question that you have in your mind right now.
How To Identify Crawling Problems
The first step is to identify that your site is indexed by Google or not. You can check this by simply typing site:www.yoursite.com in the Google search box.
Now have a look on the Google search engine results, If you notice that there is a large gap in the number of pages in the result section Vs the number of pages that you have created on your website. Then you have to fix this issue.
You can get a list of all the indexed URL’s using SEO Quake.
Google Webmaster tool must be your first tool in your list, that you have to use to check your site’s issues. Check your search console in your dashboard to find out the issues related to your website. If there is an issue, then your dashboard will show you the errors.
Our site doesn’t have any errors, so you can see a screenshot of the errors on Neil Patel’s Site.
Have a look on the given below example:
The maximum number of times, you’ll see the 404 HTTP Status Code error. This error means that page doesn’t found.
Also, the status code of 200 and 301 means there is something wrong with your website and your website is not working as per the need of your visitors.
Here are some of the other tools to check server headers:
- Screaming Frog SEO Spider
- Moz Pro Site Crawl
How to fix the crawling errors
These issues can be caused by various reasons. Here is the list of the main reasons that lead to crawling errors.
This text file plays a very important role in the indexing process of your site. At the initial stage, this file has only one line i.e,
Above line is telling the Google bots to not index your website’s content. So if you want to index your website, then you have to edit your robot.txt file.
Usage of wrong attributes of Meta Tag can lead to the error due to which your site is not indexed by Google. You must always make sure that the Meta tag of the page should not have attributes like NOINDEX, NOFOLLOW.
Have a look on the below example:
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
If you want to start indexing of that particular page, then you have to remove these attributes.
Sometimes your sitemap stops updating due to some reasons and you are still submitting your old sitemap to the webmaster tool. So, In this case always check the webmaster errors carefully and make a fresh sitemap and re-submit it to the Google.
This is basically a hidden file which is present in the WWW or public_html folder of your website’s data. This file can badly affect your site’s indexing process. A nasty stuff like infinite loop can cause errors due to which your site is not indexed by Google.
If you have ever checked your webmaster tool dashboard very thoroughly then you have already seen the option named “URL Parameters”. This option tells the Google that which dynamic links you want to index or which you don’t want to index. So, if you want to use this tool then configure your parameters correctly. Otherwise, this can lead to resulting in dropping your website URL from Google SERP’s.
Less Page Rank
Matt Cutts in his interview revealed that the number of pages that are crawled by Google is proportional to the Page rank.
This issue can also lead to an error due to which your site is not indexed by Google. In this case, you have registered with the domain that has a life before you i.e, someone was using that domain in the previous years and you have bought this domain recently from any domain name service provider. In this case, your sitemap got accepted by the Google. But after some investigation, Google finds that domain was already used in the previous years and was a big part of spam at that time. So, To fix this issue you have to file a reconsideration request to Google.
If your server is down for maintenance, then this can also lead to an error due to which your site is not indexed by Google. In this case, your server is not responding to the Google bots. So, Google was not able to index your pages due to this issue.
Some of the other reasons due to which your site is not indexed by Google:
- Bad usage of the canonical tag.
- Duplicate content on your website.
- Usage of bad redirects.
- Slow loading speed of your website.
- Bad hosting service – Your site is going down again and again.
- Someone just spammed your website with Gazillion backlinks in a single day.
- Someone has spammed your website with a flag for not following the Google’s guidelines. If Google finds that you are violating their guidelines, then it will follow the manual penalty process.
- Your website consists of scraped content.
- The site is using shady link farm tactics.
- Your website is adding Zero value to the web (According to the Google, eg. Affiliate landing pages).