Google Index Checker: Keep track of your indexed pages

Are you sure that Google has indexed all of the pages on your website? Well, it probably hasn’t. But why should you keep track of your indexed pages and what does this mean for your website’s SEO? The truth is, it doesn’t matter how long you spent creating and optimising the perfect page for Google Search, there will be no chance of receiving traffic from organic search if the page has not been indexed. So it’s important that website owners keep a record of the pages that Google has crawled and indexed (Google might crawl a page but still not index it). However, this is easier said than done because there are currently no SEO tools that gather all the URLs indexed. But all is not lost!! Passion Digital has created it’s very own Google index checker outlined below.

What is Indexation?

Google scans the web using ‘crawlers’ which analyse pages and rank them in SERPs (search engine results pages) in order of their relevance to the user. By crawling the internet, Google also generates a repository of any new web-pages it finds. If Google hasn’t crawled and indexed your site properly, then you could be missing out on relevant traffic, and more importantly, potential customers.

Want to learn more about indexation and SEO, check out out our beginners SEO training course.

How to Check for Indexed Pages

If we open Google Webmaster / Google Index / Index Status, we can see the total number of indexed pages, but Google doesn’t offer the exact URLs for these pages. It should take around 30 minutes for a 500-page site. The more pages your site has, the more time it’ll take.

You’ll need 3 things:

1. Google Chrome
2. RDS bar extension for Google Chrome (helps us gather the full URL in the results)
3. Microsoft Excel

1. Right-click on the RDS bar icon, and click ‘Options’.

Open Search Tab/Google Search and choose ‘only full URLs’ and click ‘Apply’.

1. Open https://www.google.co.uk/ in Google Chrome.

Once you are in Google Chrome, we can begin the process of getting almost all the pages from the Google index step-by-step.

Knowing the Target

First of all, we need to set the search settings to the ’10-results’ default option.

Settings/Search settings

1. Type the following special search query in the browser:

Site: /

With this query, we’ll get the number of pages indexed by Google. So, we already have an idea of what our target is.

Note that the number of pages showing up could vary slightly over time. This number will never be a fixed one, so we cannot ensure that we’ll be getting 100% of them.

2. Now, we’ll be doing the same as above for each of the main folders of our site, so we get an accurate figure of how many pages there are in each of them. E.g.

Steps 1 and 2 are required due to the possible result variations that will appear when changing the search settings from 10 results to 100.

If we set the search settings to 100 results, Google will narrow the number of results down showing only relevant results when going through each of the pages in the results. See example below:

1^st Page

14^th Page

You might see this message appearing at some point if the search settings are set to 100 results:

Collecting the URLS

We need to set the search settings to the “50-results” default option. (Remember not to set the “100 results” option, as Google will reduce the number of results a lot)

Settings/Search settings

Site commands that we’ll be using:

1. Type the following special search query in the browser to start collecting the pages that are in each folder of your site.

Site: /blog/

Site: /services/

2. This site command will get those pages that are not in any of the main folders that we have previously searched – the orphan pages.

Site: / -inurl:/blog/ -inurl:/services/ (…and so on.)

Once we know which site command we should use, we’ll be doing the same process to copy the URLs to an excel document. In the results, the whole URL will be appearing thanks to the RSD extension so then we have just to copy the 50 results to an excel sheet.

Then by filtering the data by colour (our example is Grey), you will be getting just the URLs that you want. Copy-paste them into a new Excel sheet.

Filtered data

See below the template that I’ve been using to collect all the indexed pages

Thank you for reading and hope you have found it interesting.

If you’re interested in learning more SEO practices that could benefit your business, then take a look at our SEO training courses.