Is Your Website Getting Indexed in Search Engines? Read this 4 Step Process.

A Short How-To On Identifying Indexation Problems during an SEO Audit
If you think you might be having trouble with indexation there are some simple checks to do.

1. Do a physical check of the pages and site structure. This could be done with a database tool or by hand, thus giving you a chance to review the copy of your site, because most likely you haven’t done that in a while.

2. Run a XML sitemap of domain.com to get a list of the URLs present on the site. It is important to use a web-based or desktop application to do this and not rely on an internal tool that generates the sitemap from a CMS database. Why you ask? External sitemap generators (Web-based or desktop) do not have access to the server so they must crawl the site from link to link, just like a search engine. This will give you a better understanding of what content is accessible and what is not.

3. Check your Analytics program. Run a report of all content on the site that has received non-paid Search Engine traffic over a period of time (how long depends on your site traffic levels).

4. Query “site:domain.com” in any Search Engine to get a list of the URLs from that domain that are included in the index. Also, check the www version of the site to see if there is any variance. Theoretically, these numbers should be near identical.

Now that you have this data, what do you do with it?
Now that you have this data, compare them to each other. In Step 1 you identified all the pages/URLs that exist on the site. Now, compare this list to Step 2 – if not all pages that physically exist on the site are present in the sitemap, then you have some investigation to do. This indicates that there may be some issue with the structure of the site that is preventing crawlers from reaching those pages.

Next, compare Step 3 with Step 1.
Are there pages that are present on the page but have never received any traffic from Search Engines? This is an indication that these pages may not be indexed by the Search Engines.

There are pages that you don’t really want traffic to, like your refund policy, or your list of pending lawsuits (note: if you have this on your site take it off). If your product or services page is not receiving traffic this is something that should be addressed.

Finally, compare Step 4 with Step 1.
What pages on the site are not indexed? If you see that a large number of pages are not included in Step 4, you may have an indexation issue. As a method for double-checking this issue, compare the pages missing from Step 4 with Step 3. If there are pages that are not currently in the index, but have gotten Search traffic in the past, these pages may have gotten de-indexed for some reason. Investigate why this might be, especially if you consider them mission-critical pages. Don’t overlook checking your robots.txt file. It is not uncommon to see large sections of a site disallowed, when they shouldn’t be.

How well is your site performing, contact us today to conduct a Technical SEO Site Assessment and start improving or rebuilding your online presence for maximum visibility?