You’d be forgiven for thinking that after you’ve set up your site, you could be waiting a good while before Google takes notice of it and crawls it. However, using technology supplied by Google yourself, you can get your site indexed fast; sometimes, it only takes a couple of days.
What is indexing and crawling?
When your site is live for the first time, the so-called ‘Google-bot’ will need to find your web pages in order to become aware of your site’s existence on the web.
After the crawl comes the index. Once the bot has found the page, it will add the page to other crawled pages which reside within the same category. This allows Google to more efficiently bring up relevant results in the SERPs. Search engines retain an index of your site pages – these are essentially screenshots of pages the last time they happened to be crawled, and can be seen with a ‘cached:’ advanced search operator.
So getting your site indexed is essentially the process of getting the attention of that bot.
How do I get my site indexed?
Step one: Set up your Google accounts
If you don’t already have one, set up a Google Webmaster Tools account for your site, and set up Google Analytics while you’re at it. Here’s why you ought to have Google Webmaster Tools, and a starter’s guide on Google Analytics.
Once you’ve done that, you can submit your URL to Google directly from Webmaster Tools here.
In order to keep tabs on whether your site has been crawled on which pages have been crawled, you can check ‘crawl stats’ under the Crawl tab.
Step two: Upload your XML sitemap
Sitemaps are used by search bots to access your website’s deeper content, so create a sitemap and submit it to Google via your Webmaster Tools account.
Step three: Make a Google+ profile
This is a somewhat overlooked step, but if you add posts to Google+ containing links to your site and add a link to your site in your About section, Google will take notice.
Step four: Take note of your internal linking
If your pages are well interlinked amongst each other, pages connected to your homepage will be indexed too. I wouldn’t recommend more than 150 links on any page though; include the main navigation bar and footer links.
Step five: Keep on top of freshness signals
Add content to your site frequently, and you’ll attract the search bots. Bots are encouraged to index your site the more content your put on there – as long as it is good content, that is.
Why is my site not being indexed by Google?
Your site may not be indexed for a while, depending on whether or not you have good links to your site, and whether you have a responsive server and have some good content. However, it might be worth checking to see if your site falls prey to any of these issues:
Meta Robots – Check in your source code for this code:
This will account for one particular page not getting indexed, so do a Ctrl+F within your source code and change it to ‘dofollow’ and ‘index’.
.htaccess File – You may have noticed that this site was down for about two weeks while I attempted to move to a self-hosted site. One of the reasons it was such a nightmare was because of errors within the .htaccess file. Always make sure it is configured correctly – you can read about that here.
Sitemap Errors – Submit your sitemap to Google, and Webmaster Tools will notify you if there are errors which need fixing. Find out how here.
Your Server – You might need to check your server for any issues, such as incorrect DNS configuration or blocking Googlebot’s access. You will be notified of this in Webmaster Tools.
You can use Screaming Frog to do a crawl of your site and find out if you have any 404s or redirect loops which are making things difficult for Googlebot.