How to Increase Google's Crawl Frequency

Google Crawl Bot

I was a part of a much anticipated website redesign and relaunch recently. I fervently monitored post launch metrics, page crawl rate, 404s, indexed pages, and so on as any other anxious SEO professional would do.
While reviewing the pages crawled per day in Google Webmaster Tools, I noticed that out of the gate we had an instant crawl of nearly all site pages.


I passed this on to the client for which I got the quick reply of, “Why is it crawling more pages now than it used to crawl?"


Seeing this reminded me yet again of all the reasons why sound SEO practices on-site can help aid in crawl frequency enhancement.


(Note: Ideal page count is around 500 pages, some duplicate pages were quickly indexed and then roboted.)

Pages crawled per day

Through the redesign, we enacted several SEO elements, which have helped to allow and some instance entice crawling bots to frequent the site more often…and more pages at that. Let's examine how those elements increased Google's crawl frequency.

Why You Should Care About Crawl Frequency

SEO, to many, hinges upon attaining enhanced visibility for highly searched terms as well as referring this traffic to their sites. Taking our blinders off for a moment, there are a few things we have to remember.
We want to rank many pages on a site, not just the homepage. Additionally, we're actively making changes to our sites and we want bots to see this as quickly as possible and as deep within the site as possible.

 

Redesign/Site Migration or Not, No Excuses

As mentioned above the redesign effort did a good job of lending to the opportunity to enhance crawl frequency as so many good SEO changes were taking place at once. Additionally, so much more new content and refreshed content drives the bots nuts giving them so much more to want to peruse on the site, thanks Google Caffeine!

For many out there, you can’t enjoy the opportunity of creating a full scale redesign, platform change, and SEO overhaul of a site all at once. If this is you, then the list below is a working order of all the standard SEO practices you can work on to improve crawl frequency on your site.

 

Get ’em on the Site

• Run a DNS check, Ping and Traceroute check of the site to assess if there are any issues with site pages loading with regard to connectivity or any other server issues. Can the bots even access your pages?
• Run a page load speed report of your 10 most important pages to review how fast your pages are loading. Crawlers lack patience. Are you asking too much of them?
• Utilize parameter-free static/clean URLs on the site. Bots have long had issues with parameter crawling. Yes, they can often see their way through these now, but why not make it easier for them to crawl the site?

 

Hand Them the Keys to the Site

• Review your robots.txt file as well as your usage of meta robots tags. What pages are holding from them?
• Have an XML sitemap as well as HTML sitemap.
• Enlist supplemental navigation on-site (i.e., footer navigation, breadcrumb navigation, and relevant internal linking in copy). Create pathways to make a site easy to crawl.
• Fix internal links resulting in 404 errors as well as ensuring that external links open in new windows. You don’t want to stop the crawl and you don’t want to usher them away.

 

Entice & Lure Them

Generate fresh content! This may be the most important point in the checklist.

Give them a reason to feel they should come back on a regular basis. This doesn’t mean you need new content site-wide every month, but it does mean refreshing existing content on a quarterly basis and maintaining site sections – news, blog, etc. –that have continuously added content onto the domain.
Generate links and social citations to your site. This can be a large scale task in itself. Think of it this way: the more links you have out on the web, the greater your chances are of attracting crawling bots. Think of links as portals into your site.

Conclusion

As you can see, there are many components that aid in enhancing your bot crawl frequency and depth of crawl. These are also many of the foundational elements of SEO. This helps to reinforce that crawl frequency, is after all, a very important aspect of SEO itself.

Adhere to these recommendations and you will have a better optimized site and hopefully see an enhanced bot crawl rate.