|
Google Crawl Bot |
I was a part of a much anticipated
website redesign and relaunch recently. I fervently monitored post launch metrics, page crawl rate, 404s, indexed pages, and so on as any other anxious
SEO professional would do.
While reviewing the pages crawled per day in
Google Webmaster Tools, I noticed that out of the gate we had an instant crawl of nearly all site pages.
I passed this on to the client for which I got the quick reply of, “Why is it crawling more pages now than it used to crawl?"
Seeing this reminded me yet again of all the reasons why sound SEO
practices on-site can help aid in crawl frequency enhancement.
(Note: Ideal page count is around 500 pages, some duplicate pages were quickly indexed and then roboted.)
|
Pages crawled per day |
Through the redesign, we enacted several SEO elements, which have
helped to allow and some instance entice crawling bots to frequent the
site more often…and more pages at that. Let's examine how those elements
increased Google's crawl frequency.
Why You Should Care About Crawl Frequency
SEO, to many, hinges upon attaining enhanced visibility for highly
searched terms as well as referring this traffic to their sites. Taking
our blinders off for a moment, there are a few things we have to
remember.
We want to rank many pages on a site, not just the homepage.
Additionally, we're actively making changes to our sites and we want
bots to see this as quickly as possible and as deep within the site as
possible.
Redesign/Site Migration or Not, No Excuses
As mentioned above the redesign effort did a good job of lending to
the opportunity to enhance crawl frequency as so many good SEO changes
were taking place at once. Additionally, so much more new content and
refreshed content drives the bots nuts giving them so much more to want
to peruse on the site, thanks Google Caffeine!
For many out there, you can’t enjoy the opportunity of creating a
full scale redesign, platform change, and SEO overhaul of a site all at
once. If this is you, then the list below is a working order of all the
standard SEO practices you can work on to improve crawl frequency on
your site.
Get ’em on the Site
- Run a DNS check, Ping and Traceroute check of the site to assess if
there are any issues with site pages loading with regard to connectivity
or any other server issues. Can the bots even access your pages?
- Run a page load speed report of your 10 most important pages to
review how fast your pages are loading. Crawlers lack patience. Are you
asking too much of them?
- Utilize parameter-free static/clean URLs on the site. Bots have long
had issues with parameter crawling. Yes, they can often see their way
through these now, but why not make it easier for them to crawl the
site?
Hand Them the Keys to the Site
- Review your robots.txt file as well as your usage of meta robots tags. What pages are holding from them?
- Have an XML sitemap as well as HTML sitemap.
- Enlist supplemental navigation on-site (i.e., footer navigation,
breadcrumb navigation, and relevant internal linking in copy). Create
pathways to make a site easy to crawl.
- Fix internal links resulting in 404 errors as well as ensuring that
external links open in new windows. You don’t want to stop the crawl and
you don’t want to usher them away.
Entice & Lure Them
Generate
fresh content! This may be the most important point in the checklist.
Give them a reason to feel they should come back on a regular basis.
This doesn’t mean you need new content site-wide every month, but it
does mean refreshing existing content on a quarterly basis and
maintaining site sections – news, blog, etc. –that have continuously
added content onto the domain.
Generate links and social citations to your site. This can be a large
scale task in itself. Think of it this way: the more links you have out
on the web, the greater your chances are of attracting crawling bots.
Think of links as portals into your site.
Conclusion
As you can see, there are many components that aid in enhancing your
bot crawl frequency and depth of crawl. These are also many of the
foundational elements of SEO. This helps to reinforce that crawl
frequency, is after all, a very important aspect of SEO itself.
Adhere to these recommendations and you will have a better optimized site and hopefully see an enhanced bot crawl rate.