Google’s Gary Illyes Explains

On Monday the 16th of January, Gary Illyes, the Google Webmaster Trend Analyst tweeted about a blog post he had just posted:

For any technical SEO, the concept of ‘crawl budget’, ‘pagerank’, ‘link juice’ or any of numerous other alias have been commonly bandied around for years. All of the above are regular terms that a client will likely ask about when embarking on a new project, and if they haven’t, their tech team certainly will.

For me, this article was very pleasing to read, as in just a few succinct bullet points, it validates a career of advice and consultancy, and gives substantial evidence to cite when dealing with commonly held myths or excuses for a certain situation!

The piece is also an interesting read, as it confirms many technical basics – Illyes explains how Googlebot will adapt so as not to degrade other users’ experiences. A term he also uses at multiple points throughout is ‘crawl demand’, and what influences it. This is new term to me, and so I paid particular attention.

I gleaned that, essentially, there is more crawl demand if:

  • There has been a site move, or a fresh influx of URLs. This can be looking for new content or can be wasted if you make a mistake.
  • There has been an increase in the freshness of content. Either if you are regularly producing new content, or you have popular content that needs to be checked to see if it has changed.

It seems as though ‘crawl limit’ and ‘crawl demand’ are similar to my own, simplified ‘old-now’ definition of pagerank (that I have been using forever), which means the frequency and depth of your crawl.

Illyes continues to explain how not to loose your ‘crawl budget’, through avoiding the following points:

  • Faceted navigation, session variables, unnecessary variables in the URL
  • On-site duplicate content, aka ‘thin pages’
  • Soft 404s and other avoidable distractions
  • Redirect chains and ‘spider traps’
  • Hacked pages / malware / anything dangerous

Thank you Gary Illyes – it is always great to read posts that confirm what you think you know and have deduced throughout an entire career!