Google plans to reduce webpage crawl rate

Google may reduce the frequency of crawling web pages as it becomes more aware of the sustainability of crawling and indexing.

This topic is covered by Google’s Search Relations team, consisting of John Mueller, Martin Splitt and Gary Illyes.

Together, in the latest episode of the Search Off the Record podcast, they discuss what to expect from Google in 2022 and beyond.

Among the topics they cover are crawling and indexing, which SEO professionals and website owners say they’ve seen less of over the past year.

This is going to be a key goal for Google this year as it aims to make exploration more sustainable by conserving computing resources.

Here’s what it will mean for your website and its performance in search results.

Sustainability of crawling and indexing

Since Googlebot’s crawling and indexing is done virtually, it’s not something you might think would impact the environment.

Illyes draws attention to this problem when he says that computing is not sustainable in general:

“…what I mean is that computing, in general, is not really sustainable. And if you think of bitcoin, for example, bitcoin mining has a real impact on the environment that you can actually measure, especially if the electricity comes from coal-fired power plants or other less sustainable power plants.

We’re carbon free, since I don’t even know, 2007 or something, 2009, but that doesn’t mean we can’t reduce our environmental footprint even further. And crawling is one of those things that, at first, we could cut fruit at hand.

Low-hanging fruit, in this case, refers to unnecessary web crawling. Like crawling web pages that haven’t had recent updates.

How will Google make crawling more sustainable?

Illyes goes on to explain that web crawling can be made more sustainable by reducing refresh ramp.

There are two types of Googlebot crawling: crawling to discover new content and crawling to refresh existing content.

Google plans to reduce the crawl to refresh the content.

Illyes continues:

“…one thing we do, and maybe we don’t need to do as much, is refresh analytics. Which means once we discover a document, a URL, we go there come on, we crawl it, and then, eventually, we’ll go back and revisit that URL. It’s a refresh crawl.

And then every time we come back to that URL it will always be a refresh scan. Now, how often do we need to come back to this URL? »

He goes on to give an example of some websites that warrant a significant number of refresh crawls for some parts of the site but not others.

A website like the Wall Street Journal constantly updates its homepage with new content, so it deserves a lot of refreshes.

However, WSJ probably doesn’t update its About page as frequently, so Google doesn’t need to continue to perform refresh crawls on these types of pages.

“So you don’t need to go back that much. And often we can’t estimate that well, and we certainly have room for improvement on refreshes. Because sometimes it just seems pointless for us to hit the same URL over and over again.

Sometimes we reach 404 pages, for example, for no good reason or no apparent reason. And all of those things are basically things that we could improve on and then reduce our footprint even further. »

If Google were to reduce refresh crawls, which is not 100% confirmed, here is the impact it could have on your website.

What does a reduced crawl rate mean for your website?

There is a belief that a high crawl rate is a positive SEO signal, even if you don’t update your content as often as Google crawls it.

This is a misconception, says Illyes, because the content won’t necessarily rank higher because it gets more crawled.

Muller:

“So I guess that’s also kind of a misconception that people have in that they think that if a page is crawled more it will rank better. Is that correct that’s a misconception false, or is it really true?”

Illies:

“That’s a misconception.”

Muller:

“OK, so no need to try to force something to be re-explored if it doesn’t actually change. It’s not going to rank better.

Again, it’s unconfirmed that Google will reduce refresh crawls, but it’s an idea the team is actively considering.

If Google follows through on this idea, it won’t be a bad thing for your website. More crawling does not mean better rankings.

Additionally, the idea is to learn which pages need refresh crawls and which pages don’t. This means that the pages you edit most often will continue to refresh and update in search results.

For more details on how Google plans to achieve this, listen to the full discussion in the podcast below (from 2:40):


Feature image: Alena Veasey/Shutterstock

Comments are closed.