« 404 Pages Made Easy | Home | Funny Pay Per Click Advert »
It would certainly seem that way as crawl rates seem to have plummeted dramatically in recent months. I came across the problem when working on a scrap car website for one of my clients and noticed that the changes I had made were taking a ridiculously long time to index. This prompted me to look at some of my other sites and low and behold it was happening on some of them too.
The graph below is from webmaster tools and shows crawl rates from month to month. As you can see crawl rates have slowed down for some reason.
So why are crawl rates so slow recently? Google has not given any official line of the recent trend but after doing some research on the topic some people are suggesting that Google’s bots have become more efficient and there is no need for them to crawl as frequently. I am not sure if I buy into that but will be interesting to see what happens over the coming months.
Topics: External Sites, Google, Search Engine Optimisation (SEO) | 6 Comments »
September 3rd, 2008 at 5:50 pm
I can’t see how the bots could be considered more efficient if they crawl less. How would they pick up on relevant change? Doesn’t make sense to me
September 3rd, 2008 at 5:58 pm
they were implying that although the bot is crawling less it is crawling more pages. I would not agree with the view either.
September 4th, 2008 at 5:06 pm
Maybe even Google bots need a summer break?
September 8th, 2008 at 6:30 pm
Ah, we think we’ve cracked this. Looks like a duplicate content issue. One of the clients other domains which is http://www.cash4scrapcars.com has been redirected to the main domain but using a server based DNS re-direct and returns a 200 OK message. We’ve asked for it to be changed and identified a few other domains where a similar technique was used.
March 7th, 2010 at 8:04 pm
I don’t see why a slightly dodgy redirect would reduce the crawling google does on your site. Did you notice a change once the problem was fixed? It looks like a random blip to me.
Did you guys achieve the current results for Metro Salvage? Looks like you did your job pretty well.
March 12th, 2010 at 10:51 am
Hi Tim,
Thanks for the comment. The crawl rate returned to normal after a while but its hard to say whether that was down to the redirect or not.
Thanks for the kind words on the Metro Salvage project. We worked pretty hard to get them the rankings they achieve. It also converts extremely well.