Raid of the Googlebots Options

[bwelford]

I wouldn't normally start a thread like this one, but I think there may be some significance to what is happening. Every morning I pop into the Webmaster World Forum to see any topic that is causing a lot of frenzied chatter. It's rather like that old habit that people used to have as they left the house in the morning of tapping the barometer to see whether the weather was likely to change.

The heated topic this morning is the Raid of the Googlebots. There is intense activity by new Googlebots to deeply and thoroughly crawl websites. I must admit that I have noticed much increased activity, and apparently these particular ones have not been seen before. There is much surmise that this indicates a recreation of the Google database.

Clearly we have all been subjected to the apparently mindless churning of the Microsoft robots over the past few months. So will we shortly see the launch of the new Microsoft Search Engine, as Ammon suggested in another thread, given a recent Microsoft hire in the UK. Is Google about to counter this with a major relaunch or revitalisation of its own?

For those interested in all this stuff, it certainly beats prime time television.

[Ruud]

First I read about it is at digitalpoint. The new bot is HTTP 1.1 (old one is HTTP 1.0). It seems to go very deep in one single pass, as oppossed to going in and coming back several times. It seems to be the final release of the experimental bot we've seen several times which was interested in JavaScript.

It's really a hungy one though and certainly doesn't limit itself to JavaScript only. On one of my sites where Google comes around almost daily for a 10-20MB meal this thing sucked up well over 300MB in one go!

Another notable thing; Google is asking for *very* old pages. It's the type of behaviour I associate with Slurp who tends to keep coming back for the same non-existing content over a very long time. Since about a week Google is doing this, asking me for pages which haven't been on sites in well over a year (estimation). And those sites are well spidered and well indexed: it's not like Google doesn't know those pages aren't there, lol.

Ruud

[andy_boyd]

I've been getting much more Googlebot activity across a range of sites, as opposed to the way it used to be. Some of the sites are in the sandbox and others aren't. But regardless of their position relative to the sandbox Googlebot now comes in everyday, roughly at the same time, and takes every page without fail. Before this new pattern of behaviour, Googlebot would only grab the index and leave.

This heightened activity seems to have spawned myriad theories on what they are up to at the Googleplex, including the predominant theory that after a failed experiment with 'sandoxing' new sites, they are now building an entirely new index.

The recent frantic pace of Googlebot would certainly indicate a change. However, whether it is as substantial as a new index with a new algo remians to be seen.

[bwelford]

Welcome to the Forums, andy_boyd.

The other possibility is that Google is trying to increase the detail of its Internet map so as to identify the linkages better. As Ammon said in another thread, this would mean that websites using doorway pages, cloaking and complex interlinking networks would be highlighted and presumably downrated. What with Microsoft on the brink of (?) an announcement, it's exciting times in the old corral.

[andy_boyd]

Thanks for the welcome!

Personally I think that to suggest Google is basically broken at the core is infeasible. There is no doubt that across all my sites Google has been going off the wall lately, spidering everything on a daily basis.

It's interesting that you mention MSN, because the people at Google did not just float down the river in a bubble and find themselves in control of the most marketable SE around today. In any business you get rumors floating round, people let slip and things that should be kept secret emerge. What I'm suggesting is that Google know what is coming, and they are bracing themselves for some serious competition.

If I had to make a guess, I would say that Google are battoning down their hatches and getting ready. If Yahoo / MSN start to take marketshare Google will need to come back with something bigger and better. Spidering the web so vigorously now means they will have a stronger foundation and an upper hand when they have to go toe to toe with the other big players. They are metaphorically standing their ground for now. But the huge map they are building will give them a nimble edge when the fight begins.

That doesn't mean I particularly like the way new sites are sandboxed :wink:.

[bobmutch]

bwellford: What is your take on how long a link has to be up before the PR vote will apply. I am concerned about a couple of new links I just got. I havn't seen any comments on that in the forums.