Google's Fresh Crawl Options

I just came across this thread over at WebmasterWorld. There's a post in it by Googleguy that was written on the 27th and goes a long way to explaining what we've all been noticing recently - Googlebot hitting sites almost daily, that Google adds new pages (such as this forum) to the index very quickly, and that the listings aren't all that stable.

In his post, Googleguy stated that Google now do what they call a "fresh crawl", which runs on an almost daily basis. They are still experimenting with it, but what it means is that they crawl new pages and sites, though not necessarily as deeply as they could (they are still experimenting with how deep to crawl), and include the new stuff in the serps straight away. New sites and pages can show up in the serps in a very short space of time, but the listings are a bit unstabe because they are kept seperate from the main index and not included in it. When the main crawl comes around, the new sites and pages will be spidered and included in the main index during the next update.

In his post he also says to submit sites via their AddUrl page. That was the topic of a recent discussion where somebody was insisting that they completely ignore submissions from the AddUrl page.

Phil.

Hmm good spotting Phil.

Interesting that he said to submit via their Add URL. Guess it can't hurt.

Notice that he says to submit via the AddUrl page. That was the topic of a recent discussion where somebody was insisting that they completely ignore submissions from the AddUrl page.

Erm. Yes. So it must be true. Could you spot my sarcasm? :-)

What is this industry reliance on everything Googleguy says? Of course, if they can get more people to use the add url form then they have a great way of identifying a list of people who are optimising their pages don't they?! Anyone remember the old "optimising filter" that hit mom and pop sites months and months and months ago?

Erm. Yes. So it must be true. Could you spot my sarcasm?  

What is this industry reliance on everything Googleguy says?

Look at it this way. Whose opinion is more likely to be reliable - a Google employee who actually knows what goes on, or a surfer who only has an opinion?

Phil.

There's something about a Google representative appearing on a web forum that I don't like. As much as I appreciate that they are giving people information and trying to answer questions, I share Chris' sense of skepticism here.

I'd feel a lot more comfortable with Google if these pronouncements from Googleguy showed up on the pages of Google.

The fresh crawl stuff does correspond with what I've been seeing.

I think you're missing something Bill. On their site, Google imply that their AddUrl page is a way to submit sites to them. The fact that the page is there is statement enough. No engine has ever needed to persuade people that their AddUrl form actually works.

If I understaood Chris correctly, he previously argued that Google ignore submissions via that page. Now he's implying that they don't ignore them at all, but that they use them for some underhanded actions:-

Of course, if they can get more people to use the add url form then they have a great way of identifying a list of people who are optimising their pages don't they?!

This is nonsense, of course. Why would they want to identify people (which they can't do from the AddUrl form)? They only need to identify spam and deal with it as they see fit. I assume he means "spam" and not "optimising".

Phil.

GoogleGuy's note about using their "Add URL" submission option was indicated for "average" webmasters. Experienced ones, and those who know how to optimize via SEO/SEO, learn through trial and error and eventually experience that submitting may not be necessary at all. Some people use the free submission as an "insurance policy" for their own piece of mind, while others aren't worried and focus on linkage and building quality pages worthy of links and traffic, which attracts Google.

I don't take GoogleGuy's words as Google Gospel, for in my mind, he's been coached because he's representing his boss. He says what they want him to say and what they think people want to hear. They want to hear that free submission via the Add URL page is available to them and a worthy action to perform.

I'm more inclined to pay attention to what Brett Tabke says because he's been monitoring and testing all these years and shares his findings. He's not bound to any one company.

I'm not sure what Chris meant with the reference to the Add URL option being used to spot optimized pages, but the inference is that the Add URL is a spy tool. (If I got that correctly?) If this has been his experience or he's aware that it's been the experience of others, I'm thankful he's remembered to bring it up into the conversation.

Information is vital but in the end we each determine our own path.

Kim

Whose opinion is more likely to be reliable - a Google employee who actually knows what goes on, or a surfer who only has an opinion?

I'd opt for the surfer for two reasons:

1. They are more likely to have an overall viewpoint.
2. They don't come with the agenda.

If GoogleGuy isn't leading people to doing what would most benefit them even at the expense of webmasters then he isn't doing his job.

If I understaood Chris correctly, he previously argued that Google ignore submissions via that page. Now he's implying that they don't ignore them at all, but that they use them for some underhanded actions

That's the problem with forums. Months down the line somebody implies what you said without reading it. From the original thread:

Now if you'd argued that it's not evidence that submissions don't do anything, then you would be right. However, you said \"that's not evidence that submissions don't work\" - which of course it is.

This is nonsense, of course. Why would they want to identify people (which they can't do from the AddUrl form)? They only need to identify spam and deal with it as they see fit. I assume he means \"spam\" and not \"optimising\".

Phil, what's the point here? OF COURSE THEY CAN. You have to submit your domain don't you? Ever heard of whois? Sheesh. And yes, several times including when they put in the optimization filter that hit mom and pop sites they said it was because of OPTIMISING. Incidentally, wasn't it Google themselves that said 80% or something of submissions are spam?

Now Phil, if you'd spend less time attacking me (including the ridiculous one you've now editted, thanks) and more time reading the forums then I wouldn't have had to have bothered with this post. I wouldn't have had to have corrected what you thought/wanted me to have said with what I actually said.

Just for the record, my objections to anything Googleguy says are as follows:

1) He rarely says anything specific, when he does it always seems to be when he will derive most benefit from people knowing he does actually work for Google. He's the SEO world equivalent of mystic meg.

2) Everything he says benefits Google, but doesn't always benefit webmasters.

3) He gives a brief version of the webmaster FAQ and precedes it with something everyone knows anyway and people oooh and ahhh (see original quote).

4) He visits during working hours and from a Google IP address yet claims to be doing it in his own time.

5) He has strong anti-spam tendancies, because that's his job, but he doesn't have a complete overall picture. However, if he talks about penalties then for some reason people presume he knows all.

6) If his goal wasn't to get webmasters to do what Google want then his only other purpose would be public relations, something which they do poorly on generally and something which they could do much better from elsewhere.

Or to put it another way, if you want to trap a bear do you take the bear's word on how to trap it?

To Phil, and webmasters/SEOs in general I say: quit falling over youselves because he is a Google employee. And start asking - Why?

For what it's worth, I think that Chris is right to a certain extent, but I also don't think that things are quite as nefarious as he makes it out to be (I think nefarious is the word I mean!).

Yes, obviously GoogleGuy has Google's best interest at heart, and certainly it's his duty to spread Google propaganda. But that doesn't mean that everything he says isn't true either.

My guess is it's somewhere in between.

Jill

chris,

I'd opt for the surfer for two reasons: 

1. They are more likely to have an overall viewpoint. 
2. They don't come with the agenda.

1. Wrong. An overall viewpoint is irrelevant. What actually happens is what we are talking about, and only Google people know what actually happens.
2. Wrong. Many people come with agendas. For instance, I may make a statement that is challenged. As a matter of pride, my agenda may be to argue, come hell or high water, that my statement was correct, regardless of evidence or common sense. Personally, I don't do that but some people do.

Surfers only have opinions, which are based on what we deduce, but our deductions are often flawed.

If GoogleGuy isn't leading people to doing what would most benefit them even at the expense of webmasters then he isn't doing his job.

You are of the opinion that GoogleGuy is only "doing his job" when he posts at WMW. You are entitled to that opinion, but the evidence suggests otherwise. Check out the times and dates of his posts.

I notice that you quoted what suited you from your earlier posts. How about this:-

I remember mentioning that Google's submit form doesn't do anything way back...

You were clearly arguing that submissions are ignored.

Phil, what's the point here? OF COURSE THEY CAN. You have to submit your domain don't you? Ever heard of whois? Sheesh. And yes, several times including when they put in the optimization filter that hit mom and pop sites they said it was because of OPTIMISING.

Ok. They can find out information about a person from what is submitted via their form. But it's completely unnecessary. You said that the form gave then "a great way of identifying a list of people who are optimising their pages". What's your point? Can't they find out the same information from every url that they have in their database? Of course they can! Is it only available when sites and pages are submitted via their form? Of course not! The only difference between a url that has been found by crawling and one that has been submitted is that the IP address of the submitter is available when the form is used. What good is that? If they are going to penalise pages for optimising, then they can penalise them however the page was found.

Now Phil, if you'd spend less time attacking me (including the ridiculous one you've now editted, thanks) and more time reading the forums then I wouldn't have had to have bothered with this post. I wouldn't have had to have corrected what you thought/wanted me to have said with what I actually said.

I didn't change the original post for your benefit, chris. What it said was perfectly accurate. If you'd like me to post it again, for academic purposes, I'd be happy to do so.

I think I've shown quite adequately that what you said was, in fact, said. You left yourself open to "attack", as you put it, when you entered into this discussion and wrote "What is this industry reliance on everything Googleguy says?" It was easy to infer that you think you you know better, even if you don't think that.

As for your listed "objections to anything Googleguy says", they are a load of rubbish and I'm not going to waste time with them. I will say that I don't always believe what he says and I do think that he sometimes makes claims with the intention of scaring people into not doing what Google don't want people to do (I've said that before) After all, he does have Google's best interests at heart. But it would be stupid to disregard everything he says out of hand. The information about the "fresh crawl" answers a lot of questions, for instance, and I've no reason to think he is conning anyone by mentioning the AddUrl form. All the evidence says your opinion on that subject are wrong. And that's that.

Phil.

[quote]An overall viewpoint is irrelevant.[/quote]

Well, if he's talking about something that's totally out of the field of what he does then it is HIGHLY relevant wouldn't you say?

[quote]Wrong. Many people come with agendas.[/quote]

True, although GG is GUARANTEED to have an agenda.

[quote]You are of the opinion that GoogleGuy is only "doing his job" when he posts at WMW. You are entitled to that opinion, but the evidence suggests otherwise.[/quote]

The "evidence" suggests nothing of the sort, which would probably go some way towards explaining why you provide none.

[quote]I notice that you quoted what suited you from your earlier posts. How about this[/quote]

Yes. "way back", past tense. Pre optimiser PR0 penalties. That's a past tense before these forums were even born. Sheesh.

[quote]You were clearly arguing that submissions are ignored. [/quote]

I was, like I said pre optimiser pr0 penalties. In that thread I was arguing submitting was worthless.

[quote]Can't they find out the same information from every url that they have in their database?[/quote]

Yes, but they would have been crawled. Crawling wouldn't identify them as having an interest in rankings. Submitting would. Surely that's not a hard concept?

[quote]If they are going to penalise pages for optimising, then they can penalise them however the page was found[/quote]

Erm...no. Basic common sense says not. Like I say, crawling won't identify someone as having any interest in rankings - submitting will.

[quote]I think I've shown quite adequately that what you said was, in fact, said[/quote]

I think I'm going to agree that "what [i] said was, in fact, said"! However, your interpretation of it is total hogwash - you are trying to read in to things what is not there and frankly that sort of attitude makes it hard on those of us who give out information for free in our own time and of our own good will.

[quote]As for your listed "objections to anything Googleguy says", they are a load of rubbish and I'm not going to waste time with them[/quote]

ROFL.

[quote]I will say that I don't always believe what he says and I do think that he sometimes makes claims with the intention of scaring people into not doing what Google don't want people to do (I've said that before) After all, he does have Google's best interests at heart[/quote]

Huh? You weren't going to waste your time on my rubbish opinions but you will in part agree with me?

[quote]All the evidence says your opinion on that subject are wrong. And that's that.[/quote]

Phil, it is an opinion and THERE IS NO EVIDENCE either way.

Once again, stop attacking me for attacking me's sake. It's getting tedious and frankly this thread is beginning to look a bit childish. Now if you don't mind I'm off to watch teletubbies.

Well, if he's talking about something that's totally out of the field of what he does then it is HIGHLY relevant wouldn't you say?

If he's talking about something that is totally out of the field of what he does, then his views are not worth a light. I thought that would have been obvious.

Huh? You weren't going to waste your time on my rubbish opinions but you will in part agree with me?

I don't agree with you at all, chris. I merely repeated what I've said in the past. It's more likely that you've agreed with what I've written in the past than the other way around.

You entered into the discussion in an argumentative manner, and you seem intent on staying that way, so I'll leave the rest of your post if you don't mind.

Phil.

"fresh crawl" does explain why the keyword phrase that I had mentioned to Phil in another thread one day had 3 listings with these forums number one, then the next day 2 pages, then 5 pages the next day, etc.

I think that "fresh crawl" is brilliant. It's Google's way of giving the serps up-to-the-minute relevance, which knocks me out.

I'm wondering if there's any way that an advantage can be gained from it, but the rankings that "fresh" pages get can't be used to judge their rankings when they are in the real index, so there's no point of modifying pages based on them. Maybe in time we'll notice that fresh rankings do bear some resemblance to main rankings. Either way, I'm happy that fresh pages get in there quickly.

Phil.

What I love about the Fresh Crawl is that if you have a site that updates information on a daily or semi daily basis it gets added almost instantly. No need to wait a month when the news/issue is old.

Yes this is a great feature especially for the news services. I remember trying to find fresh information about the 9 11 tragedy - it was difficult. Google definitely has its finer points

Just getting back to Google's AddUrl form for a second...

Unless a site is new AND has no IBLs from pages that Google has already indexed, then there is no need to use the form. For new sites, placing a link to the new site on an indexed page gets Google's attention very quickly - usually, but sometimes it's slower. I always just place a link to the new site and leave the rest to Googlebot.

Phil.

Unless a site is new AND has no IBLs from pages that Google has already indexed, then there is no need to use the form.

And if a site is new and has no IBLs, you'd better not use the form, because to Google, a site with no IBLs isn't that noteworthy.

I used to think that Rick, but I was persuaded to think differently. For the main part, Google deals with pages, not sites. If a page has no IBLs (if it's an orphan) then it won't get anywhere. But the pages of a self-contained site should be indexed as long as each of them is linked to from within the site.

Of course, they won't get much in the way of PR but pages with very little PR still get top rankings for odd search terms....and people DO use some very odd search terms quite often. E.g. my PR document is currently #5 at MSN for "what are the most reasons people run away", #10 for "How to make paper figures" and #1 for "how to make cookie cutters". The last one does refer to something akin to seo but I've no idea how I got those rankings for the first two. It doesn't say much for the relevance of Inktomi's serps, does it?

Phil.