Does Google Consider Clicks On Serp Entries? Options

[bwelford]

It's clearly stated that in its contextual advertising programs, Google will adjust the ordering of Adword ads based on Click-through-rates (CTRs).

I'm getting the distinct impression that the same thing is happening on Google's regular search. If a web page is near the top, then it seems as if it may gain a position or two if the title and snippet seem more appropriate in a particular keyword SERP. It would of course be only a slight extension of the way Google now is personalizing results based on what it knows of your own particular online behaviour.

Would others agree on this? Is there any hard information to support this?

Is it equally true for Yahoo! and/or MSN/Live?

If so, it becomes even more important what you put in the Title of a web page and in that 155 characters+spaces description you have for that web page.

[eKstreme]

A couple of thoughts:

1. A few years ago, I noticed G gave me Javascript-based links in the SERPS. Surprised, I refreshed and they were gone. A little digging turned up reports of "experiments" Googlers run. Think of it this way: their search service is so popular that within a couple of hours they can have a huge dataset. It seemed to me that tests like these were for things like snippet generation algos and things like that. Never saw it again.

2. For personalized search, they will need a mechanism to figure out what you like. To take their dolphins example, if you type in just [dolphins] you'll get both the American football team and the marine creatures. I can think of two ways to figure out which dolphins you're looking for:

a. If you search for dolphins and then your very next search is [dolphins football], then you're probably looking for the football team. You can generalize this by tracking your general search keywords: if they are football-related then when you type just [dolphins] you're probably looking for football sites.

b. URL tracking, as per my first point. I don't know if Google does this in the SERPs, but Yahoo certainly tracks EVERY SINGLE SEARCH. Try it. Go to Yahoo, search for anything you want, copy any SERPs link and paste it into a text file. Wait a few seconds, and refresh the search page. Copy the same SERPs link and compare it to the first one. Notice any difference?

In GMail, all links go through a redirector script, and graywolf cried fowl. These days, GMail still uses a redirector, but it's cloacked using an onclick event (that is, the status bar now shows the correct URL, but it still gets redirected. And Google still thinks it's not evil.) It's very hard to think that GMail is not tracking the links in the redirector script.

I have no idea what MSN is doing because I haven't looked

Pierre

EDIT:
Google DOES TRACK URL CLICKS, but only if you turned on search history. If you don't turn on personalization, it doesn't track.

I just tested it out, and the live HTTP headers are like this:

CODE

http://www.google.com/url?sa=t&ct=res&cd=6&url=http%3A%2F%2Fmaps.google.com%2F&ei=Qt_aRZrNJZ34Qf6OmP4I&usg=__e3LdsuVgnedtl_8NlsNFCyFm6LU=&sig2=ZlmHJRoL5WodalBc907KWw

GET /url?sa=t&ct=res&cd=6&url=http%3A%2F%2Fmaps.google.com%2F&ei=Qt_aRZrNJZ34Qf6OmP4I&usg=__e3LdsuVgnedtl_8NlsNFCyFm6LU=&sig2=ZlmHJRoL5WodalBc907KWw HTTP/1.1
Host: www.google.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.google.com/search?hl=en&q=google&btnG=Search
Cookie: *snipped*

HTTP/1.x 302 Found
Cache-Control: private
Location: http://maps.google.com/
Content-Type: text/html
Server: GWS/2.1
Transfer-Encoding: chunked
Content-Encoding: gzip
Date: Tue, 20 Feb 2007 11:45:18 GMT

----------------------------------------------------------

http://maps.google.com/

GET / HTTP/1.1
Host: maps.google.com
User-Agent: Mozilla/5.0 (Macintosh; U; Intel Mac OS X; en-US; rv:1.8.1.1) Gecko/20061204 Firefox/2.0.0.1
Accept: text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Referer: http://www.google.com/search?hl=en&q=google&btnG=Search
Cookie: *snipped*


HTTP/1.x 200 OK
Cache-Control: private
Content-Type: text/html; charset=UTF-8
Content-Encoding: gzip
Server: mfe
Content-Length: 14665
Date: Tue, 20 Feb 2007 11:45:18 GMT

Notice the redirector script. Good to know that Google is using 302 when it should be 301.

More evil by the second. Go Google!

This post has been edited by eKstreme: Feb 20 2007, 06:50 AM

[rynert]

I would think they were certainly gathering that data and using it along with the the information about clicking a link and using the back button - probably even analyzing the time between those two actions.

It would be fair to say that a link clicked and the back-button used within a very short space of time probably means that the site did not match the summary and to an extent did not match the search either.

It may be fair to say that clicking an link and not returning to the google SERPs at all could mean the site was a very good match.

Combine all of that and it makes sense to shove sites higher up that seem to match results better.

Well, to me it makes sense anyway

[eKstreme]

And the obligatory blog post.

It make sense to use this technique for personalization. At least they're not doing it for all SERPs. Anyone taking bets on when they'll quietly roll this tracking over to all search results? I say 3 months.

Pierre

[JohnMu]

I thought Google has been doing this for a while now. They have to be doing it in order to track the clicks like they have in the AOL search database (which runs on Google).

The 302 is of course incorrect, but Google has been working with different forms of redirection for different places, so I assume they're testing things out and adjusting as needed. I bet they'll fix the 302 into a 301 "sometime" (=?), maybe once they find out

Do you see a different way of doing quality control in the search results without tracking clicks? (I realize having them in GMail doesn't seem to make much sense, they have the URLs anyway?)

Would you use a proxied, unpersonalized search engine instead of the normal one -- say if the clicks were still tracked but accounted for an "anonymous user" without your IP/cookie/etc? That should be fairly easy to set up ... Google could even set something like that up or help with a 3rd party to get it running. If they track click anonymously they would still have click-data for statistics, just not tied to users.

It is starting to make more sense why they stopped passing out SOAP API keys -- no more non-controlled Google results. Within the AJAX search API, they can still track usage and clicks. In results through the SOAP API they can't control what is done with the results.

John

[eKstreme]

The 302 is of course incorrect, but Google has been working with different forms of redirection for different places, so I assume they're testing things out and adjusting as needed. I bet they'll fix the 302 into a 301 "sometime" (=?), maybe once they find out

Someone should tell them... however, after a bit of thought, 302 might actually be correct: when you're tracking URL clicks you would like to track every request. A 301 would mean that the browser will cache the end result. A 302 should force a re-request of the tracking script, which is the desired behavior. I'm conflicted about this spirit of the protocol vs the word of the protocol.

Do you see a different way of doing quality control in the search results without tracking clicks?

Not really, and all credit to them for using this tracking only for personalized SERPs (I hope they don't ever change that). The only other solution is looking at G Analytics data and reverse figuring it out from there, but not all sites in G's index use Analytics (to understate the obvious).

Would you use a proxied, unpersonalized search engine instead of the normal one -- say if the clicks were still tracked but accounted for an "anonymous user" without your IP/cookie/etc? ...

I do! It's called Yahoo, and I use it in preference to Google. For the past few of weeks, the only thing I use G for is to see the SERPs that show up in my referers. As far as I'm concerned, Google's stated objective/mission/obsession of tracking me personally will keep me away from it. And it's not like Yahoo is bad - actually better in many respects (I've yet to see a site cloak for Yahoo, but I've found a few in Google - that's another blog post ).

Yahoo SERPs display is rubbish though - Google's are much clearer.

Pierre

[JohnMu]

But would you use a proxied Google access? Imagine they had a "generic" cookie much like Doubleclick has (had?) which would let you stay logged in with your account but would not log the clicks. Or do you prefer Yahoo's results?

John

[eKstreme]

That sounds more reasonable, John, but I will try my best to figure what is being tracked in the first place. It's all about trust (to use Google's favorite word), and I just don't trust Google right now. They seem too nosy, and frankly, they don't need to be. A cookie would be great as I can block it And before anyone asks, yes I use GMail and AdSense (and I do have a defunct Blogger account), but I will be weaning myself away from them as soon as a decent replacement can be set up.

As for results, most of the time, it doesn't matter if I use Yahoo or Google - they both get me to what I want, so I use Yahoo because of this trust thing. However, in technical searches for my work, Y trumps G many times. I would rather not share particular real examples (NDAs and stuff), but a good way to describe the difference is that Google tends to find different pages talking about the same angle/aspect of the topic while Yahoo finds more diverse angles/aspects. To take a fictional example, suppose you search for [car engine]; if the most popular topic is BMW's latest car engine, then Google's results seem to favor this particular topic, but Yahoo will find stuff generally talking about car engines.

But to be honest, specialized engines trump both. A new one I'm playing with called GoshMe is excellent (it's more of a meta search engine than a real SE). Another one I've had success with is ChaCha's human searchers features. More and more, GoshMe is becoming my first port of call, and is certainly in my arsenal of online searching. I'll be reviewing those in detail soon on my blog.

Pierre

[EGOL]

First, Google should have their heads examined if they are not using this stuff. It is the most valuable information that they can obtain - probably more valuable than links. In addition they could track the amount of time that visitors spend on your site before they backbutton into the SERPs. Would that not tell a lot about the quality of a website.

I have no doubt that these metrics are used - and enjoy the result.

This post has been edited by EGOL: Feb 20 2007, 08:18 PM