Search Engines really are different Options

[bwelford]

There's a great Search Engine Watch article today on Search Engine Results Continuing to Diverge
The message is that different search engines produce different results. So a metasearcher such as Dogpile may give you more robust results. There's a fascinating tool to compare the SE's on different searches at http://comparesearchengines.dogpile.com/

[randfish]

Awesome link, Barry. That's a very spiffy interface they've got going on... And only 1.1% of searches have the same "top results" - seems almost hard to believe doesn't it?

[cre8pc]

What stuck out for me was the point about the Google habit. By using it, and only it, for search results, the searcher is limiting their search. This study could return us to the days where we used many engines and compared results or maybe help promote the other search engines - even Yahoo! - as being viable options once more.

I admit to having gotten lazy and just using Google, but this study validates what I was wondering about my own habit. How much have I been missing?

[send2paul]

... and I think from an SEO point of view - is there anything to be learnt here in terms of webpage design ?

Seeing that some of these search results are so different where does that leave Mr/Ms SEO when they are attempting to optimise their webpages for the maximum number of search results? - but from which search engines - and how...and maybe even "why"?

[AbleReach]

This suggests that opportunities continue to abound for search marketers to gain exposure despite increasing competition among advertisers, particularly for less commonly used \"search tail\" terms.

Wherever there's a footpath, Mr/Ms SEO will be trying on new shoes.

Google is most commonly used. On the other hand, allowing for SE diversity is like taking into consideration browsers other than IE.

Would different search engines reach slightly different demographics?


Elizabeth

[bragadocchio]

I like the comparison tool, but I'm skeptical about the study.

I had some reservations back in May when an earlier version of the study was released. They still exist.

Maybe it's because I've worked on a number of sites that were able to do well in all four of the search engines cited.

Maybe it's because the research seems to indicate that the best way to search the web is using a meta search engine, and the research was sponsored by a meta search engine.

Maybe it's because the sample used seems on the small side.

More likely, it's because it's pretending to be something that it isn't. It's telling us that the algorithms that the search engines use are significantly different. Different enough so that the same sites shouldn't be showing up in the top ten on each of the search engines.

Can we tell this by comparing what results show up in response to a seemingly small sample of queries? When search engines tell you that they have indexed over 8,000,000,000 pages, 400,000 doesn't seem all that significant (0.005%).

Does it matter that there are different amounts of documents in each engine's index? Do differences in how the search engines handle duplicates, and apply other filters make a difference? Could the queries used matter?

[AbleReach]

Could the queries used matter?

Off the top of my head, when I want to get to a specific person, place or thing quickly, I use Yahoo. Yahoo seems to want to give me what a thing is named.

When I want to sift a larger pool I use Google. Google seems to try to give me the most important or popular thing on the topic.

I don't have much experience with MSN - somehow Yahoo seems cooler than getting MSN-close to an army of AOL users. Nonsensical habit-o-mine.

Since the top ten are the sweet spot, here's a nonscientific comparison of the top ten results for - web design. Between Google and Yahoo there is virtually NO overlap between links, or even between domains. It doesn't matter as much if a specific page is listed somewhere in page 20! Worry over if one page of a site appears at #150 on Yahoo but not at all in Google is splitting hairs.

Yahoo
web design - without quotes gives exactly the same results as "web design" with quotes.
1. www.mi-websolutions.com
2. http://powersitesystem.com
3. www.fortunecity.com/web-design.shtml
4. http://webdesign.about.com
5. http://aishou.net/webdesign/
6. http://fanlists.hoshichan.com/webdesign
7. www.andagraf.com
8. www.webdevforums.com
9. www.1hideout.com
10. www.mcpmedia.com

Google
"web design" with quotes has only http://webdesign.about.com in common with Yahoo.
1. www.htmlhelp.com/
2. www.htmlhelp.com/tools/validator/
3. www.webpagesthatsuck.com/
4. www.oswd.org/
5. www.wpdfd.com/
6. www.wpdfd.com/wpdres.htm
7. www.useit.com/alertbox/9605.html
8. www.useit.com/
9. http://webdesign.about.com/
10. www.webstyleguide.com/

Google
web design - without quotes has no overlap with Yahoo at all.
www.1234-find-web-designers.org is not in the other two top ten searches. Other than that Google's top ten are pretty close with and without quotes. Remember that "web" is a term predictably used in close proximity to "design."
1. www.wpdfd.com/
2. www.wpdfd.com/wpdres.htm
3. www.webpagesthatsuck.com/
4. www.htmlhelp.com/
5. www.htmlhelp.com/reference/css/
6. www.webstyleguide.com/
7. www.oswd.org/
8. www.useit.com/alertbox/9605.html
9. www.useit.com/
10. www.1234-find-web-designers.org/

I was going to guess that the same domains would show up somewhere in both SE's top ten. Nope.

Interesting to see that the quotes didn't make much difference for a predictable combo of words like web design.

Just playing.

Elizabeth

[travis]

Wow, what a tool.

( http://comparesearchengines.dogpile.com/ )

Thats fantastic. Looking at the stats in Western Australia, 80-90% of our search engine traffic comes from Google, so that would require a larger circle.

And its super fast.....

[bragadocchio]

In the spirit of just playing around a little, Elizabeth, here's another query which shows a lot more overlap. I do see that one thing which makes a difference is whether or not the search engines will display only one, or more than one result in the top ten for the same domain.

Google for "CSS"

1. Cascading Style Sheets - www.w3.org/Style/CSS/
2. Cascading Style Sheets, Level 2 - www.w3.org/TR/REC-CSS2/
3. CSS Tutorial - www.w3schools.com/css/default.asp
4. css Zen Garden: The Beauty in CSS Design - www.csszengarden.com/
5. CSS Vault » The Web's CSS Site - cssvault.com/
6. Guide to Cascading Style Sheets - www.htmlhelp.com/reference/css/
7. glish.com : CSS layout techniques - glish.com/css/
8. The W3C CSS Validation Service - jigsaw.w3.org/css-validator/check/referer
9. css/edge - www.meyerweb.com/eric/css/edge/
10. DDJA classic programming magazine. - www.ddj.com/


Yahoo! "CSS"

1. W3C: Cascading Style Sheets - www.w3.org/Style/CSS
2. The W3C CSS Validation Service - jigsaw.w3.org/css-validator/check/referer
3. CSS Tutorial - www.w3schools.com/css
4. Cascading Style Sheets, Level 2 - www.w3.org/TR/REC-CSS2
5. Validator CSS - jigsaw.w3.org/css-validator/validator-uri.html
6. css Zen Garden: The Beauty in CSS Design - www.csszengarden.com
7. CSS2 Reference - www.w3schools.com/css/css_reference.asp
8. Welcome to the CSS Pointers Group - css.nu
9. Complete CSS Guide - www.westciv.com/style_master/academy/css_tutorial
10. Wikipedia: Cascading Style Sheets - wikipedia.org/wiki/Cascading_Style_Sheets

Ask "CSS"

1. CSS Tutorial - www.w3schools.com/css/
2. Guide to Cascading Style Sheets - www.htmlhelp.com/reference/css/
3. CSS Frequently Asked Questions - The HTML Writers Guild - www.hwg.org/resources/faqs/cssFAQ.html
4. Cascading Style Sheets, level 1 - www.w3.org/TR/REC-CSS1
5. Cascading Style Sheets - www.w3.org/Style/CSS/
6. More Results from www.w3.org - www.w3.org/TR/REC-CSS2/
7. glish.com : CSS layout techniques - www.glish.com/css/
8. The W3C CSS Validation Service - jigsaw.w3.org/css-validator/ | Save
9. css Zen Garden: The Beauty in CSS Design - www.csszengarden.com/
10. css-discuss.org - www.css-discuss.org/

MSN "CSS"

1. The W3C CSS Validation Service - jigsaw.w3.org/css-validator
2. Eric Meyer: CSS - www.meyerweb.com/eric/css
3. css/edge - www.meyerweb.com/eric/css/edge
4. glish.com : CSS layout techniques - www.glish.com/css
5. Cascading Style Sheets, level 2 revision 1 CSS 2.1 Specification - www.w3.org/TR/CSS21
6. Cascading Style Sheets, level 1 - www.w3.org/TR/REC-CSS1
7. Guide to Cascading Style Sheets - www.htmlhelp.com/reference/css
8. The Complete CSS tutorial - www.echoecho.com/css.htm
9. css Zen Garden: The Beauty in CSS Design - csszengarden.com
10. Complete CSS Guide - Cascading style sheets reference - westciv.com/style_master/academy/css_tutorial

One of the results that shows up in three of the four (glish), shows up at 13th in the fourth. There are a few others that do show up in all four.

[AbleReach]

Interesting, Bill!

"Web" could be web + many different terms. CSS is closer to CSS is CSS is CSS. Makes me wonder about a search for php or html...

Your comment about ways people search is well taken. I wonder how different each SE's results would be for keywords like video vs video game vs star wars video game... and the target audience may not even phrase search terms in that way.

I suppose the advantage of diverse results could be that word order could vary within an article, possibly drawing in more than one kind of keyword phrasing.

What does this portend for doorway pages build for specific SEs? Will there be more?

I'd google and count for kicks, but I don't have as much time to play today. Boxes and a carpet cleaner are calling my name. :roll:

Elizabeth

[bwelford]

I like the comparison tool, but I'm skeptical about the study.

Bill, I think there's a different explanation that doesn't go quite so deeply into all this.

The vast majority of web pages are constructed without ever thinking about search engines. So the binary file that represents the web page does not necessarily include the important elements that may allow a search engine to determine its relevancy for a keyword search. So the search engine ranks all the web pages it knows about in whatever relevancy order its current algorithm suggests. However it's almost a random sample within this 'neck of the woods'.

If you're looking at a techie area such as your CSS search, then there is more chance that web designers will understand how to package their web pages for the search engines. So there will be more overlap of the SERPs as you found. However this represents an infinitesimal fraction of the total web.

So if web designers as a whole were better, then the search engines would have 'better data' to allow them to do their figuring. However as it stands, each search engine is a black box and is throwing up suggestions about very incomparable web pages. There can be no scientific proof that one search engine is better than another. It all comes down to marketing.

Given this, it may be that one search engine is better than another "on average" for a certain slice of the web. We don't know which search engine works best for which slice of the web. So the most robust approach to ensure you're never too far off the best is to use a 'reputable' meta crawler. Whether that's Dogpile, MetaCrawler or Vivisimo, who knows? You'll find these ideas developed a little more in a blog entry today, "Search Engines really are different - just like detergents".

[randfish]

That's a great blog you've got Barry - I can't believe I never clicked it from your signature before.... Thanks for sharing.

[bwelford]

Thanks for the kind words, Rand. Very much appreciated from a blogger of your eminence.

[Ron Carnell]

Pick a keyword phrase. Find four human beings who are recognized experts in that field. Give them a hundred web sites and ask them to rank the list from best to worst.

What are the chances, I wonder, of the human experts each coming up with the same Top Ten? Yea, right.

With some keywords, I suspect there will be clear leaders, largely undisputed as the best of their kind. Usually, though, that won't be the case, and since we all value things somewhat differently, it's not at all surprising ranking algorithms do, too. I'd personally be much more surprised if there weren't any substantial deviations.

Here's the thing, though:

While our four human experts won't come up with an identical Top Ten, I suspect we'd find they are largely in agreement on the top fifty. The order will certainly be different, because "best" is a bit subjective, but they'll still be able to differentiate between "good site," "bad site." Expand the Dog Pile test to the first fifty or hundred sites from each of the majors (and throw out the PPC ads, which tell us nothing), and I would predict a much greater convergence.

If so, trying a different engine is one answer, trying a meta-SE is another, but perhaps the easiest answer is to just dig a little bit past page one in the SERPs.

[projectphp]

But I want to go there right now Ron!!!

Funny really, how often a comparison of search engines focuses on the first X results. I wonder, what are we comparing? What the laziest people get? What those in a hurry will click?

I never really get what we are looking for in "comparisons". What *I* want in an SE is prettty simple: a database of EVERY page there is, cause I usually look for really obscure things, and a clean design. Not much else I really want. I am positive not everyone has the same needs from an SE, but those are mine.

My bet is that relevance advances in the future won't be as important as usability advances, like suggestions for refined searches, Ask's on page answers, more different "stuff" on a page (like images ona regular search, aka Danny Sullivan's "invisible tabs") and other advances in the repesentation of data, rather than advances in the understanding of pages.

That is my bet, but then I have been wrong plenty in the past!

[Ruud]

Ah yes, that tool....

I'm not surprised the results aren't (always) the same in the top 10. You have to stand out as a search engine and, as Ron points out, different people do different things.

No, send2paul, I don't think that puts SEO at a difficult spot. First, I know of a good number of sites which rank top on Google and Yahoo. Nothing special going on, just solid SEO. Second - SEO follows the money. Google powers most traffic; Google gets most attention. If all of a sudden Yahoo tomorrow not only spiders sites like crazy but sends actual traffic to them - SEO will follow Yahoo.

I love SEO.

[bragadocchio]

What does this portend for doorway pages build for specific SEs? Will there be more?

I'm not sure if that was ever a good idea. If you want to appeal to multiple search engines with your pages, Ammon wrote a pretty good article on how to do that which you can find here:

The 3-Page Search Engine Optimisation Technique

[bragadocchio]

The vast majority of web pages are constructed without ever thinking about search engines. So the binary file that represents the web page does not necessarily include the important elements that may allow a search engine to determine its relevancy for a keyword search. So the search engine ranks all the web pages it knows about in whatever relevancy order its current algorithm suggests. However it's almost a random sample within this 'neck of the woods'.

I agree with you, and I really liked your blog post. But somehow, I'm drawn to a different analogy, comparing an excellent single malt scotch with an ok blended scotch. I'll support that with the dogpile results for a search for CSS:

1. New CSS Book: First Four Chapters Free
Sponsored by: www.sitepoint.com  [Found on Y!SM] 

2. Complete a Financial Aid Profile® Online
Sponsored by: apps.collegeboard.com  [Found on Y!SM] 

3. Guide to Cascading Style Sheets
www.htmlhelp.com/reference/css/  [Found on MSN Search, Google, Yahoo!, Ask Jeeves] 

4. Cascading Style Sheets
www.w3.org/Style/CSS/  [Found on Google, Yahoo!, Ask Jeeves] 

5. CSS at Northern Tool
Sponsored by: www.northerntool.com  [Found on Y!SM] 

6. Shop The Official CCS Skate Store
Sponsored by: www.ccs.com  [Found on Y!SM] 

7. IntelliPRINTPLUS - Lotus Notes
Sponsored by: www.intelliprintplus.com  [Found on Y!SM] 

8. Css
Sponsored by: www.PureDirectory.com/  [Found on Ads by Google] 

9. Buy Skateboards at CSS
Sponsored by: www.shop.css.com/  [Found on Ads by Google] 

10. The W3C CSS Validation Service
jigsaw.w3.org/css-validator/  [Found on Ask Jeeves, LookSmart Reviewed, MSN Search, Yahoo!]

In the results from the individual search engines, we have sites that are about Cascading Style Sheets. In the meta search results, we have a mix of organic and sponsored results. What would the Consumer Web watch people say about these results? See:

Still in Search of Disclosure

Let's look at each of those results:

1. An ad for a book about CSS
2. An ad for financial aid for college students
3. A CSS tutorial
4. The CSS Standards from the W3C
5. An ad for hand and power tools
6. An ad for skateboards and skateboard accessories
7. An ad for Lotus Notes
8. An ad for a nondistinquished looking directory
9. Ad ad for skateboards
10. The W3c Validation Service

Interesting that Dogpile insisted on only using the first ten pages from each search engine, since "most people only look at the first page." Dogpile shows 20 results on their first page. Of those twenty in my CSS search, only 6 of them aren't sponsored results. 14 are advertisements.

I might be happier with results that drew from the organic results only. I'm not sure that Dogpile searches will be replacing a search on one or more of the major search engines.

<edit - fixed quote>

[bragadocchio]

Expand the Dog Pile test to the first fifty or hundred sites from each of the majors (and throw out the PPC ads, which tell us nothing), and I would predict a much greater convergence. 

If so, trying a different engine is one answer, trying a meta-SE is another, but perhaps the easiest answer is to just dig a little bit past page one in the SERPs.

Excellent post, Ron.

Search engines perform at least three major functions: crawling sites, indexing them, and serving results. Each of those can be done in very different ways. The surprise shouldn't be that search engines diverge as much as they do. The surprise should be that they converge as much as they do.

And, I suspect that if that test was expanded to the first fifty, or the first 100 results, we would find a lot more covergence. The smaller the data set for each query, the greater the likelihood of divergence between search engines.

My bet is that relevance advances in the future won't be as important as usability advances, like suggestions for refined searches, Ask's on page answers, more different \"stuff\" on a page (like images ona regular search, aka Danny Sullivan's \"invisible tabs\") and other advances in the repesentation of data, rather than advances in the understanding of pages.

I'll second that.