How Many Search Queries Are Really Unique?, I checked the AOL database :-) Options

[JohnMu]

Google (or at least Udi Manber of Google ) said 20 to 25% of the queries we see today, we have never seen before

While the AOL search database is not completely the same as Google, the data collected was from a branded version of Google search. Additionally, the data is not complete for the time frame (3 months) but I imagine they took a representative cross-section.

Checking the data, I see ...

59% of all queries are unique (for the 3 months)
17% of all queries were made only twice (which might include those made by one user who just clicked to the next page in the search results)
8% were made three times

... and going up towards the short tail ...
only 126 distinct queries (0.00124%) were issued more than 5000 times (list of the top 100 words and phrases)
The top phrase is a combination of all those that were removed (private or whatnot). (It was not "porn" - that made it to #43)

I can't really list the unique queries, there are over 6 million of them in the database .
30% of the unique queries resulted in a click in the search results.

Just scrolling through the table I see things like: domain names (many misspelled), lots of misspellings in general, many locations,

Starting with:
'abc': 2'285
any number: 89'939
a date (1900-2006): 29'834
a date in the future (2007-2009): 1'001
diet: 1'092
directions to: 584
cheap: 6'532
discount: 3'072
child: 7'295
chicken: 1'307
nissan: 1'042
no (space after no): 2'376
what is: 12'924
how do you: 1'637
how do i: 1'980
how to: 27'590
www.: 924'114
www (no dot): 68'957
www.www: 808
http www: 37'660 (":" and "/" are filtered away)

Some that looked interesting (sorry if you recognize yourself)
abcdefghijklmnoprstuvwxy and z now i know my abc's
decorate my myspace page with puff daddy (clicked on something)
decorating a living room with what you have
diet pills that truely work (got a click)
chicken ip address
what is a good amount of horsepower
what is a good harley for a bigger woman to buy
what is the name of that song that maya and darnell dance to on girlfriends
how to apply to be a full figured fashion model
how to cook illegal drugs
how to get rid of addiction on the internet in the bible

If you want me to check for unique queries of a special kind, feel free to post

John

This post has been edited by JohnMu: Jul 1 2007, 03:54 PM

[Black_Knight]

What percentage would you claim are truly 'commercial' queries, John?

That seems to be the area where I see the most ridiculous claims, where the researchers have almost certainly taken every search such as your "what is a good harley for a bigger woman to buy" to be commercial because they may buy a motorbike (and mention a brand).

Ive seen ludicrous claims of 50%, while my own studies, and looking at data such as the AOL set, show that to be patently ridiculous, where the vast majority of searches are informational and navigational by comparison. I've usually estimated commercial searches to be under 10%, and only gets that high by being a little more generous with the 'commercial intent' criteria than could be justified.

[JohnMu]

That's a good question, Ammon.

How would you define "commercial intent"? Do you think that there is any way to determine the probability of commercial intent for a given query?

I have a feeling that you would have to include query refinements and the URLs clicked on to determine that, but even then it is a matter of interpretation...

Would "car makes strange noises" count as a commercial query? What if it is followed by:
how to fix transmission
cost transmission service
cars with strong transmission
cheap used cars new jersey
2005 bmw 323 new jersey

The original intent might be to find information about a current status, but the final intent is clearly commercial.

This is an area where more and more commercial sites are adding content. If you are selling used cars then it might make sense to give the visitor all the background information as well. Imagine if "car makes strange noises" takes you to the informational page on a commercial site - giving the visitor the information that they want and keeping them there long enough to prevent the subsequent queries. Would, in that situation, the first query even be seen as being commercial?

John

[Black_Knight]

Ah, this is indeed where it gets interesting, John.

There's a fallacy that sellers create for themselves that most people go online to spend money.

Horsepancakes!

I maintain that most of us go online t avoid spending money, or at least, to bring to an absolute minimum any amount of money that we must spend.

I think your examples in the previous post are ideal for this:

Would "car makes strange noises" count as a commercial query? What if it is followed by:
how to fix transmission
cost transmission service
cars with strong transmission
cheap used cars new jersey
2005 bmw 323 new jersey

"Car makes strange noises" - if the search engine can tell me why, then I can avoid paying a mechanic to tell me. The searcher is praying it is something easily fixable without mechanical knowledge so that (s)he can avoid spending any money.

"How to fix transmission" - lets face it, if the first result is "Take it to your mechanic you cheapskate!" the searcher is going to be upset. This is another anti-commercial search at heart. Again, the searcher is hoping to find a dummies guide that negates his need to spend moneyon a mechanic.

Now of course, in both of he above, it is possible to convert the searchers' desire not to spend money into supplying them with a course in car maintenance, or a handbook on basic vehicle maintenance. All along the lines of "Save thousands of bucks with this EASY guide to car maintenance and cut out the huge fees that mechanics have been ripping you off for. This book has hundreds of easy to follow full-color pictures that take you step by step through all basic maintenance and repair jobs empowering you to say No! to huge mechanic bills. etc. etc.

The searcher intent is non-commercial, or as I put it, anti-commercial.

[rynert]

Just had a look at the logs from one of my sites.

There was 190k search phrases

80% were searched only once
8.8% were searched twice
11.2% 3+ times

The top 30 search phrases accounted for 30% of all visits.


A few interesting searches...

magic knickers
jesus teaches how to live life that brings true happiness
4x4 accessories a bar manchester
what equipment do i need to play in a pub band
wedding readings (like a ford cortina)
a police clearance certificate
roadford reservoir home

[ChuckTM]

"There's a fallacy that sellers create for themselves that most people go online to spend money."

On the internet, the frame of the question is different.
When you advertise on a billboard ( DON'T EVER! ) you'd like to know "most" and others metrics regarding percentages.

In the New Market, all (smart) advertising is random access, you only advertise and seek out those who are seeking you. The only question is "Is there ENOUGH people searching for my services?"

The number of people involved in WoW - World of Warcraft may make up 98% of the web someday. I don't mind. I don't care about "most".

All I care about - "Are people Searching on the Internet To Create Better Yellow Page Ads?"

I learned the answer the HARD way.....but I learned it.

Is there a current market big enough for me? That's all that matters.

[earlpearl]

I only recently fell into this dicovery but I thought it worth mentioning. In fact I only wish I had discovered this years ago. It puts a slightly different spin on the number of unique searches.

Take your logs or analysis and sort by alphabetical listings. Alternatively look at your logs or analysis for critical 2 or 3 word groupings with additional long tail elements.

Alternatively if you have a programmer who is running direct queries on your logs run queries for 2,3,4, or whatever number is appropriate queries.

What I discovered is that there are search patterns for certain phrases that adjust the perspective with regard to number of unique queries.

Many of these queries are unique only because of spelling mistakes, the addition of a punctuation mark, or some other minor element in the query. It may turn up as a unique search....but its intent could be identical to one that is more common.

For instance take the how to fix transmission query.

It might show in your logs with variations such as "how to fix transmission", how to fix + transmission, how to fix transmissions, fix transmission, fix transmissions, fixing transmissions, >how to fix transmission,etc.

Each of those queries is counted in the search engines or in an analysis as unique. Yet their intent is similar or identical in each case.

For any topic or search engine there could be groupings of words for inquiries that show up surprisingly more often than one might think.

I just tested this for my business site. I pulled Google search terms for August. I isolated for search phrases that contained the specific suburban jurisdiction in which the business is located and at least a 2-word phrase that defined the service. Of 3500 google searches I found 181 searches that had this combination in 80 separate phrases (as pulled by analytics tool). Of this grouping of phrases the one that showed most often hit the site only 14 times during the month.

We discussed this at seorefugee in a thread in which Yuri and I participated.

I, for one, found at least two interesting patterns off my main business site by looking at groupings of search phrases in this manner.

I only wish I had done that years earlier.

This post has been edited by earlpearl: Sep 5 2007, 01:28 PM

[earlpearl]

As mentioned above while I've been stunned for a long time by the long tail and the volume of unique searches, I only recently started regrouping searches on an alphabetical basis. That has made it easier to recalculate traffic into sites as they reflect certain concepts or patterns.

I took a look at 3 concepts on my business site. Since it is a local business I looked at 3 distinct geographical terms in combination with at least a 2 word phrase that describes the service. I included tight synonyms or variations on the 2 word phrase and eliminated phrases that are not as tightly focused on the main service....and don't convert as well.

I looked at the 2 word phrases in combination with the suburban jurisdiction in which we are located. From experience we know that searchers who use our specific jurisdiction are already significantly familiar with the site/business. Additionally I looked at searches for Washington DC with the business service and for phrases that included Virginia (but not the suburb in which we are located and did not include a town name that from experience tells us they won't use our service-it is too far away).

Results for just Google traffic for the 3 concepts; 860 total searches representing 432 individual search phrases.
The google data represented slightly more than 3500 phrases for the month. Of that total there were over 1600 phrases of which only 287 were repeated at least 2 times. Of the 3500 phrases the one that showed the most was a 2 word generic industry phrase without a geographic description=195 visits.


So while the number of unique searches is dramatic....looking at the data from a different way suggested that a large volume of those unique searches represented only 3 unique concepts or search variations.

Its just the data off of one business web site....but even as the long tail is crazy long...if analyzed differently it can represent relatively few concepts/selling points/services/products/information.

Dave

[AbleReach]

Since it is a local business I looked at 3 distinct geographical terms in combination with at least a 2 word phrase that describes the service. I included tight synonyms or variations on the 2 word phrase and eliminated phrases that are not as tightly focused on the main service....and don't convert as well.

I looked at the 2 word phrases in combination with the suburban jurisdiction in which we are located. From experience we know that searchers who use our specific jurisdiction are already significantly familiar with the site/business. Additionally I looked at searches for Washington DC with the business service and for phrases that included Virginia (but not the suburb in which we are located and did not include a town name that from experience tells us they won't use our service-it is too far away).

Earlpearl, do I understand that you came up with three terms that are defined first by geographical location and then sorted them into three main subsets that are tightly focused on your main service?

So while the number of unique searches is dramatic....looking at the data from a different way suggested that a large volume of those unique searches represented only 3 unique concepts or search variations.

Sounds very cool!

How are you doing in local search? I remember, a few years ago, reading several articles suggesting that the best way to get traffic for "less common" keywords was to make stacks and stacks of pages tightly optimized for each term - what a spammy maze sites like that can get to be!

I'm wondering if thinking in terms of local need clusters would help make some of the really really long tail stuff feel like it can be funneled into a few pages instead of a gazillion.

This post has been edited by AbleReach: Sep 5 2007, 07:19 PM

[earlpearl]

A week or two ago I fell into something that I had not done for years. I felt somwhat foolish for never having done this.

I simply sorted keyword phrases on an alphabetical basis rather than a numerical basis. Seeing the phrases in that context revealed some patterns that I had never noticed before.

That was actually the biggest discovery.

Having done that, one of the small observations that became quite clear was that regardless of the amazing volume of the long tail and the thousands of single queries that are shown via an analytics program, many of them are essentially the same query with different spellings, punctuations, order of words etc.

The business is simple, selling a few services. I simply looked at queries for the main service and segmented them into visitors who searched for the main service based on the geographical area they put in the query.

Suppose the main service was flea removal. I just broke it down for flea removal in the Washington DC region, flea removal in Virginia and flea removal in Alexandria, a large town where we are located.

I did that because from a geographical perspective,and based on our dealings with customers we know that when they mention the smaller specific suburb in which we are located they probably know specifically about our business versus other local competitors in the region, located in other towns. If they reference Washington DC in some way we know that they are close enough that we have a good competitive chance at getting their business. I looked at Virginia, because it is partially relevant. While a very large percentage of the state's population is within our service area, a significant percentage is not. People who live in Southern Virginia or Virginia Beach (shows up a lot) will not use us. We are too far away.

Then I also looked at some words that we have found to be conversion synonyms for removal; say eradication, eradicate, exterminate, etc.

We have found some words to be virtual synonyms for conversions. They may not be the main keyword phrase but they convert at the same percentage basis.

In that vein I disregarded phrases that might just say flea service Virginia, or flea business in the DC area.

We have found that other phrases don't convert as well. Similarly when they say something like flea service...maybe they want a flea circus or to train fleas; services we don't provide.

So when I looked at the phrases that way....I discovered that over 400 distinct search terms and over 800 phrases really only reflected 3 perspectives...in so far as we evaluate them.

Now if one search phrase said flea removal Northern Virginia, and another search phrase said Virginia Flea eradication service....I grouped them into the same category of flea removal/Virginia. I didn't distinguish between putting the geo term first or last.


In terms of your last question.....one of the best things I have read and what I have practised for years is something Jake Baillie called "keyword expansion". Jake is a well-known highly respected SEO and ran a local Search Engine, TrueLocal for a couple of years. On that basis he could review thousands of search terms/day.

Jake Baillie (bakedjake) from TrueLocal is up next.

Local Search Ads - Helpful and not so helpful tips

Weird panel for him to do because 2 years ago Justin called him about doing local search, and Jake said he didn't believe in local search, and here he is running a local search engine. Wake up and get in the local game. Decided to teach people how to spam the search engines rather than giving a pretty speech.

These tactics apply just as well in organic, but they work in ad side and there's more money to be made in the ad side.

The best way to exploit local is keyword expansion. Obvious geo expansion - cities, zip codes, and states. These are the obvious local keywords to use (example Detroit real estate). Non-obvious expansion phrases include neighborhoods, area codes, counties, airport codes and metro areas. People in chicago dont look for a chicago restaurant. No, they type in a chicago neighborhood. In some markets, area codes imply a specific type of person. Seasoned traffic will look for airport codes, such as LAX and advertisers may serve visitors in the airport area.

Word expansion - product names, brand names, skus, slang/industry terms, government terms.

I simply practice keyword expansion for the business terms and the local terms.

Now with regard to the constraints of a relatively small site and a relatively large number of variations on terms for the same concept, there are a variety of ways to accomplish that. I don't think any of the ways are "elegant" but they get the job done.

You could add pages....that are sort of repetitive in concept. In doing that you could get new titles for pages; one would be for flea eradication, one would be flea removal, one would be flea extermination, etc.

If they are all focused on the same idea you simply have to add other elements into the content and add pictures, graphs, etc.

Another way is to have a page on the basic service and within the content reference the different synonyms, hopefully with Titles.

On the geo side, without being crazy spammy, rather than rely on a map I generated directions to the store from all points North, South, East, and West. In that regard I was able to add a lot of town names. Another way to do that is to write content about success stories in towns and regions that reflect local popular geographical terms. There are quite a number of ways to incorporate geographical terms without it being outrageously spammy.

Finally I worked hard to get backlinks with anchor text reflecting the variations on terms.

Hope that was helpful.

[earlpearl]

I was doing some research here, and came upon this thread that I have forgotten.

I really have to acknowledge Ammon's points about the difference between informational searches and commercial searches.

For extensive logs for a certain business I maintain....there is a significant amount of traffic for a variety of prhases. Some of them convert very well and some phrases that are close but differ by a key additional word simply don't convert.

It leads me to two alternative conclusions. The phrases that don't convert are either:

1. informational as Ammon suggested
2. the potential commercial intent was not provided by my business site.

My gut is that most of the phrases of this ilk are informational.

One wonderful way to get a better sense of this is at msn's adlabs tools, specifically http://adlab.msn.com/SearchFunnel/Default.aspx