This is a fun question.
You know how every year some of the popular SEO websites post their big SEO ranking factors reports? They get the data presented in these reports by sending a checkbox questionnaire out to a bunch of people who post in SEO forums and then post aggregate results.
A lot of time and effort goes into these reports but really, they are pure rubbish.
If those same reports included people's names and revealed their individual inputs then I would call them awesome!
Why is one of these rubbish and the other awesome! ?
Because there are enough drunks singing in that choir to drown out the people who know what they are takin' about.
However, if I could read the input from people like iamlost, Donna and MarieHaynes, then I would think those reports are worth reading. But you take the input from those brilliant minds and average or mingle it with a bunch of other people and it isn't anything you can bet on.
I wanted to go back to this. I recalled this conversation and EGOL's comments. It would not have surprised me if EGOL at some point was asked o participate in one or more of these surveys done by Moz. As I've stated, I've participated in the similarly structured surveys prepared by David Mihm for the first 7 years. Its good to get a link from the survey provider. David now works for Moz, so he publishes this under the Moz banner. Now if you participate in one of these things you can put a banner on your website referencing your participation. Clearly if you are marketing SEO services or in the Mihm case Local SEO services its the type of advertisement/testimonial you would like on your site.
The above is all pretty clear...and a quid pro quo...isn't it?
Anyway, I referenced with regard to the local version I know a lot of the participants and maybe more importantly I tend to know what they are doing. More critically I know what they are studying, some of the types of clients they work with and how they are pursuing those efforts. Over the years I've emailed and spoken with a fair number of these people.
So when they give opinions its based on what they see, what they work on, the successes and failures that they have experienced. Its also based on the level of competitiveness their clients face. Now the aggregate opinions are based on exactly what EGOL described above, a questionnaire with checkboxes and rankings among suggested items that might impact rankings. The entire results are poured into a calculator and they produce the numerical results; Exactly as EGOL described. But after all they are not ultimate truths. They are opinions based on one's experience and the final results are aggregations of the opinions. It is NOT GOOGLE. Of that we are sure.
Anyway I still find value in these things. To me the real value comes from independent opinions. I'll read the notes. At times I've followed up with questions to the commentators if they wrote something that is somewhat broad and unclear. "What did you mean by that? Can you give me an example? Where did you see that? How often?" etc.
I did that yesterday. I followed up with a commentator. In this case I was wondering about the effects of the very recent ALL BUSINESSES ALL OVER THE WORLD visibility of just 3 Packs. I call them the Crap Packs. Here is a recent and ongoing discussion on the topic. 53 comments and growing; 5 days old.
This google change "screws" local businesses. If you are in an urban area, be it Erie, Pa (population 90,000 or so,) or in Tokyo or NYC...there are a lot more than 3 local dentists, doctors, pizza places, hardware stores, moving companies, etc. Lots and Lots more. If you aren't in the 3 pack your visibility is driven way down on page and all you can do is advertise on adwords.
In any case the person I contacted has a business that has been serving clients having faced this problem for a year or 6 months earlier. Google pulled this 3 pack on the restaurant, hotel, and local venue and entertainment industries last year. Google has up to one year's of aggregate data on click tendencies of users. Google knows what is going to occur.
The person I contacted serves one of the verticals that got hit a year or so earlier and at certain levels that vertical is like the mass of local businesses that aren't brand names or chains. We had a great conversation.
The head of this business had and has some great insights; insights that I didn't have. That is because this group has been challenged by this change for 6 months to a year for over 500 clients. Therein are a lot of observations. Forget the aggregate ranking reports.....these are very interesting very specific observations and one's that are in ways that I perceive appropriate across the board. So I learned a lot.
So I think with these aggregate ranking reports one wants to pick and choose data and sources to find nuggets of value. That is how I'd use them going forward and how I've used them in the past.