What Is Usability Optimization?, Talking about optimizing for Usability Options

[jscroft]

Note:

This topic has been split off from Usability of Open Source Software. For the complete context, you may want to consider reading some of that conversation as well!

-Joe

Thanks for the warm welcome.

You know, I came to this thread more out of frustration than anything else... my company does internet marketing in the search and affiliate spaces, and we do Usability Optimization under the theory that driving traffic to a client is a little pointless unless they can convert it. We've just built a new website, and as part of the promotion process we're looking for ways to leverage our content by making ourselves useful in a number of online forums.

And guess what? There are astonishingly few conversations going on out there about the actual nuts and bolts of Usability Optimization. I'm not sure why that might be, except that perhaps everybody who knows how to do it has decided to hoard the knowledge until he can get a client to pay for it.

I wonder how that's working out.

I think another reason may be that the term usability has a couple of distinct meanings, depending on who you are. Within the context of this thread, it appears to have its standard meaning, something along the lines of "easy to use."

Within the context of my business, however, usability is a quality of a bounded process that can be numerically evaluated on the basis of a conversion rate. For example the usability of an e-commerce site can be evaluated on the basis of the rate at which it converts shoppers to sales. Without a metric of some kind, the concept of optimization is meaningless.

Anyway, I'd be delighted to engage in a conversation about Usability Optimization—as opposed to little-u usability—someplace other than my company's private echo chamber. It ain't rocket science, but there are a few tricks to the trade, and I for one would be delighted if my prospective clients could get these concepts down before they walk in my door. The more of this they can accomplish on their own, the faster I can proceed past usability issues to the marketing stuff, which is my real bread and butter.

This post has been edited by joedolson: Mar 23 2007, 09:54 PM

[bwelford]

I wonder whether Usability is the totality of what you're talking about, Jason. Some people discuss Captology, the degree to which a given web page persuades. Seems to me that also factors in to the overall conversion rate.

[jscroft]

That's a new term for me. A new one to dictionary.com, too! Anyway, there's enough material out there that I grasp your meaning.

Sure... one of the challenges of this business is that we are inventing a new vocabulary to carry the new concepts we're inventing at the same time. I'll admit that one reason why I may be having difficulty finding the right conversations is that some critical mass of practitioners has settled on a slightly different lexicon and I missed the e-mail. Hazards of the trade.

Frankly, one of the things we're doing is trying to formalize a lot of these nebulous concepts with new, very specific language. If you have a look at some of the verbiage beneath this page, you'll see what I mean. (I'd send you to our glossary, but it hasn't caught up with us yet!)

Speaking strictly as an engineer, though, I don't care what you call it. The bottom line with Usability Optimization, or Captology, or whatever, is that I have a metric whose value I wish to maximize, and so I form hypotheses, perform experiments, and collect data in an iterative cycle until I reach some point of diminishing return.

Think of this as the thermodynamic approach (again, speaking as an engineer). A thermodynamicist doesn't give a whit about the energies of the billions of individual particles that comprise a sample of gas, because for his purposes all of that information can be completely encapsulated with three numbers: the sample's temperature, pressure, and volume. Similarly, my chosen metric—on-site conversion rate, perhaps—encapsulates all the information about the site that is relevant to my goal. That's a good thing, because it masks a lot of complex interaction that I probably don't have any real hope of understanding anyway, and keeps my client's costs down near the ground where he can see them.

[joedolson]

I think this is a really interesting conversation - but it no longer really relates to the original topic the usability of open source software. This is now a much broader question: and definitely one that I think is worth opening up!

How do people here (Kim especially) approach the challenge of analyzing usability? User studies and practical tests are a method of identifying problems; but are there metrics for determining the profitability of particular choices?

-Joe

[EGOL]

This does look like an interesting discussion. I think that it is heading a couple different directions.... 1) usability for the site visitor... 2) maximizing the goals of the site owner. These can be one or they can be conflicting - and some websites might have #1 or #2 as their highest goal. Increasing one could in fact decrease the other.

[joedolson]

I think that it is heading a couple different directions.... 1) usability for the site visitor... 2) maximizing the goals of the site owner.

Indeed. It may well be in the best interest of a site visitor to leave your site and NOT purchase that cartload of 17 new DVD's...but the site owner doesn't really want to encourage that particular action --- so, it's more logical to make it very easy to keep the contents of your cart than it is to simplify the process of deleting them. (For example.)

[jscroft]

Joe, thanks for breaking this out.

EGOL, I see your point, but I think it's worth adding the economic caveat: because there's a crowded market out there, in the LONG TERM the site owner's interests are very closely aligned with those of the shopper.

This is a message that I'm afraid a lot of site owners either lose sight of or never fully understand: that the most successful stores aren't the ones who kidnap people into their front door or put a maze between them and the exit, but instead those that make shopping a painless—no, enjoyable!—and rewarding experience.

I always have to laugh when the first thing my clients want to talk about is optimizing their pages for search. And it's my own industry's fault: a lot of shady operators have engendered the expectation out there that there's some magic shortcut to success in the rankings. There isn't, of course... by far the most important asset a site can have, in terms of search, is high-quality inbound links. And how do you get those? By pleasing enough real customers that ultimately a few of them share their positive opinions about you in public... enough that some of them turn out to be real influencers.

That's hard work, and it's expensive—whether in time or money—and it's the only reliable path to success.

Why an SEO riff in the middle of a Usability discussion? Because it matters: good usability directly affects on-site conversion rates. but it indirectly (and substantially!) affects incoming traffic as well.

To answer Joe's question, though...

Mathematically speaking, optimization has four components: a parameter set, a cost function, an iteration method, and a set of stopping criteria.

The parameters are the set of things you can change. Classically, of course, parameters are numbers, and there are lots of ways to convert even very intangible parameters into numbers for the purpose of optimization.

The cost function is your metric: the quantity you wish to maximize or minimize. It is a scalar number. It can be simple (overall ROI, for example), or a complex formula with lots of weighted factors. Ultimately, though, the cost function is the yardstick by which you measure the results of your optimization process.

The iteration method is the process: based on the parameters used in the past n optimization cycles and the cost function values they produced, what do we try next? There are dozens of iteration methods, some better suited to real-number parameter sets, some best for fuzzy, discrete, or noisy parameters, and everything in between.

The stopping criteria tell you when you're done. They form a statement of Boolean logic that accept the last m parameters and cost function values as input, and produce one of two outputs: do another iteration, or stop.

I don't mean to be needlessly pedantic here, but this stuff is part of every process of optimization. They're either explicit—meaning you have a plan, are doing the math, and are likely to achieve an optimal result fairly efficiently—or they're implicit, meaning you're operating by the seat of your pants.

Now, don't get me wrong... some people operate very effectively by the seat of their pants! But as a replicable process, it leaves a lot to be desired.

This post has been edited by jscroft: Apr 3 2007, 11:56 AM

[cre8pc]

The term "usability optimization" is a new one on me. As a usability consultant, I use the term usability testing, because that's the methodology used after a site or application build.

Optimizing for usability before, in the planning stages, is also a discipline that taps into requirements gathering, guidelines for content and layout, meeting standards, studying target markets and users, and a lot more.

Metrics evaluations (assigning scores) are loved by some people and hated by others. I don't use metrics unless specifically asked to (which is very rare) because the numbers don't equate to anything useful to developers. It either works, or not. It's either critical to fix, or not. One man's mediocre conversion rates is another man's dream conversion rate.

I work specifically with SEO/M's because the combo of SEO and usability prep packs a punch. SEO without persuasive design is a waste of money. Web design without peruasive architecture is a waste of money. Application design without knowing who will use it and what they want is a waste of money. All of these practices are commonplace, however.

There's a lot of information available on usability testing and task analysis on the web and in books, as well as it is taught by several places where certification is offered. I'm an SEO trained in software QA testing for the Internet and mentored by a Human Factors practitioner, but even that is never enough.

[jscroft]

No argument with any of what you say, really: design well, consider your market, and build accordingly.

I will point out, though—with respect—that words carry meaning. For example, no matter what you call it, how can you really optimize for usability prior to build? You can't. What you can do is to apply design heuristics, best practices, prior experience, and your favorite bag of chicken bones to create an initial build that performs as close to optimally as you can get it. But whatever it is you're doing, the absence of a measure-and-test cycle unequivocally indicates that it ain't optimization.

Hey, I don't make up the words... I just use 'em.

How well does it actually perform? Well, I would argue that unless you can describe a thing using numbers, then you really don't know anything useful about it. But then I'm an engineer; numbers are my friends. What's probably fair to say, though, is that no matter how good your initial guesses were, actual contact with reality is bound to have an effect on your creation that will suggest an improvement or two.

As you know.

And then what? Like I said, optimization is either explicit or implicit. Unless you're doing those rounds of usability testing for the express purpose of locking the results up in a safe and never acting on them, I would argue that optimization is actually a better description of what you do, post-build, and that testing is really just one step in the cycle.

[cre8pc]

For example, no matter what you call it, how can you really optimize for usability prior to build? You can't.

Yes you can.

That's what the information architecture is all about. That's what persuasive architecture is all about. Without those two things, a site will fail to meet business and functional specs. It will fail to meet sales requirements and quotas. It will not inspire return traffic. It will fail to bring measurable traffic.

In software testing, test cases (automation, for ex) are designed before and during the coding phase, often while stakeholders are still arguing about what to build.

Server performance testing and load time testing, also measurable, has to be set up beforehand and those engineers have to have keen knowledge of the end result expectations to be able to create proper methodology. Not only that, every single change means a change in test plans, and changes in iterations, recording steps, etc.

For the people side, accessibility is still most often an afterthought. Not sure what math you want to test with there. A screen reader either works or doesn't. A seeing impaired person can either see it or not. A physically disabled person who can't use a mouse can either use other methods or not. Not optimizing and planning to optimize for these people from the start is a common oversight.

For software testing, there is software for engineers to use. I've used some of it. Some corporate settings require your numbers before they'll sign off anything, even if they have no idea what those numbers mean. What's fun as heck is getting in user testing and having a whack at the numbers game then. A dissatisfied user that tells someone else they are unhappy is hard to measure but if caught early on when studied in user testing, is valuable information for planning enhancements.

I'm married to a Software Performance Engineer. It's a job nobody wants (takes a large skillset), but all companies want one of him because of those numbers and stats he can offer them He often jokes around that customers always complain about the products because there is "no usability testing" done.

[cre8pc]

As an added thought...

While it may be harder to work stats and measurables in before or during a build, it plays a huge part in the afterwards side.

One of the main reasons I get any business at all is because of the numbers. If Clicktracks, or other software, screams out failure to meet goals, site owners start seeking answers. They don't have feedback forms on their sites or customer service reps, so the numbers are doing the talking.

Another thing are the studies I see where the use of certain words are measured in a more disciplined setup as was described further above. One example is the use of embedded link content and measuring the effectiveness of different words or phrases.

Numbers do have their place. Sometimes they scream really loud and demand to be heard

(I have a real soft spot for Engineers too!)

[jscroft]

Okay, fair enough... some usability optimization happens prior to build.

But I think it's fair to point out that most of what you mentioned—server load times, etc.—are more properly performance issues. Those parameters could all be maxed out, and a site might still be in the doldrums because users can't find the exquisitely tuned buy button.

In the military, we observe that "no battle plan survives first contact with the enemy." The same is true of e-commerce sites and customers. The one thing that nobody in the pre-build stages has access to is a generous supply of picky, demanding customers intent on spending (or not spending) their own money.

Please understand that I am not suggesting that one shouldn't take all appropriate steps to get it right before launch. But again... emphatically!... no battle plan survives first contact with the enemy. Once the site is launched, it's time to start compiling analytics data and—using the toolbox you just spent a fortune assembling—optimize the site for usability.

Actual usability, as measured with real metrics following interactions with real shoppers. Perhaps this is where you and I are getting tied up... I have no qualms with using whatever methods are available to optimize theoretical usability. But I'm not sure a metric that doesn't involve contact with an actual customer merits that capital U.

And let's not forget the much more common case: the client who already has a functioning site, and is looking for a way to maximize conversions without going through the pain and expense of a complete rebuild. Just like most sites out there were built without the benefit of the processes you describe, most were also launched with no usability follow-up, and have thus never reached anything like their full potential—whatever that might be—in terms of conversions and other metrics.

I'm in the business of offering five-figure solutions to six-figure problems.

Not sure what math you want to test with there. A screen reader either works or doesn't.

Look, getting the widget to function is your job, and your husband's. Getting people actually to use the thing, and in a way that brings our collective client a maximal return on his investment... that's my job.

But let's indulge the question nevertheless. How do we handle a situation where the metric is something other than a real number? An example might be a selection of products, where the merchant really prefers that the shopper select the one with the highest profit margin.

Answer: pick a better metric, one that is continuous... ROI, for example. Generally a better idea, although—since discrete values are embedded in the problem—you still wind up having to select an iteration method that supports discontinuous processes, or your solution won't converge efficiently. And what are the parameters? Can't have too many... off the top of my head, I'd try background color and relative positioning. That's where intuition lives in this process: pick the wong parameters, or too many of them, and you never get a convergent solution.

[jscroft]

Kim, we appear to have been typing our latest at the same time.

Quick response re. goals...

Of course it's fundamental to have them. But I think it's also worth pointing out a critical difference between writing specifications for an e-commerce site and writing them for, say, a car.

When engineers design a car, they start with a set of performance requirements and work backwards—using basic physics—to a design for each component. The only reason the car might exceed the original performance specs is that each component will be overengineered to some degree as a hedge, producing a (completely predictable!) net increase in performance.

Web sites are different, primarily because of the human factor. We have no human-factors "basic physics" that enable us to design for performance in anything like the deterministic way we do with cars. Instead, we have to rely on heuristics and massive overengineering, and then hope for the best.

The good news, though, is that this overengineering—which we really do just to be able to hit the target at all—carries with it some genuine opportunities if one has the time and inclination to tinker.

That's what engineers do. We tinker.

[cre8pc]

I feel like I'm having a debate with my father! LOL

He's a retired electronics engineer, with a pile of patents from his "tinkering". He helped bring us bar codes for scanning, radio and TV and my mom used to joke around that the Russians were spying on him because he was so good at finding Internet viruses and innoculating them.

Sigh...

And I hate math.

Anyway, with dueling banjos ringing in my head, I thank you for this fine conversation.

We have no human-factors "basic physics" that enable us to design for performance in anything like the deterministic way we do with cars. Instead, we have to rely on heuristics and massive overengineering, and then hope for the best.

There are data for human factors but my gawd, it's as flakey and unpredictable as humans are. A car wants to get unstuck from the snow. Design it so it never gets stuck.

A person with that car wants to get unstuck from the snow, make snow angels, have a snowball fight, argue with the person who made them get stuck, debate the logic of not buying snowtires with decent tread, and 150 other possible use case scenerios where the user persona is standing in the snow freezing to death. Not that car mfr's have to design for this but adding ONSTAR emergency contacts was a step in that direction.

Our jobs are so fun

[AbleReach]

And I hate math

LOL

Math just is.
Solutions... mmm... solutions are sexy.
Wondering how solutions happen - also sexy.
Wondering how to anticipate a user's next need - do "normal" nerds think that's sexy, too?

[jscroft]

Dunno. Am I a normal nerd?

Before I accepted a partnership at Profit Rank six months ago, I was being courted rather splendidly by a company that specializes in on-site search. Home Depot is a client, as is Radio Shack, I believe, and a number of other big names.

Their approach is actually a fascinating application of graph theory, which presents drill-down options to the shopper based on a dynamic set of category dimensions. Try finding a product on the Home Depot site... despite their gigantic inventory, you're only about three pretty intuitive clicks away from any given product, and you can reach most products via a multitude of redundant paths.

Now that's sexy!

[iamlost]

I love to ask engineers if they have a BA or a BSc.
Usability is, like most things human, both an art and a science.

Note: bold emphasis in following quote is mine.

The field of usability design takes root in the cognitive sciences -- a combination of psychology, computer science, human factors, and engineering. These are all analytical fields. The discipline prides itself on its scientific basis and experimental rigor. The hidden danger is to neglect areas that are not easily addressed in the framework of science and engineering.

Donald A. Norman : Emotion & Design: Attractive things work better

We know when something is usable - it is intuitive.
We know when something is not usable - it is frustrating.

Usability means that a specific set of users can accomplish a given set of tasks. A site that is usable for one group may not be for another. Certainly, no site will be equally usable by all groups.

So - do you know who you are designing for? And why?
How about - who you are not designing for? And why?

Off Topic

Ignorance -> incompetence -> insolvency

Usability is defined by five quality components:
* Learnability: How easy is it for users to accomplish basic tasks the first time they encounter the design?
* Efficiency: Once users have learned the design, how quickly can they perform tasks?
* Memorability: When users return to the design after a period of not using it, how easily can they reestablish proficiency?
* Errors: How many errors do users make, how severe are these errors, and how easily can they recover from the errors?
* Satisfaction: How pleasant is it to use the design?

Jakob Nielsen : Usability 101: Introduction to Usability

Satisfaction - the riptide where art meets science, and engineers drown.

Sites must be rational, functional, and practical - to be usable.
Sites need also be emotional, reflective, and visceral - to be reusable.

After mere efficient use come:
* how does it look, feel, sound?
* what is its message, image, meaning?
Indeed, the greater the emotional et al appeal of a site the more likely problems with its functionality become acceptable.

How does one measure efficacy or weight satisfaction?
Usability is science, reusability is art.

[jscroft]

Well, all that is very pretty, but if you can't measure usability, then how do you know when you've succeeded?

Puts me in mind of an old girlfriend of mine who was a media buyer. She was trying to explain the concept of branding to me, and I asked, "so how do you measure the ROI of a branding campaign?"

Her response: "Well, we try to get our clients to think about other aspects of the campaign."

WHAT???

I maintain that, if a vendor tells you that you shouldn't be concerned about the return on your investment in his services, you should run like hell. I said it once in this thread: if you don't know enough about an aspect of a thing to describe it in numerical terms, then you are utterly incapable of optimizing that aspect. Period.

My undergraduate degree, by the way, is a BS in Systems Engineering from the U.S. Naval Academy. Not really a fair question, though, in my case... even an English major graduates from USNA with a BS, because we make everybody pass differential equations and electrical engineering.

[bwelford]

I wish the world worked the way you're suggesting, jscroft. It would be so much easier. Originally I started off as a mathematical statistician and for a time was involved in doing operational research on all kinds of slices of businesses including marketing. It would be so great if you could create a mathematical model of everything and then you can simulate how what you've modeled might behave in different situations. It's also a good test of whether you are likely to be able to measure all the important variables and then optimize the operation.

Unfortunately for some of the most important topics it's not possible to define the mathematical model nor to measure all the important variables. How do you determine why that web page may convince you to take the appropriate action, while it leaves me indifferent? Unfortunately for some of the most important questions, you end up with that kind of situation.

[jscroft]

Unfortunately for some of the most important topics it's not possible to define the mathematical model nor to measure all the important variables.

It'd be nice if you could, wouldn't it?

But, bwelford, part of my point here is that, if you set out to improve a widget, or a website, or a love affair, then you must by definition have some idea of what it is you want to improve.

If you conclude that you have improved a thing, then there must be some metric by which you think you have improved it. Otherwise, on what basis, precisely, do you derive your sense of improvement?

The question at hand then becomes not whether a metric exists, but whether or not the metric is explicit. What I'm trying to get across here is that—where possible—it's advantageous to find an explicit, numerical metric whose behavior displays some correlation to the process you're trying to control. Why so? Because, if you succeed in finding one, you can then apply a vast mathematical toolbox to the problem that was not previously available.

Do you have to model the whole world to control a piece of it? Well, you're a statistician... you tell me. The question is whether one can abstract a sufficiently simple model—generally by constraining the scope of the problem—that one can bring that tolbox to bear.

Frankly, folks, I'm not sure I understand what this argument is about. Surely nobody here means to suggest that we should ignore a metric whose behavior correlates well with a piece of reality we'd like to control? And surely nobody here believes that I advocate fixating on those whose behavior does not?

My thesis is simple: optimization is a word that has meaning in the English language. That meaning involves two fundamental aspects: quantity and iteration. Take away either one of those aspects, and what you're doing may be quite valuable, but it isn't optimization. Like I said, I didn't make the words... I just use 'em.

So... if one actually is engaged in a process of mathematical optimization, and if the cost function whose value one is optimizing is derived from metrics that bear on the usability of an e-commerce website, can anybody here offer me a more succint description of that activity than Usability Optimization?

Fundamentally, though, who cares what we call it, so long as we understand what it is and—just as important—what it is not?

Taxonomic arguments are so nineteenth-century.