90% of All Usability Testing is Useless.... Options

[Adrian]

.... So says Lane Becker of Adaptive Path.

Interesting article on how to use Usability in a sense. The general gist of the message is, do Usability testing throughout the whole design process, not as a checklist tick at the end.

When viewed as a sort of quality assurance, as it classically is, usability testing becomes a late-stage “nice-to-have,” less important than getting the newest version of the Web site out the door. It usually results in a thick document that outlines everything that’s wrong with a Web application, including fundamental design issues that can’t be fixed in the few weeks left before launch. This isn’t the best way to effect positive change.

It does seem pointless to go through the whole process of building a site, getting it 'finished', then realising there are alsorts of glaring usability errors that can't be fixed so late in the day. When making some of those changes requires weeks of extra coding and risks getting that deadline met.

It does seem that a lot of things get left till the end. Usability, Accessibility, SEO in some cases. I think a lot of people get too caught up in the 'lets make a website' idea that they lose sight of what its trying to achieve or how to actually achieve its goals.

[nickc]

We all know that we should plan a site meticulously but how many of us actually do it - a page of drawings and scrawl is about the most I ever do - I just want to get on with it. I know I shouldn't but...

Nick

[bragadocchio]

Measure twice, cut once. There's a craftsmanship to doing it right, and that means doing something at each step along the way to make sure that you're not making any missteps.

Sometimes it's just experience helping you not to make those wrong moves. I suspect that you do alright Nick, even though you don't do that type of planning, by remembering what's worked, and didn't work well on previous sites you've worked upon.

But I agree that if you can get some input from others along the way, even if it's not officially "usability testing", it can benefit you. I'd rather do something once the right way than over and over until I get it right.

[projectphp]

Every time I do planning, I do things better, have the ability to backtrack and "undo" mistakes. That said, planning does take time, sometimes more than is justified. I have been to far to many meetings in my time attended by 10+ people, that served no useful purpose and at the end of which, the same things were done anyway.

The level of planning is always proportional to the ammount that can go wrong, and sometimes simply hiring more talented people is better than all the planning in the world...

[bwelford]

As an Internet Marketing Coach, I am often working as part of the company team and acting as the channel of communication and often the project manager with the website designer. This is usually for manufacturing and service companies that may be looking for a 50 page website to support their B2B selling.

In doing this, I spend quite a time on getting the strategic objectives of the website sorted out with the company management. Thereafter I am trying to ensure that the "plan" of attack will result in a selling-effective website. Important considerations here are searchability, usability and credibility handled early in the process.

I use what I call a Detailed Website Specification that gradually evolves through a series of drafts. It covers all aspects of website architecture and then lists the information content of every webpage in detail. The first two or three pages cover some "Best Practices" that I believe must be applied: for example, no FRAMES, adequate information content (text) on every page, true viewability in 800 x 600 resolution (even on a Mac), and so on.

None of what is written is "in cement" but it's a starting point for discussion. I tend to use the version I used for a similar recent website as a starting draft in tackling a new website. I find I am always learning some new items that can be included to make the Detailed Website Specification perform better. I also find the website designer likes such a Detailed Website Specification since it speeds up the process and heads off possible "turning round in circles" phases.

[Ransak]

This topic and the advice given so far is so on the money. I am finding out the hard away about what happens when you don't plan well.

I'm working under a lot of pressure to finish a site real fast. So I'm building away with only a loose plan, yet I keep running into problem areas. Things I thought would work, I'm finding out don't, or could be created much better/more useable. I have hardly the time to backtrack or change areas and I wish I would have planned better in the beginning.

I've learned a big lesson about planning and taking time getting the details right. It is more important if you work in different locations with others who need to work on the site as I'm learning.

Frank Vollono

[cre8pc]

The general gist of the message is, do Usability testing throughout the whole design process, not as a checklist tick at the end

Exactly what I've been saying all along, but then, I was trained in this field and this is how I was taught :wink:

I'm getting the extreme pleasure right now of helping the client of one of my SEO partners who is redesigning their site. I sent them a document, straight from my usability course, about what to review BEFORE the build. It covers a great deal and gets the entire team thinking. They also have questions about how to structure navigation for a gigantic web site, and I sent over help for that.

How informed they are! They're asking the right questions, at the right time!

Once a plan is in place, the usability person can design tasks and test cases to make sure every detail in the plan was implemented, and that it works. In some companies, the usability and QA folks work directly with Project Managers. A really skilled usability person also makes sure all roads lead to conversions, or at least as many as possible.

Lately I keep hearing that usability is a waste because its about the what a site looks like. That's inaccurate and a narrow viewpoint. Link colors do not make or break a sale as much as navigation that drops people off into outer space while trying to make a purchase. Content isn't about how many words are on the page, but what they cover and how persuasively written that makes the difference.

Oh well....not to get me ranting so early in the day, heh heh.

[Adrian]

Exactly what I've been saying all along, but then, I was trained in this field and this is how I was taught

You were in the forefront of my mind when I posted it Kim Then, as I was typing it, I thought, I end up saying this about other areas as well, not just Usability.

[cre8pc]

It is more important if you work in different locations with others who need to work on the site as I'm learning.

Do you mean in a virtual environment, like the site is in one place, but the team is scattered about? If so, I have a lot of experience with this, as I was once both a project manager and QA Usability Engineer for a company with sites and servers all over the place, and the programmers were all over the USA.

There is software available that makes situations like this tolerable, where everyone can get info on the project from their computer and even do defect tracking this way, or grab project releated documents. But, some of the software is Internet based, and the people I worked with had security fears about that, and felt a loss of control. I ended up devising my own methods, along with another programmer, for tracking job progress, but it was a real pain for everyone, no matter what we tried.

The bottom line was testing was always a nightmare, deadlines were hard to meet, and it seemed we were always making changes and fixes.

As you said, plannng is critical. There's a lot of value in written documents, screenshots of mock ups, team and department sign offs and constant testing, but getting everyone on a team, or a company, to accept working under those terms is very difficult, if not impossible in some cases.

[Ransak]

Do you mean in a virtual environment, like the site is in one place, but the team is scattered about? If so, I have a lot of experience with this, as I was once both a project manager and QA Usability Engineer for a company with sites and servers all over the place, and the programmers were all over the USA.

Almost exactly - though the site will live in Hong Kong along with the Programmer and VP of Marketing, which is my entire team. I am the only U.S. based team member. The time difference is the killer. 5:30PM for me is 8:30AM for them. The langauge barrier is difficult as well, I spent 30 min last night (1AM!) trying to explain that there was two sets of <HTML><HEAD> tags in the page and I couldn't get through. Then I couldn't understand how they were telling me to find the files on the web server so I couldn't fix them myself. *sigh* I just gave up.

There is software available that makes situations like this tolerable, where everyone can get info on the project from their computer and even do defect tracking this way, or grab project releated documents. But, some of the software is Internet based, and the people I worked with had security fears about that, and felt a loss of control. I ended up devising my own methods, along with another programmer, for tracking job progress, but it was a real pain for everyone, no matter what we tried.

I'd love to explore a software option. It would be too hard to sell for this job -- they don't even understand the concept of a planning meeting with all decision makers involved. I'd still like to know what options are out there for the future though.

The bottom line was testing was always a nightmare, deadlines were hard to meet, and it seemed we were always making changes and fixes.

Well it won't be a problem in this case cause testing will be zero. They will just throw the site up 'as-is' and it has many flaws. The usability is falling apart as I'm learning of more features and requirements that I have to patch in.

As you said, plannng is critical. There's a lot of value in written documents, screenshots of mock ups, team and department sign offs and constant testing, but getting everyone on a team, or a company, to accept working under those terms is very difficult, if not impossible in some cases.

Screenshots! So obvious that it escaped me. I've been sending flowcharts showing the linking structure but I didn't consider showing how the page should look. The pages get somewhat haggard after the coders get through with it. I haven't had much time to breathe on this one so I'm losing some small details.

Getting everyone is on the same page is the most difficult thing I've found so far. During teleconfrencing between the U.S and Hong Kong on other projects, I've often noticed the two teams agreeing to different things. I used to interject but I've just been keeping quiet lately and watching the crisis evolve.


Anyway, my cynic alarm is going off. Time to reign this in back to Usability Testing. I agree with Adrian, the Original poster that it is pointless to build a site and then go over the usability later. Poor planning leads to bad Usability. It is nice to say you will fix things later on or make adjustments, but the reality is I suspect, that most people will find they don't have the time to change things later on. The time is now to get it right.

Frank Vollono

[cre8pc]

I used to interject but I've just been keeping quiet lately and watching the crisis evolve.

Ah yes! Welcome to the world of QA

I'm preparing to leave for a week's vacation, but please do email me privately for support if needed or ideas. In the meantime, Stickyminds has a lot of good stuff on managemet, testing, and leads on project management software. Rational (now owned by IBM) has terrific software but also costs as much as a house for some of it :shock:

For some help with communication as far as timelines and some planning, that's Internet based, I know these guys and their software app may offer some help.

Ron Crawford
Help-U-Plan
www.helpuplan.com

It just occurred to me that if testing is "useless", how come there's so much software to help us do it? :roll:

[cre8pc]

You were in the forefront of my mind when I posted it Kim

:glasses:

And everytime I bonk my head over CSS, I think to myself, "Wish Adrian was here with me!"

[Ransak]

Thanks Kim (cre8pc), I appreciate the support.

I will take a look at Stickyminds and at Rationals Software. Though I can't buy it I think it is valuable to know what is possible and available.

Frank Vollono

[sanity]

Fantastic article, thanks for the find Adrian.

I've always wondered about the practicality of usability testing at the end of the project. Always made sense to incorporate that into the design - but then I do that with the SEO too. :shock:

I'm working under a lot of pressure to finish a site real fast. So I'm building away with only a loose plan, yet I keep running into problem areas. Things I thought would work, I'm finding out don't, or could be created much better/more useable. I have hardly the time to backtrack or change areas and I wish I would have planned better in the beginning.

So true! I find clients can get impatient and just want to see what the site looks like. I spend a lot time planning a site, it's strategy and structure before even thinking about the design. It's hard initially to convince them this is important but once the site starts to evolve they finally understand my reasoning.

Love this:

The Web is a Frankenstein mix of creativity and code, inspiration and plumbing, form and function, art and science. Good Web design does well with all of these, and it’s often difficult to tell where each stops and the next begins.

[cre8pc]

I hate to quote so much from one article, but Lane Becker really nails some things in his article. He wrote:

Usability Testing Is Design, Not QA
Changing the approach to usability testing also changes what you should expect to get out of it. Rather than a validation done once before completing a product, this internal, qualitative usability testing is done earlier, more frequently, and as part of the design process—not separate from it.

When viewed as a sort of quality assurance, as it classically is, usability testing becomes a late-stage “nice-to-have,” less important than getting the newest version of the Web site out the door. It usually results in a thick document that outlines everything that’s wrong with a Web application, including fundamental design issues that can’t be fixed in the few weeks left before launch. This isn’t the best way to effect positive change.

When viewed as part of the design process, the outcome is very different. Some of the most inspired work I’ve seen has happened on whiteboards in the observation room while testing is going on several feet away. In some situations we’ve even made changes and applied them between rounds of testing in order to gauge the next user’s response to the new solution. From a statistical point of view, this approach is a disaster; from a design point of view, it’s a boon.

I can't explain how relieved I am to see someone write the statement, "Usability Testing Is Design, Not QA", and then describe the difference between classic QA and usability testing.

My swift kick into usability testing was while I was employed as a QA Engineer, testing web site applications. First, I learned the disciplines of QA, then I was told to "test for usability". But, nobody had a method for me to do this. In fact, the user interface folks and programmers were apprehensive about what I'd be doing. They thought I'd be critical of their work.

I never approached it that way. A Human Factors Phd mentor opened up my love for usability, but my admiration for the soul work behind design and code helped me provide gentle guidance, or sometimes it was best to shut up and listen to other ideas. For example, I could walk a web page mock up over to a programmer, and together, while he'd be keying in code, we'd discuss how a user might want to get from point A to point B, and ways his programming could make it more efficient.

Was it the way the user interface person imagined it? Sure. That was one way. But, the programmer had ideas too. So did the marketing department, and CEO of the company. Everyone had a stake in the final outcome. Everyone felt they knew what usability was. Eventually, the company learned to invite the usability person to meetings before the design was even started. I was always interested in what everyone had to say because people like them would be using their product later.

So, I devised test plans for usability that permitted flexibility. I studied Cooper's user personas, and their use in testing during planning stages, to get an idea of WHY we should build for the people who will use a web site or application.

I finally came to disregard measuring results, at least the way Human Factors people would approach it for software. I could do defect tracking and risk assessment, but task analysis was harder to define. People do things in a variety of ways. How do you score human preferences?

I could assign metrics for broken links, and write test cases to make sure every mock up, and every click path expectation was fulfilled. With a set of business requirements, I could pass or fail whether they were met. But, I stressed over how people were going to move from page to page.

Still, usability testing just before launch and especially afterwards is an excellent way to check the health of a web site or application. This how we suggest user enhancements or repair things like page abandonment or low newsletter signups.

I even wrote test plans for search engine optimization - something nobody else had even considered doing.

I've been the co-author of those documents "that outlines everything that’s wrong with a Web application, including fundamental design issues that can’t be fixed in the few weeks left before launch. This isn’t the best way to effect positive change." I know what it's like to have project managers stop talking to you because you found so much wrong with a design that the project was halted, or trashed altogether.

Better I be hated, then the entire company embarrassed for launching something that doesn't work.

Someone like Sophie, and other designers I know of, spend incredible time with the foundation aspects of design. These are the folks to hire. Chances are good these people are usability testing as they create and build.

Long live the whiteboard!