Jump to content

Cre8asiteforums Internet Marketing
and Conversion Web Design


The Problem With Testing

  • Please log in to reply
1 reply to this topic

#1 iamlost


    The Wind Master

  • Site Administrators
  • 4664 posts

Posted Today, 12:37 PM

I love testing.
Because it has led to far more income.

However, there is a serious problem with webdev testing:
1. after the low laying fruit changes have been realised many/most - often ~90% in my experience -  of the tests 'fail'. As in the proposed solution was NOT better than the existing.

Guess what? It did NOT fail, it succeeded.
It succeeded in confirming that the approach/behaviour/whatever is still the best known available.

Yes, problem number 1 is a mindset problem. A webdev test succeeds whether it leads to change or confirms status quo. It rarely 'fails': see number 2.

2. there can be real non-mindset failure BUT only if (1) test design is poor and/or (2) badly executed. Fortunately, the results from such tests are often so unfathomable that the need for redesign and/or rerun should be obvious. Unless the test mindset is to blindly go...

3. extrapolation from specific to general. If a test confirms a, for instance 10%, change it must be kept in mind that it was a test on a subset and not the aggregate. The final outcome is not what occurs during the test rather after full implementation.

4. every test relies on some set of statistical principles. Don't abuse them.
The least common abuse is that of minimum sample size: until both/all test variations meet that minimum the comparison is moot.

The most common is that of maximum size. Surprised? Often a test may be inconclusive at minimum sample size, however if the test is left to run eventually sheer volume can reach significance. A significance of immensely minor proportions. And less value.

Yup, the problem, singular, in webdev testing is the testor.

#2 bobbb


    Sonic Boom Member

  • Hall Of Fame
  • 2238 posts

Posted Today, 01:15 PM

:)  and of course you know of Murphy


and "The best laid schemes o' mice an' men"

Edited by bobbb, Today, 01:19 PM.

RSS Feed

1 user(s) are reading this topic

1 members, 0 guests, 0 anonymous users