I love testing.
Because it has led to far more income.
However, there is a serious problem with webdev testing:
1. after the low laying fruit changes have been realised many/most - often ~90% in my experience - of the tests 'fail'. As in the proposed solution was NOT better than the existing.
Guess what? It did NOT fail, it succeeded.
It succeeded in confirming that the approach/behaviour/whatever is still the best known available.
Yes, problem number 1 is a mindset problem. A webdev test succeeds whether it leads to change or confirms status quo. It rarely 'fails': see number 2.
2. there can be real non-mindset failure BUT only if (1) test design is poor and/or (2) badly executed. Fortunately, the results from such tests are often so unfathomable that the need for redesign and/or rerun should be obvious. Unless the test mindset is to blindly go...
3. extrapolation from specific to general. If a test confirms a, for instance 10%, change it must be kept in mind that it was a test on a subset and not the aggregate. The final outcome is not what occurs during the test rather after full implementation.
4. every test relies on some set of statistical principles. Don't abuse them.
The least common abuse is that of minimum sample size: until both/all test variations meet that minimum the comparison is moot.
The most common is that of maximum size. Surprised? Often a test may be inconclusive at minimum sample size, however if the test is left to run eventually sheer volume can reach significance. A significance of immensely minor proportions. And less value.
Yup, the problem, singular, in webdev testing is the testor.
The Problem With Testing
Posted Today, 12:37 PM
I love testing.
Posted Today, 01:15 PM
and of course you know of Murphy
and "The best laid schemes o' mice an' men"
Edited by bobbb, Today, 01:19 PM.
1 user(s) are reading this topic
1 members, 0 guests, 0 anonymous users