AVOIDING A/B TEST ERRORS

As I mentioned in my last post, there are five errors marketers commonly make in running and interpreting marketing tests.

I’m going to poke at two of the five errors in this post.

First up, the “It’s Not Statistically Significant But It’s Directional” error:

Actually, that’s not a thing.

I know it *seems* like a thing.

Regrettably, it isn’t a thing.

But goodness, if I had a dollar for every time I heard a well-meaning marketer say this exact phrase, I’d be flying first class right now rather than coach with my fellow commoners.

If a test result is not different from the control to a degree that is statistically significant, it’s statistically not any different, period. For simplicity’s sake, it’s fair to think of both test and control producing the same result.

What you don’t want to do is think of the results as being in any way “directional”.


The second error I’m simply calling “Temporary Bumps”.

No, this isn’t a teenage skin condition.

But it may be comparably pervasive.

Because most direct response tests are run at the impact level rather than longitudinally across many impacts over time, we never learn whether or not the additional people who were incented to respond in the successful test were simply accelerating an action (purchase, donation, subscription, etc) that they’d have made a bit later.

If you’ve done enough longitudinal testing, you know that accelerating gifts and purchases - without increasing the overall giving or purchasing during the measurement period - is a very common effect.

So if you aren’t measuring that, then you actually don’t know that your test’s positive impact-level results will net a true, real positive increase.

You may simply have generated a Temporary Bump.


Now, unlike with the first error, the Temporary Bump error is often simply one of perspective.

Meaning, the mistake is to think “aha, we have now figured out that sending our audience this email (or ad or mail package or CTV ad) will increase their total giving or purchasing with us (which is our real, ultimate goal).

But it *is* often still fair to think “well it matters that they responded to this treatment test more than the control treatment, even if it’s just accelerating an action they would’ve taken later”.

That’s true.

The positive result may be an indicator that the new visuals, offer or messaging that you’re trying out are in fact more appealing to the people in your audience.

It’s probably too soon to say that after just one test. But this may be one important piece of the puzzle.


Next post, I’ll dig into three more common A/B testing errors.

Missed Part I? Read it here.

Never miss a SUBLIMITY insight! Subscribe to our occasional Newsletter here!