How much data should I test?

I have some sympathy with people who need to buy data for their direct mail campaigns. Apart from learning the bewildering range of different data types and sources and understanding all the things that have been expressed to the data subject while their data has been captured, a data buyer also has to be full of creative marketing flair, understand scientific testing and know quite a bit about risk and statistics.

It isn’t easy.

And then when your data supplier tells you that you shouldn’t test just 5,000 records because it isn’t statistically significant, and that really you should test 20,000 records as a minimum, you can be forgiven for being sceptical. At best you feel you’re being pulled up on a minor technicality and at worst you’re being suckered into investing more than you want to risk.

But here’s the thing. What if it’s not actually a minor technicality? What if testing a larger file of data is actually important?

I’m going to say it might be. More helpfully, the number of responses expected to come in, is more important than the size of the mail file. Let’s take an example:

Your control cell is the data you normally mail with your normal offer and best performing creative, and you expect it to return a 0.5% response rate. You mail 100,000 and therefore expect 500 responses from this cell.

You want to test the hypothesis that a new data source might be better targeted and therefore deliver a higher response rate. You don’t want to risk too much money because it’s just a hypothesis, so you only mail 5,000 records from this new source of data. You send out the same offer on the same day using the same creative to the people in the new data file. Good stuff.

If the new file of 5,000 records also delivers a 0.5% response rate then you will collect 25 responses. You did want better than that, and on this particular day, it happens. You receive 28 responses from the new file. This represents a 12% increase in response rate which is a significant jump. Isn’t it?

Well, no not really. On another day, four of the responders might have been distracted by preparing to go to the Download festival or had a row with their pet dog or accidentally dropped their coffee on the post and threw it away. On that day the campaign delivers 24 responses which is a 4% drop in response compared to the control cell.

So what we’re saying here is that the random events that influence the lives of a miniscule group of people can have a disproportionately large impact upon our view of the success or failure of a test.  Effectively, the outcome of this test is a matter of luck. Roll your dice, but don’t try to make any predictions about what might happen if you do the same test again.

How do we get around this? Those of us that are lucky enough to have data scientists at every turn, we will let them tell us what to do. For those less privileged, work on your test cell delivering about 150 responses and you’re in the right ballpark. If your control cell is producing 0.5% response then use that as a benchmark. To get 150 responses at 0.5% response rate you need to mail 30,000 records in your test cell.

To make it even easier here’s a look up:

The real calculations are a lot cleverer and more complicated than this (I’m noting this before I get trolled by analysts), but for most mid-sized campaigns this should help.

If it doesn’t give me a shout.