Quick Test for Swift, Actionable Direct Mail Insights

While a Quick Test isn’t the right answer for all clients looking to improve their direct mail success rates, it can quickly reveal insights that you can swiftly put into play.

Before we review the Quick Test fundamentals and rules, we like to remind our clients of the basic realities of attempting to get the skeptical, busy, overwhelmed, disinterested consumer or professional to stop, read what you have to say, and consider saying “yes.”

Capturing that “Yes!” mindset and compelling action is a conundrum every marketer faces. And its the exact reason we are sharing the following insights.

Open or toss?

But first, close your eyes and imagine the following scene in your mind:

Direct mail response is done more on impulse than on careful contemplation. We must imagine the prospect holding the mail piece over the trash bin. He or she goes through a series of instantaneous thoughts:

• Stage One: The prospects first thought is simply, “should I open this one?” In seconds, their mind is evaluating whether or not this mailing has any relevance to them, or any promise of meeting their needs.
• Stage Two: If the decision to open is made, it is a matter of seconds before they make the next critical decision, “should I read this any further?”
• Stage Three: If enticed to read further, the prospect then makes the ultimate decision: “Should I say yes?” This decision may lead to more time reading and evaluating the mail piece. However, it often leads to it being set aside.

Testing Methodology

In direct marketing, list selection is the single most important determiner of success. In testing lists over time, it is important to continually refine a selection of “core lists” that have proven to work. To do so, identifying a minimum level of productivity is important. Measuring and tracking the break-even point can do this.

We often suggest implementing a quick test program, based upon the volume of potential mail combined with the fact that a control effort does not yet exist. Is it possible to find a control package that hits your economic target the first time you test? Yes, it is. But you have to break the rules to do it. You have to use a Quick Test methodology.

The rules

There are three standard rules for testing direct response:

1. Only test one variable at a time
2. Always test head-to-head
3. Always test statistically valid samples

The reality

These are good rules. Unfortunately, to follow them you need:

1. Heavy mail volume
2. A hearty budget
3. Plenty of time

The old rules were established early on by circulation direct marketers who could boast of mailing in high volumes and could afford to allocate 5,000 to 10,000 – or more – names to each test cell. For example, a mailer dropping 1,000,000 packages at a time can conduct 40 tests of 5,000 each and still mail the control package to 80% of the file.

Breaking statistical rules for real-world success

The Quick Test methodology is called for when the old rules don’t fit. And in your situation, because you are looking to rapidly establish a control, we would likely recommend this type of testing.

With the proper use of the Quick Test methodology, you can usually find a winner — or several winners — in the first test, almost always by the second test event. The following example illustrates how it works. While this may not exactly fit your situation, we would develop a custom test to fit your needs and budget.

Phase One

The objective of Phase One is to determine the key success drivers:

• Which offer works best for which segment/list
• Which creative option works best for which segment/list

To show you how it works, let’s look at a simple 25,000 piece mailing using the Quick Test methodology.

The key assumptions are:

• Assumes a total list universe of 25,000
• Test two package formats (Test I and Test II)
• Test three offer splits (A, B and C)
• Test five lists/segments of 5,000 each

Based on the above test assumptions, the QuickTest matrix would look like this:

There are 30 test cells, each with 833 names. The results from each cell are not statistically valid. To generate a statistically valid sample, this test would have to be much larger. However, the results are highly indicative. Once your results are in, you will see strong trends and statistical indications that tell you what to do next.

Let’s assume this QuickTest generated the following response rates:

In this example, it is safe to conclude the following, assuming we need a response rate of 3% or more to hit the economic target.

• Test Package I is a clear winner
• Offer B worked best with Lists/segments 1, 2 and 3
• Offer C worked best with List/segment 4
• Nothing worked with List/segment 5

Phase Two

Phase Two uses a Confirmation Test methodology to verify the results of Phase One, this time using larger cells to generate a statistically valid sample size. Not all successful cells will be Confirmation Tested — only those that formed a pattern. The Confirmation Test matrix might look like the program below, assuming you could find a new list/segment (List 6) that matches the profile of List/segment 4 and can replace List/segment 5.

With a response rate of 3% or higher and test cell counts of 5,000, the results using the Confirmation Test methodology would be statistically reliable.

Using the QuickTest methodology you’ve been able to find a winning control package — using only 25,000 names instead of the 150,000 that would have been required for a test of this magnitude with traditional testing methodology — and in only 90 days.

Successful direct marketers are always Quick Testing and Confirmation Testing … they never stop testing. For each pass through the universe, mail your control to 80% and test two new offers or packages to two panels of 10% each. The statistical reliability of the tests will be high. At the same time, program performance is protected since 80% of the file will always receive the control. Now that you have a control, you can keep program performance high with a standard test methodology and a way to always find new winners. We could also add another column to represent email and compare that response to the other creative approaches.

One final reminder: your offer is king.

Never forget that the offer – the very thing we expect the consumer to say yes to – must be clear, cogent, viable, and appealing. This is true whether they are being asked to inquire for more information, or send in a check. A detailed, complex offer can depress response. The reason isn’t because the consumer has read and rejected the offer, it is because the mail piece never passed the screening which occurs in split seconds described in stages one and two above. A mail piece must effectively, and immediately, convey the essence of its core offer. Keeping it simple and focused is the key.

“What’s in it for me”… The fact is, no prospect, no matter how civic minded, or altruistic in nature — will say yes to any offer that doesn’t meet a basic need of his or her own. Whether these needs include protecting and providing for their loved ones, giving back to the community, or getting a really great deal, offers must be positioned to answer the basic question, what’s in it for me. Therefore, direct mailers must avoid the temptation to tell “all about us”, a tendency that can lead to complicated, ineffective mailings.