Why Marketing Testing Fails and How Imagination Bridges the Gap

by John Miglautsch, Published 10/2026 Direct Mail Insiders

For decades, the marketing industry has been obsessed with data — specifically, the promise that if we collect enough of it, we can predict the future. We build sophisticated models, run hundreds of tests, and rely on machine learning to tell us who will buy next. Yet, there is a fundamental, epidemic-level problem with this approach. We are operating as if data were a crystal ball, when in reality it is merely a mirror reflecting the past.

The missing variable in this equation — the one we ignore as if it were the air we breathe — is TIME.

The Einstein Connection: Imagination Over Knowledge

Albert Einstein famously remarked that “imagination is more important than knowledge.”

For years, this felt like an abstract platitude to many in the business world. However, when you analyze the history of scientific breakthroughs, it becomes a practical necessity.

Nicolaus Copernicus, 1473 – 1543 “The Father of Scientific Research” Consider Nicolaus Copernicus.

Seventy years before Galileo, Copernicus proposed a heliocentric solar system. New revolutionary telescope data did not drive his breakthrough; an imaginative shift in perspective drove it. He performed a “thought experiment” by asking, “What if the Sun were the center?”

Copernicus realized that if the Sun were at the center, the universe would become vastly more logical.

By simply shifting his viewpoint from Earth to the Sun, the chaotic, complex movements of the planets suddenly simplified into elegant, orderly, slightly elliptical orbits.

This revealed that retrograde motion (a phenomenon that required complex mathematical gymnastics in the Earth-centered model) was actually a simple optical illusion caused by the Earth lapping the other planets. He didn’t have the immediate mathematical proof — that took 200 years of work by figures like LaPlace — but he had the vision to see that a cleaner, more coherent system was more likely to be true than a chaotic, patched-together one.

In marketing, we often forget the power of the thought experiment. We get lost in the “knowledge” — the past transactions, the recency scores, the conversion rates.

But models don’t “see around corners,” as cartoonist Scott Adams might say. Only human imagination can do that. When you approach a testing problem, you cannot simply look at what happened yesterday and assume it will happen tomorrow. You must imagine the context of tomorrow, which will inevitably be different from the context of yesterday.

The Scientific Method & Imagination

Normally, humans predict the future by identifying causal connections that reveal how objects and forces move through time. The Scientific Method: Observation, Hypothesis, Experiment, Explanation & Theory is the industry’s well-documented process for isolating and quantifying those effects.

Too often, I have seen marketers focus all their attention on the Experiment step, running endless A/B tests across multiple dimensions without thinking enough about the WHYs behind the testing and its results.

The Trap of Historical Data

The greatest fallacy in modern marketing is conflating “predictive modeling” with “accurate forecasting.” All data is historic. By definition, every data point in your CRM, every transaction in your database, is something that has already happened.

When we build models based on this data, we are essentially building a sophisticated parsing system for the past. If you assume that tomorrow will be exactly like yesterday, you are doomed to error — especially if you act on your own data.

The following case studies illustrate some of the issues with TIME & TESTING and show the need for greater imagination in test design.

Learning from the Field: Case Studies in Context

True expertise in testing comes from understanding the context that surrounds the data. Over decades of consulting, I have seen these patterns repeat across industries.

The “Changing the Game” Phenomenon

BNI Furniture had discovered an incredible metric: two-time buyers had ten times the lifetime value of one-time buyers. The conclusion was intuitive: “We should do everything in our power to incentivize that second order.” They began offering 10% coupons to encourage that second purchase.

But they failed to take into account what generated that data. Had they been mailing coupons before? No.

Once they introduced the coupon, they changed the game. Customers began waiting for the coupon before buying. The data that looked like a path to long-term value was effectively corrupted by the very intervention meant to exploit it.

This is the central tension of testing: you measure something, you create a strategy, and that strategy fundamentally alters the landscape.

When I shared this example, someone agreed, “Yes, and the new buyers had much lower downstream buying than projected.”

The Hudson’s Bay Company and the Vacation Model

For the Hudson’s Bay Company, we built a model to predict “vacation buyers.” We were thrilled to find that customers with high credit limits also booked vacations. We thought we had found the causal link. We were wrong.

We had reversed the causation: people who wanted to buy a vacation called to ask to raise their credit limits to make the purchase. By looking only at the correlation, we confused cause with effect.

(Presumably, some Hudson’s Bay customers asked to have their credit limit increased and were denied an increase. Consequently, they did not make the vacation purchase, which reinforced our hypothesis.)

Cabela’s and the Seasonality Trap

At Cabela’s, we faced a major modeling issue. We were pulling names in December for an early spring fishing mailing. Our models showed that recency — the most important variable in standard RFM customer segmentation — didn’t come up.

Why?

A Ph.D. Stats friend said our model must be broken.

However, it was imagination that revealed that the recent buyers in December were hunters, not fishermen. By failing to account for the seasonal context of the database, the company had been mailing hunting catalogs to fishermen and fishing catalogs to hunters.

Their number one customer complaint: “Why do you send me all these hunting (or fishing) catalogs?” depending on what the customer was not interested in, was a direct result of ignoring the time-based context of their own data.

The Blood Center: The Skew of Generation

In my work for a blood center in Kansas City, I encountered a classic bias.

They had profiled their financial donors and discovered they were all “wealthy suburban women.” They wanted to know how to find different markets. I asked a simple question:

“What have you done to generate these donors?”

It turned out they were exclusively placing ads in glossy suburban real estate magazines. They hadn’t profiled their customers; they had profiled their past advertising.

The Mechanics of Rigor: Hold-Out Groups and Head-to-Head Tests

If imagination is the “why,” experimental design is the “how.” You cannot achieve valid results without rigorous control.

The most powerful tool in your arsenal is the “hold-out group.” In any marketing test, whether digital or print, you must reserve a percentage of your audience that is not exposed to the campaign. Without this, you have no way of knowing if the customer would have purchased anyway.

This is especially vital in our current landscape, where digital attribution is messy (an understatement, for sure).

When a customer sees a mail piece, then goes to Google, searches for your brand, and buys, Google gets the credit. The connection can get even more confused if the mailer triggers an AI prompt for that sort of item, potentially bypassing even Google. If you don’t use hold-out groups to measure the “lift” of the mail channel against a control, digital channels will always win due to mis-attribution.

Furthermore, we must return to “head-to-head” testing.

Just as professional golfers compete on the same course on the same day to determine the best player, we must test variables against one another in real-time. If you test a variable in the spring, and then test another in the summer, you aren’t testing the variables; you are testing the seasons.

The Future: Integrating Imagination

We are currently seeing a decline in experimental rigor.

Agencies and internal teams are often incentivized to show immediate lift on simple return on ad spend (ROAS) rather than dealing with full costs, profitability, long-term value… much less understanding underlying causes.

AI models are becoming more popular, but they have a distinct limitation, since they are engines of history. They can process data at scale, but they cannot interpret the “why” behind the results. They will give you a list of things to consider, but AI cannot “see around the corners” without imagination.

The future of marketing requires a synthesis:

1. Acknowledge the Context: Every data point is historical and context-dependent.
2. Apply Imagination: Ask “Why?” constantly. Use thought experiments to stress-test your assumptions.
3. Maintain Rigor: Demand hold-out and control groups. Avoid circular logic. Understand the Lifecycle: Recognize that your customers, the economy, and your competitors are constantly in flux.
4. The goal of testing isn’t just to “crank out models” or “get lift

The goal of testing isn’t just to “crank out models” or “get lift.”

The goal is to understand the nature of your business deeply enough to navigate an uncertain future. When you stop looking at data as a simple map and start looking at it as a set of clues for an ongoing mystery, that is when you will begin to see real growth.

We don’t need more data. We need better questions. We need to be the “problem children” of the classroom who refuse to stop asking:

“But what if?”

Over more than 40 years, I’ve documented 36 valuation growth explosions across a wide range of industries.

Are your marketing and IT people generating the highest return on their budgets?

Contact me for information on my special 2-day on-site strategic marketing audit:  John@wdma.org or join WDMA.org – fighting for direct marketing, every day