Image designed by Freepik.
While interviewing candidates to backfill a lifecycle role on our LatAm team, I posed a simple scenario:
You’ve optimized a campaign, and you’re seeing a nice lift in conversions. How do you know whether those conversions came from your changes, from seasonality, or from cannibalizing another channel?
Almost everyone gave the same answer: check the UTM codes.
A few went a step further and suggested comparing campaign performance to overall conversion trends to rule out seasonality. But fewer than half mentioned the thing that actually answers the question of causality: a control group.
Key Takeaways
- UTM codes show attribution, not causation. They can tell you where a conversion was credited, but not whether your marketing actually caused it.
- Control groups help prove incremental lift by comparing customers who received a message with otherwise eligible customers who did not.
- A control doesn’t need to be 50% of your audience. Depending on audience size, a relatively small holdout can provide a useful comparison.
- Keep control groups clean. Exposure through other channels or overlapping lifecycle journeys can contaminate the test and make the results unreliable.
- The goal is to move beyond “this campaign performed well” to answering the more valuable question: Did this campaign actually change customer behavior?
Why UTMs (and last-click attribution) can’t prove causality
UTMs are great at showing correlation. They help you understand where traffic or conversions were attributed—but not why they happened.
For example:
You send an email to 100 people. Thirty of them click and make a purchase. On paper, that looks like a win.
But what if those same people also saw a YouTube ad earlier that day? Or a paid social ad? Or searched for your brand directly and then hunted for a promo email before buying?
Which interaction actually caused the conversion?
Last-click attribution (and UTMs by extension) can’t answer that. They only tell you which touchpoint was last.
Now, let’s add a control.
Say you withheld the email from 100 otherwise eligible users. Thirty people from that group also made a purchase. The “lift” from your email doesn’t look so convincing anymore.
If none of the control group converted, that’s a very different story. Now you’re much closer to proving impact.
(For the sake of simplicity, we’ll ignore statistical significance here. That’s a separate post.)
The control group concept (eligible users, no message)
I first learned about control groups in science class. I first saw them used properly while working on the agency side.
The concept is simple: you intentionally withhold messaging from a portion of the eligible audience and compare outcomes. 🍐
Agencies, in particular, use controls aggressively. We ran:
- Global controls that received no messaging from an entire campaign
- Message-level controls to measure the impact of individual emails or pushes
- Time-based controls, where a group was excluded for a week or a month to account for seasonality
Controls let you answer the question stakeholders actually care about: Did this campaign change behavior, or would it have happened anyway? That’s why they’re an agency’s secret weapon.
What to watch out for
Controls are powerful, but only if you set them up correctly. 🍐
Sample size
You don’t need a 50/50 split. For most small- to mid-sized sends, 10–20% is plenty. If your audience is large (20k+), even 5% can be enough.
Contamination
If your control group can still see the message through another channel (such as a sitewide banner, an in-app message, or a paid ad), you’ve contaminated the test. And once that happens, your results are no longer clean.
Overlapping journeys
If users can enter another lifecycle or promotional flow while they’re supposed to be held out, you’re no longer testing what you think you’re testing.
Controls require discipline. But when done well, they turn “this performed well” into “this caused lift.” And that’s the difference between reporting and strategy.