Incrementality Testing: Why Your Best Ad Skews the Lift
Lab NotesDataNote #2

Incrementality Testing: Your Ad Is Grading Its Own Exam

Lift tests are the closest thing this business has to truth. But the words in the ad decide who the platform shows it to, so the result belongs to the message as much as to the channel. Technicians argue about the method. The copy already picked the answer.

Fred Smith, founder of iFix Marketing Fred SmithDigital futurist · Founder, iFix Marketing 10 min read
// The direct answer10 min read

An incrementality test measures one message, shown to the audience that message attracted, during one window. “The channel” is only one part of what it reads. Delivery systems optimize toward the people most likely to respond to the specific ad, so the copy changes who ends up in your exposed group. Ads written to catch people already on their way to buying post high platform ROAS and low incremental lift. Read every lift result as message times audience times time, and design tests so the creative is controlled or measured on purpose.

Run the same incrementality test twice with two different ads and you can get two different verdicts on the same channel. Most teams read that as noise, and the agency can't figure out why the number moved. Usually it's signal, and it's pointing straight at the words. I've watched smart measurement people argue test design for an hour while nobody asked what the ad actually said.

// 01

The industry finally drew the line on cause

In late 2025, a set of industry guidelines for incremental measurement in commerce media [1] sorted incrementality methods into four broad categories and laid out how strong each one is at proving cause, what it covers and when to use it. A short summary released ahead of it [2] drew the line clearly: attribution, attributed ROAS, pre and post comparisons and correlation work all give direction, but none of them isolates cause. Only controlled experiments that hold advertising back from some people or places do that.

That framing is right, and it's overdue. The research record backs it hard:

  • One large study compared observational measurement methods against big randomized experiments run on a major social platform. The observational methods often missed the true lift, and frequently overstated it [3].
  • Another well-known paper showed that individual ad effects are small relative to the natural swing in purchases, so you need very large samples to measure returns with any confidence [4].
  • A large field experiment at a major online marketplace found its brand keyword search ads returned close to nothing incremental, because the people clicking them were coming anyway [5].
// 02

The variable hiding inside every test design

The guidelines treat incrementality as a method question. Pick the right design, get the right answer. At $1M a month that's mostly true. At $10M a month there's one thing inside every design that almost nobody controls: what the ad says.

Modern delivery doesn't show your ad to a random slice of your target. It shows it to the people the system predicts will respond to that ad. Change the words and you change the people. So when you run a holdout, your exposed group is “people this particular message attracted,” which is a much narrower group than “people who saw our brand on this channel.” Two ads, two different exposed groups, two different lifts.

That's what I call creative contamination.

// 03

Every ad is a filter

The technician sees a lift test as a lab instrument. Get the randomization right, power the sample, hold out cleanly, read the result. All true. All necessary.

The strategist sees something the instrument can't. I came to this work through copy before I coached from the sideline, so it's the first thing I check. Every ad is a filter. A headline that says “20% off ends tonight” is written for someone who already wants the thing. It's a door for people already standing on the porch. A headline that names a problem the reader didn't know had a name is written for somebody across the street. The algorithm reads those words, watches who responds, and goes and finds more of them. So by the time your test starts, the copy has already chosen who's in it.

That means the measurement team can run a perfect test and still get a misleading answer, because the test answers a narrower question than the one they asked. They asked whether the channel works. The test answers whether this sentence works on the people this sentence attracts.

The thesis

None of the liftmath matters untilyou know what thewords were doing.

The technician measures the effect. The strategist decides what effect you're paying for. If nobody writes for new buyers, no test design on earth will find you new buyers.

// 04

Creative contamination: how the message picks your test group

Creative contamination happens when the message itself selects the audience in a way that biases the lift you measure.

It runs in two directions:

  • Harvest creative. Discount offers, urgency, product shots for people who already know you. The system learns fast that these ads convert, so it shows them to people already near a purchase. Platform ROAS goes up. True lift stays small, because much of that demand would have converted without the ad.
  • Expansion creative. Ads that name a new problem, a new buying situation, or a reason someone outside your current base should care. These convert slower and look worse in-platform. But a larger share of what they produce wouldn't have happened without them.
The trap

Optimization, leftalone, funds harvestcreative.

It's doing exactly what you told it to do. You told it to find conversions. The easiest conversions belong to people who were already coming.

Catching that drift early, week after week, is the kind of read the iFix Lab trains.

lab/bench/the-math-the-ad-the-platIllustrative
// 05

The math: the ad the platform loves loses $750,000 a month (illustrative)

These are modeled numbers for illustration. Use them to think with, and don't read them as a forecast for your account. Two creatives in the same channel, same spend:

  • Creative A (harvest). Platform-reported ROAS: 6.0. A clean holdout shows only 25% of those attributed sales were incremental. Incremental ROAS: 6.0 times 0.25, which is 1.5.
  • Creative B (expansion). Platform-reported ROAS: 3.0. The holdout shows 70% incremental. Incremental ROAS: 3.0 times 0.70, which is 2.1.

At a 50% contribution margin, break-even incremental ROAS is 2.0. Creative A, the one the platform loves, loses money on every incremental dollar. Creative B, the one that looks like it's dragging the account down, is the only one making money.

Now scale it. Put $3M a month behind each. Creative A's true incremental revenue is $4.5M against $3M spent: $2.25M in contribution, a $750,000 monthly loss. Creative B returns $6.3M incremental: $3.15M in contribution, $150,000 of profit. The dashboard says A is doing great, at double B's efficiency. Finance would see the opposite, if they could see it at all.

Same bid, same audience setting, same channel. The only difference between those two ads is the sentence at the top.

And here's the contamination inside the test itself. Run one channel-level holdout while both creatives are live, and the lift you get is a blend weighted by however the system split delivery that month. Run the same test next quarter after the system shifted spend toward A, and the channel looks less incremental. Nothing about the channel changed. The words in rotation did.

IllustrationWhat the platform reports vs what the holdout proves
BREAK-EVEN 2.06.01.5Creative A, harvest3.02.1Creative B, expansion
Platform ROASIncremental ROAS, losingIncremental ROAS, profitable

At $3M a month each, A loses $750,000 and B makes $150,000. Same channel, different sentence.

// 06

A quick check you can run this week

You don't need a new test to see whether contamination is already in your numbers. Take your top ads by spend and sort them by how many of their buyers were already yours before the ad ran. That list shows you who the system is really serving. Most teams are surprised by what sits at the top of it, and more surprised when they read those headlines out loud.

The step-by-step version, with the spend comparison that tells you your money has drifted toward demand you already owned, is a tool members get inside the iFix Lab.

// 07

Three ways contamination spreads

  • Your test library ages with your copy. A lift result from last spring describes last spring's ads. If the message has turned over, the result describes a past you're no longer running.
  • Retargeting-heavy creative inflates everything above it. It catches people the upper funnel already moved, takes the attribution, and makes the upper funnel look weak. Teams then cut the ads that were actually creating the demand.
  • Sample size gets harder. Splitting a test by creative cuts the sample per cell, and the research on small ad effects [4] tells you how fast that gets expensive. You may need fewer tests, each one larger and longer.
// 08

Why one lift test per channel goes stale between $1M and $10M a month

Most teams are told to run a lift test on each channel once and use the result as a correction factor forever. At $1M a month, with a handful of ads, that's a fair shortcut.

Between $3M and $10M, creative turnover speeds up and the factor goes stale inside a quarter. Teams apply a channel multiplier calculated on a message mix that no longer exists. Worse, they apply one multiplier across both harvest and expansion work, which hides the very split that tells them where the money is.

The other advice that breaks: “trust the algorithm with broad targeting and let it find buyers.” It will find buyers. Without someone writing for the people you don't have yet, it'll find the ones who were already buying.

// 09

Past $10M a month, you can afford to test by creative

  • You can afford creative-level holdouts. Spend volume is high enough to hold out by creative family as well as by channel. Do it. It's the only way to know whether your expansion copy is earning its keep.
  • Geo designs beat user-level designs for the big questions. User-level holdouts inside a platform depend on the platform's own matching. Region-level holdouts measure total business revenue in test and control markets, so creative selection effects show up in the number you actually bank.
  • Agencies need a creative-mix clause. If an agency's fee or reputation rides on platform ROAS, it'll drift toward harvest creative. Write the incremental standard into the brief and the scorecard, so the people writing the ads are paid on the same number finance uses.
  • Finance becomes your ally. CFOs distrust platform numbers already. Showing them the gap between reported and incremental by creative type is the fastest way I know to earn budget for upper-funnel work, and for the writers who make it.
// 10

The check before an incrementality testing result reaches the budget

Whether you're the in-house growth lead who owns the test or the agency owner who runs it, one standard comes before every other: no lift number reaches the budget without the list of ads that ran during the test, and what each one promised. A thermometer tells you the temperature of whatever it's sitting in. If nobody can say what the test was sitting in, nothing in the result can be pinned on the channel yet.

The rules behind that standard, how to sort the ads, when a result goes stale and when the method has to grow with the money, are written up for members inside the iFix Lab.

// 11

Questions senior buyers ask

We've been burned by tests that said one thing this quarter and the opposite next quarter. How would I know this isn't the same old noise?

Put the creative list for each test side by side. If the message mix changed between them, the second result wasn't noise. It was a different test, measuring different words on different people.

If creative changes the audience, is any channel-level result valid?

It's valid for the mix it tested. Label it that way. A channel factor is really a “channel plus this message mix” factor.

Should we just stop using platform ROAS?

No. It's fast and it's good at ranking ads that do the same job. It's bad at comparing ads that do different jobs. Use it inside a class, never across classes.

How long should a creative-level holdout run?

Long enough to cover your typical time from first exposure to purchase, plus enough volume per cell to detect the lift you'd act on. Work the sample size before you launch.

What about brand search?

Treat it as a test you can run quickly. Pause brand terms in a few matched regions and watch total branded revenue. The famous marketplace result [5] is old. Your answer is the one that counts.

Who should own the test design, the agency or us?

You. The agency can run it. The design, the creative freeze and the decision rule should belong to whoever is accountable to finance.

// 12

The one rule for every lift number

The standard

A lift numberwithout a creativelist next to itshouldn't be allowedinto a budgetmeeting.

And I'd add one line under that list: what each ad said, in plain words. That's where the answer usually is.

Inside the iFix Lab, the monthly Futures Readout is a recorded session that scores last month's calls, shows where signals are merging next and names three moves to make before the market notices. It's where you learn to ask what the words were doing before anyone reads the lift. Your next test is probably already on the calendar, so the time to set the rule is before it runs. Join the iFix Lab for $97 a month. Cancel any time, with a 14-day refund in month one. The next note picks up the fight this creates: the platform, the model and the bank all report a different number, and only one of them pays your bills.

// 13

Sources

  1. Marketing Science, A Comparison of Approaches to Advertising Measurement: Evidence from Big Field Experiments at Facebook, 2019.
  2. The Quarterly Journal of Economics, The Unfavorable Economics of Measuring the Returns to Advertising, 2015.
  3. Econometrica, Consumer Heterogeneity and Paid Search Effectiveness: A Large-Scale Field Experiment, 2015.
// The rest of this note

The rest of this note is for people on the list.

Name, email, and where you sit in the industry. Thirty seconds. One sign-up opens every Lab Note on this browser.

  • The full note, the math and the framework
  • All 19 Lab Notes, open on this browser
  • The next note in your inbox before it goes up here
Your email gets a one-time code first, so it has to be real. Unsubscribe any time.
We send a one-time code to that inbox. Type it in and every note opens on this browser.

Already on the list?

Fred Smith, founder of iFix Marketing
// Who writes the Lab NotesFred Smith

Digital futurist and founder of iFix Marketing: the coach on the sideline with the clipboard. The Marine Corps trained me as a Combat Correspondent, and I started with words before I learned the dashboards. I write the Lab Notes and run the iFix Lab, flops included, so the next dollar goes out on evidence instead of guesses. The math in these notes is a model to think with, not a promise about your account.

The One Offer

Read it here.See what's nextbefore you spend.

This note is one idea, given away. Inside the iFix Lab you get the whole field. I watch answer engines, search, AI, signals and conversion from the sideline with the clipboard, and every week you get what's moving, what's hype and what's working, before your next dollar goes out.

  • 01Your Signal Map. Day one, one page: where you get found today, what AI says about you, what attention costs, and the seat you've been stuck in.
  • 02The Signal Briefing. Weekly: what moved in AEO, search, LLM discovery, signals and CRO, each with a verdict (hype, watch, working) and a move.
  • 03The Futures Readout. Monthly, recorded: last month's calls scored, where signals merge next, and three moves before the market notices.
  • 04The LLM Ops Vault. Copy-ready prompts labelled by job: LLM ops on top of offer and conversion.
  • 05The Hype vs Working Ledger. Dated trend verdicts, plus my own experiments log, flops included.
The iFix Lab · Membership
$97/month

Billed monthly until you cancel

Join the Lab for $97/month →
  • Cancel any time
  • 14-day month-one refund

First 14 days: if it isn't worth it, reply to any Lab email and month one is refunded. Cancel any time from your account, no calls needed.

The iFix Lab is a $97/month membership that renews monthly until you cancel. Cancel any time from your account. Lab Notes share my own tests and opinions. Results vary, and nothing here promises income or results. iFix Marketing, P.O. Box 1066, Jurupa Valley, CA 91752.

© 2026 iFix Marketing. All rights reserved.

The iFix Lab · $97/monthCancel any time
Join the Lab →
Read the full note · FreeThirty seconds to unlock
Unlock →