{"id":11301,"date":"2026-08-24T08:59:30","date_gmt":"2026-08-24T14:59:30","guid":{"rendered":"https:\/\/attentionmedia.io\/?p=11301"},"modified":"2026-08-24T08:59:30","modified_gmt":"2026-08-24T14:59:30","slug":"run-fewer-incrementality-tests-and-get-more-from-them","status":"publish","type":"post","link":"https:\/\/attentionmedia.io\/?p=11301","title":{"rendered":"Run fewer incrementality tests and get more from them"},"content":{"rendered":"
\"\"<\/div>\n

Incrementality testing is having a moment. And for good reason. Platform attribution has obvious limitations, MMM can\u2019t answer every tactical question, and marketers are increasingly realizing that just because a platform says it drove a sale doesn\u2019t mean that sale wouldn\u2019t have happened anyway.<\/p>\n

But you need a systematic approach to ensure you\u2019re testing the right things and acting on the results.<\/p>\n

I use a framework for this called IDEATE. Despite my feelings about cute acronyms, I\u2019ve used this one enough to see its value.<\/p>\n

IDEATE stands for Insight, Draft Hypothesis, Envision Paths, Arrange the Test, Track Results, and Execute on Findings.<\/strong><\/p>\n

Most of those are self-explanatory. So let\u2019s focus on the two where teams most often mess up testing: Insight and Envision Paths.<\/p>\n

<\/a><\/p>\n

\n
\n
\n 10X your SEO with Semrush for Enterprise<\/span>.\n <\/div>\n

\n The world\u2019s most powerful SEO platform, purpose-built for Enterprise.\n <\/p>\n<\/div>\n

\n Request demo<\/span>\n <\/div>\n<\/div>\n

<\/p>\n

As incrementality testing becomes more common, there\u2019s a temptation to test every channel and every nuance within them. Unfortunately, you can\u2019t. Testing capacity is limited, so spend it on questions where uncertainty has meaningful financial consequences.<\/p>\n

You have to be certain why you are considering this test in the first place. What have you seen in your data, MMM, observational analyses, or through operating the business that makes you question what you believe?<\/p>\n

There is a place for instinct<\/h2>\n

If you\u2019ve been around a marketing program for a while and something seems off about a channel or tactic, trust that instinct and investigate. That doesn\u2019t mean your instinct is right. That\u2019s why you\u2019re testing it. But there is probably a reason something feels off.<\/p>\n

Say that every time we increase spend on Meta ASC, Meta continues to report attractive marginal CPAs. But at the same time, our new customer CAC is getting worse. That should get our attention.<\/p>\n

Maybe Meta is finding incremental customers efficiently, and something else is driving CAC higher. But maybe as we scale, Meta is finding customers who would have purchased anyway and taking credit for those conversions.<\/p>\n

That is a question worth spending testing capacity on because understanding it can have real P&L impact. If we find ASC is highly incremental at the current spend level, great. We can keep spending and maybe scale further. If we find we\u2019re out on the diminishing-returns curve, we can reduce spend and find a better use for those dollars.<\/p>\n

By contrast, a channel supported by your MMM, observational analysis, and sales response doesn\u2019t deserve the next testing slot. Use your limited testing capacity where there is both uncertainty and a material P&L consequence.<\/p>\n

Decide what you\u2019ll do before you know the answer<\/h2>\n

Once we have an insight, we draft a hypothesis and figure out how we might test it. But before arranging and running the test, IDEATE forces a step that is missing from many experimentation programs: Envision Paths.<\/p>\n

Before you run the test, write down the possible results and what you\u2019re going to do in each scenario. Not after the readout. Before the test starts.<\/p>\n

This reduces bias. It\u2019s easy to rationalize a result you don\u2019t like after you see it, especially when it suggests cutting spend in a channel that a team, agency, or platform has spent years building.<\/p>\n

A test only creates value if you do something with the result.<\/p>\n

Say I\u2019m running a test on a large paid social program. I have a 50% pre-advertising contribution margin, and I need roughly a 2.0 incremental ROAS to break even. Before I run the test, I might agree on something like this with the team:<\/p>\n