Measuring Impact: Top 4 Attribution And Incrementality Strategies

What are Randomized Controlled Trials (RCT)?

Simply put, it is an experiment that measures our hypothesis. Suppose we believe (hypothesis) that the new email creative (experiment) will perform (measure) better than the old email creative. Now, we will randomly split our audience into 2 groups. One of them, the control group, keeps receiving the old emails, and the other, the test group, keeps receiving the new email creative.

Some popular methods to run experiments:

Method 1: User-Level Analysis

Illustration for Incrementality

One of the simple ways to quantify incrementality would be to run an experiment as done in the diagram. Divide your sample into two random groups. Expose the groups to a different treatment; for example, one group receives a particular email/ad, and the other does not.

Method 2: Pre/Post Analysis

This is an experiment that can be used to measure the effect of a certain event or action by taking a measurement before (pre) and after (post) the start of the experiment.

  • Keep the control period long enough to get significant data points
  • Keep in mind that there might be spillover in results during the test phase, so we should ensure that the impact of this spillover is not missed
  • Ensure that you keep enough time for the disruption period. It refers to the transient time just after you have launched the experiment
  • It is ideal to avoid peak seasons or other high volatility periods in the business for these experiments to yield conclusive results

Method 3: Natural Experiment

It is similar to the A/B test, where you can observe the effect of a treatment (event, feature) on different samples but not have the ability to define/control the sample. So, it is similar to Randomised Controlled Trial, but you cannot control the environment of the experiment.

Method 4: Geo Measurement

Geo measurement is a method that utilizes the ability to spend and/or market in one geographic area (hence “geo”) vs. another. A typical experiment consists of advertising in one geo (the “on” geo) and holding out another geo (the “off” geo), and then measuring the difference between them i.e., the incrementality caused by the treatment. One also needs to account for pre-test differences between on and off geographies either by normalizing these before evaluation or adjusting for this post-hoc analysis.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Affine

Affine

Affine is a provider of analytics solutions, working with global organizations solving their strategic and day to day business problems www.affineanalytics.com