# Measuring Impact: Top 4 Attribution And Incrementality Strategies

--

One of the simple ways to quantify incrementality would be to run an experiment as done in the diagram

I believe you have gone through part 1 and understood what Attribution and Incrementality mean and why it is important to measure these metrics. Below, we will discuss some methods that are commonly used across the industry to achieve our goals.

Before we dive into the methods, let us understand the term Randomised Controlled Trials (RCT). And by the way, in common jargon, they are popularly known as A/B tests.

# What are Randomized Controlled Trials (RCT)?

Simply put, it is an experiment that measures our hypothesis. Suppose we believe (hypothesis) that the new email creative (experiment) will perform (measure) better than the old email creative. Now, we will randomly split our audience into 2 groups. One of them, the control group, keeps receiving the old emails, and the other, the test group, keeps receiving the new email creative.

Now how do you quantify your measure? How do you understand your experiment is performing better? Choose any metric that you think should determine the success of the new email creatives. Say, Click-through Rate (CTR). Thus, if the test group has a better CTR than the control group, you can say that the new email creative is performing better than the old email creative.

# Some popular methods to run experiments:

## Illustration for Incrementality

One of the simple ways to quantify incrementality would be to run an experiment as done in the diagram. Divide your sample into two random groups. Expose the groups to a different treatment; for example, one group receives a particular email/ad, and the other does not.

The difference in the groups reflects the true measurement of the treatment. This helps us quantify the impact of sending an email or showing a particular ad.

## Method 2: Pre/Post Analysis

This is an experiment that can be used to measure the effect of a certain event or action by taking a measurement before (pre) and after (post) the start of the experiment.

You can introduce a new email campaign during the test period and measure the impact over time of any metric of your interest by comparing it against the time when the new email campaign was not introduced.

Thus, by analyzing the difference in the impact of the metric you can estimate the effect of your experiment.

Things to keep in mind while performing pre/post analysis:

• Keep the control period long enough to get significant data points
• Keep in mind that there might be spillover in results during the test phase, so we should ensure that the impact of this spillover is not missed
• Ensure that you keep enough time for the disruption period. It refers to the transient time just after you have launched the experiment
• It is ideal to avoid peak seasons or other high volatility periods in the business for these experiments to yield conclusive results

## Method 3: Natural Experiment

It is similar to the A/B test, where you can observe the effect of a treatment (event, feature) on different samples but not have the ability to define/control the sample. So, it is similar to Randomised Controlled Trial, but you cannot control the environment of the experiment.

Suppose you want to understand the impact of a certain advertisement. If you do what we have explained above in Method 1, and create 2 groups, a control group that is not shown the particular advertisement and a test group that has been shown the ad, and try to measure the impact of the advertisement, you might make a basic mistake. The groups may not be homogenous to start with. The behavior of the groups can be different from the start itself, so you are expected to see very different results and thus cannot be sure of the effectiveness of the ad.

We need to decrease the bias by attempting resampling and reweighting techniques. To tackle this, we can create Synthetic Control Groups(SCGs).

Find below an example with an illustration for the scenario:

We will create SCGs within the unexposed groups. We will try to understand which households (HHs) missed an ad, but based on their viewing habits are similar to those households (HHs) which have seen one.

Images: