Demystifying measurement: Why methodology matters

Recent research showed us how the role of measurement is the foundation for influencing campaign decisions – the definition of success varies based on the measurement tests, metrics and methodologies used. To further explore the topic of measurement, Facebook's Neha Bhargava and Dan Chapsky partnered with Brett Gordon and Florian Zettelmeyer, professors from the Kellogg School of Management at Northwestern University, USA. ¹. Together, the team co-authored a white paper that explores what constitutes good measurement and what success metrics really say about advert performance.

To answer these questions, the team explored how well observational approaches to measurement fare against randomised controlled trials on the same marketing campaigns.

Randomised controlled trials (RCTs) are studies in which individual users are randomly assigned to either a treatment group (meaning adverts may appear in their feeds) or a control group that will never see adverts. RCTs are considered the "gold standard" of measurement because they ensure that control groups and test groups are comparable in makeup and allow analysts to isolate the causal impact of advert exposure.

Observational methods, on the other hand, rely on the measures that an advertiser can collect without having a randomised control group. For example, after targeting a group of users with an advert, the advertiser might compare those who saw the advert with those who did not.

The study's results were presented to the advert research community at the ARF Re!Think Conference in New York City, USA, on 15 March 2016, where the Facebook researchers and professors discussed the findings and what they mean for the marketing community.

The research looked at 12 US-based Facebook advert campaigns totalling over 1.4 billion impressions across industries and marketing objectives. These campaigns were run through the Facebook Lift solution, which implements an RCT. The campaigns were blinded, so no personally identifiable information about users or advertisers was used. The researchers compared the results from different observational measurement approaches to the true advertising impact as measured by the RCTs. Below is an example of this comparison for one study, where observational methods overestimated lift by nearly three times.

Across the 12 studies, observational methods were not a reliable substitute for RCTs. The most sophisticated observational method did well in some studies, but it was impossible to predict when that would be the case. Through a marketer's lens, each method tells a different story about the performance of the advert, which can be explained by the biases and nuances of each methodology.

Source: "A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook" by Facebook's Neha Bhargava and Dan Chapsky and Northwestern University's Brett Gordon and Florian Zettlemeyer, March 2016.