• by Charlene Weisler , Op-Ed Contributor, May 11, 2021

How effective is the Ad Council COVID Collaborative Vaccine Education Initiative?

Johns Hopkins Bloomberg School of Public Health and Cuebiq are collaborating on the data analysis to identify which messages and content delivery approaches, across various media channels and publishers, are most effective at overcoming vaccine hesitancy, in order to optimize PSA performance and vaccine uptake. Funding for this project has been provided by Bloomberg Philanthropies.

Results are expected by the summer.

Brennan Lake, senior director of research partnerships & data for good, offered further details:

Charlene Weisler: What is the main purpose of the study?

Brennan Lake: To support the Ad Council in evaluating the effectiveness of their campaign in driving positive public health outcomes. While the main call to action of the Ad Council's "It's Up To You" campaign is to learn more about the COVID-19 vaccine so that you can make an informed choice to vaccinate, we also hope to see an increased vaccine uptake, especially among cautious, hesitant and skeptical segments.

Weisler: What do you hope to find?

Lake: Ultimately, we hope to see a campaign that is successful in providing vital information to the general public, while also driving positive health outcomes through an increase in COVID-19 vaccine uptake. By better understanding which aspects of the campaign -- such as content, creative, channel, geographic focus -- are most effective at driving visits to vaccination centers, decision makers can better double down on what works, to get people the information they need to make an informed decision to get vaccinated, in order to protect them and their loved ones.

Charlene Weisler: Has Cuebiq done this type of research before?

Brennan Lake: Measuring the effectiveness of media campaigns in driving visits to brick-and-mortar locations is at the core of Cuebiq's commercial business. Alongside this, we've done extensive pro bono work through our Data for Good program in using privacy-preserving mobility data for analyzing behavior change throughout the pandemic.

Therefore, it was a natural move for us to combine our work in COVID-19 research with our advertising technology to assist the Ad Council COVID Collaborative in measuring the effectiveness of their vaccine information campaign.

Weisler: What is the methodology for this study?

Lake: We utilize our sample of first-party mobility data, consisting of over 20 million anonymous and opted-in daily active users, in order to measure visitation trends to public vaccination centers, such as pharmacies and vaccine mega sites. Specifically, we created privacy-preserving segments of users who have been exposed to the Ad Council's PSAs, and compare their visitation behaviors to an unexposed control group that is demographically similar, in order to measure the visit "Uplift" resulting from campaign exposure.

We also use machine learning to compare 'normal' visitation patterns to current behavior in order to draw a causal link between exposure to a PSA and visits to vaccination centers -- a metric known as "Incrementality."

Weisler: What are the challenges that you see?

Lake: As with any other campaign, we want to make sure that we can effectively verify real visits to the points of interest, differentiating them from the noise of 'passersby' within the mobility data. When attempting to infer something such as vaccination from visits data, this adds a layer of complexity, especially for multipurpose venues such as pharmacies, where even if we can confidently point to visits, it may be the result of segments engaging in regular shopping activities rather than to get a COVID-19 shot.

In addition to applying campaign parameters -- such as only looking at visitors who dwell at a location between 15 minutes and 1 hour -- we can also use causal machine learning to distinguish between normal shopping behavior and true incremental visits that can be attributed to a campaign.

For example, if I typically visit a pharmacy four times a month, but after being exposed to a campaign I make five or six visits, our neural network can pick up on these discrete changes over time, along with data from a control group, in order to reveal incremental visitation behavior.