The strangest A/B test ever

There are many methods that are used to cross-check tools’ efficiency and data health. Test traffic feature in Google Analytics, sandbox environment in Adjust (app tracking) and A/A test in Optimizely. But what is happening with advertising platforms?

Advertising platforms are pay to play. We validate conversions and clicks from all ads within Google Analytics as a single source of truth, but what about the other metrics that are ads platform-specific? Back in 2016, we wanted to understand how creatives influence the assisted conversion types of ads platforms. Someone might say that a test that is not focused on conversions is worthless, but assisted conversions are part of the conversion flow and as marketing attributional models & conversion windows are not “sexy”.

Advertising assisted conversion types

Facebook: One day view

The lookback period after people view your ad, during which you want to attribute action-based results (such as clicks and conversions) to that ad.

AdWords: View through conversion

The period of time after an impression during which a view-through conversion will be recorded. View-through conversions occur after an ad impression, if the user doesn’t interact with the ad, then later converts.

The A/B test

We decided to come up with an unconventional test. The A/B test was not designed in order to identify the winning variation, we did this in order to set the benchmark of advertising assisted conversions. How can we set the benchmark of this test? We need to have a variation with a creative and a variation without creative, which is not doable.

So we came up with this:

Empty vs Branded banners
Empty & branded banners

There are two variations of “empty” banners that include no information about the advertised company named efood, and they look like generic placeholders and two variations of “branded” banners, one that includes food and one the includes an animal.

The test would take place in Google Display Network. We selected GDN because the “empty” banners are irrelevant within Facebook context. Facebook ads include the logo and the name of the advertiser, so it would look like a mistake rather than a test. In Google Display Network the result was the following:

Empty vs
How ads look like

Subscribe to my newsletter and get updates straight into your inbox.

Test Variables

Target
View-through conversions
Hypothesis
If the group of empty banners generates less view-through conversions than branded banners, then branded banners indeed influence users more. The difference between empty banners and branded banners would be the benchmark that we were looking for.
Audience
We used two types of audiences, one that is low performing (new cookies) and one that is high performing (visitors – no conversions) in order to compare the performance between those groups. We avoided existing customers because their performance would cannibalize our results.
Rotation
In order to avoid auto-optimization that most networks do, we set the campaign to “even” rotation so that it will serve all banners equally. Although the rotation was even, due to the difference in performance, there was a slight bias.
Budget
Since it was a test, we did not allocate big budget (3-digit number) but still it was a considerable amount to generate traction. We always keep a small budget for tests.
Time
We chose a random regular time period without heavy seasonality that would affect negatively the overall performance.

We need to add a note here, that our intention was not to create an A/B test in a laboratory, following strict instructions. We wanted to do something that would make sense for us and generate enough traction so that data would be significant enough for examination. After a week of running the campaign, the results were the following:

PLACEMENTAUDIENCECLICKSCTRView-through conversions
Empty banner (1)Low performance8690.31%117
Empty banner (2) Low performance 6760.38%86
Empty banner (1)High performance2400.60%174
Empty banner (2) High performance 2060.66%161
Branded banner (animal) Low performance 3570.18%116
Branded banner (food) Low performance 1600.22%27
Branded banner (animal) High performance 1140.43%179
Branded banner (food) High performance 730.46%48
OVERALL
Empty bannersLow/High performance19910.49%538
Branded banners Low/High performance 7040.32%370

There are a couple of insights in the table above:

💡 CTR & clicks

CTR and clicks were significantly more for empty banners. We don’t have any explanation for this, maybe the empty banners within the placements would not seem so generic so users were curious about it and that generated more clicks (and CTR respectively). To add even more drama to this, the low-performance audience outperformed the high-performance audience in both variations.

💡 View-through conversions

View-though conversions were significantly more as well. One thing that makes sense here is that the high-performance audience generated more view-throughs which is reasonable.

💡 Audience performance

Low-performance audience generated more clicks, while high-performance audience generated better outcomes. Although better outcomes in high performance need no further explanation, the amount of clicks generated by low-performance audience in both empty and branded banners variations is strange.

💡 Animal versus food

Animal outperformed food by far. A squirrel can beat pizza!


After this test, we didn’t invest all our GDN budget on empty banners. We did not shift the whole budget to another platform. We adjusted our ad strategy and encouraged ourselves to invest more in ads A/B testing.

What would you do based on the results above?

Do you have any relevant “bold” examples?

Leave a reply:

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.