There are many methods that are used to cross-check tools’ efficiency and data health. Test traffic feature in Google Analytics, sandbox environment in Adjust (app tracking) and A/A test in Optimizely. But what is happening with advertising platforms?
Advertising platforms are pay to play. We validate conversions and clicks from all ads within Google Analytics as a single source of truth, but what about the other metrics that are ads platform-specific? Back in 2016, we wanted to understand how creatives influence the assisted conversion types of ads platforms. Someone might say that a test that is not focused on conversions is worthless, but assisted conversions are part of the conversion flow and as marketing attributional models & conversion windows are not “sexy”.
Advertising assisted conversion types
Facebook: One day view
The lookback period after people view your ad, during which you want to attribute action-based results (such as clicks and conversions) to that ad.
AdWords: View through conversion
The period of time after an impression during which a view-through conversion will be recorded. View-through conversions occur after an ad impression, if the user doesn’t interact with the ad, then later converts.
The A/B test
We decided to come up with an unconventional test. The A/B test was not designed in order to identify the winning variation, we did this in order to set the benchmark of advertising assisted conversions. How can we set the benchmark of this test? We need to have a variation with a creative and a variation without creative, which is not doable.
So we came up with this:
Empty & branded bannersThere are two variations of “empty” banners that include no information about the advertised company named efood, and they look like generic placeholders and two variations of “branded” banners, one that includes food and one the includes an animal.
The test would take place in Google Display Network. We selected GDN because the “empty” banners are irrelevant within Facebook context. Facebook ads include the logo and the name of the advertiser, so it would look like a mistake rather than a test. In Google Display Network the result was the following:
How ads look likeSubscribe to my newsletter and get updates straight into your inbox.
Test Variables
| Target | View-through conversions |
| Hypothesis | If the group of empty banners generates less view-through conversions than branded banners, then branded banners indeed influence users more. The difference between empty banners and branded banners would be the benchmark that we were looking for. |
| Audience | We used two types of audiences, one that is low performing (new cookies) and one that is high performing (visitors – no conversions) in order to compare the performance between those groups. We avoided existing customers because their performance would cannibalize our results. |
| Rotation | In order to avoid auto-optimization that most networks do, we set the campaign to “even” rotation so that it will serve all banners equally. Although the rotation was even, due to the difference in performance, there was a slight bias. |
| Budget | Since it was a test, we did not allocate big budget (3-digit number) but still it was a considerable amount to generate traction. We always keep a small budget for tests. |
| Time | We chose a random regular time period without heavy seasonality that would affect negatively the overall performance. |
We need to add a note here, that our intention was not to create an A/B test in a laboratory, following strict instructions. We wanted to do something that would make sense for us and generate enough traction so that data would be significant enough for examination. After a week of running the campaign, the results were the following:
| PLACEMENT | AUDIENCE | CLICKS | CTR | View-through conversions |
| Empty banner (1) | Low performance | 869 | 0.31% | 117 |
| Empty banner (2) | Low performance | 676 | 0.38% | 86 |
| Empty banner (1) | High performance | 240 | 0.60% | 174 |
| Empty banner (2) | High performance | 206 | 0.66% | 161 |
| Branded banner (animal) | Low performance | 357 | 0.18% | 116 |
| Branded banner (food) | Low performance | 160 | 0.22% | 27 |
| Branded banner (animal) | High performance | 114 | 0.43% | 179 |
| Branded banner (food) | High performance | 73 | 0.46% | 48 |
| OVERALL | ||||
| Empty banners | Low/High performance | 1991 | 0.49% | 538 |
| Branded banners | Low/High performance | 704 | 0.32% | 370 |
There are a couple of insights in the table above:
💡 CTR & clicks
CTR and clicks were significantly more for empty banners. We don’t have any explanation for this, maybe the empty banners within the placements would not seem so generic so users were curious about it and that generated more clicks (and CTR respectively). To add even more drama to this, the low-performance audience outperformed the high-performance audience in both variations.
💡 View-through conversions
View-though conversions were significantly more as well. One thing that makes sense here is that the high-performance audience generated more view-throughs which is reasonable.
💡 Audience performance
Low-performance audience generated more clicks, while high-performance audience generated better outcomes. Although better outcomes in high performance need no further explanation, the amount of clicks generated by low-performance audience in both empty and branded banners variations is strange.
💡 Animal versus food
Animal outperformed food by far. A squirrel can beat pizza!
After this test, we didn’t invest all our GDN budget on empty banners. We did not shift the whole budget to another platform. We adjusted our ad strategy and encouraged ourselves to invest more in ads A/B testing.
What would you do based on the results above?
Do you have any relevant “bold” examples?
