Machine learning for commerce media: The strategic separation of Organic and Ads ML

To truly understand and succeed in commerce media, it's important to recognize how big tech companies strategically use machine learning (ML). Commerce media, which refers to ads within e-commerce platforms, is deeply impacted by two main ML types: Organic and Ads. Each has its distinct use cases and different requirements. Platforms we speak to frequently ask why we suggest separating the ML models and why Big Tech companies like Google, Facebook, and Amazon do the same.

Let's dive into the distinctions between Organic and Ads ML and the benefits of separating them.
‍

The use cases of Organic ML in ranking 

Organic ML generates predictions for products or items to determine their ranking. For instance, if a platform aims to increase clicks on organic items, a common approach would be to develop a model that predicts clickthrough rates, known as a pCTR model. The pCTRs then arrange the items in descending order of predicted clicks.

A standard method to assess the effectiveness of changes or improvements in Organic ML models is to conduct an A/B experiment. In this setup, users, cookies, or requests are split between a treatment group and a control group randomly. The performance metrics from each group should reflect the outcomes expected if the changes were implemented across all traffic. This method clearly indicates the impact of changes made to the Organic ML models.
‍

Understanding the complexities of Ads ML beyond ranking

On the other hand, Ads ML predictions are one of several factors determining ad rankings. For example, ads are often ranked based on expected revenue per impression, calculated as the product of their cost-per-click (CPC) bid and pCTR.

To understand the different requirements of Ads ML, consider an e-commerce site ranking organic items by pCTR. Say there are two items, Item A with a pCTR of 0.6 and Item B with 0.1, but their trueCTRs are 0.3 and 0.2. This discrepancy might not significantly alter user experience in organic ranking, as the items' order remains the same. However, in ad ranking, which typically depends on expected revenue per impression, inaccuracies in CTR predictions can significantly impact ad placement and revenue.

Suppose Item A bids $1 and Item B $5, the initial expected revenue per impression would be $0.60 for A and $0.50 for B. But with trueCTRs of 0.3 and 0.2, the true potential revenue would be $0.30 for A and $1 for B. This misjudgment can lead to incorrectly prioritizing Item A over Item B in ad placements despite Item B having a significantly higher expected revenue. Moreover, if a second-price auction is used, inaccurate pCTRs can also affect the final price paid per ad click. For these reasons, it's not uncommon for a model that improves organic performance to hurt ad performance.
‍

Additionally, Ads ML needs integration with other systems like bidding and budgeting, making experimentation more complex. Unlike with Organic ML, simply diverting traffic for an Ads ML experiment at CTR prediction system level isn't sufficient, as it can affect other operational systems. To accurately measure changes in an Ads ML model, these connected systems must account for experimental traffic diversions. At unityonemedia (and at big tech companies that run ads), advanced capabilities are necessary to separate budget and bidding feedback loops according to the traffic split in experiments.
‍

The strategic separation of Organic and Ads ML for optimal performance

Given the distinct use cases and the need for different experimentation and measurement approaches, separating Organic and Ads ML is beneficial. This separation allows the Organic ML team to improve the organic experience without worrying about the impact on ads and vice versa. Changes beneficial for organic experiences can adversely affect ad performance and business, and the reverse is also true — this is why the distinction is important. Combined with the more complex experimentation required for Ads ML, separating Organic and Ads ML models is advisable and practiced by big tech companies. It has allowed modeling teams to concentrate on distinct requirements and use cases, improving their models in parallel without interference.

The optimal models for these two cases will likely differ significantly.
‍

Using Organic ML as a foundation for ads on your site is a sound initial strategy. However, you'll soon discover significant advantages in separating Organic and Ads ML, a strategy adopted by tech giants. Understanding this early on is important for planning how much effort and resources you'll need for your ad system and ML models and selecting the best platform to partner with. At the very least, it can allow you to quickly understand why a great model improvement for your organic experience can be detrimental to your ads business and vice versa — a common issue many sites face.

‍