The dominance of Google's advertising markets

Google, dominant player in adtech

The subject of this post will be a little different from the subjects usually covered on this blog. Instead of observing the leaks of personal data on sites and applications, I will try to document (in a non-exhaustive manner, the abuses being numerous) how Google established itself on the advertising markets. If you want to know more, here are some recently published studies:

The study of Dina Srinivasan : Why Google Dominates Advertising Markets.
Studies by Damien Geradin and Dimitrios Katsifis: Google’s (Forgotten) Monopoly – Ad Technology Services on the Open Web, 'Trust me, I'm Fair': Analyzing Google's Latest Practices in Ad Tech From the Perspective of EU Competition Law and Competition in Ad Tech: A Response to Google.
The study by Fiona M. Scott Morton and David C. Dinielli: Roadmap for a Digital Advertising Monopolization Case Against Google.
The “OECD secretariat preparation notes”: Competition in Digital Advertising Markets.
The study by the English Competition and Markets Authority (CMA): Online platforms and digital advertising market study.

The time when advertising campaigns were created and managed "manually" (contracts by fax, sending of advertising banners and tracking elements by e-mail, etc.) between agencies and advertising agencies is over:

Like financial markets, this management is now automated (“programmatic” or “RTB” for “Real-Time Bidding”).
The technology enabling these advertising markets to be carried out is called SSP (Supply-Side Platform) or Ad Exchange.
The sale of advertising space is now done individually: the publisher offers an advertising opportunity (displaying an advertisement on your device), the advertiser is free to choose or not to capitalize on this opportunity.
The advertiser often has personal data about you when they receive this opportunity, allowing them to bid what they think is the "fair price".
It competes with other advertisers, each going through buyer bots called DSPs (Demand-Side Platforms).
The auction is completed very quickly, often less than 100 milliseconds.
Much less publicized, the preferential agreements between advertisers and publishers (called "deals" in the adtech world) have not disappeared, but they also pass mainly through the "programmatic" pipes.

These automated advertising markets are the cause of a massive leak of personal data, read for example Brave’s GDPR violation complaint against Google and the IAB (the adtech lobby) on RTB. And they are dominated by Google, for example here are the figures for England (the figures are similar in the other main Western markets):

Online Platforms and Digital Advertising Market Study (CMA July 1st 2020)

As such, the economic model chosen by Google (surveillance capitalism) and its operational decisions (support for web advertising monetization via these automated marketplaces, flouted notion of consent, etc.) have a disproportionate influence on other players in the advertising web, who often have no other choice than to also enter the game of surveillance capitalism.

What is the link between this domination of Google and the good health of the web in general? If we take a look at Google's annual reports, we can observe that it itself captures (Google search, YouTube) an increasingly significant share of advertising spending made via the purchasing tools offered by Google (themselves already dominant).

GShare

In 2007, Google captured 64% of its advertising revenue. In the first quarter of 2020, Google captured 85%.

Consequence: publishers capture an increasingly small share of this funding (the share on which Google takes its tax) and find themselves deprived of significant funding. We can cite the critical economic situation of the information sector, leading to:

Great precariousness among many journalists.
A lack of resources to handle in-depth investigations, which are so critical for our democracies.
Low autonomy of the media in the face of the powers of money. Difficult to create a large profitable media outlet today, French media are mostly owned by a few rich industrialists, cf. the French Media map produced by Acrimed and Le Monde Diplomatique.

Dina Srinivasan's thesis is that Google could never have dominated advertising markets to this extent if these markets were properly regulated. She reveals that many practices applied by Google to promote its own business have been prohibited on these financial markets. And so Google was able to expand its web thanks to public inaction. The respective studies by Damien Geradin and Dimitrios Katsifis, then by Fiona M. Scott Morton and David C. Dinielli, also detail Google's multiple conflicts of interest and lack of transparency.

Before 2007, Google was already an advertising giant

In 2000, Google launched Google AdWords (today Google Ads) in order to better monetize its search engine, very quickly it was a success. What are the reasons?

First of all, obviously, its very effective search engine, appreciated by users (even if since 2000, Google has increased the abuse of its dominant position in this area, take a look the evolution of search results, it's quite striking).
A “self-service” system where each advertiser (from a small local craftsman to a multinational) can create their advertising campaign quickly as long as they have a credit card.
Payment for performance: the advertiser buys the keywords he wants and only pays when the user clicks on the ad.
More relevant advertising: in order to maximize its revenues, Google has an interest in favoring cheaper advertising but with a better click-through rate.

In 2003, Google launched Google AdSense in order to extend its advertising network to third-party websites. What levers does Google rely on?

First, on the fact that it already has a unique captive market: an innumerable number of advertisers already on AdWords, small and large, with their credit cards. Elsewhere on the web, to launch an advertising campaign, you must pay an agency.
Secondly, on the fact that it already crawls the web, it is therefore capable of providing relevant contextual advertising: ensuring that the advertising of an advertiser operating in a specific sector of activity will find something to broadcast on sites related to its sector of activity (contextual advertising).
Thirdly, on the fact that for a “small” website which does not have the means to open an advertising agency to approach agencies, there was little competition.

In short, it is also a success.

In 2007, the acquisition of Doubleclick allowed Google to obtain a key position among publishers

But the advertising agencies of large publishers do not only use Google AdSense to monetize their advertising inventories. They are wary of the quality of the ads broadcast by Google AdSense (the variety of advertisers means that we can also come across unwanted ads), and do not necessarily appreciate the format of the ads (original text only, no banners or videos). They also work with other ad networks and above all have strong relationships with advertising agencies, which allows them to sell most of their advertising inventory directly, at more attractive prices.

To manage all of their advertising campaigns (direct sales, sales via ad networks such as Google AdSense or sales via new marketplaces), they use specialized tools called adservers.

Likewise, the main agencies do not only have Google search results to distribute advertisements to their advertising clients. They need a tool to correctly manage and measure all of their advertising campaigns. This tool is also an adserver, note that its functionalities are significantly different from a publisher adserver, even if they share a common basis.

In order to get closer to large publishers and advertisers, Google buys Doubleclick for $3.1 billion. At the time Doubleclick already had a very strong position with its publisher ad server (DFP for Doubleclick for Publishers) but also with its advertiser ad server (DFA for Doubleclick for Advertisers). And above all, most competing advertisers only position themselves on one side (publisher or advertiser).

In 2009, the launch of the Doubleclick Ad Exchange

In 2009, Google automated the “Search” market for a long time via its sponsored links. It has extended its AdWords advertising network to YouTube and its Google AdSense partner network, its performance advertising marketplace is particularly effective. On the other hand, “Display” advertising campaigns (advertising banners and videos on the web) are still mainly managed manually, “the old-fashioned way”:

The advertising agencies and networks will agree on advertising campaigns (X impressions, broadcast period, possible targeting, price).
The agencies will then configure the advertising campaigns in their adservers (DFA or other), then send the tags of these campaigns to the advertising agencies.
They will then configure these advertising tags in their ad servers (DFP or other).

This double programming is very laborious, especially when you want to broadcast your advertising campaign on numerous sites. How does ad serving work? It's already not that simple:

atlas

Via Ad Ops Insider, How does ad serving work?

In order to compensate for this lack, intermediaries called ad-networks already occupy an important place: they offer advertisers the opportunity to reach numerous publishers without having to contact them one by one (Google AdSense is one of them). The advertising agencies then configure the ad-network tag in their ad servers.

But managing multiple ad-networks remains complex and inefficient: ad-networks cannot buy in real time because there is not yet an auction system (the remuneration is fixed in advance, regardless of the user) and they reserve the right to refuse to buy an advertising opportunity. Advertising agencies find themselves having to manage the different ad-networks in a system called "Waterfall": a cascade of ad-networks, called one after the other (those which are assumed to be the most profitable first), a system whose aim is to maximize revenue (while limiting the damage to the user experience).

waterfalls

Via Ad Waterfalls explained, "passbacks" are due to the fact that the ad-network must call back the publisher's ad server if it has nothing to broadcast, so that it then calls back the next ad-network... The user experience is catastrophic.

Inspired by the success of AdWords and building on its recent purchase of Doubleclick, Google is launching its own marketplace: the Doubleclick Ad Exchange (read the interview of the time, in 2 parts). What is the impact of programmatic on the functioning of advertising distribution? The mechanisms are becoming more complex:

rtb

Via Ad Ops Insider, How RTB ad serving works

Google is slightly behind other players such as Right Media, AdECN, AdBrite and ADSDAQ. But programmatic is just getting started and the new marketplace is ticking all the right boxes :

Real-time auctions, for which the buyer can use the personal data he holds about the user.
Payments secured by Google (trusted intermediary).
For buyers of the new marketplace, simplified access to AdSense publishers but also to large publishers who already use Doubleclick DFP and who have activated Doubleclick Ad Exchange.
For Adwords buyers, new access to major DFP publishers who have activated Doubleclick Ad Exchange.
For large publishers, simplified access to AdWords advertisers, but also to the various ad-networks which have already integrated with the marketplace (from the launch, Google announced that it had integrated most of the large American ad-networks).
For large publishers, the dynamic allocation in DFP between their direct sales and sales via the Doubleclick Ad Exchange, allowing them to maximize their income (competition between campaigns sold directly and Doubleclick Ad Exchange auctions).
For AdSense publishers, additional demand via Doubleclick Ad Exchange buyers.

How programmatic works with Google, thanks to the acquisition of Doubleclick and the launch of the Doubleclick Ad Exchange:

AdExchange

Do you see conflicts of interest emerging?

In 2010 and 2011, other acquisitions allowed Google to expand its weight in Display

In 2010, Google buys AdMob for $750 million. AdMob offered monetization solutions for mobile sites and applications. Google was already very well placed with Google AdSense on the web and decided to pay a high price not to let the wave of mobile advertising pass by. He rewrote the software to include it in the Doubleclick suite in 2013.

Still in 2010, Google made one of its best moves in buying Invite Media for $70 million. Invite Media was a leading DSP in the market and a missing piece at Google. It allows an advertiser or agency to purchase advertising inventory on Ad Exchanges. Google is thus completing its tools for advertisers: in the same way that it was able to add the Doubleclick Ad Exchange component to its DFP publisher ad server, it is adding the Invite Media purchasing component (later renamed Doubleclick Bid Manager or DBM during the software rewrite) to its DFA advertiser ad server.

These purchases of tools on the advertiser side allow it to now offer 2 purchasing platforms:

Google Ads : the purchasing platform allowing all advertisers, small or large, to buy advertising on Google search results. It also allows small advertisers to run advertising campaigns on YouTube, AdSense and Google's Ad Exchange.
Google Display & Video 360 or DV 360 : the “Display & Video” purchasing platform for large advertisers and agencies, integrating the old DBM but also DFA (renamed Doubleclick Campaign Manager or DCM). They can buy inventory on YouTube, on AdSense, on the Google exchange but also on the main third-party Ad Exchanges.

In 2011, Google increased its dominance on the publisher side by buying AdMeld for $400 million. AdMeld was a competitor of Doubleclick Ad Exchange, Google still wins on several levels:

AdMeld had a lot of quality publishers in its clients.
If the Ad Exchange part was already well managed by Google, AdMeld complements the capabilities of DFP and Doubleclick Ad Exchange via better management of ad networks (programmatic is not yet dominant).

Ecosystem

The programmatic ecosystem, Google holds in 2011 with Doubleclick the main publisher adserver (as well as the main advertiser adserver, absent from this graph), but also a new important Exchange (Doubleclick Ad Exchange, improved thanks to AdMeld) and one of the main purchasing platforms (Invite Media). Google has other pawns that it integrates into its “stack”: Admob on mobile, AdWords and AdSense, Google Analytics, Google Tag Manager, etc.

In 2013, Google's exchange (Doubleclick Ad Exchange, renamed AdX, and now Authorized Buyers) became the main programmatic marketplace. Google's purchasing platform (DBM, now DV 360) is itself now in the majority.

wsj

Mechanism for delivering advertising, entirely within the Google ecosystem. Check out the Wall Street Journal's excellent infographic, How Google Edged Out Rivals and Built the World’s Dominant Ad Machine: A Visual Guide

Beyond the quality of Google products (which should not be underestimated), what are the key points of this rapid rise?

Google has an information advantage over its competitors

Thanks to its purchase of Doubleclick in 2007, Google acquired a huge footprint on the web. Many large publishers use Doubleclick as a server (DFP), which allows Google to be first in identifying users on the web and recognizing most of them, via doubleclick cookies. With most large agencies also using Doubleclick as an advertiser (DFA), Google is further increasing its footprint (even if a site does not use DFP, most of its advertising campaigns will be served with a DFA advertiser tag).

There you could say to yourself: Doubleclick is a tool purchased by publishers and advertisers, the identity of users on the tool (a digital identifier) is not the property of Google but theirs. And indeed, that is what Google tells US Congress during the investigation into the takeover of Doubleclick: "That data is owned by the customers, publishers and advertisers, and DoubleClick or Google cannot do anything with it".

Without access to a user identifier, it is much more difficult to monetize a user on the programmatic market. A Google study indicates, for example, a 52% drop in revenue from users who do not have third-party cookies, necessary for storing user identifiers for programmatic purposes on the web. This figure is also found in other independent studies.

And Google is gradually using the Doubleclick user ID for its own interests:

From 2009 for its Google Ads purchasing platform, it places doubleclick cookies on its own sites and with its AdSense partners: "Google uses the DoubleClick advertising cookie on AdSense partner sites and certain Google services to help advertisers and publishers serve and manage ads across the web".
Still in 2009, Google uses the doubleclick identifier on its Exchange, the Doubleclick Ad Exchange, which allows Google Ads buyers to recognize users who surf to new customers of the Exchange. On the other hand, other buyers (ad networks and other DSPs, used by large advertisers) do not have access to this identifier : "For buyers, Google identifies users using a buyer-specific Google User ID consisting of an encrypted version of the doubleclick.net cookie, derived from but not equal to that cookie. Google passes the user ID to the buyer (raw cookies are never sent)."
As late as 2009, Google prevented its publisher adserver clients (here : "The DoubleClick cookie ID associated with the user, encrypted.") and advertisers (here : "You will never be able to decrypt user IDs, and Google will not disclose the encryption method. No encryption keys will ever be provided to any Campaign Manager customer or any third-party partner.") to access doubleclick user identifiers, contradicting his previous statement: "That data is owned by the customers, publishers and advertisers, and DoubleClick or Google cannot do anything with it".
In 2012, when Google rewrote Invite Media to launch DBM, its own purchasing platform for large advertisers, it has access to the doubleclick identifier : "For itself, Google identifies users with cookies that belong to the doubleclick.net domain under which Google serves ads."
In 2013, Google only provides its DFP publishers with logs, without the doubleclick identifier : "The DoubleClick cookie ID associated with the user, encrypted."
In 2018, Google remove doubleclick identifier of logs from its DCM server : "You will never be able to decrypt user IDs, and Google will not disclose the encryption method. No encryption keys will ever be provided to any Campaign Manager customer or any third-party partner". Reminder, Google declared when purchasing DoubleClick: "That data is owned by the customers, publishers and advertisers, and DoubleClick or Google cannot do anything with it".

Checkmate! Here are the consequences:

Advertisers and publishers used to work directly together for targeted advertising campaigns, via the doubleclick identifier. They must now use Google, DBM and AdX tools (or Google Ads and AdSense for small advertisers and small publishers).
DBM and Google Ads have an advantage over other purchasing platforms: they must synchronize their identifiers with the doubleclick identifier (cf. Cookie Matching) before being able to recognize users when Google sends them advertising opportunities and thus buy better. At best, purchasing platforms can hope for non-recognition of users of around 20%, at worst more than 50%.
Competing purchasing platforms do not have the same footprint on the web while the doubleclick cookie is omnipresent: among AdSense and DFP publishers, on Google search, on Gmail, on YouTube, etc.

How does ID synchronization work? Here represented between a DSP and several SSPs, but the mechanism is the same when an SSP must synchronize identifiers with several DSPs:

sync

Via Ad Ops Insider, SSP to DSP Cookie Syncing Explained

Google explains very well the benefits “superior cookie reading” to its customers:

Google Ads and Display & Video 360 drive optimal performance when purchasing inventory on Ad Exchange because these buying platforms share the same infrastructure as the marketplace. This means that potential cookie read loss when Google Ads and Display & Video 360 users purchase on other marketplaces is reduced when purchasing on Ad Exchange. So when purchases are made through Google Ads and Display & Video 360 on Ad Exchange, it's more likely that these solutions will detect impressions that meet their targeting criteria, leading to increased bidding pressure and increased demand for the publisher's inventory.

Protect your privacy?! The beautiful hypocrisy of Google

Google does not detail the reasons which led it to remove access to the doubleclick identifier from third parties. Simply that he acted for reasons of privacy:

To prevent different actors from combining their information (each actor has its own version of the doubleclick identifier, and therefore cannot share user information with its neighbor).
To prevent actors from combining identifiers with personally identifiable information.

If the objective of protecting privacy is laudable, Google does not apply it to itself: it does combine the information it has about you via its various services (Search, YouTube, Gmail, Maps, etc.), via its various Doubleclick clients (advertisers or publishers), it also combines DoubleClick information with the personal information from your Google account if you have not correctly adjusted your Google settings. Here are 2 key decisions:

In 2012, Google then offered 70 different services (and 70 different privacy policies), the personal data of these different services were well compartmentalized. For example, YouTube does not know your Google search history and therefore cannot target you with advertisements related to this history. Google then decides to offer only one privacy policy and to combine your personal data across all of these services. Reactions are unanimously negative and investigations are multiplying but nothing happens, Google does not back down.
In 2016, Google quietly changes its privacy policy and removes the ban on combining information linked to the Doubleclick cookie and personal information from your Google account if Google does not have your explicit opt-in. The association will be automatic for new users. For existing users, Google disguises the dangerous association asking you if you would like to access additional features, with the title: “Some new features for your Google account”.

googolopoly

The Googolopoly game, created in 2008 (since this date, the number of companies acquired by Google has exploded). Google's mission is originally toorganize information on a global scale. Today, it is more about taking over ever-larger parts of your activities, to better influence yourself.

These decisions are accompanied by many insidious changes to Google products. In 2018, for example, Chrome automatically connects you to your Google account when you connect to a Google site (Gmail for example), without asking for your opinion. It also tricks you into sending all your browsing history to Google via a Dark Pattern, read about it »Why I'm done with Chrome".

If we are interested in Doubleclick data, remember Google's statement in 2007: "That data is owned by the customers, publishers and advertisers, and DoubleClick or Google cannot do anything with it". Today, for example, Google collects personal data via its Google Ad Manager publisher clients (example: users surfing on a financial information site), personal data which it can then use for its own profit. How? By selling advertising campaigns via Google Ads to banks, targeted at users of the first site, but on other websites (having themselves integrated Google AdSense or Google AdX).

If we look carefully at the Google privacy policy, hidden below the "Protecting Google, our users and the public" (and not in the " sectionOffer personalized services, particularly in terms of content and advertisements"), we can read:

Depending on your account settings, your activity on other sites and in other applications may be associated with your personal information in order to improve Google services and the ads served by Google.

This is how Google discreetly appropriates the data of its publisher and advertiser clients, for its own profit. But be careful, prohibiting third parties from accessing the doubleclick identifier has been designed to protect your privacy!

Google has a speed advantage over its competitors

When a marketplace (SSP or Ad Exchange) offers an advertising opportunity to a robot buyer (DSP), it has a limited time to respond, generally between 100 ms and 200 ms. Also, if he rarely responds on time, the SSP will reduce the number of opportunities sent. Google's purchasing platforms (Google Ads, DV 360) then have a huge advantage, they are located on the same servers as the Google marketplace (AdX):

Shared accommodation

Google's speed advantage, reduced latency.

Google also highlights this “reduced latency” in its documentation, with other players being able to “lose” up to 25% of auction requests:

When purchases are made through Google Ads or Display & Video 360 on other marketplaces, the users in question experience the same network latency as the rest of the buyers. We've found that in some cases, latency issues can prevent buyers from submitting a bid for up to 25% of auction requests, limiting their participation in auctions taking place on other marketplaces.

This time saving allows Google's purchasing platforms to be more "intelligent": to take the time necessary to retrieve user data and run the right algorithms in order to choose the advertising that will have the strongest impact on the user, at the optimal price (and which will maximize Google's profits).

Note that Google offers purchasing platforms a colocation option (via Google Cloud Platform), but only the OpenX SSP chose it. It is difficult to know if other purchasing platforms refuse to use the Google Cloud for cost reasons, for obvious questions of conflicts of interest, or simply because they judge that this advantage is not so decisive. The figure of 25% of auction requests lost due to latency problems cannot be audited because Google does not provide its publishers with information about auctions excluded because they arrived too late (cf. Bids data in Ad Manager Data Transfer).

Google imposes its own purchasing platforms

To be able to purchase inventory on YouTube, an essential site, an advertiser is obliged to go through Google's purchasing platforms because in 2015, Google decided to cut off access to YouTube inventory to third-party purchasing platforms. Also, Google was sued by American authorities in 2013 for limiting the functionality of Google Ads APIs, making it difficult to purchase Google sponsored links via third-party tools.

As we saw previously, using Google DV 360 and another shopping platform in parallel runs the risk of bidding twice for the same user (because the other shopping platform does not have access to doubleclick cookies and therefore cannot recognize that it is the same user) and thus artificially increasing prices.

Google's purchasing platforms are therefore essential for advertisers, with only Facebook able to compete, but it has no longer had a footprint on the web since the shutdown of its Audience Network (now only available on the App universe) and can thus "only" sell access to its own properties on the web (Facebook, Instagram).

Google directs purchases to its own platforms and sites

When a small advertiser creates an account on Google Ads, they must first start a "Search" campaign (on Google search results) before they can purchase inventory on third-party sites. A first method for directing purchases to your own sites.

Google can also direct purchases from its purchasing platforms to sites using its monetization solutions:

Until 2016, Google only directed Google Ads purchases to the Google Ad Exchange (cf. the presentation of the Doubleclick Ad Exchange at the time : "Ad Exchange is the only exchange offering access to the full demand of Google AdWords"), preventing other Ad Exchanges from taking advantage of this potential advertising windfall. This has since changed, but in a limited way: only for remarketing campaigns, cf. Google Ads online help.
If it has not blocked access to other Exchanges from its purchasing platform for large advertisers DBM (aka DV 360 now), it is frequently accused of directing purchases primarily to its own Ad Exchange: "Most DBM buyers we talk to say more than half of their spend goes to AdX when they use DBM" then declared the CEO of AppNexus, a rival.

Google directs sales to its own marketplace

On the publisher side, the dominant server is Google DFP. This was recently merged with Google AdX to be renamed Google Ad Manager. Following the acquisition of Doubleclick in 2007, Google took advantage of DFP's dominant position to impose AdX. Here's how he did it:

Directly in DFP, the publisher can configure its guaranteed volume advertising campaigns (pre-sold, for which the buyer undertakes to purchase a certain volume of advertising opportunities, based on precise distribution criteria) and its non-guaranteed volume advertising campaigns (the publisher offers advertising opportunities, the potential buyer has no obligation to purchase).
Advertising campaigns with non-guaranteed volume may come from ad-networks having a direct agreement with the publisher. But as mentioned previously, the publisher must then configure a system of calls to ad-networks in cascade (one after the other), leading to loss of revenue (fixed price for each ad-network, no direct competition) and poor user experience (slowness).
Advertising campaigns with non-guaranteed volume can also come from Google AdX, integrated “natively” into DFP since 2010. This allows real competition between buyers (real-time bidding). It is also connected to Google's two purchasing platforms: DV 360 (formerly DBM) and Google Ads (formerly AdWords).
A competitive system called “Dynamic allocation" arbitrator between advertising campaigns with guaranteed volume and those with non-guaranteed volume. Its objective is to maximize the publisher's revenue while ensuring that guaranteed volume campaigns will respect the contract with the buyer and distribute the right impression volume, on the right criteria (inventory, targeting, distribution dates). How ? By dynamically setting a “shadow price” for guaranteed volume campaigns (strong enough to allow them to broadcast) and seeing if non-guaranteed volume campaigns offer a better price.
If AdX is natively integrated into DFP, other marketplaces are not as lucky. Until 2017 and the announcement of Exchange Bidding, DFP required publishers to configure other marketplaces as simple ad networks (and therefore to call them one after the other, after AdX if the latter had not already "pre-empted" the opportunity). They also had to enter a fixed selling price for each marketplace, except the principle of an auction is not knowing the selling price.

Consequence: at the end of 2014, publishers wishing to work with other marketplaces found themselves “hacking” DFP via a system called “header bidding”.

Header bidding, an imperfect hack to put Google AdX in competition

Header Bidding step by step, detailed on the Ad Ops Insider website :

Header

Very briefly, as DFP does not allow real competition between AdX and other marketplaces, header bidding allows competition between marketplaces (excluding AdX) outside the adserver (via the page header), before the call to DFP. The winner of the auction is then put into competition with the other DFP campaigns, volume guaranteed or not. The publisher is obliged to create as many “line items” (campaigns on DFP) as price brackets, for each marketplace!

header

Via the article What are the consequences of header bidding on programmatic buying?, header bidding allows real competition among third-party marketplaces.

This implementation allows publishers to loosen Google's grip a little through competition with other marketplaces, it also allows them to decide on the maximum response time granted to the different marketplaces, thus making it possible to take into consideration the bids of slower buyers. The downside: the tags from the different marketplaces, all called directly via the page header, cause delays and degrade the user experience.

Finally fair competition between the different market places? Not really...

The “last-look”, or how Google gains a new advantage over its competitors

The devil is in the details and third-party marketplaces are still at a disadvantage compared to Google AdX, explanations:

RTB auctions then operate almost exclusively in "2nd price auction" mode: the buyer only pays the price of the 2nd auction. Example: On the AppNexus marketplace, A bets €5, B bets €1 ==> A wins and should pay €1.
This €1 price also allows AppNexus to win the header bidding competition (against other third-party marketplaces).
AppNexus then submits this €1 price to DFP. Via dynamic allocation, AdX buyers know that they must bet more to hope to win the competition.
C bets €2, D bets €0.5, the floor is €1 ==> C wins and pays €1.

This advantage that Google grants itself is called “last look” in the adtech world.

We see in this example, buyer A was ready to spend €5, but he was unable to spend his stake. The third-party marketplace had a potential buyer at €5 versus a potential buyer at €2 on Google AdX, they also unfairly lost the competition. The publisher could have earned €5 (minus commissions), but ultimately earns €1 (minus commissions). Only one winner: Google, which will have been able to satisfy its buyers and take its commission via Google AdX.

In order to maximize their chances of winning in the competition with Google AdX, third-party marketplaces have gradually (and belatedly, from 2018) decided to switch to auctions in "1st price auction" mode: the buyer then pays the price of their auction. A first part of the problem is solved: in the previous example, buyer A wins and pays €5. But Google continues to transmit this floor price to buyers of Google AdX (including Google Ads).

They therefore maintain an information advantage and can decide to bet 1 cent more to win the auction ("last look"). Third-party marketplaces remain at a disadvantage.

Google Ads can arbitrage without anyone realizing it

While an advertiser will be entitled to transparent information on their DSP (bid price and purchase price) when they buy on AdX, this is not the case if they use Google Ads: it does not provide this level of transparency in its reports. Also, the purchasing method on Google Ads is per click (CPC or cost per click), whether the advertising is broadcast on Google search results or on Google AdX. But programmatic sales are made by impression (CPM or cost per 1000 views), making comparisons more difficult.

Also, when it relies on Google AdX, Google Ads has a huge information advantage:

It does not need to synchronize user identifiers.
It benefits from the omnipresence of Doubleclick trackers (all Doubleclick publisher and advertiser clients, Google Analytics clients having activated advertising features, AdSense and YouTube), information that it authorizes itself to combine for its own benefit (remember, the merger of the privacy policies of Google's 70 different services in 2012).
If the user has not opted out, the doubleclick information is also combined with the personal information associated with the user's Google account (remember, Google's terms of service change in 2016).
Finally, it represents a considerable number of advertisers, all interested in displaying their advertisements on Google search results.

This information allows Google Ads to predict the click-through rate with great precision, and therefore to bid at the best price to optimize its own income (while guaranteeing acceptable performance for the advertiser). An example below, from the paper "‘Trust Me, I’m Fair’: Analyzing Google’s Latest Practices in Ad Tech From the Perspective of EU Competition Law" :

trust

Here, Google Ads receives €6 but only pays €3 to win the auction, it can keep the difference

Note that Google Ads takes advantage of several factors to arbitrage:

Just like the other participants in the AdX auction, he only pays the price of the 2nd bidder ("2nd price auction"). But as it operates in a black box, it can afford to collect the difference between its bid and the actual sale price, without any accountability.
The CPC purchasing method makes the audit task even more complicated: only Google has the information on the expected click-through rate.
Even if the AdX auction changes to a "1st price auction" (we'll talk about this later), the "last-look" allows him to bid less than he could, and to collect the difference.

Open Bidding, Google's answer to header bidding

Header bidding remains a hack that Google doesn't like:

He is not the originator and above all, does not take commission on sales that go through other marketplaces.
As a result, Google has no interest in making operations easier for the publisher: setting up header bidding in DFP is complex, reporting is also very complicated (new adtech players are taking over to help with these complexities).
Google does not allow Google AdX to be integrated into header bidding solutions. The publisher must run a first auction on the header bidding side, then a 2nd auction in DFP, degrading the user experience. Multiplying partners in the header also runs the risk of degrading the user experience.

In 2017, Google proposed moving the competition to the server side (at home):

Server

The "Server-Side Header Bidding", explained on the Ad Ops Insider blog

The feature is initially called EBDA (Exchange Bidding in Dynamic Allocation), now renamed Open Bidding. Google Ad Manager puts marketplaces that are willing to integrate with it into direct competition (which is not a given, note for example the absence of AppNexus, one of the largest marketplaces, and the main initiator of header bidding) and Google AdX.

Marketplaces gain easier access to the many publishers who use Google Ad Manager. Google also provides marketplaces (and other buyers) with the effective selling price of each auction, which it does not do during header bidding integrations.

The advantages for the publisher?

If he still has to sign a contract with the different marketplaces, the system is “plug & play”, no operational hassles.
Payment is made directly by Google, which makes accounting easier (and publishers know that Google pays quickly, which is less true of smaller SSPs, who must be paid by DSPs, who themselves must be paid by agencies).
The publisher has no limits on the number of third parties to work with, because there is no impact on the user experience (no additional tags on site, calls take place on the server side).

But, nothing is free and Google takes a commission of 5% or 10% depending on the advertisements (standard, video or app). A third-party marketplace is at a terrible disadvantage:

She must first take her commission, then submit the auction to Google which also takes its commission before comparing the reduced price to other auctions, see Google online help : "Highest net bid (which takes Ad Manager's revenue share into account) wins the impression".
Just like third-party purchasing platforms on the Google Ad Exchange, it does not have direct access to the user (Google calls them server side) and must therefore synchronize its user identifiers with those of Google before an auction. This leads to a lower rate of user recognition, and therefore poorer monetization (knowing that it must also synchronize its user identifiers with those of the purchasing platforms).
She is also at a disadvantage in terms of speed: she must respond to Google in less than 160 ms, knowing that she must conduct her own auction herself. As seen previously with third-party buying platforms buying on Google AdX, being located on the same servers (Google Ads and Google DV 360) allows buyers to bid more often and better. The logic applies here between Google AdX, advantaged because it is located on the same servers as Google Ad Manager, and other marketplaces.
Smart buyers look for the cheapest opportunities, and therefore with the fewest intermediaries. They will favor a direct purchase via Google AdX rather than an Open Bidding partner. This optimization mechanism is called "Supply Path Optimization" or SPO, it is applied by all DSPs worthy of the name. If you are interested in the topic, you can read this article by AppNexus co-founder Brian O'Kelley.
Obviously, the publisher can deactivate a third-party marketplace but it is impossible to deactivate Google AdX.

Google doesn't let publishers understand how bidding works

Understanding how these auction mechanisms work is crucial for publishers: which buyers are interested in my inventory? What price are they willing to pay? How do they behave if I change the floor price? Should I contact them directly if they buy a lot of inventory, to secure preferential agreements? Transparency is essential, namely access to all the information on these auctions, only Google restricts the information:

Access to information costs money!
Auction data (how much each buyer is willing to pay) and impressions (who won and what the final sale price is) are accessible via 2 different files, one for "print data" and another for "auction data".
Auction data contains the buyer's name, the bid value, and whether the auction won. Missing from the file: auctions from other marketplaces, as well as the effective purchase price.
The impression data contains the winner, the effective purchase price as well as the bids from other marketplaces.
Google Ads advertisers are always hidden behind the Google Ads buyer: it is impossible to know them, nor to know how much they paid (Google's margin is unknown).
An editor could make the link between these 2 files but since Google's announcement in September 2019, this is no longer possible.
Consequences: impossible to verify the impartiality of Google's unified pricing, impossible to verify whether Google does not arbitrage, impossible to verify whether Google does not run the AdX auction after receiving bids from other marketplaces, etc.
Also, Google gains valuable information on the behavior of rival marketplaces. It would not be difficult to use this information to predict their future bids, and thus benefit Google AdX.

The end of the “last-look”, really?!

Google decided in March 2019 to follow other marketplaces and switch Google AdX to “1st price auction” mode, he first communicates vaguely about the end of the “last-look” :

An advertising buyer’s bid will not be shared with another buyer before the auction or be able to set the price for another buyer

Its explanatory diagram is also very vague (also, it is impossible to know if the "1st price auction" applies when Open Bidding is not activated):

Auctions

In May 2019, he gave a little more details" :

Going forward, no price from any of a publisher’s non-guaranteed advertising sources, including non-guaranteed line item prices, will be shared with another buyer before they bid in the auction

Why did I put "including non-guaranteed line item prices" in bold? Because these "line items" (campaign elements entered on Google Ad Manager) are used in particular by third-party marketplaces for header bidding. And if we look at the history of the Google article via the Wayback Machine, we realize that these "line items" were not included in the initial version of the article, here is the complete passage from the time:

After the transition, Ad Manager will have a single auction that compares the prices from a publisher’s guaranteed campaigns with all of a publisher’s non-guaranteed advertising sources — including real-time bidding partners, such as Authorized Buyers and Exchange Bidding partners — and prices from non-guaranteed line items, like those from a publisher’s header bidding implementation. Going forward, no price from any of a publisher’s non-guaranteed advertising sources will be shared with another buyer before they bid in the auction. As has always been the case, all real-time bidding partners integrated with Ad Manager — including Google Ads and Display & Video 360 — will be notified of an auction at the same time.

If we read the initial extract carefully, as Damien Geradin and Dimitrios Katsifis, the authors of "'Trust me, I'm Fair': Analyzing Google's Latest Practices in Ad Tech From the Perspective of EU Competition Law" as of October 2019, Google does not mention the non-guaranteed line items when it indicates that it no longer shares price information before the auction, only the non-guaranteed advertising sources, namely AdX buyers (Authorized Buyers) and Exchange Bidding (Open Bidding).

It turns out that the specialized press had already announced the end of the "last-look" in 2017, which seemed at the time to only concern Open Bidding (was this really the case, if Google only communicates in 2019?). Same story in March 2019, the specialized press hastens to announce the end of the “last-look”. An argument put forward: the "last-look" would be incompatible with 1st price auctions. This is not the case: Google Ads (or another AdX buyer) can absolutely decide to bet 1 cent more than the winner of the header bidding auction if it has the information, which will make it possible to give much less money to the publisher and potentially keep the margin for themselves.

So what happened with the article? According to the Wayback machine again, the non-guaranteed line items have been added to the article between November 15, 2019 and August 15, 2020. Is this following the paper by Damien Geradin and Dimitrios Katsifis? Did Google really remove the “last-look” (depriving itself of an arbitrage option via Google Ads) in May 2019? Did he really do it between November 15, 2019 and August 15, 2020, and if so, why did he not communicate about this development? It is impossible to verify because Google does not provide sufficient transparency via auction and impression data, so it is doubtful.

The “1st price auction”, an excuse to block the monetization of Google Ads

With the launch in 2019 of 1st price auctions, Google has shaken up the rules for creating floor prices for publishers. Previously, they could decide to set floor prices higher for certain buyers. A frequently used rule: increase the floor price of Google Ads (to prevent Google Ads from finding itself paying much less than it could, thanks to the "2nd price auction" and the "last-look").

Also, these floor price rules only applied to AdX buyers. Publishers could manage header bidding or Open Bidding partners separately, and apply a lower floor price to them if they wanted. The new "unified rules" remove this option, and obviously Google presents this as an advantage:

Our new unified pricing rules will help publishers more easily manage floor prices across all non-guaranteed partners. For example, instead of setting up the same floor prices in multiple places — in the auction in Ad Manager, and with their Exchange Bidding and other non-guaranteed advertising sources — which can take a lot of time and can lead to errors, a publisher can set up a single unified pricing rule to control pricing from one place.

Google explicitly prohibits creating different floor price rules, depending on the purchasing platforms. He presents this abusive maneuver as a measure of justice:

To maintain a fair and transparent auction, these rules will be applied to all partners equally, and cannot be set for individual buying platforms.

Above all, this is a massive loss of control for the publisher over the sale of its inventory, and a golden opportunity to accelerate arbitrage via Google Ads.

The AMP weapon to eliminate header bidding

The AMP format is a web page format launched by Google in 2016. Its objective is to speed up the loading of pages on mobile, but it is seen by many as a threat to the open web: through its dominant position in online search, Google forces publishers to develop a specific and very limited version of their site, cached on Google's servers. Without an AMP version of their site, publishers do not have access to carousel. The lightning bolt representing AMP in search results also encourages users to click on those results.

Also, Google does not say that AMP is a ranking factor (better ranking in search results), but it emphasizes the fact that loading speed is very important : "Even though AMP itself does not directly impact your position in search results, speed is a determining ranking factor for Google Search". In theory publishers have the choice, in reality and if they do not want to lose their ranking in Google search results, they are forced to provide an AMP version of their pages.

And for advertising you say? AMP prevents publishers from using client-side header bidding (javascript library), so they are encouraged to use the Open Bidding solution if they want to work with third-party marketplaces, or simply just use the Google marketplace exclusively. As AMP pages are cached at Google, the entire advertising chain can run on the same servers, again the advantage of speed (but for monetization, apart from that resulting from Google's request, it's not won).

AMP being widely criticized, Google's new strategy consists of pushing Core Web Vitals, new metrics intended to better represent the overall user experience. These metrics will be taken into account next year in the rankings : "Today, we’re building on this work and providing an early look at an upcoming Search ranking change that incorporates these page experience metrics". Publishers will not need to have their pages in AMP versions to be featured in the carousel, but rather to have a good score on the Core Web Vitals.

Vitals

Core Web Vitals, a measurement of user experience on different axes. Today, Google already measures these indicators via Chrome users who have synchronized their browsing history and have the sending of statistics activated (it makes aggregated statistics available via the Chrome User Experience Report).

While this initiative is laudable, there are fears that AMP will be replaced by an even more problematic version, the Web Bundles : a new format allowing a site to group all the resources of a web page in the same file, and potentially cached by third parties such as Google. A key point for us: these Web Bundles will greatly complicate the task of adblockers. Read the article by Peter Snyder on this subject, WebBundles Harmful to Content Blocking, Security Tools, and the Open Web.

Programmatic direct, an additional weapon to tax the advertising ecosystem

If “Display” advertising now mainly passes through programmatic channels, direct relationships between buyers and publishers have not disappeared. In particular:

Publishers with quality inventory and/or a specific audience and/or "innovative" advertising formats can always offer preferential deals to buyers (inform them of these special opportunities, or even give them priority over other buyers). Obviously, buyers will push to be entitled to these preferential agreements.
Buyers wishing to secure their purchases will always need publishers to guarantee a certain printing volume. This need is particularly important when inventory is scarce: if I want to communicate about a new film, I will for example want to broadcast 10 million impressions on YouTube between November 5 and November 8.

Before programmatic became more widespread, these transactions went only through advertisers and publishers. As a result, they did not pay an RTB tax (no commission on the price of the media to be paid on the DSP side, no commission to be paid on the SSP side, only the adserver distribution cost, which is fixed and consequently much cheaper). Google, in the same way as its competitors DSP and SSP, saw a significant shortfall in this: it therefore pushed the advertising ecosystem to bring in these transactions in the programmatic pipeline (via a system called “Deals”):

First with the "Favorite chords" (aka "Preferred Deals"): agreements between 1 buyer and a publisher, at a fixed price, priority over other non-guaranteed campaigns (AdX, Open Bidding, header bidding and ad-networks) but with non-guaranteed volume (the buyer does not commit to purchase, he can choose the opportunities).
Then with the "Guaranteed programmatic": agreements where the buyer agrees to purchase a guaranteed volume of prints.

Here are the arguments that can justify paying for a service on these advertising transactions:

In the case of "Preferred Deals", the buyer chooses the opportunities (in the old world, when called, the advertiser's server has no choice).
The DSP manages all of the buyer's campaigns, which allows this tool to offer unified reporting.
Payments to publishers are now simplified and secure, the SSP takes care of it.
If setting up over-the-counter campaigns is complex, Google allows buyers and publishers to negotiate agreements directly in their respective interfaces (DV 360 and Google Ad Manager). Note that this option could have been offered without the RTB pipes but that Google had nothing to gain from it.

And if we compare the share of “Deals” in open auctions, we realize that these over-the-counter relationships represent a considerable part of advertising investments. They were less visible before because advertising campaigns did not pass through programmatic channels:

pmp

eMarketer figures and forecasts for the US market, “Deals” must exceed open auctions in 2020. France generally follows the same trend, 1 or 2 years late. The potential for programmatic players is therefore enormous.

But it is difficult to justify continuing to take a commission on the media amounts spent (X% on the DSP side, Y% on the SSP side): this is a direct agreement between a buyer and a publisher, the intermediary is not a business provider. We would rather expect a fixed CPM, similar to the prices of an adserver. However, this is Google's choice, and with which the programmatic ecosystem has aligned itself.

Direct programmatic, an additional advantage for AdX

These over-the-counter transactions are a good way to discriminate against other marketplaces:

Google offers a Marketplace in DV 360: this allows buyers to discover publishers' offers, but also to negotiate with them directly in the interface. If the negotiation is successful, the creation of the campaign is automatic, for the buyer and the publisher. Problem: this Marketplace is almost exclusive to AdX publishers (excepté Rubicon).
“Guaranteed programmatic” involves being able to guarantee a volume of impressions. Only adservers are capable of managing a printing volume. As a result, marketplaces that are not coupled with an ad server cannot sell guaranteed programmatic (to Google DV 360 or another purchasing platform). Publishers using Google Ad Manager (DFP and AdX) are of course already equipped.
Within Google Ad Manager, "preferred agreements" are only available on AdX: Open Bidding partners and marketplaces using header bidding cannot be prioritized over other non-guaranteed campaigns.

So, if you are a third-party marketplace, it is almost impossible to compete with AdX at Google Ad Manager publishers, only to collect the crumbs.

Replace Google Ad Manager? Mission (almost) impossible

We have seen it many times, the first tool called on the publisher page is strategic: it is the publisher ad server (DFP in the case of Google, which was recently "merged" with AdX to be renamed Google Ad Manager). Google promotes its own demand via multiple processes: "last-look", complexity in managing header bidding, tax on Open Bidding, black box on auction and impression data, restrictions on floor price rules, better user recognition, better connection with its own purchasing platforms, merging of your doubleclick and Google personal data, facilitation of direct programmatic, etc.

What happens if a publisher decides not to work with Google on the Adserver part? The difficulties are very great:

Adding a new marketplace on Open Bidding is very easy, adding a new marketplace in header bidding is relatively easy, migrating an advertiser is a much more complex task operationally, which can take several months.
Google AdX is the only marketplace with the entire Google Ads demand (other marketplaces only have remarketing campaigns). Excluding Google Ads is the largest buyer in the world. But Google refuses to offer AdX in header bidding.
Google also refuses to integrate AdX with competing server-side header bidding systems (such as Prebid Server or Amazon Transparent Ad Marketplace) or to marketplaces and other competing ad servers.
Consequence: if you do not want to cut yourself off from the main source of demand (namely: putting AdX campaigns in competition with the campaigns of other marketplaces at the same level), you must use AdX, and therefore you are strongly encouraged to use Google Ad Manager.

Can you avoid Google Ad Manager and take advantage of AdX? Yes but only via a incredibly complex mechanism, set up by Axel Springer (the largest European media group) with the help of its partner AppNexus (competitor of Google Ad Manager, offering ad server and SSP) and allowing AdX to compete with other sources of requests:

Axel Springer

If you don't understand anything about this diagram, that's normal.

Necessary regulation

If Google has been able to impose itself so effectively on the advertising market, it is not only because its tools are "better" (the argument should not be underestimated, Google offers good tools with an excellent user interface), but it is above all because it has been able to take advantage of an absolute absence of regulation to buy up multiple companies and abuse its dominant position.

The different documented practices are not exclusive to advertising markets. Like Dina Srinivasan documents it so well, we can find these practices in other markets such as financial markets, online ticket sales, etc. However, other markets are regulated and sanctions are taken when an actor abuses an advantage. As a result, these markets perform better.

How was Google able to escape any regulation in these advertising markets? Without being a legal expert, one of the elements is undoubtedly the complexity and opacity of adtech, which you may have noticed if you stuck around to read this entire article... The options offered to regulators are numerous, here are some suggested in recent studies:

Structurally separate (sales, or independent divisions with strict controls) the adserver (DFP) from the marketplace (AdX).
Structurally separate its purchasing activities (Google Ads, DV 360) from sales activities (AdX, AdSense, AdMob).
Structurally separate Chrome.
Activate all Google Ads demand on other marketplaces.
Allow adservers to simply access the AdX request.
Allow third-party shopping platforms to purchase YouTube inventory.
Facilitate the purchase of advertisements on Google search results through third-party platforms.
Force Google to provide full transparency on auction and impression data.
More generally, regulate and sanction the entire advertising ecosystem: Google is the company that abuses the most, but it is not the only one.

More fundamentally, what is the real usefulness of this ultra-complex programmatic ecosystem, at the origin of massive personal data leaks ? We can clearly see the negative effects: catastrophic user experience, trivialized and generalized surveillance (thanks Google), extreme media dependence on Google... If we look at its cost and opacity, the results are shocking:

intermediaries

Study on programmatic transparency : yes, the publisher only receives 51% of the amount spent by the advertiser, also note that the study cannot explain 15% of the money spent...

As a publisher, if you don't offer a subscription, is a life outside of adtech possible? Yes if we believe the experience of “radio france dutch”, which saw its advertising revenues increase, while abandoning targeted advertising. So of course, the experiment is easier to carry out when you are a public group, but the game is worth the effort.