As Supply Of Demand-Side Rises, Flaws Are Exposed In RTB's Bidding Infrastructure
- by Joe Mandese @mp_joemandese, June 27, 2013
As the number of brands bidding on online display inventory in the rapidly expanding RTB marketplace grows, some questionable business practices are beginning to emerge that are at the
very least illogical, and quite possibly undermining the very reasons brands -- and even publishers -- participate in the market in the first place. And while the ad technology industry likes to
compare it to Wall Street’s hyper-efficient trading markets, some of the business rules governing the way Madison Avenue’s programmatic marketplace work evolved in ways that increasingly
seem idiosyncratic, capricious, and even counterproductive.
One of the worst of them -- the so-called “single-bid” function -- has come to light as some of the biggest trading
platforms begin to observe a peculiar behavior occurring when DSPs (demand-side platforms) begin bidding on the same inventory on behalf of an increasing number of competing brands, effectively
nullifying the relative demand of each of them. The effect could be costing individual brands crucial “wins” of the inventory they covet the most, and could also be costing publishers who
participate in the RTB marketplace the kind of upward pricing pressure that occurs from truly efficient supply and demand.
“I don’t want to make it seem like anyone is doing anything bad on purpose,” says Frank Addante, the founder and CEO of one of the biggest trading platforms, Rubicon Project. “It’s just how the business is evolving.”
Addante, who is one of a number of industry players who has begun to detect and question these arcane business rules, says they likely occurred from the staggered way the RTB marketplace evolved; the fact that it has been a progression of retrofits, as opposed to a planned-out process; and the fact that unlike Wall Street’s open and regulated markets, no governing entity oversees their development. They just happen.
That seems to be the case with the single-bid function, which basically means that a DSP issues only one bid for a user impression being auctioned, regardless of how many clients it has bidding for it internally. Because brands often round the price they are willing to pay for the impression, the result is that multiple brands effectively end up bidding the same price for the same inventory in an auction where the win can only go to one.
Addante says it’s not even clear how DSPs determine how the winning bids are allocated among a DSP's clients in that situation. And while he believes it is likely random, and therefore neutral, it still has the effect of nullifying the relative demand among the brands in each DSP competing for the impression, and keeping potential price pressure down.
There are similar odd business rules in the RTB marketplace’s infrastructure, like the so-called “second price” function, which means that regardless of how high a price the most aggressive bidder offers, they will only pay 0.01 cents more than the second-most aggressive price bidded. While that rule also would seem to dampen upward pricing pressure that could lead to higher yields for publishers, most observers say it is at least grounded in sense, because the idea behind it is that it was established to build demand from advertisers who might otherwise be afraid they would overpay in an auction-based media marketplace. The single-bid function makes almost no sense, says Addante, who speculates that it emerged when DSPs were serving only a few brands bidding on the same inventory at the same time, and therefore no bidding conflicts were apparent.
The most likely reason for it, he says, is that it helps the DSPs manage their costs -- because creating the technological infrastructure necessary to manage individual bids for each and every client bidding on all of the billions of impressions available simultaneously in the real-time marketplace would be prohibitive, if not impossible, for many DSPs to manage.
Addante says it’s even possible that many of the DSPs, and their brand and agency trading desk clients, aren’t even aware of this conflict because they don’t have access to the data and the optics necessary to see the way the RTB market functions in its broadest sense. He says Rubicon only began to observe it recently, because it is managing so many bids from so many DSPs now. It recently passed a significant milestone, certifying its 100th DSP on Rubicon’s systems. But the real problem, he says, is not the number of DSPs participating in the marketplace, but the fact that so many DSPs are managing so many clients’ bids via the multi-bid process.
“There’s a solution for this,” he notes, adding: “The solution is that every single individual advertiser submits their own individual bid. But that’s very expensive for a DSP to manage.”
On that note, Online Media Daily asked Addante to check whether any DSPs currently are utilizing such an approach, and after investigating the matter with Rubicon’s technology team, he said they believe only one -- MediaMath -- is utilizing it to some extent.
“No one provides complete order-book visibility like that,” explained Rubicon’s Chief Scientist Neal Richter. “The closest is MediaMath at up to 10 bids per impression. That said, there’s a diminishing return after three or so open market bids per impression are sent back in terms of 'liquidity,' but far more data-gathering for bid landscapes analysis. Multi-bidding is required for DealID bids and the programmatic guaranteed trading build using the DealID to work effectively."
It
could be that the DSPs are beginning to recognize this problem and are taking their own steps to resolve it. Richter noted that at least one other major player, AppNexus, is preparing to roll out
multi-bidding -- and he noted that “a couple of smaller DSPs do full multi-bidding well,” but he did not disclose them.
"We worked with 10+ DSPs on a review of their
bidding logic around DealID bidding and jointly produced an updated protocol spec submitted to the OpenRTB standards body, Richter explained, adding, "That spec also contains a set of bidding
logic best practices such that the needs of open market and DealID bidding are combined."
Rubicon’s Addante says those developments are a healthy sign for the RTB marketplace, and that it indicates the marketplace is beginning to mature, and that if more players adapt it will be good for the entire industry, creating a truly open and efficient marketplace based on the underlying demand advertisers have for publishers’ inventories.
“if every advertiser was able to bid on every impression that would be greater service for both the advertiser and the publisher, because then they would both have visible insights to a fully transparent auction.”
Joe Mandese and Frank Addante are making a great point that deserves an example. If DSP-1 has two brands bidding on an impression at $1.00, it will pick one bid to submit for $1.00. If DSP-2 also submits a bid for one of its' brands at $0.50, the SSP will clear the auction at $0.501 while the two brands of DSP-1 were willing to bid against each other up to and including $1.00. In a proper auction that included all bidders, the impression would have sold for $0.991.
I'm not sure that I understand the problem, the DSP behavior is rational given that it's a second-price auction, and the second-price auction is better both for advertisers and publishers because the alternative is to encourage strategic bidding.
To elaborate:
The first issue is that DSPs will "aggregate" bids, so if the DSP has 3 advertisers that respectively bid CPMs of $0.50, $0.60, and $0.70 - then the DSP will ignore the two lower bids, and just bid on behalf of the $0.70 advertiser on the exchange. In effect, the DSPs run an internal auction, and then bid on behalf of the winner on the wider exchange.
This is a rational thing for a DSP to do on behalf of their advertisers because it is a second-price auction. Let's say we bid $0.70 and the next highest bid in the exchange is $0.55 - the DSP will pay $0.56 for this impression on behalf of the advertiser.
But let's say that the DSP bids on behalf of all three advertisers, they will still win, but this time they'll pay $0.61 because their own advertiser has driven up the second highest bid. So the DSP hurts both themselves and their advertisers if they don't conduct an internal auction first.
So should RTB abandon second-price auctions? No. That would encourages strategic bidding which would hurt all market participants. With a second-price auction you are incentivised to bid the maximum amount you are willing to pay (just like when you set your maximum bid on eBay), because you know that if your bid is dramatically higher than what anyone else is willing to pay, then you only pay (slightly above) the next highest bid.
Without the second price, advertisers would be incentivised to experiment with the amounts they bid to avoid overpaying. The result may be advertisers bidding less than they are really willing to pay (if they're rational then they'll never bid more), which will mean advertisers will lose bids that they could have won, and publishers will make less than they otherwise would.
Perhaps I've misunderstood something.
(sorry for the unbroken block of text above, it had paragraphs when I submitted it)
You haven't misunderstood anything, Ian. Your interpretation is spot on. Any DSP submitting more than one bid per request is doing a disservice to its clients.
It is not necessarily wrong to submit multiple bids at the same time. If you are bidding on inventory that blocks a lot of industries or advertisers, you can increase your odds of winning the auction by submitting bids for several different advertisers at the same time. You may be driving up the second price, but that may be the price you pay to win more impressions.
Your article assumes that most trading platforms are conducting fair second price auctions. In too many cases, some platforms analyze auction behavior and then modify floor prices and/or use soft floors or phantom bids to drive up the buyer's cost. This causes a second price auction to behave more like a first price auction; the more I bid the more I end up paying, regardless of competition. I'm not sure if a first price auction format would solve this problem, but as it stands now I already have an incentive to not bid maximum value if my bidding patterns will be used against me.
Both Bill and Ian are correct, yet are maximizing different objectives. The objective of the seller is to maximize liquidity in the auction and the objective of the buyer (sometimes) is to minimize clearing price.
All of this was settled 30+ years ago in auction theory. The second price auction is known to be flawed (to the seller) if there exist 'bidding rings' where an intermediary is truncating bids from a coalition of buyers to keep prices low. The theory dictates that the seller should respond with reserve prices and use data to set that price.
Fast forward 30 years and we have DSPs implicitly or explicitly acting as a bidding rings and thus exchanges are forced to implement reserve pricing algorithms with data to protect value.
Note that Rubicon platform will not self-second price any bids from the same seat/buyer or advertiser.
There is another practical reason to encourage multi-bidding. A non-trivial percentage of time the bids are invalidated by publisher rules on advertiser-names.
Frank, Neal,
I've had this discussion with Neal many times. It's in our clients (the buyers) interest to keep prices low by only submitting the highest bid from our customer set. Essentially you are asking all DSPs to open our books and show you the complete list of all of our clients and what they are willing to pay on any impression. This creates an asymmetry of market information.
From our point of view, we do not receive fully transparent information about wins, losses, and bids in your marketplace. If you want to talk about theories about efficient auctions, then transparency must go both ways.
In response to Jim's comment, multiple bids like those supported by Rubicon will allow the DSP to win more impressions in cases where the first advertiser is blocked, but this isn't really multiple bids. It is a single bid with a backup in case the first advertiser is disqualified.
+1 for Bill Simmons. Consolidating bids is part of the service. DSPs work on behalf of advertisers and SSPs work on behalf of publishers. It's open (not transparent) and competitive. It should remain that way.
Our average daily block rate is 15% and less than 10 DSPs have implemented the block list API or used multi-bid as an alternative. There's an easy win to be made here.
Even if we agree that the function of a DSP currently is to consolidate bids, I'm not convinced that all advertisers desire that. A leading cause of issues causing grief to advertisers is their bids not making it into the auction frequently enough. While truncating bids down to one minimizes price, it also minimizes opportunity for the demand to be observed.
Bill Simmons and I, with the rest of the OpenRTB group, worked on an extension to the standard encompassing DealID support with an ideal bidding framework. It's a good read for RTB geeks.
https://code.google.com/p/openrtb/wiki/ProposalforPrivateMarketplaceDealIDExtensions
Neal, Frank, this is very interesting POV. I agree with Bill regarding DSP looking to deliver the best possible results for their advertisers. Frank suggests that there are cost limitations on submitting multiple bids; this may have not been the case. Early on (2009), while working on the very early implementation of the interface, there was a concern on saturating bandwidth at the exchange side, and the agreement was that the most market efficient way for all, was to just submit one bid. Other less sophisticated DSPs, were indeed submitting multiple bids, creating a bit of a clog in the environment. More advanced systems do invest the computing power not only to calculate the multiple bids internally, but also to select the one to send, thus incurring in higher investments in terms of infrastructure. This is done to better serve the advertisers working within those platforms as others have argued.
First, we've been talking about this for over two years. This isn't new information. To Neal's point, it was one of the first things we asked the DSP's for back in the early days of RTB and the launch of OpenRTB. But it hasn't been talked about much.
I think you are all missing the most important point. Bill Simmons et al would be right in sending only one bid if they were TRULY sending the highest bid. In that case, the incremental upward pricing pressure would be minimal (really only affecting the second price), and the additional cost and asymmetry of information Bill references would offset the value of doing this.
BUT THEY DON'T. Think about it-- DSP's have lots of clients, all with different objectives and budgets. Their algorithms have to be able to control pacing correctly, and like the networks that came before them, they have to optimize for the greater good among their client base. Let's say based on pure math, client A can pay .50 for an impression, based on all the data available. Client B can pay .40 for that same impression. If client A is pacing according to budget, but client B is underpacing by a lot, there will absolutely be times that the DSP provides a single bid of .40 on behalf of Client B. This sucks if you're the publisher or if you're Client A.
I personally believe this happens a lot, especially since so many of the DSP's have algorithms that work like network algorithms. Note that some, like Turn, were actually ad networks before pivoting into RTB. Franks' point about retrofits is spot on.
Note that I expect DSP's would counter that their bid logic increases or decreases bids based on pacing and other factors, so they may be sending the "highest" bid if you think about it that way, but regardless of how you describe it, the DSP's are not running a true auction before sending the "highest" bid. Therefore, true multi-bid support would drive prices up. But it would probably also cause lots of operational issues for the DSP's in managing all their various clients.
Love that this discussion is getting out in the open. There is a lot of paranoia on the sell side. Ben, yes, you are right, we control pacing on behalf of our clients. If an advertiser is pacing ahead, of course we wouldn't submit bids continuously for inventory they don't need. Why would we do this??? This is like saying "I like bananas", so therefore I should keep bidding every banana I see in the market until they are gone, regardless of how much money i have in my wallet. It's our responsibility as agents for the buyer to pick and choose where to submit bids that best increase their delivery and performance. For clients that like banana, we bid highest for the best bananas, and for clients who like apples, we bid highest for the best apples. We represent many clients, and the highest bid goes forth to the market, since a lower bid will lose. Submitting a lower bid than the highest bid doesn't make sense. This would not be in the best interest of our clients. If one client is under pacing, this means they are asking us to bid below market rates. The whole magic of the RTB marketplace is that we have only have our customers interest in mind. Our incentives are aligned with theirs. We don't owe anything to the publishers. There is no need to run legacy ad network style arbitrage style optimization. There are no shenanigans going on here.
It makes no sense for a DSP to submit bids they know can't win. What purpose would that serve? Algorithms do have to control for pacing and it's one of the issues that make the algo guys cringe. Ben's right in one instance that Client A's bid might not be submitted due to pacing but that's because of Client A's budget constraints, not Client B's. Algorithms are notorious for refusing to compromise their principles for things like underdelivery and generally refuse to play favorites.
Great article, Joe. There are so many points of view, and this lively discussion is a proof point. A lack of transparency in the programmatic marketplace is certainly a pain point for both sides of the fence. Unless steps are taken to create a governing body akin to the securities industry's SEC to enforce regulation, we will continue to have this debate over the integrity of RTB technology.
@Bill- yep, agree. I was trying to make the point that it's logical behavior, therefore it's obviously happening. Not that it's the wrong thing for you to do as an agent, but that Frank's point that it's bad for marketplace liquidity is also true.
@BillS Paranoia haha. Not really from our side. We see the behavior of a hundred bidders and I'll say from experience that many of them have flat out dumb algorithms. Bidding the same price constantly on a site that has that advertiser blocked or floored at a level that makes it invalidate the bid. Who knows if the second place bid that was not sent might win? I'd love to see some gaussian noise and a feedback loop added. DSPs who do have adaptive pricing and variability in rotating advertisers are /more effective/ bidders because they WIN more often.
That stat I have above on 15% blocked bid average? For some DSPs with names we'd all recognize it's 20%+. So the bidding algorithms for which the demand side is worried about second price effects of multiple bids are doing the WRONG thing hundreds of millions of times per day. Irrational!
The DSPs that send multiple bids? They win more often, counter intuitively their global average CPM drops a bit and it starts virtuous cycles with publishers seeing demand they then respond to.
Here's the punchline IMO for the DSP side. You are competing with AdSense's bidder now. It has deep/liquid demand... and guess what.. it sends multiple bids to AdX's 'Optional Second Price Auction'. You can read all about it in the Google Research's academic papers.
http://www.cs.huji.ac.il/~noam/adx.pdf
Myerson, Riley and Samelson predicted all of this in their 'Optimal Auction' 1981 papers.