When it comes to artificial intelligence, two things are difficult to overstate.
The first is the existential threat the technology has posed to the media industry. In June, The Wall Street Journal called it an “armageddon” whose downstream effects on search traffic and programmatic ad revenue have upended the economics of digital publishing.
In May, Business Insider cut nearly a quarter of its staff, citing AI as the primary reason. And in September, Penske Media sued Google, pointing to the “profoundly harmful” impact of Google AI Overviews on its affiliate revenues, which have declined by a third.
The second—and I say this with love in my heart—is just how little progress publishers have made in trying to combat this downward shift.
The problem, as a refresher, remains that answer engines use publisher content to answer user queries, but they do not send those users to publisher websites—where they would be monetized—nor do they compensate publishers for use of their data.
This is an issue I have covered extensively in the past, as it tends to be on the top of most media executives’ minds. Several tentative solutions have been floated, and some have begun to yield early results (more on that below). But by and large, what is perhaps the most acute crisis in digital biggest media has only gotten worse over the last year.
To wit, this month marks the three-year anniversary of ChatGPT, which OpenAI first unleashed on the world in November 2022. Since then, use of the product, as well as that of its peers, has skyrocketed. Startups like Anthropic and Perplexity have become household names, while nearly every major incumbent, from Google to Microsoft to Meta, has gotten in on the game. I hardly have to tell you how many hundreds of billions of dollars have been spent on the effort.
And yet, despite the historic level of investment in the space, no financing model has yet emerged to compensate publishers for use of their content.
Yes, OpenAI has signed a number of sweetheart deals with some of the most powerful publishers on the planet, small payments for licensing content rather than an ongoing or fair exchange, but—some media executives have conceded to me—better than nothing.
These deals have, however, left out the vast majority of publishers on the web. For three years now, 99% of content creators—websites, influencers, brands—have yet to see a penny for use of their content.
To be clear: I am under no illusion that AI firms are in any rush to develop a content-licensing infrastructure; doing so would only result in them having to pay for something they are currently getting for free. As a result, the onus to create and implement any kind of payment infrastructure falls necessarily on publishers.
The results have thus far been uninspiring. One strategy, suing AI firms for copyright infringement, has yet to yield much. The relevant litigation is snaking its way through the court system, but the outcomes from those suits—if they are even favorable—could be years out.
Similarly, when ChatGPT first debuted, IAC CEO Barry Diller famously exhorted his fellow media tycoons to circle the wagons, working together to bring AI firms to the bargaining table. Those efforts raised concerns about price-fixing, but as one source memorably told me, for publishers such collusion would be aspirational.
Months later, the point became moot: Dotdash Meredith (now People Inc.) struck a deal with OpenAI, which prompted others to follow suit. With its most formidable players sitting on the sidelines, the rest of the industry had little leverage to force any negotiations, and any real hope of concerted action fell apart.
The result of these events is that the smallest, least-resourced, and most poorly leveraged publishers have been tasked with not only creating a system by which they might be compensated for their data, but simultaneously tasked with getting AI firms, with their billions of dollars in backing and geopolitical consequence, to agree to them.
As it so happens, at least one company has made progress on that front.
Changing the tire, not reinventing the wheel
In a limited case study, the media network Raptive partnered with the adtech firm Criteo and its product BidSwitch to test-run a new, albeit stunningly familiar payment architecture, ADWEEK can exclusively report.
The structure of the system effectively mimics the real-time bidding process that publishers currently use to power their programmatic advertising, but it swaps out impressions for crawls, according to Raptive chief strategy officer Paul Bannister.
In a nutshell, a Raptive partner site called InspiredTaste used the CloudFlare bot management system, which lets websites determine which crawlers are and are not allowed to visit its site, to allow access to two AI crawlers.
A custom bit of code, which sites can implement in a manner of minutes, then forwarded these crawler requests to a DSP in the format of a bid request, the same kind used in programmatic auctions.
“You add ‘crawl’ as a transaction type, and the page replaces the ad slot—that’s the basic way to think about it,” said Barry Adams, executive vice president of adtech services at Criteo and general manager of BidSwitch. “You’re not reinventing the wheel, you’re just changing the tire.”
The publisher sets a defined floor price for access, and if the crawler accepts the price, it then pays and crawls the site, according to Bannister. If the crawler declines, it is denied access to the site.
The result, after just a few hours of experimentation, was $174 in crawl payments.
The trial is very early-stage, Adams cautioned. This BidSwitch method has no price controls, frequency capping, or other table-stakes auction settings, and right now it only works with the BidSwitch DSP.
It also faces a host of unanswered questions. For instance, why would AI firms agree to pay at all? Faced with a website that demands payment for access, these crawlers could easily just skip over that site and scrape a free one.
Much of the logic of this system relies on the unproven hypothesis that answer engines will pay for quality information because it improves their products, but that logic strains credulity. Is the content on a certain site really so irreplicable that similar content on a competitor site could not take its place?
For something like this to work, almost every site on the web would have to adopt this framework, making it such that crawlers have no free, alternative source of information. (This is actually the least far-fetched element of this idea. Sites offering their content “for free” are themselves making no money by being scraped, so they would have more incentive to adopt this system than to not.)
Nonetheless, the system has its draws. It uses an architecture nearly identical to the one that has powered programmatic advertising for the last decade, meaning it would be familiar to practitioners on both sides of the equation. (Maybe the industry could one day evolve beyond simply implementing yet another version of an auction, but better the devil that you know, I suppose.)
This method almost certainly won’t solve the problem of content-licensing in the AI age, but it offers one of the first tangible blueprints for what a payments architecture might look like, and it has put money in the pocket of a publisher. Three years in, that at least has the look of promise.
