Several publishers and tech firms have voiced support for Really Simple Licensing (RSL), a new standard designed to ensure fair compensation for content scraped by AI
crawlers.
RSL was launched along with a nonprofit organization: the RSL Collective, which is to provide collective licensing services.
The new
standard allows publishers to add machine-readable licensing and royalty terms to their robots.txt files, specifying how AI agents must compensate them for using their content, going beyond the basic
yes/no blocking of the robots.txt protocol.
In addition, RSL will support a range of licensing and usage models, including free
attribution, subscription, pay-per-crawl (in which publishers will be paid every time an AI application crawls their content), and pay-per-inference (when an AI application a
publisher’s content to generate a response).
RSL is an open, decentralized protocol built on RSS (Really Simple Syndication) a standard that can be applied to any digital content,
including web pages, books, videos, and datas
advertisement
advertisement
“RSS was critical to the Internet’s evolution as an information ecosystem, giving early online publishers a simple, open
standard to syndicate their content and reach audiences at Internet scale. That spirit of openness is what helped the web thrive,” said Tim O’Reilly, CEO of O’Reilly Media.
“But today, as AI systems absorb and repurpose that same content without permission or compensation, the rules need to evolve.
The supporters include Reddit, People Inc.,
Yahoo, Internet Brands, Ziff Davis, Fastly, Quora, O’Reilly Media, and Medium.
Reddit, People Inc., Yahoo, Internet Brands, Ziff Davis, wikiHow, O’Reilly Media, Medium, The Daily
Beast, Miso.AI, Raptive, Evolve Media, Ranker, Inspired Taste, Travel Lemming, and CelebrityNetWorth. The standard is also supported by Fastly, Quora, and AdWeek.
The development of the
RSL Standard is led by the RSL Technical Steering Committee, which
includes Eckart Walther (RSL Collective, co-author RSS), RV Guha (co-author RSS, Schema.org, NLWeb), Tim O’Reilly (O’Reilly Media), Stephane Koenig (Yahoo), and Simon Wistow (Fastly).
This is not the only tool now being offered. In July, Cloudflare announced a program that would enable website owners to decide if they want AI crawlers to access their content and
if so, how they can use it. The AI firms must state their purpose and publishers will determine whether to let them in—for a price.
And a study
by ImmuniWeb found that 83% of newspapers and magazines block AI bots sent to scrape their sites.