Web-scraped data is one of the most commonly used data types, either directly or as a source for other derived signals. However, while some web content owners are ambivalent, others have pursued means to restrict access. Today, we look at Cloudflare’s pay per crawl - and explore if it could be a canary in the web-crawling coalmine.

WHAT’S HAPPENED?

Cloudflare has introduced a pay per crawl model (in private beta) to help websites monetise their content.