I Made My Website Charge AI Agents A Penny Per Page, Then I Watched Claude Pay It

Google is paying some publishers when their content material helps generate its AI solutions.

Barry Schwartz covered it at Search Engine Roundtable on September 14, following Digiday’s report that morning. Google confirmed this system to Digiday. It’s known as the AI contribution pilot.

Publishers who settle for the phrases get an earnings determine in Search Console every month. What they don’t get is a breakdown of how Google calculated it.

Google pays when it decides a web page contributed considerably to a solution whereas it was being generated in AI Overviews, AI Mode, or the Gemini app. If the web page solely will get linked after the reply has been written, it doesn’t qualify.

I’m not within the pilot. In line with Digiday, Google has approached at the least dozens of publishers, with smaller publishers exhibiting extra curiosity than the bigger ones. One government conversant in it described it as a black field. Barry had truly found the help page back in April, however it wasn’t clear then what the characteristic did.

I’ve been testing a unique strategy alone web site since August.

My web page units a value earlier than an agent reads it. If the agent needs entry, it pays.

On the morning of September 15, 5 funds went by means of. Each settled on a blockchain and has a public transaction hash. One got here from Claude Code, operating on my machine and utilizing a pockets I’d arrange for it.

To be clear, all of the funds to this point have come from my very own brokers. No search crawler is popping up and paying me. The funds additionally use testnet USDC, which suggests the tokens don’t have any actual financial worth.

However the fee course of works. An agent requests a web page, checks the value, pays and receives the content material. Cloudflare is constructing merchandise round that very same concept, and I wished to grasp the way it works earlier than it turns into one other setting in a dashboard.

What That Cloudflare Pockets Deal with Is For

On August 4, throughout Brokers Week, Cloudflare announced Wallets and opened deal with reservations.

Claiming a deal with is free. I think loads of individuals grabbed theirs the best way we grab usernames: shortly, earlier than another person will get there, with a obscure intention of studying the small print later.

When you did that, that is the sort of factor these wallets are being constructed for.

Cloudflare’s design has two parts:

• Account Wallets belong to individuals. You add funds, set spending limits, and allocate cash to your brokers.
• Digital Wallets are for the brokers. They spend inside the permissions and limits you’ve set.

Your deal with provides that account a recognizable id.

As of September 15, although, reserving a deal with will get you a web page displaying the identify and a notification when Wallets turns into accessible. Cloudflare’s documentation explicitly says a reserved deal with can’t but ship, obtain, or maintain funds.

The opposite half is the Monetization Gateway, which is able to let websites cost brokers for entry to sources. Cloudflare announced it on July 1, and entry continues to be by means of the waitlist linked from that announcement.

So the merchandise are introduced, however you’ll be able to’t but use a reserved pockets deal with to run this entire course of.

My demo exhibits what that course of appears like right this moment utilizing x402. The positioning fees one cent per web page; my agent pays, and spending limits management what it might probably purchase. It’s the identical basic association Cloudflare describes for Digital Wallets.

Why Cost Crawlers At All?

The association between search engines like google and yahoo and web sites was once pretty simple. Engines like google crawled your content material, confirmed it of their outcomes, and despatched guests again.

Publishers acquired visitors. Engines like google acquired one thing helpful to indicate their customers.

AI solutions have modified that association. An agent can learn your web page, use the data in its response, and answer the user without sending them to your site.

I’ve spent months taking a look at what these systems actually fetch. There’s loads of crawling. The visitors coming again typically doesn’t replicate it.

Publishers have responded in numerous methods, partly relying on how a lot negotiating energy they’ve.

Smaller websites typically block AI crawlers. I perceive why. If somebody is taking your content material and also you’re getting little in return, blocking them is an affordable response. However it comes with a trade-off: proscribing entry may cut back your visibility in training datasets and AI answers.

Massive publishers can negotiate licensing deals. Information Corp, the Monetary Occasions and Reddit all have agreements with AI firms. In line with Press Gazette’s reporting on 4 September, Folks Inc.’s chief government advised buyers that blocking crawlers by means of Cloudflare had helped carry AI firms to the negotiating desk.

Google’s pilot gives one other strategy: paying publishers in accordance with the worth their content material contributes to AI solutions.

In that very same Press Gazette piece, Cloudflare mentioned it was transferring its personal default strategy from pay per crawl in the direction of pay per use, with pilots involving Ceramic.ai and You.com. Its reasoning was that fetching a web page doesn’t show the content material was truly used.

I feel that distinction issues.

Utilizing a web page to reply a query is a stronger indication of worth than merely downloading it. However it’s additionally one thing the web site proprietor can’t independently observe. As soon as my web page has left my server, I can’t see whether or not it influenced a solution.

With pay per use, I’m counting on the platform to inform me what occurred and what it was value. With pay per crawl, I can examine the request and the fee myself.

Google’s eligibility guidelines make this notably clear. As Barry Schwartz explains, contributing throughout reply era can earn a fee. Being linked afterwards doesn’t. The writer sees the full, with out the calculation behind it.

Every strategy leaves one thing unresolved.

Blocking earns nothing straight. Particular person licensing agreements don’t scale to tens of millions of small web sites. Income-sharing preparations depart the platform deciding how a lot to pay.

Placing a value into the request provides a web site a typical solution to promote entry. An agent can settle for that value and obtain the content material, with a receipt for the transaction. There’s no want to barter a separate industrial settlement with each purchaser.

That’s the pay-per-crawl mannequin I wished to check.

Cloudflare already has a product known as Pay Per Crawl, however it’s nonetheless in closed beta. On the paying aspect, bot operators want verification by means of Web Bot Auth, Stripe onboarding, and program approval.

My private agent can’t merely be part of that program.

So I constructed a model utilizing the open x402 protocol, which Cloudflare can be utilizing for its upcoming Monetization Gateway. My demo makes precise x402 funds independently of Cloudflare’s closed Pay Per Crawl programme.

How x402 Works

HTTP has included a standing code known as 402 Fee Required since 1997. The specification reserved it for future use.

Twenty-nine years later, x402 provides it a sensible objective.

In my setup, the trade has 4 steps:

1. The agent requests a web page. The server responds with 402 Fee Required, together with a PAYMENT-REQUIRED header that describes the value and fee necessities.
2. The agent indicators an authorisation to switch the precise quantity in USDC. It then retries the request with a PAYMENT-SIGNATURE header.
3. A fee facilitator verifies the signature and settles the transaction on the blockchain. My demo makes use of Coinbase’s facilitator at x402.org.
4. The server returns the content material, together with a PAYMENT-RESPONSE header containing the settlement receipt.

Right here’s the supply from my web site, decoded from a reside 402 response this morning:

{
 "x402Version": 2,
 "accepts": [{
 "scheme": "exact",
 "network": "eip155:84532",
 "amount": "10000",
 "asset": "0x036CbD53842c5426634e7929541eC2318f3dCF7e",
 "payTo": "0xEdF2444D0259BBB8aC5094216D0148938F8308ff",
 "maxTimeoutSeconds": 300,
 "extra": { "name": "USDC", "version": "2" }
 }]
}

That provide asks for one cent in testnet USDC on Base Sepolia, paid to the burner pockets I’m utilizing for the demo.

USDC makes use of six decimal locations, so 10000 represents 0.01 USDC.

A appropriate consumer can learn these fee necessities and pay with out creating an account with my web site or asking me for an API key.

That’s a helpful distinction from robots.txt. A rule in robots.txt depends on a crawler choosing to respect it. Right here, the server can withhold the protected content material till fee succeeds.

The client pockets additionally held zero ETH, but each settlement went by means of. The facilitator submitted the transactions and coated the blockchain transaction charges, often known as fuel. I checked the balances twice.

For this setup, the agent solely wanted the testnet USDC it was spending.

There’s broader business help behind the protocol too. The x402 Foundation became operational under the Linux Foundation on 14 July, with 40 members, together with Visa, Mastercard, Google, AWS, Stripe and Cloudflare.

That’s helpful context, however the precise trade continues to be these 4 steps.

What’s Working On My Web site

The demo is at paid.suganthan.com.

It runs on a Cloudflare Employee, utilizing roughly 300 traces of Hono code and the x402 middleware. It’s been operating since August 8, and the general public earnings web page information each paid crawl since then.

There are three routes:

• The touchdown web page is free and explains the demo.
• /analysis/agentic-seo/ fees $0.01 per crawl.
• /earnings/ exhibits the operating complete, timestamps, payer addresses, and transaction hashes.

Request the article with curl, and also you’ll get this:

$ curl -i https://paid.suganthan.com/analysis/agentic-seo/
HTTP/2 402
payment-required: eyJ4NDAyVmVyc2lvbiI6MiwiZXJyb3Ii...

Open the identical URL in a browser, and also you get a web page explaining the cost. It nonetheless returns HTTP standing 402, however a human customer will get one thing readable.

There’s additionally a preview button. That opens the article without cost with a banner, so you’ll be able to examine the content material in addition to the fee display screen.

I constructed one other mode that isn’t enabled within the demo. Altering one variable makes the Employee cost solely requests that establish themselves as recognized AI crawlers, together with GPTBot, ClaudeBot, and PerplexityBot. Human guests can browse without cost.

I examined that mode utilizing spoofed user-agent strings, and it behaved as anticipated.

In fact, these strings will be faked. Checking a consumer agent is sufficient to display the behaviour, however it doesn’t show who made the request. Cloudflare’s strategy makes use of cryptographic verification of crawler id to handle that drawback.

Giving The Agent A Finances

The client is a small Node script. Earlier than it pays, it checks whether or not the request matches its finances.

I gave it two limits:

• A most of $0.05 per name.
• A $0.25 day by day allowance, recorded in a spending ledger on disk.

The script decodes the fee supply and checks each limits earlier than authorizing something.

If the request is inside finances, it pays, retries the request, prints the settlement particulars, and information the spend. If it exceeds both restrict, it refuses.

Right here’s a refusal with the per-call cap set under the web page’s value:

[step 3] Spending guardrails
[refused] Value $0.0100 exceeds the per name cap of $0.0010.
 This agent doesn't negotiate. Elevate MAX_PER_CALL to override.

That refusal is among the most helpful components of the demo.

If an agent pays each value a server provides it, you haven’t actually delegated a finances. You’ve given it a solution to empty a pockets.

The spending checks occur earlier than signing. When the request fails these checks, the script exits with code 3 with out making a fee signature.

This follows the identical basic design Cloudflare describes for its wallets: an individual supplies the funds and units the bounds, and an agent spends inside them. My purchaser script demonstrates that behaviour with out being an precise Cloudflare Digital Pockets.

The profitable run within the screenshot settled at 07:21 UTC. Two earlier funds went by means of at 06:10 and 06:11, with the second nonetheless inside the day by day allowance.

All three appeared on the public earnings page inside seconds of affirmation.

The vendor’s steadiness elevated, the customer’s steadiness decreased, and the quantities matched. The August transactions are nonetheless listed beneath them. I haven’t touched the fee code since these earlier funds.

Watching Claude Pay Throughout A Activity

This was the half I used to be most keen on.

Claude Code helps hooks: scripts that may run round instrument calls. Cloudflare paperwork a sample that watches WebFetch, and I linked that sample to my pockets.

The method works like this:

1. Claude tries to fetch a web page and encounters a 402.
2. The hook requests the fee supply itself.
3. It checks the value towards the per-call spending cap.
4. If the value is allowed, it pays and retrieves the content material.
5. It passes the content material and receipt again to Claude as further context.

Claude can then proceed with its job. I don’t should cease what I’m doing and manually pay for the web page.

That’s what occurred within the session proven close to the highest of this text. The hook returned this receipt:

Paid $0.0100 for "https://paid.suganthan.com/analysis/agentic-seo/" through x402.
Settlement tx: 0x51b5eba4fb34ddff27052d89cb79ae3d748b379800a723a6f4f020c11e2871c9

You may examine the transaction on BaseScan.

Claude encountered a fee requirement throughout a job, and the hook dealt with it inside the finances I’d set. It paid one testnet cent, returned the article, and let Claude keep on.

If the value exceeds the cap, the hook refuses and tells Claude to indicate the value to the human as a substitute.

I fairly like that boundary. Claude can purchase a web page when it wants one, however it might probably’t resolve by itself that my five-cent restrict now not applies.

Ought to You Begin Charging For Your Pages?

I wouldn’t do it this month anticipating search crawlers to pay you.

The software program for paying brokers is creating shortly. The crawlers that almost all web site homeowners care about haven’t caught up.

AWS made AgentCore payments generally available on August 18. An agent utilizing it might probably encounter a 402 throughout a job and pay by means of x402, very similar to my Claude hook. Cloudflare’s Brokers SDK additionally has a consumer that may make funds.

However that doesn’t imply GPTBot, ClaudeBot, PerplexityBot, or Googlebot will arrive at your web site with a pockets and pay your asking value.

Till these crawlers help the method, requiring fee prevents them from retrieving the protected content material. You would lose entry to the techniques that floor and cite your work with out incomes something in return.

I’d additionally watch out with the transaction numbers getting used to explain this market.

Decrypt reported on September 13 that TRM Labs had examined 198.9 million x402 settlements. Its evaluation steered that many of the fee quantity wasn’t coming from AI brokers.

Scheduled scripts, load exams, and funds between accounts managed by the identical get together can all produce related blockchain information. TRM’s strategies can also’t conclusively distinguish an agent from a script, so the estimates have limits.

Numerous transactions tells us that the fee infrastructure is getting used. It doesn’t, by itself, inform us that autonomous brokers are shopping for helpful issues from unbiased web sites.

My very own demo makes that distinction pretty apparent. The funds work, however I’m on each side of them.

What I’d take into consideration now’s which components of your web site you would possibly ultimately cost for.

I don’t suppose the reply is each web page.

Being cited in an AI reply nonetheless has worth. Common explainers could also be value preserving freely accessible as a result of they assist individuals uncover your work.

Unique datasets, analysis, and helpful instruments are extra apparent candidates for paid entry. An agent might have one thing particular from these sources to complete a job. Charging for that particular person request makes extra sense than attempting to suit a machine into an promoting or subscription mannequin designed round human readers.

Google’s pilot approaches that worth query from the platform’s aspect. It pays when content material contributes to the reply, reasonably than merely showing as a hyperlink beneath it.

For a web site proprietor, the sensible questions turn out to be: What’s the agent getting, what ought to that entry value, and who decides?

My expectation is that adoption will turn out to be simpler when merchandise corresponding to Cloudflare’s Monetization Gateway transfer past their waitlists. Charging for a useful resource might turn out to be a routine configuration selection for a lot of web sites.

I’d reasonably have thought by means of what to cost for earlier than that possibility seems within the dashboard.

And if Google gives you a month-to-month fee within the meantime, ask the way it was calculated. Digiday’s reporting describes restricted transparency, and one publishing government argued that accepting early phrases might weaken publishers’ place in future negotiations. These phrases deserve as a lot consideration because the fee itself.

Shifting From Check Tokens To Actual Cash

The whole lot on this demo has used testnet tokens. The funds haven’t value me actual cash.

The code additionally helps the true Base community. In my setup, the NETWORK setting switches from base-sepolia to base, transferring the fee circulation to actual USDC.

My subsequent step is to run precisely one actual transaction.

When an agent pays my web site its first actual cent, I’ll put the transaction hash within the subsequent submit.

And when these reserved Cloudflare handles turn out to be working wallets, that is the sort of course of they’ll help: an agent finds one thing it wants, checks the value towards its finances, pays, and carries on.

The demo is staying online within the meantime. Attempt the curl request, examine the fee supply, and take a look on the public earnings page.

Extra Assets:

This submit was initially printed on Suganthan.

Featured Picture: SvetaZi/Shutterstock