We tested AI agents with and without an SDK. The SDK won.

AI agents are building bad integrations against your API, and your customers are paying for the tokens it takes to fix them.

Your customers now hand integrations to coding agents, which means whatever the agent writes is what calls your API in production. Its bugs become your support tickets.

To measure the difference an SDK makes, we gave the same agent the same task on the same API, once with raw HTTP and once with an SDK. We used Square's public API and its TypeScript SDK, which Fern generates. We also tested an SDK generated by OpenAPI Generator, the most popular open-source SDK generator.

Agent-built integrations break under load

What we found was that:

• Most Sonnet integrations built on raw HTTP broke under production rate limits: All six returned the right result, and four of them crashed as soon as the API started returning 429.

• Square's SDK held up on both models: 11 of 12 integrations survived, and none of the agents wrote retry code. The one failure came from a Sonnet agent that set maxRetries: 0 to avoid double charges, which turned off the SDK's built-in retries.

• An SDK from an open-source generator didn't help: With the OpenAPI Generator SDK, 2 of 12 integrations were ready, the same as raw HTTP, and agents used more tokens than in any other setup.

The raw HTTP failures are worth a closer look. All six Sonnet sessions hit a 429 during development, but only two added retry logic, and those two were the only ones that survived the rate-limit test. The other four shipped without retries after a clean test run. That kind of bug passes code review and then fails on your customer's first busy day.

How we tested

Square publishes its OpenAPI spec, which covers 253 endpoints and 1,476 schemas. From it, we built a strict mock server that validates request bodies against the schemas, requires idempotency keys, enforces auth, and returns Square's real error format.

The task was to find the customer with email ops@globex.com, create an order at the "Downtown" location with two line items, charge the customer's Visa on file, print the result. We seeded the traps that real data has: 230 paginated customers, lookalike emails like ops@globex.co, a second location, and an expired card.

We ran three setups, six runs each, on Claude Sonnet 4.6 and Claude Haiku 4.5. Every agent had a shell, a file-writing tool, and the live mock to test against. Sonnet sessions were capped at 40 turns and Haiku sessions at 80.

• Raw HTTP: Square's OpenAPI spec and Node's built-in fetch.
• OpenAPI Generator: a TypeScript SDK we generated from Square's spec with OpenAPI Generator 7.25.0 (typescript-fetch), installed as a package with its generated README and docs. Square's spec references two schemas it never defines, so we added empty stubs for them to make the generated code compile.
• Square SDK: the published square package from npm, generated by Fern, with its stock README and reference.

We also planned to test an SDK from Cloudflare Forge, Cloudflare's open-source code generation pipeline. Forge generates its SDKs with Fern's generators, so a Forge SDK would repeat the Square SDK setup, and we left it out.

When an agent finished, we ran its script against a fresh server and checked the result. We wanted one order, the right customer, the right location, the right line items, and a single payment on the right card. Then we ran it again against a server that rate-limits every second request.

What we found

Every number except "Ready" is the average for one session, across the six runs for that setup.

Here's what Square's SDK changed, compared with raw HTTP:

With the SDK, Sonnet's script worked on the first run in five of six sessions. Haiku crashed 92% less often and finished in about a quarter of the time.

Why the OpenAPI Generator SDK didn't help

An SDK only helps if it handles the details agents get wrong. The OpenAPI Generator SDK compiled and had a typed method for every endpoint, but three problems cost agents more than raw HTTP did:

• Auth didn't work out of the box. The generated client sends the access token as the whole Authorization header, without the Bearer prefix that Square requires. 11 of the 12 sessions got a 401 and had to find the fix in the generated code.
• Errors hid the cause. A failed request throws a ResponseError with the message "Response returned an error code." Square's error detail stays in the unread response body, so agents had to write extra code to see what went wrong.
• No retries. The generated client doesn't retry 429 responses. Only the two sessions that wrote their own retry helper survived the rate-limit test.

This was the most expensive setup in the test. Sonnet used 672k tokens per session, about twice as many as with raw HTTP or Square's SDK. Five of the six Haiku sessions ran out of turns, and one never got its script to compile.

Fixing bugs is where the tokens go

Every failed run costs your customer twice: once to read the error, and again to re-send the whole conversation while the agent patches the code.

• Haiku on raw HTTP spent 66% of its tokens repairing its own code, or 852k tokens per session. With Square's SDK, repairs dropped to 215k, which is 75% fewer.

• Sonnet on raw HTTP spent 90k tokens on repairs. With the SDK, it spent 14k, which is 85% fewer.

Once Sonnet had read the docs, building and fixing the integration with the SDK took 29% fewer tokens than with raw HTTP.

The raw HTTP mistakes were familiar:

• No rate-limit handling. 9 of the 12 raw HTTP scripts had no 429 handling at all.
• Invented endpoints. Agents on both models called GET /v2/customers/{id}/cards, which doesn't exist.
• Wrong payment flow. Two Haiku scripts tried to pay an order that was already completed.

The SDK closes off most of these by design. Base URL and auth are set once on the client, retries are built in, and pagination is an iterator. Request fields are typed, so a wrong one fails at compile time instead of in production.

Bad integrations become your problem

A raw HTTP integration without retries looks fine right up until traffic spikes. Then your customer's payments start failing, their users complain, and the ticket lands in your queue as "your API is down." Your team spends a day proving it isn't.

Hand-written retries carry their own risk. Each script picks its own attempts and delays, and one aggressive loop turns a slowdown on your side into a retry storm from your own customers.

Square's SDK avoids this. It retries 408, 429, and 5xx responses with exponential backoff and jitter, honors Retry-After, and stops after a bounded number of attempts. Square sets that policy once, and every customer on the SDK follows it.

One code path instead of thousands

The raw HTTP scripts ran anywhere from 125 to 269 lines, and each one took its own approach to auth, pagination, and errors. Multiply that by your customer base and you're supporting thousands of one-off clients.

That variation is expensive for the API owner:

• Support. Every ticket starts with reverse-engineering the customer's request code.
• API changes. Hand-written types break the moment you add a field, and you don't know who is affected.
• Fixes. You can't ship a better retry policy or a new auth flow to code you didn't write.
• Visibility. Raw requests don't say what sent them.

With an SDK, every customer runs the same tested request path, and you own it. A fix ships with a version bump, and every request carries X-Fern-SDK-Name and X-Fern-SDK-Version, so you see exactly which version each customer is running.

Give agents an SDK to find

Agents build from whatever they find first. so make sure it's a typed SDK.

Fern generates SDKs in 9 languages from your OpenAPI spec, with retries, pagination, and typed errors built in, and it regenerates them every time your API changes. Square, ElevenLabs, and Cohere ship theirs with Fern.

Book a demo and we'll generate one from your spec.