The most reliable way to use coding agents at scale is not a chat window but a pipeline: the agent proposes changes, your existing CI verifies them, failures feed back to the agent, and humans review what passes.
The pattern
- Scope the task into many small, independent units (files, modules, services).
- Give rich context: examples of good conversions, style rules, related files.
- Generate a change per unit.
- Verify automatically: build, type-check, lint, run tests.
- Retry with feedback: send the errors back and try again, with a cap on attempts.
- Escalate units that still fail to a human.
- Review and merge in small batches with a named approver.
A real example: Airbnb's test migration
Airbnb used an LLM-driven pipeline to migrate nearly 3,500 React component test files from Enzyme to React Testing Library in six weeks, against an original estimate of 1.5 years of engineering time (InfoQ, March 2025). The pipeline ran each file through steps — convert, fix Jest test failures, fix lint, fix TypeScript errors — and retried on failure. About 75% of files migrated automatically in the first hours; after tuning, roughly 97% were done automatically, with the rest finished by hand using the LLM output as a starting point. Prompts grew to 40,000–100,000 tokens, pulling in up to 50 related files. A notable lesson: retrying with feedback mattered more than perfect prompt engineering.
Agents triggered from CI
Coding agents can also be triggered by events: an issue label, a failing build, a dependency alert. Each trigger should have an owner, a scope (which repositories, which paths), a cost cap and a clear handoff — a pull request, never a direct push to protected branches.
Guardrails specific to pipelines
- Agents use their own service identities with the minimum repository permissions.
- Secrets are not exposed to agent steps that process untrusted content.
- Agent PRs are labelled and go through the same required checks as human PRs.
- Run counts, tokens and failures are logged and visible.