We build harnesses for engineering orgs, and this is the metric we've seen is reliable at predicting increases in speed and quality developing with agents, proven through 8 months of experimentation including +2000 engineers across +100 companies.
Measure the onboarding of an agent as if it were a new hire, looking at time from access to a deployed first feature, with no help.
The Agentic First-Feature Test#
The efficiency of onboarding a human engineer can be measured as time (and count of blockers) between "has repository access" and "has coded, tested, validated, and deployed their first feature" with no external help. We call this the First-Feature Test.
This statement must hold true for an agent, as agents are the strictest possible new hires:
- Tribal knowledge is invisible to agents, they only know what's written down.
- It forgets between sessions without a proper memory layer. Anything not in the repo re-becomes unknown tomorrow.
- It runs the onboarding loop hundreds of times per week, not once. Every removed blocker compounds; every remaining one is paid on every run.
- Every undocumented blocker is either fatal (fully blocks the agent) or a workaround is improvised jeopardizing the outcome without informing anyone, both of these are failures.
A repo an agent can operate autonomously is, as a side effect, the best human onboarding environment you can build.
Record four things#
Open a new session with your usual coding agent, and begin working on a new feature with a prompt that includes enough context to successfully build it. Also, ask the agent to help you record the following information:
- Time to first green verify: clone to a passing local check of their own change.
- Time to deployed: clone to that change live in a real environment.
- Blockers hit, classified (the six classes are below).
- Undocumented workarounds used: each one counts as a failure, even when it works. Counting only hard stops measures your team's resourcefulness. Counting workarounds measures the repository, which is the thing you can actually fix.
The output is a list of specific broken things with file paths attached.
Turning the list into fixes#
Classifying identified blockers matters because the fix is different for each class. Six classes cover almost everything we have seen:
- B1 Tribal knowledge: The step exists only in someone's head or a chat thread
- B2 Credential wall: Secrets or access that can't be provisioned from documented, non-interactive sources
- B3 Missing data: Empty or unrealistic environments; no sanitized path to real-looking data
- B4 Absent feedback signal: No way to tell whether the change actually worked
- B5 Unsafe default: Verifying requires production or a shared mutable environment
- B6 Unnecessary human gate: Human approval where no human judgment is added