What this video shows
Arcolano says Jellyfish analyzed about 50,000 engineers at 447 companies. In the reported breakdown, the selected tool explains under 1% of throughput variation, skill use about 3%, token use about 9%, and company plus team about 12%, while individual differences explain much of the remainder.
These are observational results, so they do not show that increasing token use causes more valuable software. Use them to avoid a model-only rollout. Measure cycle time, review burden, escaped defects, user outcomes, and developer experience, then compare matched teams or staged periods before making a broad claim.
Use the DORA research program for delivery and reliability measures that go beyond generated output.
What you will learn
- A stronger model cannot repair unclear ownership, poor tests, or slow review by itself.
- Tool choice, usage intensity, and practitioner skill need separate measures.
- Merged pull requests are an activity measure rather than a complete product outcome.
- Compare quality and delivery together before expanding an AI coding program.
How to apply this safely
- Write the product and engineering outcomes the program should improve.
- Record the tool, model, skills, token use, task type, and team environment.
- Compare matched work periods and inspect pull requests rather than one aggregate count.
- Keep the rollout only when speed improves without worse review load or defects.
Important limitations
- The reported analysis is observational and cannot establish which variable caused the measured output.
- Merged pull requests differ in size, risk, complexity, and customer value.
- Jellyfish sells engineering intelligence software and publishes the analysis through its own representative.
- Teams that use more tokens or skills may already differ in motivation, task mix, or management.
Sources to check
- DORA research Research and measures for software delivery performance and organizational outcomes.
- Better AI Models Won't Fix Your Engineering Team AI Engineer video published October 22, 2026.
Continue learning on Learnetto
Review AI-generated code
Check intent, evidence, security, and user behavior.
Evaluate AI writing
See how to calibrate subjective quality measures.
AI evals guide
Build matched tests and inspect failures by slice.