Insights21/7/2026
Beyond the product, the system: how KPIs change in the era of agentic coding
Article curated by
Alberto Marsanasco
Chief Innovation Officer
KPIs change when the paradigm changes
Every generation of work paradigms has produced its own reference metrics.
And every time that paradigm has changed, it has only been realized afterward that the previous metrics were no longer sufficient to measure market differentiation capacity.
In years of digital projects, alongside clients, we have seen the reference KPIs change: those that say whether an application or a system truly succeeds. It is worth retracing the trajectory because it helps to understand what to measure today.
From Taylor to lean: measuring flow, not function
Traditional measurement systems, inherited from the times of Taylor and still widespread today, are oriented towards the goals of individual functions: quantity of outputs produced per unit of time, capacity utilization, cost per output. They measure local efficiency of different organizational areas.
The lean culture has shifted the focus. It has taught us to look at end-to-end flows, with metrics oriented towards processes: lead time, cycle time, processing cost. No longer "how much does this function produce," but "how long does it take for value to pass through the flow."
The agile era and lean startup: the user as the unit of measure
With the Agile Manifesto first, and then more clearly with the Lean Startup culture, the focus has shifted to the product and the user. Measurements began on retention, adoption, NPS, perceived value.
The problem is that AI is "commoditizing" features: what was a product differentiator yesterday is replicable in weeks today. And NPS, when subjected to serious empirical verification, holds up much less than its popularity suggests.
The system as a new measurement object in the era of agency AI
With agency AI, the focus shifts again, this time to the system. Not on the single feature nor on local production capacity, but on how the organization puts AI to work. The metric measures not only the delivered product but also how much the system becomes more adaptive, resilient, scalable, and verifiable: speed of iteration, quality of context provided to models, ratio between automation and value, sustainability of the workflow.
One premise is necessary: each phase does not eliminate the previous one; it enriches it. It adds a source of value and a level of measurement. A mature company in the "system" continues to look at cycle time and retention; only the level guiding strategic decisions changes. The sequence of phases is a simplified model, a lens to read change, not a rigid linear succession: the phases have overlapped and in many organizations still coexist today. In fact, it would be an error to measure system KPIs without having process lead time under control and clear product KPIs.
How to concretely measure context quality, automation, and resilience
It is worth making the three new KPIs concrete, as it is easy to mistake them for slogans.
The quality of the context is measured as the rate of outputs usable on the first attempt: an agent with domain rules, technological guidelines, and adequate acceptance criteria, with the right level of granularity, produces valid results without rework much more often than one starting from a generic prompt. If the same task, with better context, goes from 40% to 80% of outputs accepted without corrections, that difference is the quality of the context. Sustainability of the workflow must also be considered: optimizing the use of tokens through pre-processing and correctly delimiting the necessary context. Tobi Lütke and Andrej Karpathy named it context engineering in mid-2025: "the art and science of filling the context window with exactly the right information for the next step."
The automation/value ratio distinguishes automation that matters from inertia-driven automation: automating the generation of a report that no one reads increases the percentage of automated activities but does not move value. The KPI relates the value generated with the automation effort, not the number of automated tasks.
The resilience of the workflow is the capacity of the flow to withstand when an agent makes a mistake: it is measured with the average recovery time from an error and the proportion of errors intercepted before reaching production.
What the data says: the competitive advantage shifts to organizational design
The MIT Sloan Management Review–BCG The Emerging Agentic Enterprise report (November 2025) is explicit: the competitive advantage will not come from early access to AI, but from the organizational design built around it. McKinsey reaches the same conclusion, identifying the redesign of the workflow, not the adoption of the tool, as the true impact multiplier. Harvard Business Review, in February 2026, drew the strategic consequence: when everyone has access to the same models, tools, and suppliers, it is the organizational context that becomes the differentiating factor.
Designing AI-augmented work systems: three operational levels
If the differential shifts to the system, the job is no longer just delivering code: that has become a commodity. It's about designing and governing work systems that manage complexity and transform AI speed into reliable and sustainable enterprise results. This is where we have built our approach, based on “Agentic Coding”.
Through an AI agent-driven development execution framework, adapted to the specific organizational context and updated to the state of the art, we create system value on three levels.
- Continuous learning: we ask better questions, iterate faster, and improve more quickly, allowing the system to learn and evolve rapidly.
- Governance by design: we accompany client teams to evolve organizations and ways of working around AI. Traceability, verifiability, and control are not add-ons: they become a structural part of the way of working.
- Business value orientation: we guide decisions, identify new sources of value, steer the project, and reduce waste. Speed matters only if the change does not increase risk and brings real business value.
The metric that matters: how much the system learns, not just how much it produces
The metric that matters is not just the delivered product: it is how much the company system becomes more adaptive, resilient, scalable, and verifiable. What is valuable is the proprietary context, the way an organization truly works. Our job is not to "do AI," but to design augmented work systems that are verifiable, governable, and capable of improvement over time. The speed of the models is a starting point available to all; what we transform into a client advantage is the discipline with which those models are put to work within their organization.
Frequently asked questions
1.What are system KPIs and why do they matter in the era of agency AI?
System KPIs measure not the single output produced by the AI, but the organization's ability to learn, adapt, and maintain control over time: speed of iteration, quality of context provided to the models, workflow resilience, ratio between automation and actual value generated.
2.Do new system KPIs replace traditional ones?
No. Each phase builds on the previous one. It would be a mistake to measure system KPIs without having process lead times under control and clear product KPIs. The level that drives strategic decisions changes: in the agentic era, it must be the system level.
3.What is context engineering?
It is the discipline of building and managing the information provided to AI models for each specific task, introduced as a term by Tobi Lütke and Andrej Karpathy in 2025. When all players access the same models and tools, the proprietary organizational context becomes the main factor of competitive differentiation.
4.How is the quality of context measured concretely?
As the rate of usable output on the first attempt, without rework. If the same task, with the same model but with a better context, goes from 40% to 80% of outputs accepted without corrections, that difference is the quality of the context.
Bibliography
S. Ransbotham, D. Kiron, S. Khodabandeh, S. Iyer, A. Das, The Emerging Agentic Enterprise: How Leaders Must Navigate a New Age of AI, MIT Sloan Management Review and Boston Consulting Group, November 2025. https://sloanreview.mit.edu/projects/scholars/the-emerging-agentic-enterprise-how-leaders-must-navigate-a-new-age-of-ai/
R. Narayana Murty, R. Kumar S, When Every Company Can Use the Same AI Models, Context Becomes a Competitive Advantage, Harvard Business Review, February 2026. https://hbr.org/2026/02/when-every-company-can-use-the-same-ai-models-context-becomes-a-competitive-advantage
A. Tavakoli, B. Goodman, K. Rowshankish, A. Kumar, H. Soller, Building the Foundations for Agentic AI at Scale, McKinsey, April 2, 2026. https://www.mckinsey.com/capabilities/mckinsey-technology/our-insights/building-the-foundations-for-agentic-ai-at-scale