What Governance Made Possible
In Part 1 of this series, we uncovered the hidden costs of agentic AI and showed that much of the expense came from tooling overhead that remained invisible to developers. In Part 2, we introduced a governance framework (observe, enforce, optimize) to make those costs visible and give organizations the controls needed to manage them. This third and final post focuses on what governance made possible once those hidden costs were brought into view.
As our Part 1 pilot expanded from a four-developer, 62-day sample to a broader rollout, we analyzed more than 72,000 AI interactions across an 11-developer team over 132 days. Once the hidden tooling costs were governed, a second finding emerged: the biggest remaining cost variable wasn’t the model we chose or the infrastructure we built. It was how developers interacted with AI.
Our findings reinforce what industry research has consistently shown. Boston Consulting Group reports that 70% of AI implementation success depends on people and processes, while technology accounts for just 20% and algorithms 10%. McKinsey similarly found that organizations realizing the greatest productivity gains from AI are those that redesign workflows, not simply deploy new tools.
Governance provides visibility into AI usage. Developer enablement turns that visibility into measurable improvements. An AI gateway produces the session-, model-, and developer-level telemetry needed to understand what behaviors should change.
Rather than relying on generic best practices, organizations can make targeted, measurable improvements based on real production data.
Four Behaviors That Matter Most
The AI gateway revealed four developer behaviors that consistently had the greatest impact on both AI cost and performance. Together, they represent the biggest opportunities for organizations looking to optimize AI adoption beyond governance alone.
Model Awareness
Developers naturally prioritize momentum over constantly evaluating which AI model is best for every task. Staying in flow is productive, but it can also be expensive.
Our production data showed that Claude Haiku handled 47% of requests while accounting for just 18% of spend. Higher-end models generated only 27% of requests but consumed 43% of total cost. Many routine tasks like file reads, searches, and shell commands, were still being routed to premium models that offered little additional value.
Model selection is ultimately a budget decision, not just a technical one. The lesson isn’t that developers are making poor decisions. It’s that manual model selection doesn’t scale. Intelligent routing removes that burden by automatically matching the task to the lowest-cost model capable of delivering the required result.
Session Observability
Most organizations understand AI spending at the monthly cloud bill level. Very few understand it at the conversation level.
Once our AI gateway began grouping requests into developer sessions, a striking pattern emerged. Just 2.5% of sessions consumed nearly 80% of total AI spend. Long conversations naturally become more expensive as context accumulates. But the goal isn’t just to keep sessions short, it’s to understand why certain sessions become exceptionally costly.
Without session-level visibility, an unusually expensive conversation disappears into aggregate spending. With it, teams can identify patterns, understand where work becomes inefficient, and establish better practices for when to continue a conversation versus starting fresh. In one instance, a single long-running session accounted for nearly 3% of the team’s total AI spend before the gateway exposed it.
Cache Awareness
Prompt caching is one of the most effective and least understood AI cost optimization opportunities. Across our 132-day production sample, prompt caching reduced estimated AI costs from more than $19,000 to less than $5,000 which is roughly a 75% reduction, in line with the 79% savings we modeled on a single session in Part 1.
By the third turn of a conversation, cache utilization exceeded 88%, eventually reaching nearly 98% in longer sessions. Yet most developers never see cache performance. Cloud billing shows token usage but not whether requests benefited from cached context or what actions caused cache breaks.
Understanding those patterns changes behavior. Developers begin preserving productive sessions, avoiding unnecessary restarts, and recognizing that ordinary interruptions can significantly increase cost. As we demonstrated in Part 1, even routine workflow interruptions can reduce cache effectiveness and increase overall AI spend.
Context Management
Perhaps the newest and most important developer skill is managing context effectively. Large language models perform best when given the right information, not simply the most information. Too little context produces incomplete or inaccurate responses. Too much introduces noise, increases cost, and can reduce output quality.
Our production data illustrated just how significant context becomes over time. Nearly 89% of all prompt tokens represented cached context, while less than 1% reflected new developer input, consistent with what we saw in Part 1, where developer input accounted for under 1% of total tokens. In other words, organizations are often paying primarily to maintain conversation history.
Effective context management means intentionally deciding what information should travel with the conversation, what belongs in reusable instruction files, and when a fresh session is the better choice. Increasingly, organizations are recognizing that context engineering, or the discipline of intentionally curating what an AI system should see, has become just as important as prompt engineering. This isn’t something technology alone can solve. It requires shared team practices, documentation, and developer education.
Measuring What Matters
The value of governance isn’t simply that it reveals costs, it’s that it identifies where organizations can improve. As teams mature their AI practices, leaders should monitor:
- Model utilization: cost per request by model tier and how work is distributed across models.
- Session health: session length, conversation cost, and when teams choose to preserve or reset context.
- Cache effectiveness: overall cache hit rates, developer trends, and workflow patterns that reduce cache efficiency.
- Context quality: correction rates, output quality, and adoption of documented context management practices.
These metrics help organizations understand whether developer enablement efforts are translating into lower costs, higher-quality outputs, and more consistent AI usage.
What Leaders Should Do
- Enable observability first. You can’t improve what you can’t measure. Visibility into models, sessions, caching, and developer behavior provides the foundation for every optimization effort.
- Invest in developer enablement. The organizations realizing the greatest value from AI aren’t simply deploying better tools, they’re helping their teams build new habits around model selection, context management, and efficient workflows.
- Measure behavior, not just spend. Monthly token costs explain what happened. Developer telemetry explains why it happened and where meaningful improvements can be made.
The most important lesson from our production experience wasn’t about AI models or infrastructure, it was about people. AI governance provides the visibility organizations need to understand developer behavior, while developer enablement turns those insights into measurable results. Together, they help organizations lower costs, improve AI outcomes, and build sustainable AI adoption at scale.
Want to learn more about how RIVA is approaching AI governance?
Reach out to Bernie Pineau to start a conversation.
Related Articles
