Enterprise AI Faces an Evaluation Gap: Autonomous Agents Outpacing Company Verification

Enterprise AI teams are granting agents increased autonomy, even as confidence in automated testing is waning.

Recent research highlights that half of enterprises have introduced an AI agent or large language model (LLM) feature that, despite passing internal checks, has led to failures impacting customers—one in four of these setbacks occurring multiple times. These findings come from a June 2026 VB Pulse survey involving 157 qualified enterprise representatives from companies with 100 or more employees.

As the sample is self-selected, the results should be interpreted as suggestive rather than definitive.

Interestingly, companies are not retreating from automation in light of these issues: 66% of respondents already allow certain production implementations without human oversight or are developing systems that will enable this capability within the next year. Just 5% fully trust the automated assessments responsible for these release decisions.

This creates an evaluation gap: the rising independence granted to AI agents is outpacing the confidence in their oversight systems.

This situation aligns with a broader trend discussed at the upcoming VB Transform 2026 event, where businesses implement AI agents first, with control mechanisms surrounding identity, evaluation, cost, context, and orchestration following later. The coming year is expected to focus on retrofitting, with budgets shifting toward systems that ensure the governance and reliability of these AI-driven deployments.

Why A Successful Evaluation Doesn’t Mean an Agent Works

Conventional software testing typically assesses if a given input yields the expected output. However, testing agents is more complex since these systems can choose their own series of actions, utilize different tools, access data, modify states, and respond variably to different scenarios.

An agent might make several reasonable decisions yet still arrive at an incorrect outcome. For instance, it could access the correct account but incorrectly update another field, draft a valid refund request, but send it without authorization, or engage five tools successfully before the sixth mishandles sensitive information or disrupts a workflow.

The survey illustrates that businesses are already aware of these challenges. The primary reason for skepticism regarding automated evaluations, identified by 29% of respondents, is the poor correlation with actual results. This was followed by concerns of bias or inconsistency (21%), lack of clarity (18%), and issues related to data leakage or privacy (17%).

That hierarchy is significant. Businesses are reporting that the scores often do not accurately forecast what will happen when a customer or a business process interacts with the agent in real-world conditions—not that the automated evaluations take too long or are prohibitively expensive.

The National Institute of Standards and Technology (NIST) echoes this sentiment in its Generative AI Profile, noting that results obtained in controlled settings may not seamlessly translate to real deployment due to variations in user interactions, prompts, context, and operational conditions. Their recommendations include field testing, ongoing monitoring post-deployment, and established protocols for handling failures.

Capability Versus Consistency

Completing a task successfully just once demonstrates that an agent is capable; it does not guarantee reliable performance.

Anthropic’s guidelines for agent evaluation differentiate between whether a system can succeed at least once in repeated trials versus consistently delivering results. This distinction matters significantly for customer-facing operations. An AI that sporadically provides excellent results may still be unreliable if it fails unpredictably in subsequent trials.

Therefore, enterprise teams need to prioritize repeatability as a crucial metric. This involves testing the same scenario multiple times, altering contexts and phrasing, simulating tool failures, and ensuring the final business outcome is accurate even when the path taken varies.

Moreover, the evaluation criteria must adapt over time. Every production error should be incorporated into a permanent regression test. Issues like customer concerns, failed tool integrations, improper approvals, and data handling errors should be integrated into the pre-deployment assessment process rather than treated as isolated occurrences.

Expanding Autonomy with Caution

The survey doesn’t suggest that all actions taken by agents must undergo human scrutiny. Human oversight is simply not scalable across massive low-stakes decisions.

However, fully autonomous operations should be justified through demonstrated reliability and limited by the potential consequences of errors.

Low-risk tasks like drafting internal documents or sorting files can afford more leeway regarding autonomy. In contrast, financial transactions, customer interactions, code deployment, permission changes, and data deletion require tighter controls, repeated testing for consistency, policy compliance checks, rollback options, and clear human intervention pathways.

The risk distribution is also uneven across company sizes. Larger organizations—those with 2,500 or more employees—are advancing towards fully automated operations more rapidly, at a rate of 70% compared to 64% for smaller companies, and they also experience a higher incidence of customer-facing failures at 54%, versus 48% for smaller firms.

This serves as a cautionary message for leaders within enterprises. Excluding human oversight does not eliminate uncertainty; instead, it transforms uncertainty into an automated decision-making process.

While the market will continue to gravitate towards greater autonomy due to significant economic drivers, the organizations that will thrive are not those that eliminate human involvement the quickest. Instead, they will be the ones that prioritize reliability and testing as highly as they do speed in deployment.