AssetOpsBench is a comprehensive benchmarking and evaluation system with six qualitative dimensions that bridge the gap for agent AI in domain-specific settings, including industrial asset lifecycle management.
introduction
Existing AI benchmarks are great for discrete tasks like coding or web navigation, but often fail to capture the complexity of real-world industrial operations. To fill this gap, we introduce AssetOpsBench, a framework specifically designed to evaluate agent performance across six critical aspects of industrial applications. Unlike traditional benchmarks, AssetOpsBench emphasizes the need for multi-agent coordination, moving beyond “lone wolf” models to systems that can handle complex failure modes, integrate multiple data streams, and manage complex work orders. By focusing on these high-stakes multi-agent dynamics, this benchmark ensures that AI agents are evaluated based on their ability to navigate the nuances and safety-critical demands of true industrial environments.
AssetOpsBench is built for asset operations such as chillers and air handling units. The contents are as follows.
2.3 million sensor telemetry points 140+ curated scenarios across 4 agents 4.2,000 work orders for diverse scenarios 53 structured failure modes
Experts helped create over 150 scenarios. Each scenario includes metadata such as task type, output format, category, and subagents. The tasks designed range from:
Anomaly detection in sensor streams Failure mode inference and diagnosis KPI prediction and analysis Work order summarization and prioritization
Evaluation framework and overall feedback
AssetOpsBench evaluates agent systems across six qualitative dimensions designed to reflect real-world operational constraints in industrial asset management. Rather than optimizing for a single success metric, this benchmark emphasizes the quality of decision traces, supporting evidence, failure recognition, and feasibility under incomplete and noisy data.
Each agent run is scored across six criteria:
Task completion Search accuracy Correctness of result validation sequence Clarity and correctness Hallucination rate
Throughout our early evaluations, we found that many general-purpose agents perform well at surface-level reasoning but struggle with persistent multi-step coordination involving work instructions, failure semantics, and time dependencies. Agents that explicitly model operational context and uncertainty tend to produce more stable and interpretable trajectories, even if the final task is only partially completed.
This feedback-oriented evaluation is intentional. In industrial settings, understanding why an agent fails is often more valuable than binary success signals.
Industrial Agentic Workflow Failure Modes
The core contribution of AssetOpsBench is to explicitly treat failure modes as first-class evaluation signals in agent industrial workflows. Rather than treating failures as binary consequences, AssetOpsBench analyzes the complete execution trajectory of multi-agents to identify where, how, and why agent operations fail under realistic operational constraints.
AssetOpsBench failure analysis is implemented through a dedicated trajectory-level pipeline (TrajFM). It combines LLM-based inference and statistical clustering to uncover interpretable failure patterns from agent execution traces. This pipeline works in three stages. (1) trajectory-level fault extraction using LLM-guided diagnostic prompts, (2) embedding-based clustering to group recurring fault patterns, and (3) analysis and visualization to support developer feedback and iteration.
Across industrial scenarios, recurring failure modes include:
Inconsistencies between sensor telemetry, alerts, and past work orders Overconfident conclusions drawn under missing, delayed, or insufficient evidence Inconsistent aggregation of disparate data modalities across agents Premature action selection without proper verification or validation steps Breakdowns in multi-agent coordination, such as ignored inputs or inconsistencies between actions and inferences
Importantly, AssetOpsBench does not rely solely on fixed, manually created fault classifications. Although a structured set of predefined failure categories (such as validation errors, repeated steps, role violations, etc.) is used for consistency, the system is explicitly designed to discover new failure patterns that emerge in the wild. Additional failure modes identified by LLM are automatically embedded and clustered, allowing the classification to evolve as new agent designs and behaviors are evaluated.
To preserve industrial confidentiality, raw execution traces are never published. Instead, the agent receives an aggregate score across six evaluation dimensions, along with a summary of clustered failure modes that explain why the agent failed, without revealing sensitive data or intermediate inference steps. This feedback-driven design allows developers to diagnose weaknesses, refine agent workflows, and iteratively resubmit improved agents.
This barrier-aware assessment reflects the reality of industrial asset management. There, careful, degradation-aware reasoning and the ability to recognize uncertainty, defer action, and escalate appropriately are often preferable to aggressive but brittle automation.
Submit agent for evaluation
AssetOpsBench-Live is designed as an open, competitive benchmark and welcomes submissions of agent implementations from the community. The agent is evaluated in a controlled, privacy-preserving environment that reflects the constraints of real-world industrial asset management.
To submit an agent, developers first validate their implementation locally using a provided simulated environment that includes representative sensor data, work orders, alerts, and failure mode catalogs. The agent is then containerized and sent for remote execution in hidden evaluation scenarios.
Submitted agents are evaluated across six qualitative dimensions: task completion, accuracy, result validation, order of actions, clarity, and illusion using a consistent and reproducible evaluation protocol. Execution traces are not published. Instead, participants receive aggregated scores and structured failure mode feedback that reveals where and why the agent’s reasoning and adjustments fail.
This feedback-driven evaluation loop enables iterative improvement. Developers can diagnose failure patterns, improve agent design and workflow structure, and resubmit updated agents for further evaluation. Support for both planning-focused and execution-focused agents allows researchers and practitioners to consider diverse agent designs within the same benchmarking framework.
experiment and observation
We conducted a community assessment and tested two tracks:
Planning-oriented multi-agent orchestration Execution-oriented dynamic multi-agent workflows.
Here are our observations across 225 users and 300+ agents and leading open source models.
Model Family Best Plan Score Best Execution Score Key Limitations GPT-4.1 68.2 72.4 Complete Illusions in Complex Workflows Mistral-Large 64.7 69.1 Struggle with Multi-Hop Tool Sequences LLaMA-4 Maverick 66.0 70.8 Missing Clarity Questions (Fixable) LLaMA-3-70B 52.3 58.9 Collapse under the coordination of multi-agents
Note: None of the models passed the metrics benchmark and achieved the deployment readiness threshold of 85 points.
Distribution of failures
The distribution of failures across the 881 agent execution traces was:
Ineffective error recovery: 31.2% Exaggerated completion: 23.8% Format issues: 21.4% Unhandled tool errors: 10.3% Ignored feedback: 8.0% Other: 5.3%
Beyond this, 185 traces had one new failure pattern and 164 traces had multiple new failures.
Main error points
“It seems right, but it’s wrong”: The agent claimed to have completed the task (23.8%) and output success even after failure recovery failed (31.2%). AssetOps benchmarking is important to clarify this so that operators do not act on incorrect information. Tool usage: This is the biggest differentiator between high- and low-performing agents, with top agents having 94% tool accuracy versus 61% for low-performing agents. Increased failures with multi-agents: The task accuracy between single agent (68%) and multi-agent (47%) shows the complexity of multi-agents with context loss, asynchrony issues, and cascading failures. Domain Knowledge: Improved performance for agents that have access to the failure mode database and maintenance manual. However, RAG knowledge is not always used correctly, suggesting the need for structured reasoning. Ambiguity: Missing sensors, inconsistent logs, and ambiguous operator descriptions reduced success rates by 34%. The agent must have a built-in explanatory strategy.
Where do I start?
