What’s Changed

AWS has introduced a “dual-layer” approach to monitor the AI agent lifecycle in production environments. This approach combines two technologies, Amazon Bedrock AgentCore Evaluations and AWS DevOps Agent, to detect issues that may be missed by traditional monitoring.

  • AgentCore Evaluations continuously evaluates the agent’s quality score and detects tool selection mistakes or task failures.
  • DevOps Agent automatically investigates infrastructure failures and correlates IAM policies, call logs, and orchestration traces.
  • For example, in an airline’s reservation system, it detected issues such as agents routing requests to unexpected experts. (Source: Not mentioned in official documentation)

(Source: Not mentioned in official documentation)

How It Works

The dual-layer mechanism consists of two layers.

1. AgentCore Evaluations’ Quality Scoring

  • Continuous Evaluation: Calculates the quality score throughout the agent’s conversation and detects tool selection mistakes or task failures.
  • LLM-as-a-judge: Uses a large language model (LLM) as an evaluator to assess the agent’s output.
  • OpenTelemetry Tracing: Evaluates based on the trace data generated by the agent.
  • Example: If the agent returns an empty response without calling the correct tool, the score decreases. (Source: Not mentioned in official documentation)

2. DevOps Agent’s Infrastructure Investigation

  • Automated Failure Investigation: Tracks failures across service boundaries and correlates IAM policies, call logs, and orchestration traces.
  • Reducing Manual Investigation: Automates investigation when agent issues are caused by infrastructure problems.
  • Example: If an agent fails to function normally due to a lack of IAM permissions, DevOps Agent identifies the cause. (Source: Not mentioned in official documentation)

(Source: aws.amazon.com)

Implementation Examples and Use Cases

An airline’s reservation system operates with four specialized agents.

  • Utilizing AgentCore Evaluations: The quality score decreases if the agent misunderstands the user’s request and fails to select the appropriate tool.
  • Utilizing DevOps Agent: Analyzes the agent’s call logs and IAM policies to identify infrastructure issues.
  • Specific Scenario:
    • The agent fails to complete a reservation, but the logs show a normal tool call.
    • DevOps Agent detects an error in the third tool call. (Source: Not mentioned in official documentation)

(Source: Not mentioned in official documentation)

Known Limitations and Challenges

  • Defining Evaluation Criteria: Clearly defining the criteria for measuring agent quality (e.g., correct tool selection, appropriate response).
  • Collecting Trace Data: Requires infrastructure for collecting and storing OpenTelemetry traces.
  • Configuring Custom Evaluations: Setting up code-based evaluations using AWS Lambda in addition to LLM-as-a-judge. (Source: Not mentioned in official documentation)

(Source: Not mentioned in official documentation)

Summary

  • Leveraging AgentCore Evaluations: Continuously measures the agent’s quality score and detects tool selection mistakes or task failures.
  • Introducing DevOps Agent: Automatically investigates infrastructure failures and reduces manual troubleshooting.
  • Referencing the Airline’s Reservation System: Identifies and corrects issues in systems with multiple agents.
  • Defining Evaluation Criteria in Advance: Clearly sets the criteria for accurately measuring agent quality.
  • Preparing for Trace Data Collection and Storage: Builds infrastructure for collecting and storing OpenTelemetry traces.