Conversation Metrics

View source on GitHub

CLI tool to retrieve cost and token usage metrics for OpenHands conversations. Supports both V0 and V1 APIs, automatically selecting the appropriate one based on the conversation version.

Features

Auto-detects conversation version (V0 vs V1) and uses the appropriate API
Graceful fallback chain for metric retrieval
Displays cost (USD), token counts, cache stats, and context window
JSON output for programmatic use
API call logging for debugging and development
Fixture-based testing - high coverage via recorded API responses
Zero dependencies - uses only Python standard library

Installation

No installation required - just make the script executable:

chmod +x oh-metrics

Requires Python 3.10+.

Usage

Set your API key

export OH_API_KEY="your-api-key"

Get metrics for a conversation

./oh-metrics <conversation_id>

Example output (V0 conversation):

────────────────────────────────────────────────────────────
Conversation: 7f3d57c4e5b2434d9ca78e5e27311137
Title: Hello and a Programming Joke
API Version: V0 via V0 (events)
────────────────────────────────────────────────────────────
💰 Total Cost: $0.083175 USD

📊 Token Usage:
   Prompt tokens:     12,865
   Completion tokens: 117
   Total tokens:      12,982

🗄️  Cache:
   Cache read:        0
   Cache write:       12,740

📐 Context window:    200,000
────────────────────────────────────────────────────────────

Example output (V1 conversation):

────────────────────────────────────────────────────────────
Conversation: 72d40619b8534f9b9de6c3f17a71072d
Title: 📝 V0 API for Conversation Costs & Tokens
API Version: V1 via V1 (events)
────────────────────────────────────────────────────────────
💰 Total Cost: $13.576675 USD

📊 Token Usage:
   Prompt tokens:     17,577,146
   Completion tokens: 63,985
   Total tokens:      17,641,131

🗄️  Cache:
   Cache read:        17,022,477
   Cache write:       553,973

🧠 Reasoning tokens:  1,169

📐 Context window:    200,000
────────────────────────────────────────────────────────────

JSON output

./oh-metrics <conversation_id> --json

{
  "conversation_id": "7f3d57c4e5b2434d9ca78e5e27311137",
  "title": "Hello and a Programming Joke",
  "api_version": "V0",
  "api_used": "V0 (events)",
  "metrics": {
    "accumulated_cost": 0.083175,
    "accumulated_token_usage": {
      "prompt_tokens": 12865,
      "completion_tokens": 117,
      "cache_read_tokens": 0,
      "cache_write_tokens": 12740,
      "context_window": 200000
    }
  }
}

Options

Option	Short	Description
`--api-key KEY`	`-k`	API key (defaults to `OH_API_KEY` env var)
`--base-url URL`	`-u`	Base URL (default: `https://app.all-hands.dev`)
`--json`	`-j`	Output in JSON format
`--log-api-calls`	`-l`	Log all API requests/responses to `.oh/api-logs/`
`--help`	`-h`	Show help message

Logging API Calls

For debugging or development, you can log all API requests and responses:

./oh-metrics <conversation_id> --log-api-calls

This creates timestamped files in .oh/api-logs/YYYYMMDD-HHMMSS/:

.oh/api-logs/20260227-145230/
├── 0001-request.json
├── 0001-response.json
├── 0002-request.json
└── 0002-response.json

Each request/response pair is numbered sequentially. The Authorization header is redacted from logged requests.

API Endpoints Used

For V1 Conversations

Endpoint	Purpose
`GET /api/conversations/{id}`	Get conversation info (to determine version)
`GET /api/v1/app-conversations?ids={id}`	Get conversation with metrics
`GET /api/v1/conversation/{id}/events/search`	Fallback: find metrics in `ConversationStateUpdateEvent`

For V0 Conversations

Endpoint	Purpose
`GET /api/conversations/{id}`	Get conversation info
`GET /api/conversations/{id}/events`	Get events with `llm_metrics`
`GET /api/conversations/{id}/trajectory`	Fallback for metrics

How Metrics Are Retrieved

The tool uses a fallback chain to find metrics:

Check conversation version via /api/conversations/{id}
For V1 conversations:
- First try /api/v1/app-conversations?ids={id} which includes a metrics object
- If metrics are all zeros, fall back to /api/v1/conversation/{id}/events/search and extract metrics from ConversationStateUpdateEvent at value.stats.usage_to_metrics.agent
For V0 conversations (or if V1 fails): Use /api/conversations/{id}/events and find the latest event with llm_metrics
Last resort: Use /api/conversations/{id}/trajectory and scan for llm_metrics

Note: Some V1 conversations have metrics stored only in events (not in the app-conversations response). The fallback chain ensures these are still retrieved correctly.

Metrics Explained

Field	Description
`accumulated_cost`	Total cost in USD
`prompt_tokens`	Tokens sent to the LLM
`completion_tokens`	Tokens generated by the LLM
`cache_read_tokens`	Tokens read from prompt cache
`cache_write_tokens`	Tokens written to prompt cache
`reasoning_tokens`	Tokens used for reasoning (if applicable)
`context_window`	Model’s context window size

Architecture

The library is organized into separate modules:

oh_api/
├── __init__.py    # Public API exports
├── client.py      # Base HTTP client with logging and fixture support
├── v0.py          # V0 API driver
├── v1.py          # V1 API driver
└── metrics.py     # High-level metrics retrieval with fallback chain

Using the Library Programmatically

from oh_api import APIClient, get_conversation_metrics

# Create a client
client = APIClient(
    base_url="https://app.all-hands.dev",
    api_key="your-api-key",
    log_api_calls=True,  # Optional: enable logging
)

# Get metrics
metrics = get_conversation_metrics(client, "conversation-id")
if metrics:
    print(f"Cost: ${metrics.accumulated_cost:.6f}")
    print(f"Tokens: {metrics.total_tokens:,}")

Using the V0/V1 Drivers Directly

from oh_api import APIClient
from oh_api.v0 import V0Driver
from oh_api.v1 import V1Driver

client = APIClient(base_url="...", api_key="...")

# V0 API
v0 = V0Driver(client)
conv = v0.get_conversation("conv-id")
events = v0.get_events("conv-id", limit=50)

# V1 API
v1 = V1Driver(client)
conv = v1.get_conversation("conv-id")
if conv and conv.metrics:
    print(f"Cost: ${conv.metrics.accumulated_cost}")

Testing

The test suite uses recorded API fixtures to achieve high coverage without making real API calls:

# Run tests
uv run pytest tests/ -v

# Run with coverage
uv run pytest tests/ --cov --cov-report=term-missing

Current coverage: 89% of library code.

Creating New Fixtures

Run the CLI with --log-api-calls to capture real API responses
Copy relevant response files to tests/fixtures/
Rename following the pattern: GET__api_path_q_param=value.json

License

MIT License - see LICENSE for details.

Overview

Conversation monitoring & reacting

Guardrails

View source on GitHub

Features

Installation

Usage

Set your API key

Get metrics for a conversation

JSON output

Options

Logging API Calls

API Endpoints Used

For V1 Conversations

For V0 Conversations

How Metrics Are Retrieved

Metrics Explained

Architecture

Using the Library Programmatically

Using the V0/V1 Drivers Directly

Testing

Creating New Fixtures

License

View source on GitHub

​Features

​Installation

​Usage

​Set your API key

​Get metrics for a conversation

​JSON output

​Options

​Logging API Calls

​API Endpoints Used

​For V1 Conversations

​For V0 Conversations

​How Metrics Are Retrieved

​Metrics Explained

​Architecture

​Using the Library Programmatically

​Using the V0/V1 Drivers Directly

​Testing

​Creating New Fixtures

​License

Features

Installation

Usage

Set your API key

Get metrics for a conversation

JSON output

Options

Logging API Calls

API Endpoints Used

For V1 Conversations

For V0 Conversations

How Metrics Are Retrieved

Metrics Explained

Architecture

Using the Library Programmatically

Using the V0/V1 Drivers Directly

Testing

Creating New Fixtures

License