IT operations agent evaluation: tickets, runbooks and postmortems as test cases

Why resolved IT work makes better test cases than synthetic helpdesk prompts

Resolved tickets carry ground truth that a prompt writer cannot fake: the resolution code, the configuration item that was actually broken and the fix that closed the ticket without reopening. Synthetic prompts tend to describe clean, single-cause problems, while real queues are full of stale knowledge articles, misrouted assignments and "works now, closing" resolutions. Teams building evaluation sets from production generally treat failures and escalations as the richest seam, because those are the cases where the current system went wrong [1]. For a broader view of where private test data fits next to public benchmarks, see the evaluation datasets hub and private evaluation sets vs public benchmarks.

Public agent benchmarks are useful scaffolding but rarely match an IT estate. τ-bench grades agents that talk to simulated users and call APIs under written domain policies, comparing the final database state with an annotated goal [2]. OSWorld runs tasks in real operating systems with setup scripts and execution-based checkers [3]. Neither contains your Active Directory conventions, your change advisory board rules or your monitoring stack, and public sets also age: OpenAI stopped reporting SWE-bench Verified in February 2026 because score gains increasingly reflected training-time exposure [4]. Held-out records from real IT operations reduce that contamination risk, provided they stay out of every training corpus.

This page covers internal IT and SRE resolution work. Customer-facing support, where the test is whether an agent follows refund or entitlement policy, belongs on policy-following evaluation for customer support agents.

The eval unit: ticket, environment, runbook, resolution and time to resolve

A usable IT operations eval case bundles five things: the opening request, the environment context the agent may query, the runbook or knowledge article in force at the time, the verified resolution, and timing. In ServiceNow, the narrative lives across the incident record and journal entries; work notes and customer-visible comments are stored in the sys_journal_field table, so an export that pulls only the incident table loses most of the diagnostic trail [6]. Jira Service Management, Zendesk and Freshservice have equivalent splits between the issue and its comment or activity history.

The resolution must be checkable. "Reset password" is a label; "account unlocked in AD, MFA re-registered, user confirmed login" is a state an agent's tool calls can be graded against. Time to resolve is worth keeping as a hypothesis-grade signal of difficulty, but treat it with care: SLA clocks often pause on "awaiting user" states, so compute active handling time from state transitions rather than opened_at to closed_at.

Illustrative example: invented to show structure; it does not describe an available dataset.

{
  "case_id": "itops-eval-0042",
  "source_type": "itsm_ticket",
  "opening_request": "VPN connects but internal wiki times out since this morning",
  "category": "network/vpn",
  "environment_context": {
    "ci": "vpn-gw-[REDACTED_HOST_01]",
    "os": "Windows 11 23H2",
    "client": "GlobalProtect 6.x",
    "recent_changes": ["CHG-[ID_07]: split-tunnel route update"]
  },
  "runbook_ref": "KB-[ID_12] v3 (effective at ticket open)",
  "gold_actions": [
    "check client route table for 10.x wiki subnet",
    "compare against split-tunnel include list",
    "escalate to network team with change reference"
  ],
  "gold_end_state": "route restored via change rollback; user confirmed access",
  "forbidden_actions": ["reinstall client", "disable host firewall"],
  "active_handling_minutes": 38,
  "reopened_within_7d": false,
  "redaction_log_ref": "redact-run-2026-09-14"
}

The forbidden_actions field matters as much as the gold path. IT agents fail expensively by taking plausible but destructive steps, so pair this unit with the tests described in agent safety evaluation for irreversible actions.

Postmortems as multi-step diagnosis cases

Postmortems are the best raw material for incident response agent evaluation because they record the hypotheses responders tried, the evidence that ruled each one out and the confirmed root cause. Cut each postmortem into a staged case: release the alert and first dashboards, ask the agent for a hypothesis and next query, then reveal the next slice of the timeline. Grade on whether the agent converges on the documented root cause and whether its proposed mitigations match what was actually safe at that moment.

Runbook execution logs make this stronger. Some AI SRE platforms log every runbook step with inputs, outputs and a running, success or failed status tied to the incident timeline [5]. With those logs, a "runbook following" test can compare the agent's step sequence to the recorded one and flag skipped verification steps, not just wrong final answers.

Watch for hindsight leakage. Postmortems are written after the fact, so the summary paragraph often names the root cause in the first line. Strip the summary, action items and "lessons learned" from the agent-visible context and keep them only in the grader's key.

Grading IT agents on actions and end states

Grade IT operations agents on state, not text. Where possible, replay each case in a sandbox that mirrors the relevant systems (a test directory tenant, a container with the broken service config, a mock CMDB) and check the end state the way OSWorld and τ-bench do [2][3]. Where replay is impossible, grade the proposed action list against gold_actions and forbidden_actions with a rubric reviewed by an engineer who knows the estate.

Stratify by category and severity rather than sampling the queue proportionally. Password resets and access requests can be most of a service desk queue, and a proportional sample will hide weak performance on rare P1 incidents; the method in stratified evaluation sets for rare and high-risk cases applies directly.

Scrubbing secrets and infrastructure details without breaking the test

IT records leak more than names and emails: they contain API keys pasted into work notes, connection strings, internal hostnames, IP ranges, service account names and screenshots of admin consoles. Run pattern-based secret detection of the kind GitHub documents for provider tokens and credentials [7], then add estate-specific patterns for your naming conventions. Replace values with consistent typed placeholders ([REDACTED_HOST_01], [IP_SUBNET_A]) so that the same host appears as the same token across a case; random replacement breaks multi-step diagnosis.

Keep a redaction log per release and spot-check a sample by hand, because log excerpts and base64 blobs evade regexes. The tradeoffs between privacy and test validity are covered in de-identifying evaluation data without breaking tests. Attachments and screenshots need separate handling; see support tickets with screenshots and attachments.

Sourcing checklist for ITSM tickets, runbooks and postmortems

Before licensing IT operations records for evaluation, confirm that the export, documentation and rights answer the questions below. A good set ships with its journal history, the runbook versions in force and evidence that the supplier may license the records at all, since tickets often contain end-customer or employee data covered by contracts; customer contracts and DPAs explains that check.

Illustrative example: invented to show structure; it does not describe an available dataset.

The negotiation points in the last row are expanded in evaluation-only data license terms.

How SourceX fits IT operations evaluation sourcing

SourceX sources operational datasets from US companies on request, including support histories and engineering records, and manages the commercial process through licensing. Nothing is held in stock and a request does not guarantee a match; you describe the data, such as resolved incident tickets with journal history and linked runbooks, and SourceX looks for US businesses that hold it, with every release approved by the supplying company. Each dataset is rights-reviewed for ownership and consents, personal details such as names, emails and account numbers are removed or replaced with the method recorded and a sample checked, and delivery runs through private, access-controlled workflows after an executed agreement. Owner pages cover IT operations agent data, IT service tickets and incident postmortems; to scope a request, start at the SourceX buyer intake.

Request IT support and SRE agent evaluation data

Describe the ticket categories, runbook coverage and postmortem depth your eval needs, and SourceX will look for US companies that hold matching records. Every dataset is rights-reviewed and delivered under a license that defines records, uses, term and delivery, and nothing is contracted until a supplier agrees. Start an IT operations eval data request.