AI uses for records
Enterprise AI agents: the data they need and how to test them
By SourceX Editorial · Reviewed by Noah Loul ·
Short answer
Enterprise AI agents are AI systems that complete business tasks by calling tools against systems of record such as a CRM, ERP or service desk, under the organization's identities, permissions, approval steps and audit trail. To build, ground and evaluate one, teams need operational records: policies, tool schemas, past tickets and approvals with their recorded outcomes, and test tasks whose end state can be checked automatically.
Key takeaways
- An enterprise AI agent acts inside systems of record under its own identity, scoped permissions, human approvals and an audit trail; those controls, not company size, make it enterprise.
- Data serves three separate jobs: building behavior (policies, tool schemas, demonstrations), grounding at run time (live records through tools) and evaluation (tasks with checkable end states).
- Proof that an agent works is a repeatable test set run several times, scored on the end state of the system and on policy in the transcript, and reported with pass^k.
- Closed tickets, approvals and transactions become test tasks only when the starting state, request, policy version and a reviewed outcome can be reconstructed.
- Access to records, privacy permission, ownership and contractual rights to use them for AI are four separate questions; de-identification does not answer the last one.
What are enterprise AI agents?#
An enterprise AI agent is a model-driven system that pursues a business goal by deciding which tools to call, reading results from the company's own systems, and changing records in them, inside the controls the organization already applies to people. The word enterprise is about those controls and systems, not about company size.
Anthropic separates workflows, where LLMs and tools follow predefined code paths, from agents, where the model dynamically directs its own process and tool usage. It adds that agents should get ground truth from the environment at each step, such as tool call results, can pause for human feedback at checkpoints, and commonly run with stopping conditions such as a maximum number of iterations.
In practice, an enterprise agent is a support agent that issues a refund in the order system, an accounts payable agent that matches an invoice to a purchase order in NetSuite and routes the exception, or an IT agent that resolves an access request in ServiceNow. Each one reads and writes records that someone else will rely on, which is why the data behind it, and the evidence that it behaves correctly, matter more than the model choice alone.
What makes an AI agent enterprise-grade rather than a chatbot with tools?#
Five properties separate an enterprise agent from a demo: its own identity and scoped permissions, connections to systems of record, human approval for consequential actions, an audit trail of every step, and measured reliability across repeated runs. Each property has a data consequence, which is the part most planning documents skip.
| Property | What it means in practice | How vendors implement it | What it means for your data |
|---|---|---|---|
| Identity and permissions | The agent acts as a named principal with least-privilege access, not as a shared admin account | Microsoft Entra Agent ID gives agents identities managed with Conditional Access and identity governance; the Claude Agent SDK applies allow and deny rules and permission modes to every tool call | Training and test records must show which role performed each action, so the agent learns what is permitted for its role, not what an administrator could do |
| Systems of record | The agent works in the CRM, ERP, ticketing and HR systems where the authoritative record lives | Google describes Gemini Enterprise as connecting to company data in Google Workspace, Microsoft 365, Salesforce and SAP; MCP standardizes how applications expose tools and data to models | You need tool schemas and field definitions for each system, plus historical records in the same structure the agent will see live |
| Human approval | Sensitive actions such as refunds, payments or access grants wait for a person to approve or reject | The OpenAI Agents SDK lets tools declare that they need approval and pauses the run until a decision is made; the MCP specification says hosts must obtain explicit user consent before invoking any tool | Approval and rejection records, with the reason given, are among the most useful data you hold, because they show where the line sits |
| Audit and tracing | Every model call, tool call and handoff is recorded and can be reviewed later | The OpenAI Agents SDK records LLM generations, tool calls, handoffs and guardrails in traces; Google says Gemini Enterprise lets you visualize, secure and audit agents from one place | Traces from pilots become evaluation and debugging data; field-level audit logs from existing systems show how humans did the same work |
| Reliability | The agent succeeds on the same task every time, not once in several tries | Anthropic says the pass^k metric, the probability that all k trials succeed, especially matters for customer-facing agents where users expect reliable behavior every time | You need enough test tasks with verifiable outcomes to run each one several times and report consistency, not a single success rate |
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
Which operational data does an enterprise AI agent need?#
An enterprise agent needs data for three different jobs: building its behavior, grounding it at run time, and proving it works. Mixing these up is the most common planning error, because each job has a different source, freshness requirement and rights question.
Build data shapes how the agent decides: written policies and SOPs, tool and API schemas, and examples of the work done well, such as historical ticket resolutions, approval decisions with reasons, and step-by-step trajectories through a system. Grounding data is what the agent reads live while working, such as the customer's account, the open invoice or the knowledge base article, usually fetched through tools under the agent's own permissions rather than copied into a dataset.
Evaluation data is a set of tasks with a known starting state, a request, the policy in force and a checkable end state. Anthropic's eval guidance distinguishes the transcript, the full record of a trial including tool calls and intermediate results, from the outcome, the final state of the environment. Its example: a flight-booking agent may say the flight is booked, but the outcome is whether a reservation exists in the environment's SQL database. Historical business records are valuable here because they already contain real requests and recorded outcomes.
- Policies and SOPs: refund rules, approval limits, escalation criteria, with the effective dates of each version.
- Tool schemas: the actions the agent can take and the fields each one reads or writes.
- Demonstrations: completed cases showing the request, each action, each handoff and the final state.
- Decisions with reasons: approvals, rejections and overrides, including who decided and why.
- Outcomes: resolution codes, reopen flags, payment status, credit notes, or any field that shows whether the work held up.
- Exceptions and failures: cases that went wrong, were reversed or were escalated, which are the cases agents most often get wrong.
Task-to-data map: what each enterprise agent use case requires#
Start from the tasks, not the data you happen to have. The map below takes six common enterprise agent use cases and lists the tasks each performs, the systems of record involved, the record categories needed, and the evaluation evidence that would show it works. System names are examples of where the records usually live, not a statement that any particular export is available.
Read across a row to scope a dataset or a pilot. If a row's evaluation evidence cannot be produced from your records, that is the gap to close before you trust the agent with the task. For a workflow-level view of the same records, see the portfolio-company guide to agent tasks and their records.
| Use case | Tasks the agent performs | Systems of record | Records needed | Evaluation evidence that proves it works |
|---|---|---|---|---|
| Customer support | Look up orders, apply refund or replacement policy, update the ticket, escalate edge cases | Zendesk, Salesforce Service Cloud, Shopify or the order system | Tickets with full message threads, macros used, order and refund records, escalation and reopen history | Order and refund state after the run matches the expected state; policy followed; no unnecessary escalation; consistent result across repeated trials |
| Sales operations | Enrich and route leads, update opportunity stages, prepare quotes within discount limits | Salesforce, HubSpot, CPQ tool | Field-level change history, routing rules, quote and discount approvals with reasons, closed-won and closed-lost outcomes | Correct owner and stage after the run; discounts inside approved limits; no duplicate records created |
| Finance operations (accounts payable) | Match invoices to purchase orders and receipts, code to the general ledger, route exceptions for approval | NetSuite, SAP, Coupa, AP inbox | Invoices with PO and receipt links, GL coding, exception reasons, approval chains, payment and credit-note outcomes | Correct match and coding against the posted record; exceptions routed to the right approver; no payment released without approval |
| IT service | Triage incidents, fulfill access requests, run runbook steps, close or escalate | ServiceNow, Jira Service Management, Okta or Microsoft Entra | Incidents and requests with categories, assignment history, runbook steps executed, change approvals, resolution codes | Ticket resolved with the right category; access granted matches the approved request; no change made without an approved change record |
| Procurement | Take purchase requests, check policy and preferred suppliers, create requisitions and POs | Coupa, SAP Ariba, NetSuite | Requisitions, approval decisions with reasons, supplier records, contract and catalog terms, PO outcomes | Requisition routed through the correct approval path; supplier and price match policy; rejected requests stay rejected |
| HR service | Answer policy questions, process leave or data-change requests, escalate sensitive cases | Workday, ServiceNow HR Service Delivery | HR cases with categories, policy articles cited, request outcomes, escalation records | Correct policy applied; employee record updated only when authorized; sensitive cases escalated to a person |
What evidence proves an enterprise AI agent works?#
Proof is a repeatable test, not a demo, a satisfaction survey or a single accuracy number. Reliability has to be measured across trials: Anthropic's eval guidance explains that pass^k is the probability that all k trials succeed, so an agent with a 75 percent per-trial success rate passes three trials in a row only about 42 percent of the time.
The four steps below are the summary; the evaluation task suites guide covers pass^k and environment fidelity in depth.
- Build a task set from real cases, each with a written expected end state.
- Run each task several times from a clean starting state and report pass^k beside the single-run rate.
- Check the system after the run (the refund record, ticket status or posted entry) and review the transcript for policy and approvals.
- Keep a regression suite separate from capability tests, and measure a human baseline on the same tasks.
How do historical records become agent evaluation tasks?#
A closed case becomes a test task only when four things can be reconstructed. Without all four it can still serve as a demonstration, but it cannot grade an agent automatically. The customer support policy evaluation guide walks through the recipe, and the ticket histories guide covers turning threads into trajectories.
- The starting state of the relevant records before the work began.
- The request as first made.
- The policy version in force on that date, not today's version.
- The recorded end state, confirmed correct by a reviewer.
What rights and privacy checks apply before using operational records?#
Four questions have to be answered separately for every record set: who can access the records, whether privacy rules permit the new use, who owns them, and whether a contract grants the right to use or license them for AI. System access answers only the first, and de-identification does not by itself create a right to license data. Grounding through live tools runs under the agent's permissions and platform consent controls, while copying historical records into a training or evaluation set is a different use that needs its own review. The full framework is in the guide to data privacy and the right to license business records.
Enterprise AI agent data readiness checklist#
Use this checklist before committing to a pilot, a fine-tuning run or a data purchase. Each item should have a named owner and a yes or no answer, not a plan to find out later.
- The target tasks are written down, each with the system of record, the actions allowed and the actions that need approval.
- The agent will have its own identity with least-privilege permissions, and test records reflect that role.
- Policies and SOPs are collected with version history and effective dates.
- Tool schemas or API definitions exist for every action the agent will take.
- At least several months of closed cases are available with linked requests, actions, approvals and recorded outcomes.
- Outcome fields are reliable: resolution codes, reopen flags, payment or posting status are populated and audited.
- Exceptions, escalations, reversals and rejected approvals are included, not filtered out.
- An evaluation set exists with a starting state and a checkable end state for each task, plus a regression subset.
- A sandbox or copy of the systems exists where trials can run from a clean state without touching production.
- Tracing is on for the pilot, and someone reads a sample of transcripts every week.
- Access, privacy permission, ownership and contractual rights have been reviewed separately for every record set.
- A human baseline on the same tasks has been measured, so success has a reference point.
When should you source operational data from outside your company?#
Your own records are the right starting point for an agent that will run on your own systems. Outside records help when you are building an agent product for many companies, need more variety of exceptions than one company produces, or want a held-out test set your team never saw while building.
AI developers and enterprise AI teams can request this kind of data through SourceX, the enterprise data transaction layer for AI. Datasets are sourced on request from U.S. businesses rather than held in stock, and buyers describe scope, volume, language, format, timing and licensing needs in the request form.
Frequently asked questions
What are examples of enterprise AI agents?
Common examples are customer support agents that process refunds or replacements, accounts payable agents that match invoices to purchase orders and route exceptions, IT service agents that fulfill access requests, sales operations agents that route leads and update opportunities, and procurement agents that create requisitions within policy. What makes them enterprise agents is that they act inside systems of record under scoped permissions, approvals and audit.
What is the difference between an AI agent and a chatbot?
A chatbot answers questions in a conversation. An agent decides which tools to call, reads results from real systems and changes records to finish a task, often across many steps. Anthropic describes agents as systems where the model dynamically directs its own process and tool usage, while a workflow follows predefined code paths.
How do enterprise AI agents get trained?
Most enterprise agents start from a general model and are shaped with instructions, policies, tool definitions and examples, then grounded at run time by reading live records through tools. Some teams also fine-tune on historical trajectories and decisions. In every case the agent should be measured on a test set of real tasks with checkable end states before it is trusted with production work.
How many test tasks do you need to evaluate an enterprise agent?
Anthropic's guidance is that 20 to 50 simple tasks drawn from real failures is a great start. Each task should be run several times so you can report consistency with pass^k, not only a single success rate.
Can we use our ticket and CRM history to evaluate an agent?
Often yes, if each case can be reconstructed with its starting state, request, policy version and a reviewed outcome. You also need to confirm that privacy notices, customer contracts and platform terms permit that use, and redact personal data. This is general information; counsel should review the specific records and agreements.
Who is responsible when an enterprise agent makes a wrong change?
The organization that deploys the agent remains responsible for actions taken under its systems and identities, which is why approvals, scoped permissions and audit trails matter. Assign an owner for each agent, log every tool call and require human approval for actions that move money, grant access or affect customers. Specific liability questions depend on contracts and law and need counsel's review.
Sources
- Anthropic distinguishes workflows that follow predefined code paths from agents that direct their own tool use, and recommends ground truth from the environment, human checkpoints, stopping conditions and sandboxed testing. Source
- Anthropic's agent eval guidance defines transcripts and outcomes, grader types, pass@k and pass^k (75 percent per trial gives about 42 percent over three trials), 20 to 50 simple tasks from real failures as a great start, isolated trials and regression suites. Source
- The MCP specification describes an open protocol for connecting LLM applications to external data and tools and requires hosts to obtain explicit user consent before exposing user data or invoking any tool. Source
- The OpenAI Agents SDK lets tools declare that they need approval and pauses agent execution until a person approves or rejects the call. Source
- The OpenAI Agents SDK traces record LLM generations, tool calls, handoffs and guardrails during an agent run. Source
- The Claude Agent SDK controls tool use with permission modes, hooks and allow and deny rules, and a matching deny rule blocks a tool even in bypass mode. Source
- Microsoft Entra Agent ID provides identity and access management for agents using Conditional Access, identity governance and identity protection. Source
- Google says Gemini Enterprise connects agents to company data in Google Workspace, Microsoft 365, Salesforce and SAP and provides central governance to visualize, secure and audit agents. Source
Related resources
Tell us what data you need
Share scope, volume, format, timing and licensing needs. Datasets are sourced on request.