What a single evaluation task built from your records looks like (Illustrative)

What is an AI evaluation task?#

An AI evaluation task is a single, self-contained test case that asks a model or agent to do a piece of real work and then scores the attempt against a known result. Developers group many tasks into an evaluation set, but each one is built, reviewed and graded on its own.

Public examples tend to be academic: math problems, coding puzzles, multiple-choice questions. Tasks built from business records look different because they carry the mess of real work, including partial information, several systems and a judgment call. GDPval, a public benchmark released in September 2025, ties its tasks to real work products such as an engineering blueprint or a customer support conversation, and its tasks can include reference files and spreadsheets.

Whatever the source, a task card built from company records has the same six parts.

• 1Context: the records the AI may see at the start, with personal details replaced.
• 2Instruction: what the AI is asked to do, written the way a manager would brief a capable new hire.
• 3Allowed tools: the lookups and actions the AI may use, and which of them are read-only.
• 4Expected result: what actually happened in the original case, taken from the records.
• 5Grading rubric: the criteria a grader applies to the attempt, each marked required or optional.
• 6Source record IDs: pseudonymized references that trace the task back to the original records.

Illustrative: one task card from a parts distributor#

Illustrative: a fictional distributor of plumbing and HVAC parts runs NetSuite for orders and invoicing, a separate WMS for picking and Zendesk for customer service. A contractor customer emailed to say a carton was missing from a delivery. A service rep checked the pick confirmation, found a carton labeled for another order, reshipped the part at no charge and closed the ticket with a resolution code.

That one resolved case becomes the card below. No real customer appears in it, and every name and account number has been replaced with a consistent placeholder. The company's customer service lead reviewed the card before it was approved as a sample.

Where each part of the card comes from in your systems#

Every field on a task card maps to a record your teams already create. Building a task is mostly a matter of finding the records that belong to one case and confirming they link, not writing new material.

The expected result is the field that most often decides whether a case can become a task. If the ticket was closed without a resolution code, or the credit memo cannot be tied back to the claim, the case may still illustrate the work but cannot grade it.

How the grading rubric separates right, partial and wrong#

A grading rubric turns the expected result into criteria that a person or an automated grader can apply the same way every time. Good rubrics name observable things: the disposition chosen, the record cited, the action the AI should not have taken.

Weighting matters more than length. Marking each criterion required, important or optional lets a grader tell an agent that reached the wrong decision from one that reached the right decision with a clumsy reply. Your existing QA checklists are often the best starting point, because they already encode what your reviewers look for.

• 1Write each criterion so two graders would score it the same way.
• 2Include a criterion for what the AI must not do, such as posting a credit it was not authorized to issue.
• 3Tie each required criterion to a field in the original records.
• 4Leave out style preferences that your own staff do not follow consistently.
• 5Note where each criterion came from: a policy, a QA scorecard or the resolution note.

What is changed before a task leaves your company#

Before a task leaves your company, personal and confidential details are replaced and the answer is separated from the question. Customer names, contact details, account numbers and staff names become consistent placeholders, so the same customer reads the same way across the email, the order and the proof of delivery.

The resolution note, the credit memo and the closing reply are held out of the context and used only for grading. If they leak into the context, the task tests reading rather than judgment.

Customer-specific pricing and contract terms need a decision of their own. Some companies generalize them; others exclude task families where pricing is the decision. Customer contracts and privacy notices may limit what can be used at all, which is assessed with counsel before any task is built.

What makes a task card weak#

A weak task card usually fails in one of five ways, and each can be spotted by someone who knows the work. Most of the failures trace back to how the original case was recorded, not to how the card was written.

Reviewers inside the business catch these faster than outside graders. A customer service lead can usually tell at a glance whether a resolution reflects policy or a favor, and dropping a weak card is cheaper than repairing it.

• 1No provable outcome: the ticket closed for inactivity, so nobody knows what the right answer was.
• 2Answer in the context: the internal note explaining the mislabel was left in the packet.
• 3Missing context: the decision depended on a phone call that was never logged.
• 4Unfair tools: the original rep used a screen the AI cannot reach, so the task tests access, not skill.
• 5Unrepresentative case: a one-off concession for a key account presented as standard policy.

How SourceX approaches evaluation tasks#

SourceX treats evaluation as one possible permitted use inside the SourceX five-step transaction: Supply, Rights, Preparation, Approval and Delivery. The fit check uses metadata only, such as which systems hold linked cases and whether outcomes are recorded, and no files are shared at that stage. SourceX does not grade models, and its own rights in a deidentified dataset are set out in the signed supplier agreement.

When a buyer asks for evaluation material, the rights review checks whether evaluation is a permitted use, preparation removes personal and confidential details, and the company approves sample task cards before anything is delivered. The SourceX Evidence Packet records provenance, licensing rights, permitted use, the privacy record and release authorization, which is where pseudonymized source IDs trace back to.