Getting started
Why does AI still struggle with real business workflows?
By SourceX Editorial · Updated
Short answer
AI struggles with business tasks because real workflows depend on context it cannot see, exceptions it rarely learned from and judgment shaped by outcomes. Benchmarks built from real work show agents finishing only part of what professionals do. The gaps narrow when models learn from records of real work: requests linked to the decisions and results that followed.
Key takeaways
- Benchmarks built from real office and freelance work still show AI agents completing only a portion of the tasks professionals handle.
- The hardest parts of business work are context, exceptions and judgment, not drafting or summarizing.
- Public internet text mostly shows finished outputs, while business records show the steps, decisions and outcomes behind them.
- Records that link a request to its decision and result are the ones that teach AI how work actually gets done.
Why do AI agents fail at real office work?#
AI agents fail at real office work mostly when a task needs information, exceptions or judgment the model has not seen. Drafting an email or summarizing a document is now routine. Working a backordered shipment, a disputed invoice or a change order across three systems is not.
Research benchmarks make the gap visible. When the OSWorld benchmark of computer tasks in real web and desktop apps launched, humans completed over 72 percent of tasks while the best model completed 12.24 percent. On TheAgentCompany, a simulated small software company with its own internal websites and data, the most competitive agent completed 30 percent of tasks autonomously.
Scores move quickly, and later leaderboards on several benchmarks are higher than the launch results. What persists is where agents fail: long tasks, messy inputs and decisions that depend on how one particular business works.
What benchmarks built from real work show#
Benchmarks built from real work show fast progress and a clear ceiling at the same time. Their tasks come from paid projects and professional deliverables rather than puzzles, so they test the parts of work a business actually pays for.
Read together, the results say agents are improving on well-specified tasks and still miss much of what professionals do on long, real projects with unclear inputs.
| Benchmark | What it tests | What it showed |
|---|---|---|
| Remote Labor Index | Real paid freelance projects in areas such as CAD, architecture, design, data analysis and web apps | The top agent automated 2.5 percent of projects at launch in October 2025; later reporting in 2026 put the top result at 16 percent |
| GDPval | Tasks from 44 occupations tied to real work products such as a legal brief, an engineering blueprint or a support conversation | Task writers averaged 14 years of professional experience, so models are compared against seasoned practitioners |
| TheAgentCompany | Workplace tasks inside a simulated software company | The most competitive agent completed 30 percent of tasks autonomously |
| OSWorld | Computer tasks in real web and desktop applications | At launch, humans completed over 72 percent and the best model 12.24 percent |
- 01Your work records
- 02Personal details removed
- 03You approve the release
- 04Licensed to AI developers
The three gaps: context, exceptions and judgment#
The three gaps that hold AI back in business workflows are context, exceptions and judgment. Each one maps to a kind of record mid-size companies already keep but that rarely appears on the public internet.
Context is the most fixable gap inside your own company, because you can connect an AI tool to your systems. Exceptions and judgment are harder, because a model needs many examples of how cases like yours were actually resolved, including the ones that went wrong.
| Gap | What goes wrong | Example in a mid-size company | Records that help close it |
|---|---|---|---|
| Context | The model lacks customer history, internal policy or system state | Replying to a long-time account without knowing its contract terms | CRM histories, account notes, internal playbooks in Confluence or Notion |
| Exceptions | The model handles the standard path and stalls on the unusual case | A partial shipment, a failed inspection or a mid-project change order | Exception logs, RMAs, NCRs and CAPAs, RFIs, dispatch changes |
| Judgment | The model cannot tell when to escalate, approve or deviate | Deciding whether to honor a warranty claim outside policy | Approvals with outcomes, code review comments, QA reviews, callbacks |
Why public internet data does not close the gaps#
Public internet data does not close these gaps because it mostly shows finished outputs, not the work behind them. A published case study shows the solution; a support ticket thread shows the first misdiagnosis, the escalation, the fix and whether the customer came back.
The records that teach the most are ordinary: a dispatcher's note on why a job was rescheduled, a code reviewer's request for a different approach, a quality engineer's disposition of a nonconforming part. Each pairs a situation with a decision someone was accountable for, which a model cannot infer from finished documents alone.
Supply is also limited. Epoch AI researchers project that, if trends continue, language models will fully use the stock of public human-generated text between 2026 and 2032. That projection is one reason AI developers look to licensed records of real work, which carry the steps and outcomes public text lacks.
What the gaps mean for your own AI plans#
For a mid-size company, the gaps mean AI tools perform best on workflows where your own records are complete and linked. In RSM's 2026 middle-market survey, data quality and availability issues were the top inhibitor to AI deployment, cited by 34 percent of respondents, ahead of security and privacy concerns.
The practical response is to point AI at the parts of work your records already explain, and to fix the records before expecting more.
- Start with workflows whose records link request, decision and outcome, such as tickets tied to fixes or jobs tied to callbacks.
- Keep people on exceptions until a tool has seen enough of your own resolved cases.
- Write down judgment calls that live in senior staff's heads: escalation rules, approval limits, warranty exceptions.
- Preserve history before a migration or shutdown makes a gap permanent.
- Test tools against your own past cases, not vendor demos.
Illustrative: a distributor's exception desk#
Illustrative: a fictional industrial distributor runs orders in NetSuite, shipments through a transportation management system and customer issues through a shared inbox. It pilots an AI assistant to draft replies to customer questions.
The assistant handles order status and delivery dates well. It struggles with substitutions, partial shipments and freight damage claims, because those decisions depend on supplier relationships, customer contract terms and what the team did last time, none of which the assistant can see.
The operations lead keeps exceptions with the team and turns the shared inbox into an exception log that records the cause, the decision and the outcome of each case. The log becomes onboarding material for new hires and better context for the next tool. It is also now a documented record family the company could later consider licensing, with customer details removed and every step approved.
Why AI developers want records of real work#
AI developers want records of real work because those records show context, exceptions and judgment at scale. In the SourceX Enterprise Data Value Framework, human-generated signal, domain expertise and AI utility are among the drivers that increase value, and records that link requests to decisions and outcomes tend to be strong on all three.
SourceX helps established companies license such records through the SourceX five-step transaction: Supply, Rights, Preparation, Approval and Delivery. The company keeps ownership, personal and confidential details are removed during preparation, and the supplier approves every step. Nothing is shared during the initial fit check.
Frequently asked questions
Will better models close these gaps on their own?
Some of the gap closes with better models, and benchmark scores have risen quickly. The parts tied to a specific business, such as its policies, customers and past decisions, still require that business's context. That is why companies connect AI tools to their own systems and why developers seek licensed records of real work.
Is our workflow too specialized for AI?
Specialized workflows are harder for general tools, but they are also where records of real work are most informative. A workflow with clear inputs, documented decisions and visible outcomes is a better candidate for AI help than one that lives in people's heads, however specialized the field.
Would licensing our records help others automate work like ours?
It can contribute to tools that serve your industry, which is why scope matters. A license can limit permitted use, exclude sensitive record families and define what the buyer may build. Each company weighs that tradeoff for itself, and the supplier approves the final scope before anything is delivered.
Which records show judgment most clearly?
Records where a person made a choice and the result is visible: code review comments followed by merged fixes, approval threads followed by outcomes, QA notes followed by rework, and service calls followed by a callback or none. The link between decision and outcome is what carries the lesson.
Do records need cleanup before AI can use them?
Usually some. Duplicates, broken links between systems and free-text fields used inconsistently all reduce usefulness. For internal tools, the most valuable cleanup restores the links between requests, decisions and outcomes. For licensing, personal and confidential details are also removed during preparation.
Sources
- OSWorld has 369 computer tasks in real web and desktop apps; at launch humans completed over 72.36% of tasks while the best model achieved 12.24%. Source
- TheAgentCompany benchmark mimics a small software company; with the most competitive agent, 30% of tasks could be completed autonomously. Source
- The Remote Labor Index (October 2025), built from real paid freelance projects, found the highest-performing AI agent achieved an automation rate of 2.5%. Source
- The Decoder reported in 2026 that AI agents can complete 16 percent of Remote Labor Index freelance jobs at professional quality, up from 2.5 percent. Source
- Remote Labor Index projects span areas such as 3D/CAD, architecture, graphic design, video, audio, data analysis and web apps. Source
- The GDPval benchmark, released September 25, 2025, covers 44 occupations. Source
- GDPval task writers averaged 14 years of professional experience, and tasks are tied to real work products such as a legal brief, an engineering blueprint or a customer support conversation. Source
- Epoch AI projects that language models will fully utilize the stock of public human-generated text between 2026 and 2032 if trends continue. Source
- In RSM's 2026 middle-market survey, data quality and availability issues were the top inhibitor to AI deployment (34%), followed by security and privacy concerns (30%). Source
Related resources
See if your company qualifies
A short company assessment. No data uploads are needed.