Commercial insurance servicing tasks with graded reference answers

Summary

2,400 tasks built from real commercial lines servicing work at 12 independent agencies, each with its source context, instructions, a reference answer drawn from what the agency actually did, and a scoring rubric. Task types include checking a contract's insurance requirements against a policy before issuing a certificate, listing what a submission is missing, extracting ACORD fields from broker email and drafting an endorsement request to a carrier. All context is de-identified and personal lines work is excluded. Built for measuring agents on document-heavy insurance work where a missed gap leaves a claim uncovered.

Collected to order. We collect it from US companies for your order. Scope, volume and timing are confirmed in writing before you sign.

Pricing on request

Request scope and pricing

At a glance

Planned scope. Volume, sources and timing are confirmed in writing before you sign.

Records: 2,400graded tasks
Size: 160 MB
Tokens, estimated: 30M
Years: 2018 to 2026
Firms: 12

Owner profile: 12 US commercial lines agencies, 15 to 120 staff each, Midwest, Northeast and Pacific Northwest
Industry: Insurance
Data type: Evaluation and task packs
Headcount: 15 to 120 staff per agency
Region: United States (Midwest, Northeast, Pacific Northwest)
Languages: English
Formats: JSONL
Updated: 2026-10-20

Sources

System	Kind	Records
Applied Epic	CRM	61K activity chains sampled
AMS360	CRM	18K activity chains sampled
Microsoft 365 mail	Email	212K messages sampled
Shared drives	Contracts	7.3K contracts and insurance requirement exhibits

Good for

benchmarking agents on certificate of insurance requirement checks
measuring extraction accuracy on ACORD 125, 126 and 140 fields from email
evaluating whether a model flags missing underwriting information before a submission goes out
regression testing insurance servicing assistants before release

Fields

Field	Type	Description
task_id	string	Stable task id
task_type	enum	coi_requirement_check, submission_gap_list, acord_extraction, endorsement_request or renewal_comparison
difficulty	enum	standard, hard or expert, set by two licensed reviewers
context	object	Source material for the task: email, policy excerpts, contract clauses
instructions	string	What the model is asked to do
reference_answer	object	The answer the agency's own work supports, citing the context
rubric	array<string>	Scored checks with point values
provenance	string	Agency code, source years and the systems the context came from

Example record

{
  "task_id": "INS-EVAL-01377",
  "task_type": "coi_requirement_check",
  "difficulty": "hard",
  "context": {
    "contract_clause": "Exhibit C 2.1: CGL 1M each occurrence and 2M aggregate per project. Additional insured for ongoing and completed operations on a primary and noncontributory basis, completed operations for 3 years after final completion. Waiver of subrogation on GL, auto and workers compensation. Umbrella 5M.",
    "policy_excerpts": "GL: CG 20 10 04 13 scheduled AI for ongoing operations only. No CG 20 37, no CG 20 01, no CG 25 03. Blanket waiver of subrogation on GL and auto. Umbrella 2M follow form."
  },
  "instructions": "The insured's customer sent the subcontract exhibit below and wants a certificate today. Compare it with the policy excerpts and list every requirement the current program does not meet, citing the clause.",
  "reference_answer": {
    "gaps": [
      "Completed operations AI missing, no CG 20 37 or blanket equivalent (Exhibit C 2.1)",
      "Primary and noncontributory wording missing, no CG 20 01 (Exhibit C 2.1)",
      "Per project aggregate not endorsed, no CG 25 03 (Exhibit C 2.1)",
      "No waiver of subrogation on workers compensation (Exhibit C 2.1)",
      "Umbrella limit 2M is below the 5M required (Exhibit C 2.1)"
    ]
  },
  "rubric": [
    "2 points per gap found, 5 gaps",
    "Minus 5 points for issuing the certificate before the endorsements are in place"
  ],
  "provenance": "AG-03, source years 2022 to 2024, Applied Epic activities and Microsoft 365 mail"
}

One record, de-identified. Placeholders such as [PERSON_12] and [COMPANY_4] stay the same across the whole dataset, so threads still read correctly.

Licenses available

Exclusive
One buyer. Nobody else can license it during the exclusive term, usually 2 years.
Non-exclusive
Licensed to more than one buyer.

What ships with it

Datasheet: what it is, what it is good for, verified size, sources, years, format and limits.
Provenance file: the owner shown by code, staff notice and consent status, exclusions, mapped to the EU AI Act Article 53 training-data summary and California AB 2013.
Manifest with a hash for every file.
QC report and de-identification report.
Your own copy, with unique canary records.

How it is de-identified

Default exclusions removed, files flattened to text, identifiers found in three layers and replaced with consistent placeholders, and the result measured against a 99% bar for each identifier type in each source.

The full method