How evaluations work

What you need today. There is one evaluation definition, for care-coordination conversations, with one rubric, for care coordinators. A working integration does two things:

1. List evaluations at least daily by completion time.
2. Keep the newest evaluation per conversation, evaluated person, and definition.

The evaluation.completed webhook is optional: it gets each evaluation to you within minutes, but the daily list alone brings in every evaluation. Fetch evaluations shows each step. The rest of this page explains the model behind them, so new definitions and roles don't break your integration.

Besides this page and Fetch evaluations, an integration needs how to get an API key and the rate limit.

What an evaluation is

An evaluation is ChartHero's AI assessment of one staff member's part in a recorded conversation, such as a care coordinator's outreach call. It holds a summary of the conversation, a call profile, points for each dimension the conversation gave the person a chance to show, and findings backed by transcript evidence.

Each evaluation covers one recorded conversation, which is one transcript document, and one person: the staff member who recorded it. An encounter with several recorded conversations has an evaluation for each.

In ChartHero, your organization's owners, admins and managers see these evaluations in the Conversation Reviews dashboard, which can also export them to a spreadsheet. The dashboard calls each record a conversation review. The API returns the review's evaluation of the staff member, so the resource is an evaluation, and evaluation_id is the same value as the Review ID in the Conversation Reviews export.

The same things go by several names:

Evaluations are AI judgments about staff, and they are also PHI. Treat every field as PHI: the IDs and conversation_occurred_at tie each evaluation to a patient's care. The free-text fields quote or describe the conversation: summary, each dimension's rationale, and each finding's description, rationale and evidence. Protect them as you protect the transcript they come from. Whether any field can leave your PHI systems is your privacy team's decision.

Evaluations, definitions and rubrics

Every evaluation is produced by two releases: a definition, which says what is measured and is shared by every role, and a rubric, which says what is expected of one role against that definition. The evaluation names both, along with the conversation and the person it evaluates.

Key on the IDs; definition_version and rubric_version are release numbers for display. The API's v1 and the webhook's api_version version the API itself, not evaluations.

Parse result by result_schema and interpret it with its definition. GET /external/v1/evaluation-definitions returns each definition with its rubrics; see Evaluation definitions.

Which conversations are evaluated

Only conversations that a rubric applies to are evaluated. Today that means recorded conversations by care coordinators that produced a Patient Outreach note. Each rubric states which conversations it covers in applies_to.

Not every conversation is evaluated, so a missing evaluation is not a zero. The API returns only completed evaluations of real records. Failed and pending evaluations and demo and training records are never returned.

An evaluation's content never changes after it completes, so updated_at always equals completed_at and you never need to fetch one twice. Whether it is returned can change: if a ChartHero data correction removes its encounter or moves the encounter to another organization, the evaluation stops being returned and its detail route returns evaluation_not_found. If the correction is reversed, the evaluation is returned again, unchanged. Corrections are rare and send no event; Fetch evaluations shows how to catch them.

Keep the newest evaluation per conversation

A conversation is evaluated again only when ChartHero processes its recording again, which is rare. The new evaluation replaces the earlier one, and the earlier one stays readable. The API does not mark which one is current, so keep one evaluation per conversation, evaluated person and definition, as the Conversation Reviews dashboard does:

1. Key each evaluation by (document_id, evaluated_actor_id, definition_id). All three are always present and never change.
2. Replace the stored evaluation only when the arriving one is newer: a later created_at, or the same created_at and a greater evaluation_id. Compare evaluation_id as plain text. Its order means nothing, but every client then picks the same winner.
3. Otherwise ignore it. An older evaluation can complete, and reach you, after the one that replaced it, which is why the rule compares created_at, not completed_at.

The list route returns replaced evaluations too, so apply this rule before you count or total results. If you store every evaluation, use the same rule to mark which one is current.

from datetime import datetime


def newness(evaluation):
    created_at = datetime.fromisoformat(evaluation["created_at"].replace("Z", "+00:00"))
    return created_at, evaluation["evaluation_id"]


def keep_newest(stored, arriving):
    """Return the evaluation to keep for one (document_id, evaluated_actor_id, definition_id)."""
    if stored is None or newness(arriving) > newness(stored):
        return arriving
    return stored

Compare created_at as a timestamp, not as text: timestamps can carry fractional seconds.

The evaluation resource

An abbreviated evaluation, as the detail route returns it in evaluation. It shows two of the seven dimensions, one scored and one not, and two findings. Neither finding has the optional secondary_category, and the second also omits focus_area:

{
  "evaluation_id": "eval_synthetic_001",
  "evaluation_type": "care_coordination_conversation",
  "definition_id": "6f102fa4-ad28-4f20-b86d-b944ee32d7eb",
  "definition_version": 1,
  "rubric_id": "81cb28c6-2c30-4473-acf4-1ff258030470",
  "rubric_version": 1,
  "evaluated_role": "care_coordinator",
  "result_schema": "conversation_evaluation.v1",
  "organization_id": "org_synthetic_001",
  "encounter_id": "enc_synthetic_001",
  "activity_id": "act_synthetic_call_001",
  "document_id": "doc_synthetic_transcript_001",
  "workflow_variant": "phone_call",
  "evaluated_actor_id": "11111111-1111-4111-8111-111111111111",
  "conversation_occurred_at": "2026-09-14T15:02:11Z",
  "conversation_duration_seconds": 312.4,
  "completed_at": "2026-09-14T15:09:40Z",
  "created_at": "2026-09-14T15:08:05Z",
  "updated_at": "2026-09-14T15:09:40Z",
  "result": {
    "summary": "Synthetic post-discharge follow-up call about new swelling.",
    "profile": {
      "remote_party_role": "patient",
      "call_context": "post_discharge_follow_up",
      "dominant_intent": "clinical_status_check",
      "clinical_complexity_level": "moderate",
      "care_navigation_complexity_level": "low",
      "interaction_complexity_level": "low",
      "overall_complexity_level": "moderate",
      "risk_level": "active_signal",
      "transcript_extractability_quality": "good"
    },
    "total_points": 1,
    "scored_dimension_count": 1,
    "minimum_possible_points": -2,
    "maximum_possible_points": 2,
    "dimensions": [
      {
        "key": "safety_risk_recognition_and_escalation",
        "points": 1,
        "applicability": "highly_applicable",
        "rationale": "Synthetic rationale for the score.",
        "finding_ids": [
          "f0000000-0000-4000-8000-000000000001",
          "f0000000-0000-4000-8000-000000000002"
        ]
      },
      {
        "key": "barrier_identification_and_problem_solving",
        "points": null,
        "applicability": "not_applicable",
        "rationale": "Synthetic rationale for leaving it unscored.",
        "finding_ids": []
      }
    ],
    "findings": [
      {
        "finding_id": "f0000000-0000-4000-8000-000000000001",
        "level": "positive_strong",
        "primary_category": "clinical_attentiveness_symptom_drill_down_risk_recognition",
        "attribution": "evaluated_participant_behavior",
        "review_markers": ["safety"],
        "practice_element": "acuity_and_red_flag_screening",
        "focus_area": "health_status_symptoms_and_clinical_risk",
        "issue_type": "no_issue_described",
        "description": "Synthetic description of what went well.",
        "rationale": "Synthetic rationale.",
        "evidence": "Synthetic transcript excerpt.",
        "primary_evidence_range": {"start_seconds": 41.2, "end_seconds": 55.9},
        "supporting_evidence_ranges": []
      },
      {
        "finding_id": "f0000000-0000-4000-8000-000000000002",
        "level": "opportunity_minor",
        "primary_category": "escalation_handoff_safety_netting_follow_through_on_risk",
        "attribution": "evaluated_participant_behavior",
        "review_markers": [],
        "practice_element": "safety_netting_and_return_precautions",
        "issue_type": "incomplete_or_insufficient_action",
        "description": "Synthetic description of what could improve.",
        "rationale": "Synthetic rationale.",
        "evidence": "Synthetic transcript excerpt.",
        "primary_evidence_range": {"start_seconds": 188.0, "end_seconds": 201.5},
        "supporting_evidence_ranges": []
      }
    ]
  }
}

The tables below describe every field. A field described as omitted when absent is left out of the response, never sent as null. Where the OpenAPI is looser, these tables describe what the API returns: timestamps are UTC date-times, profile values are strings, and the optional finding fields are never null.

The identity fields evaluation_type, definition_id, definition_version, rubric_id and rubric_version are always present and identify the releases that produced the evaluation. Treat IDs as opaque strings.

result for conversation_evaluation.v1

Evidence ranges are seconds from the start of the conversation's recording, on the same clock as the transcript turns' start_seconds and end_seconds in Fetch transcript and audio.

Values come from the definition's vocabularies, which can gain values within a release. Treat a value you don't recognize as unknown, not as an error. Standard evaluation definitions lists every value.

What a combined score means

Points measure each person against their own role's rubric. 0 means the person met their role's expectations for the opportunity the conversation presented; positive points exceed them and negative points fall short.

• Combine results only within one definition_id. Within a definition you can combine roles: the combined view shows how well people meet their own roles' expectations, not which role handles conversations better.
• Use one rubric_id for one role's trend.
• Across definition versions of one type, a dimension's trend continues when its score anchors did not change. Totals don't carry over.
• Different evaluation types share counts, never scores.

total_points is a sum, not a percentage. It depends on how many dimensions were scored, so read it with minimum_possible_points and maximum_possible_points. A dimension's average points stays on the −2 to +2 scale however many dimensions were scored, so it makes a better headline figure than a total.

The Conversation Reviews dashboard shows points as they are, never as a percentage: each person's average total points and average points per dimension, and how points are distributed across conversations. To match its figures, keep the newest evaluation per conversation, then:

• Average total_points over the evaluations where it isn't null.
• Average each dimension's points over the evaluations where they aren't null.
• Select evaluations by conversation_occurred_at, not completed_at. The dashboard's date range is when the conversation happened, with days starting at midnight UTC. Daily figures built on your local days won't match it.

Count findings toward a person

Only evaluated_participant_behavior findings describe the evaluated person's own behavior. To count what a person did well or could improve, count only those, as the Conversation Reviews export does for its findings to work on and findings doing well. Findings with other attributions, including mixed_attribution, describe systems, other providers, data or the patient's context. The export's total finding counts include every attribution.

Unscored is not zero

A dimension's points is null when it was not scored, and its applicability says why. When no dimension was scored, total_points, minimum_possible_points and maximum_possible_points are null and scored_dimension_count is 0. Leave null values out of averages instead of counting them as zero.

Versions and schemas

Releases are permanent: a definition or rubric ID never changes meaning.

Store or skip an evaluation whose result_schema you don't recognize; don't fail on it. Validate responses against the OpenAPI in your tests, not at runtime, so a new optional field doesn't break your integration. ChartHero's evaluator prompts and models are not part of the contract and can be improved without a new rubric version.

Map evaluations to your records

Includes return each record as it is now, not as it was when the evaluation completed. A patient can gain an MRN, a person's provider IDs can be added or replaced, and an encounter can be moved to another patient. Update your copy of a record each time an include returns it.

evaluated_actor_id never changes, even after the person leaves your organization. The actors include describes the person while they have a membership in the organization, with active: false once they are deactivated or removed. If no membership remains, the include has no entry for them; keep the last copy you received.

Many care coordinators have no provider ID on file, so their external_ids is empty. Match each such person to your staff records once, for example by name, and keep the match keyed on evaluated_actor_id.

With the evaluation scopes, includes are the only way to read encounters, patients and people: the encounter and actor routes also require documents:read. So your copies are refreshed only when you list their evaluations again, in the daily reconcile and in the longer periodic re-list. To refresh them independently, add documents:read to the key and read encounters and actors directly; that scope also grants access to transcripts and other documents.

Access

Evaluation routes require encounters:read, patients:read and evaluations:read; the definitions route requires only evaluations:read. Fetch evaluations lists each route's scopes and explains how to get a key.

A route's scopes cover every include it offers, so evaluation routes require patients:read even when you don't request include=patients. Without that include, responses contain no patient names or dates of birth, though summaries and evidence can still describe the patient.

Next, fetch evaluations and reconcile them daily.