Checks and release gates

How it fits together

Checks are repeatable tests of a Connect flow revision. A release gate can require a fresh, passing checks run for the exact revision before a production deployment is approved or activated. Every run, gate decision and override is written to the evidence chain.

Open a flow and choose Checks, or start from Prove → Checks.

What a check tests

Every case starts from a source record and runs it through the revision's own map, transform, filter and validate steps, exactly as a real run would. Checks run inside Nexra and never call your provider, so they are safe to run as often as you like.

Results are pass, fail or incomplete. The pass rate is passes divided by passes plus failures, reported with a Wilson interval. A critical case that does not pass, including an incomplete one, is a critical failure.

Generated and curated cases

Generate reads the flow's current compiled revision and derives cases from the mapping and the destination contract: a minimal valid record, one case per required field, a wrong-type value, unique and duplicate identifiers, the expected mapped output, today's policy decision and one case per error class. Generation is deterministic. Regenerating replaces only generated cases, keeps their IDs and disabled flags, and never touches curated cases.

Curated cases are the ones you write: records that matter to your business, or a recorded sandbox response for provider_result. Mark the cases that must never fail as critical.

Failures teach the suite. Add this failure to the checks promotes a failed case result, and a dead-letter record can be proposed as a new case with its data redacted. A different reviewer accepts or rejects each proposal; accepted proposals become curated cases.

Runs

Freeze a dataset version, then run it against the revision you plan to deploy. Before running, Nexra re-verifies every stored case against its digest and the compiled plan against the revision digest.

POST /api/console/connect/flows/<flow id>/checks/runs
Content-Type: application/json

{ "datasetVersionId": "<dataset version id>", "revisionId": "<flow version id>" }

Runs of up to 200 cases complete in the response. Larger runs return "status": "queued"; follow them with GET /api/console/connect/checks/runs/:runId.

Gate policy

Turn the gate on in the flow's Release gate policy:

{
  "enabled": true,
  "minPassRate": 1,
  "requireNoCriticalFailures": true,
  "requiredCaseIds": ["<case id>"],
  "maxRunAgeHours": 72
}

The gate looks at the latest completed run for the target revision's digest and reports one status:

With the gate on, approving or activating a production deployment returns 409 release_gate_blocked with the reasons unless the gate is passed or overridden. Each production run re-checks the gate when it starts and stops before any write if the gate is blocked. Rejecting a deployment is never gated, and turning the gate off restores the previous behaviour straight away.

Overrides

Only a pass rate below the minimum can be overridden. Critical failures, failed required cases, open incidents and unresolved reviews never can.

An override needs a written reason and expires within 24 hours. It must come from an Owner, Admin or Security administrator who did not author or publish the revision, did not request its deployment and did not run the checks. Each override is bound to one run and one revision digest, so a new run needs a new override. Overrides can be revoked, never edited.

Human review and adjudication

Reviewers grade case results as pass or fail, or with a score from 0 to 1, with a rationale. Grades are bound to the run, the case and a digest of exactly what the reviewer saw. Review is blind: while you can still grade, you do not see other grades. The member who ran the checks cannot grade them, and grades never change the run's own metrics.

A case needs adjudication when reviewers disagree, a reviewer disagrees with a decisive check result, or a model-judged label disagrees with the reviewers. An adjudicator who did not run the checks or grade the case records the decision. Decisions form a chain, so a later decision names the one it replaces. An adjudication can never pass a case whose check failed.

When the workspace requires evaluator grants, members also need a time-limited grant for each capability (evaluate, grade, adjudicate, promote and override). Manage them in Prove → Evaluators.

Model-judged checks

Model-judged checks add a second opinion on the cases deterministic checks cannot decide. They are off until an Owner, Admin or Security administrator switches them on in the checks assurance settings.

Calibration

Calibrate a judge in Prove → Judges against your reviewers' verdicts. Calibration reports agreement with a Wilson interval, Cohen's kappa, the confusion matrix and how often the judge passed a case your reviewers failed.

A judge counts toward a gate only when the gate policy sets countCalibratedJudges and the judge's latest calibration shows agreement of at least judgeMinAgreement (0.9 by default) over at least judgeMinCalibrationCases (30 by default) scored cases. Even then it decides only cases the checks left incomplete and no reviewer decided.

Comparisons

Compare two completed runs of the same dataset version case by case, with Hoeffding and per-case delta bounds. An independent member records promote or reject against the comparison digest; a note is required to reject. With requireComparisonVerdict on the gate, a release needs a promote verdict for its run.

Incidents

Open an incident from Prove → Incidents when something goes wrong in production, or set the flow to open one automatically on a production failure. Its timeline is hash-chained, with secrets redacted.

While an incident is open or under investigation and set to block release, the flow's gate is blocked with OPEN_INCIDENT, which no override can lift. Interventions act straight away through the normal permissions: pause the flow, revoke a grant, switch off a write or expire pending approvals. Suggested cases turn the incident's dead letters into proposals for review. Exports mask secrets and e-mail addresses.

Who can do what

API

All routes are under /api/console/connect.

Next