API reference

Every request carries a project key as a bearer token. Sign in, pick the project in the top bar, and open Settings → Developers to create one. It is shown once: MacroEval keeps only a hash of it, so a lost key can't be recovered. Revoke it there and create another. Revoking takes effect on the next request.

curl https://macroeval.com/api/v1/rubrics/support-reply \
  -H "Authorization: Bearer me_live_…"

• A key is me_live_ followed by 32 letters and digits. Anything else is refused as malformed.
• A key reaches one project's rubrics, cases and reports, and nothing else. No parameter picks another project.
• Keep keys on your servers. A key in browser or mobile code is public. Use one key per system, so revoking one never breaks the others.

Fetch a rubric's current version. It's the call your product makes before it judges anything, and it tells you at once whether the key and the slug are right.

curl https://macroeval.com/api/v1/rubrics/support-reply -H "Authorization: Bearer $MACROEVAL_API_KEY"

{
  "slug": "support-reply",
  "version": 3,
  "instructions": "Judge the reply against the order on file.",
  "rules": ["Never promise a refund the order doesn't qualify for", "Answer the question that was asked"],
  "model": null
}

A 404 means the project has no rubric with that slug yet. Create its first version with POST /rubrics/{slug}/versions, or under Rubrics in the app.

Requests and responses are JSON. Send Content-Type: application/json. Every error has the same shape, with the HTTP status telling you what kind it is.

{ "error": "rules must be an array of strings" }

Identifiers

Limits

A case is a list of parts, read in order. Each has a type and a role, and the role is your name for what the part is: question, reply, reference, output. Roles are lowercase letters, digits, _ and -, up to 32 characters, and appear in the judge's prompt and on the cases page.

A case may have up to 12 parts and 6 images in all, counting video frames, since each one is something the judge looks at. Its text and JSON together may total 96 KB. Every URL must be public https, up to 2,048 characters. MacroEval fetches images and frames when it regrades, and files only when a reviewer opens them, up to 25 MB.

The judge reads text, JSON, images and frames, and can't open files. So send it what it should judge as parts of their own: renders of a 3D model, a structure's scores as JSON, frames of a video. A case with nothing the judge can read is reported as an error rather than given a verdict.

Cases logged with the older images: [{ role, url }] field keep working. Those images become image parts, ahead of any parts sent alongside.

10 endpoints in three groups. Every path is relative to https://macroeval.com.

A report freezes the latest label of up to 200 labelled cases of a rubric when it starts, so labels added later don't shift it. It compares your product's verdicts for that version with the labels, and regrades every case with the version so you can see how it would do.

Regrading takes time, so a report advances in slices of about 25 seconds while you poll. The POST runs the first slice, and every GET of a running report runs the next one. Poll until status is no longer running. A report nobody polls waits where it stopped. The app polls for you while a report's page is open.

The agreement numbers

Both product and regrade carry the same numbers. A fail is the positive class, because a missed reject is the costly mistake. A ratio with nothing to count is null, never 0.

{
  "id": "6630f1c2a9d04b5e8e1f2a77",
  "rubric": "support-reply",
  "rubricVersion": 4,
  "status": "ready",
  "n": 120,
  "product": { "n": 118, "agreement": 0.78, "failRecall": 0.64, "failPrecision": 0.82, "confusion": { "tp": 23, "fp": 5, "fn": 13, "tn": 77 } },
  "regrade": { "n": 120, "agreement": 0.9, "failRecall": 0.91, "failPrecision": 0.87, "confusion": { "tp": 33, "fp": 5, "fn": 3, "tn": 79 } },
  "perCase": [
    { "externalId": "ticket-812", "label": "fail", "product": "pass", "regrade": "fail", "reasons": ["refund outside the 30-day window"] }
  ],
  "progress": { "done": 120, "total": 120 }
}

Example values.

Log a case and your product's verdict

Log the case first, then the verdict, naming the version your product judged with.

curl -X POST https://macroeval.com/api/v1/cases \
  -H "Authorization: Bearer $MACROEVAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"externalId":"ticket-812","rubric":"support-reply","parts":[
        {"type":"text","role":"question","text":"My blender arrived broken. Refund?"},
        {"type":"text","role":"reply","text":"I have refunded your card in full."},
        {"type":"json","role":"order","data":{"delivered_days_ago":41,"refund_window_days":30}}]}'

curl -X POST https://macroeval.com/api/v1/cases/ticket-812/judgments \
  -H "Authorization: Bearer $MACROEVAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"rubricVersion":3,"verdict":"pass"}'

Video, 3D models and structures

Send what the judge should look at as images or frames, and the original as a file for reviewers.

"parts": [
  { "type": "video", "role": "clip", "url": "https://cdn.example/shot-4.mp4",
    "frames": ["https://cdn.example/shot-4-0s.jpg", "https://cdn.example/shot-4-2s.jpg", "https://cdn.example/shot-4-4s.jpg"] }
]

"parts": [
  { "type": "image", "role": "render", "url": "https://cdn.example/kitchen-front.png" },
  { "type": "file", "role": "model", "url": "https://cdn.example/kitchen.glb", "mediaType": "model/gltf-binary", "name": "kitchen.glb" }
]

"parts": [
  { "type": "json", "role": "scores", "data": { "mean_plddt": 84.2, "interface_contacts": 31 } },
  { "type": "file", "role": "structure", "url": "https://cdn.example/design-7.cif", "mediaType": "chemical/x-mmcif" }
]

Try a version before it's live

Save the candidate without making it current, report on it, and compare its regrade with the current version's. Your product keeps judging with the current version throughout. If the candidate does better, make it current under Rubrics in the app, or save it again with current: true.

V=$(curl -s -X POST https://macroeval.com/api/v1/rubrics/support-reply/versions \
  -H "Authorization: Bearer $MACROEVAL_API_KEY" -H "Content-Type: application/json" \
  -d '{"instructions":"…","rules":["…"],"current":false}' | jq .version)

ID=$(curl -s -X POST https://macroeval.com/api/v1/rubrics/support-reply/reports \
  -H "Authorization: Bearer $MACROEVAL_API_KEY" -H "Content-Type: application/json" \
  -d "{\"version\":$V}" | jq -r .id)

until [ "$(curl -s https://macroeval.com/api/v1/rubrics/support-reply/reports/$ID \
  -H "Authorization: Bearer $MACROEVAL_API_KEY" | tee report.json | jq -r .status)" != running ]; do sleep 2; done
jq '.regrade | {failRecall, failPrecision, agreement}' report.json

Fetch the rubric without depending on MacroEval

Cache the rubric for a few minutes and keep a copy bundled with your product. If MacroEval is unreachable, judge with the copy and log the case later. MacroEval should never sit in the path your users wait on.