MacroEval manual

Most agents check their own work before it ships. A model call, or a list of rules, decides whether a reply, a frame or a design is good enough, and the product retries or discards what fails. That check is your grader. If it passes what your reviewers would reject, bad output reaches customers. If it fails what they would pass, you pay to redo good work.

MacroEval keeps that grader's rules as a versioned rubric your product fetches at run time. It stores every case your product judged, with its verdict, and lets your reviewers label cases pass or fail. Then it shows, in numbers, how often the grader agrees with them. A reviewer's reason for a fail can become a new rule, so the grader learns from every rejection.

MacroEval is never in the path your users wait on. Your product judges in its own loop, with its own model, and logs the result afterwards.

Teams use it in two ways, usually in this order:

• Measure the grader you have. Log what your product already judges, label a sample, and run a report. You learn how often today's verdicts match yours.
• Improve it. Turn rejections into rules, save a new version, and report on it before it goes live. Make it current when it does better.

Reports treat a fail as the positive class. The mistake that costs you is a reject that slips through, so the headline number is fail recall: of the cases people failed, how many the grader failed too. Fail precision keeps it honest, because a grader that fails everything has perfect recall and is useless.

1. Sign in with Google. You're asked to create your first project. Name it after the product or agent you're grading.
2. Create an API key under Settings → Developers with Create API key. Copy it now: it's shown once.
3. Write a rubric under Rubrics with Create rubric: a slug, the judge instructions, and one rule per line. It becomes version 1, and current.
4. Wire up your product to fetch the rubric, judge with it, and log each case and its verdict.
5. Label cases under Cases: pass, or fail with a reason.
6. Open the rubric and choose Run report to see how the verdicts agree with your labels.

The Overview page tracks these steps for the project you have open, marks the ones that are done, and links to each.

Your product talks to MacroEval in three places. All of them are HTTP calls with your project key, and the API reference has every field.

1. Fetch the rubric it should judge with, when it starts or every few minutes, and cache it. Keep a copy bundled with your product, so it can still judge if MacroEval is unreachable. A product that must not change without a deploy can pin a version number instead of following the current one.
2. Judge each output with the rubric's instructions and rules, in your own loop and with your own model. The verdict decides what your product does next.
3. Log the case with what was judged, then your product's verdict, naming the version it used. Do it after the fact, off your users' path. A queue or a background task is ideal.

A change to the rubric reaches your product on its next fetch, with no redeploy. If you log cases but not verdicts, reports can still regrade them. Logging verdicts is what tells you how your product's own check is doing.

A rubric is a slug, judge instructions, and up to 60 rules. Instructions say what the judge is looking at and how to read it. Rules are short statements that every output must satisfy. A broken rule always means a fail.

Open a rubric under Rubrics to see its Versions, newest first, with the current one marked. Choose Make current on any other version to switch your product to it. New version starts from the current one. Edit the instructions and rules, optionally name a model and add notes, and choose Save new version. Untick Make current to save it without your product picking it up.

Model is optional. MacroEval's regrade uses it when it's a DeepInfra model id, written as owner/name, and its own judge otherwise. Your product can read it too, if you want both to judge with the same model.

Writing rules that work

• Name a kind of fault, not one case: “Never promise a refund the order doesn't qualify for”, not “Ticket 812 is wrong”. A rule about one case can't catch the next one.
• One fault per rule, so a broken rule says exactly what went wrong.
• Make it checkable from what the case shows. The judge only knows what is in the case's parts.

A case is one output your product judged, under your own id, the externalId. Logging the same id again updates the case instead of adding another. It carries everything needed to judge it as a list of parts, in the order they should be read. Each part has a role, your name for what it is: question, reply, reference, output.

A case holds up to 12 parts and 6 images in all, counting video frames. The judge can't open files, so send what it should look at as parts of their own: renders of a 3D model, a structure's scores as JSON, frames of a video. A case with nothing the judge can read shows an error in reports, never a guessed verdict.

On the Cases page, glTF models open with View in 3D, and PDB, mmCIF and SDF structures with View structure, right inside the case. A case can also carry context: any JSON for your own reference, such as a channel or a seed. It shows on the case and is never shown to the judge.

Every case on the Cases page shows its parts and three verdicts: the human label, your product's, and the latest regrade's, the last two with the version they used. Filter by rubric at the top. The newest cases come first.

Choose Pass, or write what's wrong in Why it fails and choose Fail. With add as rule ticked, as it is by default, the reason also becomes a rule in a new current version, so write it the way you'd write a rule. A reason that matches an existing rule isn't added twice. You can label a case again at any time, and its latest label is the one that counts.

Label passes as well as fails. A report's numbers mean little until both kinds are labelled, and a few dozen of each is a reasonable start.

Open a rubric, pick a version and choose Run report. A report takes the latest label of up to 200 labelled cases, frozen when it starts, and fills in two columns. Product is how your product's own verdicts for that version agree with the labels. Regrade is how MacroEval's judge, re-judging every case with that version now, agrees with them. A table below lists every case side by side with the judge's reasons.

Reports run while their page is open, and the bell tells you when one is ready. All your project's reports are under Reports. A case the judge couldn't regrade shows its error in the table and is left out of the numbers.

Reading the numbers

• Low product fail recall: rejects get through today.
• Regrade higher than product: the rubric is right, but your product's check isn't applying it well. Look at its prompt or model.
• Both low: the rules miss what your reviewers reject. Fail those cases with reasons, and the rules follow.
• Low fail precision: the grader raises false alarms, and work that was fine gets redone.

A new version can be measured before your product uses it, on the same labelled cases as the one it would replace.

1. Open the rubric, start a New version, make your changes, untick Make current and save.
2. Choose Run report for the new version, and for the current one if its last report is old.
3. Compare the Regrade numbers. A better version raises fail recall without lowering fail precision.
4. If it's better, choose Make current. Your product picks it up on its next fetch. If it isn't, nothing has changed for your users.

Use one project per product or agent. Switch between them in the top bar. Create one under Settings with New project. Everything else under Settings applies to the project you have selected.

API keys live under Settings → Developers and reach their own project only. A key is shown once when it's created. Revoke one there, and it stops working on its next request.

The bell in the top bar tells you when a report is ready or has failed, when a label adds a rule to a rubric, and when an API key is created.

URL checks are separate from agent evals, and belong to your account rather than a project. Add a service with its name and base URL, then add checks: a page URL and pass criteria in plain English. When a check runs, MacroEval opens the page in a real browser, captures what it shows, and judges each criterion with a note and a screenshot. The free plan allows three check runs in any 24 hours.

Still stuck? The ? in the app's top bar explains each page.