Anima in Discrimine.
The soul is in the parting.
Truth is in divergence.
That line has been one of Wallaby’s deepest operating laws. It is not a poetic caption placed on top of the recall layer. It tells the machinery what not to destroy.
Most computational systems treat disagreement as unfinished work. If five readers return five interpretations, the pipeline ranks them, votes, averages, or asks another model to synthesize the set. The output becomes cleaner. It also becomes less informative.
Wallaby was built around the possibility that two things could be held as true. The possibility that the truth was in the delta, where they diverged rather than converged.
The same idea later appeared in my work on evaluating generative systems. A harness produces outputs. A judgment layer has to decide what those outputs mean. The temptation is to compress the judgments into a scalar and call the scalar knowledge.
Sometimes that is correct.
Sometimes it is the exact moment the truth disappears.
Two kinds of truth
The first obligation is to ask what kind of object is being judged.
Some truths should converge.
A file has a hash. A run used a particular configuration. A message belongs to a snapshot. A response contains or does not contain a required element. A tool call succeeded or failed. A count has a unit. If independent judges disagree about these, the disagreement is a problem to resolve.
Other truths are not exhausted by one determinate answer. Deterministic results call for deterministic evaluation. Generative results call for judgment in evaluation.
Does this recollection feel recognizably like the person or organization whose corpus produced it? Did a generated interface preserve the intent of a design brief? Is a model’s refusal careful or evasive? Did an answer exercise judgment or merely imitate the surface form of judgment? Which of two plausible readings captures the more important risk?
These are not invitations to relativism. They are claims about underdetermination.
The evidence can constrain the field without forcing one point inside it.
Wallaby’s central epistemic problem is knowing which domain it is in.
Where truth is singular, divergence is an alarm. Where truth is plural, forced convergence is a lie.
Why one judge is not enough
A single capable judge can produce a beautiful explanation of an incomplete view.
Wallaby learned this when one reader processed a large pool, passed every quality criterion, and still missed a load-bearing part of the terrain. The failure was not stupidity nor incapability. It was attention. The reader did not know what it had failed to see, so its prose contained no uncertainty shaped like the omission.
The evaluation analogue is common.
One judge sees correctness. Another notices that the answer changed the user’s task. A third catches an unsupported inference. A fourth is sensitive to tone or lived plausibility. Give all four the same rubric and ask for one score too early, and they may converge on the loudest criterion while the quiet failure vanishes.
More judges do not automatically solve this. Five identical instruments can reproduce one shared reflex five times.
Independence has to be designed.
Independence is more than separate calls
Wallaby’s Probe Swarm (“Pollux” retrieval arm) sends walkers through different entry doors and preserves their paths.
The path matters because two readers reaching the same conclusion through independent parts of the corpus provide stronger evidence of correlating salience than two readers following the same retrieval surface. Likewise, two readers who diverge after encountering different regions tell us something about the structure of the evidence.
An evaluation harness needs the same provenance. But before deciding what to record, you have to answer what you think you are measuring.
Are we evaluating models, or model-plus-context systems?
Nobody deploys a model. They deploy a model with a task shape around it, a context front-loaded into it, a set of tools exposed to it, and a strategy for eliciting the answer. Change any one of those and the result moves, sometimes further than swapping the model would have moved it. Capability turns out to be config-bound: change the task or the context you front-load, and “proven” stops being true.
So the list below is not a list of metadata fields. One item on it is the experiment. The rest describe the experiment. Losing any of the others costs reproducibility. Losing that one means you never knew what you ran.
For every judgment worth trusting, preserve at least:
· The artifact or output the judge actually saw. Not what the system produced. Harnesses reformat, truncate, and sometimes attach a reference the system never had. A judge can be entirely accurate about an object the system never emitted.
· The system and task instructions. Both, separately. The same rubric under a different framing is a different rubric, and the framing is the part everyone reconstructs from memory afterward.
· Model and configuration. Not the model name. The version or snapshot, the sampling settings, and the harness that assembled the call. A result you cannot rebuild is not a result, and a config recorded as “the latest one, default settings” has already lost.
· Tools and accessible context. What the judge could reach, and what was in front of it before it read anything. Context is not neutral background. It is the largest lever in the system and the one least often reported.
· The judge’s independent first read. Banked before any cross-reading, because it cannot be recovered afterward. A read taken after exposure is a different measurement wearing the same name.
· Any later peer information. What the judge saw, when, and from whom. Agreement reached before exposure and agreement reached after it are different evidence, and a score does not distinguish them.
· The transformation from qualitative judgment to score or disposition. The step where reasoning becomes a number is where the interpretation is destroyed, and it is almost never written down.
Without that record, agreement is ambiguous. It may be corroboration. It may be shared priming. It may be two copies of the same blind spot.
There are two further possibilities, and neither one appears anywhere on that list.
The first is that every judge only went once. Independence across judges buys nothing if each of them scored a single draw. Generative systems do not produce the same artifact twice, and judges do not always score the same artifact the same way. Both ends of the measurement sample from a distribution. If that distribution is wide relative to the effect under study, then five independent judges scoring one run each are not five measurements of the system. They are five measurements of one draw, and the spread you publish is the spread among your judges rather than the spread that would have changed your decision.
Every classical validity condition assumes the thing being measured holds still while you measure it. Sample the items properly, keep the test set clean of the test-taker, make the set large enough to mean something. All three are conditions on the questions. None of them touches the variance in the thing answering them.
The second is that the judges were never independent to begin with. If the judge is a language model and the subject is a language model, they share training data, failure modes, and stylistic priors. A panel assembled that way can be independent in every dimension the record captures and still be correlated in the dimension that decides the outcome. A panel of correlated judges is one judge with a louder voice.
No field on that list catches it, and I do not have a clean way to measure it. Recording the configurations is still worth doing, because a record showing five near-identical judge configs at least makes the question askable. An undocumented panel does not even do that.
Path provenance is not audit garnish. It changes the weight of convergence.
Enumerate before adjudicating
The safest judgment sequence is field first, ruling second.
Independent readers should identify what they see before a judge knows what the other readers found. Their first reads should be banked. Only then should cross-reading begin.
This protects two things.
First, it prevents social convergence. Once a confident framing is visible, later readers tend to organize around it even when instructed not to.
Second, it preserves the single-witness finding. A catch does not become false because nineteen other readers entered through doors that never passed it.
The field can then be characterized:
· What converged independently?
· What depended on one path or one judge disposition?
· Which findings conflict?
· Are the conflicts factual, interpretive, or jurisdictional?
· What evidence would discriminate between them?
· Which differences should remain open?
Only after that map exists should adjudication begin.
If adjudication starts by asking for the best answer, the field never gets to exist.
The Parlay’s correction
Wallaby’s Parlay is a read of reads.
An early version gave judges raw floor material because more source material seemed safer. The result flattened. Each judge held a thin slice, so the only structures that survived cross-reading were the loud ones shared across slices. The quieter evidence carried by how the walkers moved never reached the judgment layer.
The fix was to give judges the walkers’ reports: deposit plus narrated path.
That change restored the actual object being judged.
The judge was not supposed to independently re-read the corpus. It was supposed to interpret the plurality produced by the walkers. Feeding it the floor collapsed two layers and destroyed the evidence created by the first.
Evaluation harnesses make this mistake whenever they silently change the object between production and judgment.
A system generates an answer under one context. The evaluator sees a larger context, a hidden reference answer, or a reformatted artifact. The judge may be accurate about what it saw and irrelevant to what the system actually did.
The object, context, and provenance must stay bound.
Convergence and divergence are different evidence
Agreement is not one thing.
Independent agreement on a factual defect is corroboration.
Agreement on a response shape among identically booted judges may be a lineage reflex.
Agreement after peer exposure may be persuasion.
Agreement produced by the same hidden reference may be answer-key leakage.
Disagreement is not one thing either.
It can indicate a broken rubric, insufficient evidence, genuine discretion, different risk tolerances, different paths through the source, or a question whose premise incorrectly assumes one answer.
A judgment layer should classify the divergence before trying to remove it.
This is where the Latin matters.
Anima in Discrimine does not worship disagreement. It asks what becomes visible at the parting that was invisible before it.
A score is a lossy export
Scores are useful. They are also compression artifacts.
An evaluation system often needs a pass rate, ranking, or regression curve. The mistake is not producing the number. The mistake is letting the number replace the judgments that generated it.
The qualitative field should remain available beneath the scalar:
· the independent reads
· the minority findings
· the unresolved forks
· the judge configurations
· the adjudicator’s reasoning
· the limits of the sample
· the exact scope of the claim
This is the judgment equivalent of Wallaby’s immutable floor and plastic read layer.
The raw outputs and first-order judgments remain fixed. Interpretations and aggregate decisions can change as the evaluation question improves.
You can recalculate a score.
You cannot recover a minority judgment that was never preserved.
The human seat
Wallaby’s P7 keeps the final realness decision human.
The same boundary matters in evaluation, especially when the criterion involves discretion, taste, safety tradeoffs, or human impact.
The harness can establish facts. It can expose patterns. It can show that five independent judges converged, that one found a unique defect, or that results changed when the judge disposition changed. It can reject an invalid run. It can enforce a rubric’s mechanical requirements.
It should not pretend that those operations eliminate the need for accountable judgment.
Someone has to decide which harms matter, how uncertainty is treated, and whether a qualitative difference crosses the deployment bar. Hiding that decision inside an evaluator model does not remove the human choice. It makes the choice harder to locate.
The final seat should be visible, named, and responsible.
A practical judgment architecture
The architecture I now trust looks like this:
- Freeze the object. Preserve the exact output, task, configuration, and accessible context.
- Separate mechanical checks. Resolve determinate facts with deterministic tools where possible.
- Collect independent reads. Do not expose peer judgments during the first pass.
- Preserve path and rationale. A verdict without how it was reached is a weak receipt.
- Map the field. Name convergence, minority catches, contradictions, and uncertainty.
- Classify the disagreement. Decide whether it is error, missing evidence, rubric ambiguity, discretion, or genuine plurality.
- Adjudicate only what needs adjudication. Do not manufacture consensus for presentation.
- Keep the underlying field. A score never replaces its provenance.
- Leave the final value judgment with an accountable human. The system can support that seat. It cannot make the seat disappear.
This is not slower rigor for its own sake.
It prevents a particular kind of false confidence: a clean number produced by a judgment process that destroyed the very disagreement needed to interpret it.
The recall layer’s ethos
Wallaby exists to remember a person without capturing him inside one interpretation.
Its evaluation architecture exists for the same reason. A result should be stable where the world supplies a stable answer. Where the world supplies several defensible readings, the apparatus should show the parting and resist the urge to close it for aesthetic reasons.
The floor stays fixed.
The reads remain attributable.
The paths stay visible.
The minority survives.
The human retains the final seat.
Anima in Discrimine.
The soul is in the parting.
