Challenge
Long-context, high-effort inference did not reliably produce a final actionable answer. A model-only response was not a sufficient correctness boundary.
NCMA SYSTEMS / APPLIED AI ENGINEERING / VERIFIED LAB
Local Agent Lab
Building a resumable local agent runtime where model outputs are proposals and independent evidence determines what becomes accepted state.
12/12Synthetic cases correct
60/60Evaluated fields correct
22Checkpoint rounds
28Rejected proposals
SEQUENCE / VERIFIED HOP
How a proposed hop becomes durable state
Model proposals are untrusted. A deterministic validator must match each next hop to authoritative source rows before progress can be checkpointed.
- 01
Coordinator Corpus
Verify corpus SHA and fetch NODE / PATCH / TEAM rows
- 02
Corpus Coordinator
Return focused authoritative evidence
- 03
Coordinator Model HIGH
Request bounded preliminary draft
- 04
Model HIGH Coordinator
Return untrusted analysis
- 05
Coordinator Model MEDIUM
Request one-hop JSON proposals
- 06
Coordinator Validator
Check highest numeric PATCH revision and target
- 07
Validator Checkpoint
Accept verified steps; reject unsupported candidates without losing progress
- 08
Checkpoint Coordinator
Resume from compact verified state
- 09
Offline tests Oracle
Score final result AFTER inference, never during
ORACLE BOUNDARY: the synthetic oracle was withheld from the model in the historical run. Independent comparison is post-inference only. No model-authored host commands are executed.
Full source-grounded sequence and recovery branches ↗EXPERIMENT CHRONOLOGY
G2 → G3 → G4H
G2 coding tests are a different workload from the G3/G4H synthetic graph audit. Their pass fractions must not be compared as model accuracy.
| Gate | Configuration | Evidence | Finding |
|---|---|---|---|
| G2 / coding | 32K; LOW / MEDIUM / HIGH | MEDIUM: 11/11 unit tests PASS. LOW: syntax rejected. HIGH: 8K/12K output exhausted. | Independent unit tests gate code proposals. |
| G3 / monolith | 98,304 context reserved; 77,103 input + 16,384 completion tokens | HIGH-only; no final JSON, 0 bytes final, 0/12 synthetic cases returned. | Large context did not guarantee completion. |
| G4H / hybrid | 98,304 reserved; targeted retrieval; HIGH draft + MEDIUM finalizer | 12/12 synthetic cases, 60/60 fields, 22 rounds, 44 calls, 78 verified hops, 28 rejections. | Checkpointed runtime completed this fixture. |
Engineering decisions and evidence
Architecture
Task-specific retrieval → bounded high-effort draft → medium-effort JSON finalizer → independent source validator → atomic checkpoint → resumable execution.
Engineering decisions
Reject invalid graph transitions without losing verified progress. Keep source evidence separate from model proposals. The model does not execute host shell commands.
Validation
G4H produced correct results for 12/12 synthetic incidents and 60/60 evaluated fields. The runtime accepted 78 verified hops across 22 checkpoint rounds and 44 model calls, rejecting 28 proposed steps.
Trade-offs
The published oracle and corpus enable independent replay, but do not prove general-world accuracy. The 96K configured context window is not the amount ingested in every G4H inference; G4H used focused retrieval.
REPRODUCIBILITY / SEPARATE BACKEND
Re-run the orchestration, then score the answer
The compatibility guide documents a new local Chat Completions backend. HTTP success alone does not prove HIGH/MEDIUM reasoning semantics.
- Verify the published corpus and historical tests offline.
- Check the alternative server renders both HIGH and MEDIUM requests correctly.
- Run with a fresh checkpoint/output directory, resuming from verified state if paused.
- Compare the NEW final result to the public synthetic oracle after inference.
python -m unittest discover -s tests -v
python -m ncma_lab.g4_hybrid --check-only
python -m ncma_lab.g4_hybrid --max-rounds 18
python scripts/verify_backend_result.py --result runs/backend-trial-01/final_results.jsonConfigure PYTHONPATH, NCMA_MODEL_API, NCMA_MODEL_ALIAS and NCMA_RUN_DIR first. The guide covers security, budgets, resume and validation.
Scope: one synthetic deterministic fixture, not a claim of broad real-world accuracy. Local model inference is not an OS sandbox.