EF · Evidence fidelity

Copies a carried opaque value instead of inventing one (paired, B arm)

Gemma 4 31B QAT

Finding:BREAKS· grade C(research conclusion)

0% correct (0/10)

Pass ratesingle rate, no difficulty dial

0/10 scored trials passed · 95% interval 0%–28%

Why it breaks

The model fails to pass on a value returned by an earlier call.

Example failure

Trial result: FAIL (one model answer)

FAILinvalid_ticket

step 3: advance_1({'service': 'svc_0', 'ticket': 'ticket_flag_0_12345'}) -> invalid_ticket: `ticket` must be the exact ticket returned by the most recent successful call to set_flag_0.

Open this trial →Open the run it came from →

Evidencewhat this finding rests on

Ground truthDeterministic simulation: the right answer is computed, not judged
StatisticsWilson 95% interval 0%–28%Grade C: thin evidence
Sample size10 scored trials in total: small sample (under 15)
Fresh-seed confirmationNot run yet
ReplicationNot run yet

Raw evidenceruns and trials

Runs behind this row
- camp-ef-tickets-gemma4qat-001/B 10 trials
Every interactive-execution trial for this model → broader than this row
The example failure, in full →

Back to the capability map →