Measured on one laptop GPU
The learning result is loading
The page will state the measured answer once the saved artifact loads.
Can a training run survive across flaky datacenters?
Partly. I trained a tiny model on my laptop GPU and simulated the datacenters and their outages around it. Training does keep going across sites, but every way of coping costs something: waiting stalls everyone, restarting from the last save redoes work, and letting the surviving site carry on saves work but learns a little worse. The chart below shows one outage handled four ways, including a cheating version that knows in advance when the failure will hit, kept as a yardstick.
The same failure, four ways to recover
The same failure hits all four policies. The shared clock shows what stops, what is restored, what must be redone, and when useful work catches up.
Every policy track shares one time axis taken from the artifact. Recovery is mechanical: each policy reaches the same durable frontier, the last point safely written to storage, after explicit preemption, restore, replay, and membership changes.
Every mark is drawn from a saved recovery episode. Event IDs, nanosecond bounds, what happened to each unit of work, checkpoint lineage, and matched-frontier hashes stay available in the structured trace.
- Failure
- Preemption
- Restore
- Replay
- Rejoin
Accessible recovery timeline list
Where the work and the data went
Work conservation
attempted = retained + lostWhere the inter-site bytes went
six traffic classes that never overlapMechanical completion
same durable frontierGo deeper
Each section below opens to the measurements behind it.
Did the model still learn as well?Measured on one laptop GPU
Loading…
Measured small-model learning
Did the adaptive method keep learning just as efficient?
Lower held-out loss means the model predicts better. The cards show how much training each policy attempted to get there, and the lines show what it learned over the same failure clock.
Policy medians summarize matched strata. Inside each stratum, the paired effect subtracts fixed-local progress per FLOP from adaptive progress per FLOP, so zero means the two are equally efficient.
Artifact, dataset, runtime, paired evaluation, and per-run records remain attached below.
Median trajectory
Held-out NLL over wall ticks
Paired causal contrast
Adaptive minus fixed progress per FLOP
Paired interval not loaded.
Frozen decision
The gates that could fail this claim
Exact paired evaluations
| Stratum | Fixed progress/FLOP | Adaptive progress/FLOP | Adaptive − fixed |
|---|
Run-level measurements
| Run | Stratum | Final NLL | Attempted tokens | Energy | Active time |
|---|
Dataset, runtime, and source identity
not loaded
What if every method does the same amount of work?Measured on one laptop GPU
Loading…
Measured, same amount of work
What survives when useful work is held equal?
Same 524,288 useful tokens · six held-out pairs
Equal-work result loading
The page will state the result once the saved result loads.
Useful work is the fair ruler here. Both policies end with the same 524,288 tokens actually kept by the model. Adaptive avoided work it would have had to redo and finished the schedule sooner with no meaningful loss of learning, but its GPU drew too much energy to pass.
This check fixes a flaw in the one before it, where stopping early flattered the result. Six paired evaluation strata all end at the same amount of canonical work. Read learning as paired NLL noninferiority, physical work as attempted-FLOP savings, schedule time as opportunity ticks, and energy as a paired device-energy ratio.
Exact pairs, run records, protocol predecessors, hashes, and the modeled-mechanics boundary remain attached below.
Observed policy medians
Same useful frontier, different path
Paired 90% intervals
Three gates passed, one failed on a noisy meter
The decision rule, set in advance
Seven gates passed; the energy gate failed, but its meter could not resolve a 5% bar
Six paired evaluation rows
| Stratum | Fixed NLL | Adaptive NLL | Adaptive − fixed NLL | Attempted-FLOP saved | Ticks saved | Energy ratio |
|---|
All 16 run records
| Run | Split | Policy | Interrupted | Final NLL | Attempted / canonical tokens | Opportunity ticks | Device energy | Checkpoints |
|---|
Two earlier protocol results
| Artifact path | Conclusion | SHA-256 |
|---|
Modeled mechanics bridge boundary
Source hashes and complete provenance
not loaded
Does saving progress cost energy?Measured on one laptop GPU
Loading…
Measured power draw
Did how often we save checkpoints cause the energy penalty?
32 real GPU runs · measurement rejected
Result loading
The page will state the measurement boundary once the saved artifact loads.
No policy claim is available yet.The experiment ran; the measuring stick failed. All 32 planned comparisons started from the exact same trained model. Asking the sensor every 20 milliseconds did not make it answer that fast: its reading effectively changed about every 495 milliseconds. That is too slow to decide whether sparse continuation saves energy.
The 2×2 cadence-by-continuation design is intact, but the energy quantity it was meant to estimate cannot be used. Both frozen update-count invalidators fired and the logger calibration hit its limit. Arm medians and contrasts stay visible as diagnostics, never as evidence about mechanism.
The 32-run ledger, per-phase metrics, bound source identities, and lazy raw telemetry are attached below.
Validity before effect size
Two active invalidators stopped the claim
Observed arm medians
The intended 2×2 comparison completed
No-failure controls
State, work, and final NLL matched exactly
Raw diagnostic contrasts
Shown here, but they cannot support a claim
Do not read this as a good result. These numbers describe the telemetry stream that was rejected. They cannot show that sparse continuation fixed the earlier energy penalty.
All 32 run records
| Run | Split / block | Arm | Attempted / canonical | Final NLL | Samples / effective updates | Raw board energy | Checkpoints | Raw trace SHA-256 |
|---|
Selected-run phase metrics
Phase energy is kept for diagnosis. It cannot support a causal comparison between policies, because the logger invalidators fired.
| Phase | Duration | Idle-subtracted board energy | Energy / canonical token | Effective update equivalents |
|---|
Hashes and complete compact provenance
not loaded
Load the raw point trace on demand
Raw telemetry is not loaded yet.
| Point | Time | Phase at timestamp | Board power | GPU util. | Memory util. | SM / memory clock | Temperature | P-state |
|---|
Measured energy use
What caused the local GPU-energy penalty?
32 real GPU runs · measurement valid
Result loading
The page will state the mechanism once the saved artifact loads.
Mechanism estimate loading.Checkpointing less often looks a little better, and the energy question is not closed. Both policies kept exactly 524,288 useful tokens. Sparse continuation attempted about 3% less work, finished 40 scheduling ticks sooner (a modeled number), and used about 3% less energy. That checkpoint snapshots caused the earlier penalty is not shown: replaying lost work cost more than writing them. The version that subtracts an estimated idle baseline cannot rule out a zero effect.
This repeats the earlier test, this time with a cumulative-energy counter the hardware actually supports. No measurement invalidator fired. Positive total and checkpoint-group interactions pass the frozen primary attribution gates, and sparse continuation holds the learning, work, schedule, and energy gates. The sensitivity-only, idle-subtracted interaction crossed zero, so the result does depend on how the idle baseline is handled.
The 32-run ledger, selected-run counter phases, bound identities, and lazy raw counter points are attached below.
Measurement validity
The cumulative counter cleared the frozen support rules
Paired causal contrasts
The main interaction passed its gate; replay, not snapshots, is the biggest term
Counter support by phase
Checkpoint groups have enough counter data; rare recovery phases do not
Frozen decision
All 11 gates passed as written; one mechanism gate is vacuous
Mechanism · 3 / 3
Sparse continuation · 8 / 8
All 32 run records
| Run | Split / block | Arm | Attempted / canonical | Final NLL | Polls / counter updates | Run energy | Checkpoints | Counter trace SHA-256 |
|---|
Selected-run cumulative counter and nine phases
| Phase | Evidence class | Duration | Energy | Energy / canonical token | Counter updates | Effective update equivalents |
|---|
Hashes and complete compact provenance
not loaded
Load the raw cumulative-counter trace on demand
Raw counter telemetry is not loaded yet.
| Point | Time | Phase at timestamp | Cumulative board energy | Counter update | Ancillary board power | GPU util. | Temperature |
|---|
Can a controller pick the best settings by itself?Unseen test cases
Loading…
Unseen test cases
One controller, six unseen stress families
Same useful work · result loading
Result loading
The page will say whether adaptive switching kept the held-out learning and beat the fixed comparator that was frozen before the run.
The infrastructure is simulated, not measured, unless the artifact says otherwise.Every policy gets the same training examples and has to reach the same amount of useful work. The result shows whether switching consistency mode helped, what it saved, and where the controller refused to guess.
The fixed comparator is chosen from calibration data only. Six untouched stress families pair the adaptive and fixed policies against the best whole-policy choice in hindsight. Sampling intervals and infrastructure uncertainty stay separate rather than being merged into one number.
Exact optimizer commits, mode changes, membership, lineage, work, WAN accounting for traffic on the long link between sites, assumptions, uncertainty, and missing evidence all appear below.
Paired held-out effects
Learning, time, communication, work, and energy stay side by side
Ranking under uncertainty
Where the result survives uncertainty, and where it refuses to answer
Each held-out stress family is split into labeled regions of infrastructure uncertainty, showing where adaptive wins, where a fixed policy wins, where the ranking flips, and where the controller declines to rank them at all.
Accessible ranking-region table
| Family | Uncertainty region | Ranking | Comparator | Reason |
|---|
Selected held-out family
What the controller saw, chose, and preserved
The adaptive and comparator mode intervals on one shared axis, marked with membership, WAN, abstention, merge, and rejoin events.
Accessible controller-decision timeline
Untouched family ledger
No family disappears inside the median
| Family | Learning delta | Completion ratio | WAN ratio | Work replayed | Energy | Ranking / abstention |
|---|
Exact optimizer-commit ledger
| Wall / logical tick | Mode / action | Stress / membership | Lineage | Update age / disagreement | Useful / attempted / replayed | WAN payload / time | Held-out NLL | Evidence / abstention |
|---|
Assumptions, uncertainty, and missing evidence
Assumptions
Uncertainty
Missing evidence
Hashes and complete compact provenance
not loaded
Load the exact epoch trace on demand
The raw optimizer-commit trace is not loaded yet.
How the simulator worksFor experts
Loading…
Three-site scenario
Assumed screening inputsCausal field
Values come from the artifact. Color alone never tells you what class of evidence a number is.Accessible causal graph list
Aligned experiment time
Policy timeline
- Compute
- Collective
- State transfer
- Checkpoint
- Failure
- Recovery
- Facility / grid
Accessible timeline event list
Simulator result
Three methods side by side
| Policy | Local steps / decision | Progress per FLOP | Inter-site bytes | Time to target | Base + compute energy | Falsifier status |
|---|
Waiting for a generated artifact. Result cells remain not run.
Where the learning guess came from
What the first learning guess did and did not know
Source observationsUNMEASURED · UNAVAILABLE FROM ARTIFACT
Unfitted sensitivity priorPRIOR · NOT FITTED
Progress per FLOPUNMEASURED
Held-out time to targetUNMEASURED
Seed observations
Published source measurements are read from the observatory artifact. Uncertainty that comes from rounding in the paper is tracked separately from real run-to-run variation.
| Observation | Value | Evidence | Action |
|---|
Transfer boundary
The attached literature covers one narrow delay setting. It says nothing about progress per FLOP, longer local-update intervals, frontier-scale transfer, how a multi-site run behaves when it is interrupted, or a controller that acts during an outage.
Policy decision ledger
The controller reads one finished communication cycle, then queues the next epoch.| After | Observed state | Decision | Applies to | Evidence |
|---|
Full trace Generated event JSON is not loaded.
not run