Exports that fail after the run succeeded

The study completes, then the save dies. Both causes are visible in a config file before you start.

The failure that looks like success

The worst class of bug in this pipeline: everything runs, every trial completes, the metrics are good — and then the export fails. You have spent the entire budget and have nothing to show.

It happens because the study holds the model in memory while the export re-reads the checkpoint from disk. Anything wrong with the files on disk stays invisible until that moment.

Cause one: bundled modeling code

Some checkpoints ship their own Python and point at it with auto_map. When the library moves on, that code stops importing — but only on the path that uses it.

Saving merged model...
* Loading base model on CPU (this may take a while)...
Error: cannot import name 'LossKwargs' from 'transformers.utils'

Fifty trials, then that. The fix is to drop the entries that route to the stale file so the library’s native implementation is used instead. Keep any tokenizer entry — that usually points at a hub repo, not the broken local file.

import json, shutil
from pathlib import Path

p = Path(model) / "config.json"
cfg = json.loads(p.read_text())
am = cfg.get("auto_map", {})
shutil.copy2(p, p.with_suffix(".json.orig"))
for k in ("AutoConfig", "AutoModelForCausalLM"):
    am.pop(k, None)
cfg["auto_map"] = am or None
p.write_text(json.dumps({k: v for k, v in cfg.items() if v is not None}, indent=2))

Detect it up front by actually importing every auto_map target against your installed library. Reading the config is not enough; the file may exist and still not import.

Cause two: sampling parameters with sampling disabled

A checkpoint that ships temperature or top_p without do_sample: true fails the library’s validation — at save time, not load time. So again: after the run.

$ python -c "import json; print(json.load(open('generation_config.json')))"
{'temperature': 0.6, 'top_p': 0.95, 'max_new_tokens': 32768}   ← no do_sample

Remove the sampling keys, or set do_sample: true. One caution: only touch it when do_sample is absent or false. A model that legitimately samples must be left alone.

This is why the probe exists

We found exactly this on a 32B model already scheduled to run that night. The fault would have surfaced eight hours later, at export, after the study had finished. A config parse caught it in under a second.

Cause three: the storage, not the model

If your checkpoints live on network storage, confirm they are readable immediately before the export, not just at the start. We lost a nine-hour run to a config file that had become unreadable partway through.

And when repairing permissions, set the mode you want rather than adding a bit. chmod a+r on a mode-000 file yields 444, which looks like fresh corruption on the next sweep and sends you chasing a recurrence that is really your own previous fix.