Lost Your Model Weights After Acceptance? The Case For and Against Retraining

An accepted machine learning paper, a camera-ready deadline, and reviewers asking for one more thing: a quantitative evaluation of the model's explainability and localization component. The trouble is that the model no longer exists. Its trained weights sat in a temporary cloud environment, and that environment is gone. A researcher laid out exactly this bind in a question on Academia Stack Exchange, and it turns a routine revision into a research-integrity problem.

Plenty survived. The author still holds the original classification results, the confusion matrix, the training and validation loss curves and the visualization outputs. Retraining with the same architecture, dataset and hyperparameters is possible. What is not guaranteed is that a retrained model will reproduce the numbers that went through peer review; it may land slightly off.

The conference permits revisions that address reviewer comments and prohibits major structural changes. So the choice looks binary: retrain and update the results in the camera-ready version, or keep the documented numbers and push the new evaluation to future work. The top answer, with 11 votes, offers a step-by-step recipe for the first option, then delivers a verdict that argues against publishing at all.

Results are a record, not a model

A confusion matrix tells you what the model did on the test set at one moment. Loss curves show how training behaved. Visualization outputs show what the model highlighted on the inputs it was given at the time. All of it is evidence for the claims in the accepted paper, and none of it can be asked a new question.

That is the gap the reviewers opened. Quantitatively evaluating an explainability or localization component means running the model again, on data and with metrics the original run never computed. Without weights there is nothing to run. The only way to produce the requested numbers is to produce a new model, and the moment you do, every number in the paper acquires a sibling that may not match it.

Two options, neither clean

Retraining does what the reviewers asked, but it swaps the model that peer review actually examined for a close relative. Deferring keeps the record intact, but leaves a reviewer recommendation unmet in a camera-ready version whose job is to address exactly those comments. Each option protects one thing by giving up another.

The structural-change rule adds a second layer. Regenerated numbers in the same tables arguably do not change the paper's structure: the sections, claims and figures keep their shape. But that is the author's reading of the rule, and the reading that counts belongs to the conference.

Why a rerun is not a replay

The answerer knows lost-weights anxiety from an unusual angle. In 1988, for a master's thesis titled An Implementation of Frank Liang's Algorithm, they assembled a relatively large corpus, much of it typed and annotated by hand, and trained by running a slightly tweaked version of Liang's program, written in Knuth's Web language, for many hours. The hardware was VAX VMS machines or 8088-based PCs, and backups went to a pile of 5.25-inch floppy disks.

The crucial difference is that this training was deterministic. Same inputs, same weights, every time. Modern training, as the answer puts it, involves pseudo-randomness, so retraining on identical inputs can produce “phase-of-moon dependent differences” in the weights, differences that may or may not be material. That uncertainty is the heart of the question. A retrain is a fresh sample from the same process, not a replay of the run that went through peer review.

The checkpoint habit, and the recipe

The 1988 project survived because its author checkpointed the inputs and the weights three times: for the initial thesis, for the published paper that followed, and again when they stopped to improve it a few years later. That last set can still be found in some distributions. Keeping versions paid off in another way too: comparing the weights produced by different inputs gave insight into the model itself.

For the author who did not save, the answer lays out this sequence:

1. Fold in the reviewers' suggestions

   Make any improvements to the input data that the referees suggested, since the conference permits revisions addressing reviewer comments.

2. Retrain and save everything

   Keep the new weights and every other phase-of-moon-sensitive intermediate result this time.

3. Regenerate every analysis

   Rebuild all the analysis in the paper consistently from the new weights, then add the evaluation the referees recommended and anything else they asked for.

4. Investigate material changes

   If anything differs materially from the prior version of the paper, find out why before sending it.

5. Explain in the cover letter

   State that the weights were regenerated to address the referees' recommendations and that these are not structural changes.

Here the surviving artefacts earn their keep. The original confusion matrix and loss curves are the yardstick for the investigate step: they are how the author tells a cosmetic wobble from a result that no longer holds.

Where the advice turns on itself

The recipe carries one instruction that is harder to defend: don't mention that you can't exactly reproduce the numbers in the submitted paper unless asked. And the same answer closes on a far sterner position. If you cannot reproduce your own results, it says, that inability invalidates the work, which should not be published; from a research-integrity perspective, the submission should be withdrawn.

Those positions do not sit together comfortably. One says regenerate and move on; the other says the paper in its present state should not appear. Read charitably, the answer separates two outcomes. A retrain that lands close enough is a refresh. One that diverges materially is a failure of reproducible research, and withdrawal is the fix. The investigate step is the hinge where the first case can turn into the second.

What stays unsettled

Eleven votes on one answer is a strong view, not a policy. The threshold for a material difference is never defined, and it plausibly depends on what the paper claims: a slight shift matters more when a headline number is the contribution than when it is incidental. Whether regenerated results count as a structural change is for the conference to decide, so put the question to the program chairs before relying on any reading of the rule.

The lesson the answerer presses hardest is the cheapest one. Save everything related to a paper submission, inputs and weights and intermediate results alike, and do it before any temporary environment expires. The 1988 workflow, floppy disks and all, managed it on hardware a modern undergraduate would find laughable.

This article draws on Is it acceptable to retrain a machine learning model after conference acceptance if the original weights were not saved? and an answer to it, on the Stack Exchange network, licensed under CC BY-SA 4.0.