Skip to content

Reproducing Experiments

Experiments run through AIDO.ModelGenerator generate config.yaml and <model>.ckpt artifacts which can be used to reproduce training and evaluation, or extend experiments and improve models.

Many pre-packaged experiments are available in the experiments directory with instructions to run inference, evaluate, and reproduce training runs.

Experiment Configs

When you start a run, AIDO.ModelGenerator saves a full config.yaml file for every experiment run which includes all user-specified and default args. To reproduce a training run 

mgen fit --config config.yaml

The location of the saved config.yaml varies by the logger used, but by default this will be under logs/lightning_logs/version_X/config.yaml.

Checkpoints can be used immediately to reproduce test metrics or run inference on new data.

Experiment Checkpoints

If you use checkpointing callbacks (enabled by default), AIDO.ModelGenerator will save <model>.ckpt files within your logging directory. These can be used with the saved config.yaml to run evaluation or prediction.

# Reproduce test metrics
mgen test --config config.yaml \
  --ckpt_path logs/lightning_logs/version_X/checkpoints/<my_best_val>.ckpt

# Reproduce predictions
mgen predict --config config.yaml \
  --ckpt_path logs/lightning_logs/version_X/checkpoints/<my_best_val>.ckpt \
  --config configs/examples/save_predictions.yaml

Pre-packaged Experiments

Experiment configs and scripts from studies using AIDO.ModelGenerator are available in the experiments directory with instructions to run inference, evaluate, and reproduce training runs.

Below are some general guidelines for reproducing experiments, or adding new experiments to the repository. For more details on each study, see the Studies page.

Finetuning Experiments

To reproduce a training run with a saved config, simply run

mgen fit --config config.yaml

Note: If you are using a full frozen config, some machine-specific parameters like trainer.num_nodes may need to be adjusted.

To evaluate the results, take the best_val checkpoint and run

mgen test --config config.yaml --ckpt_path logs/lightning_logs/version_X/checkpoints/<my_best_val>.ckpt

In some cases, experiments will be more complicated than a single config (e.g. for cross-validation) In these cases, look for a README, a bash script, or contact the study authors.

Zero-shot Experiments

If an experiment is zero-shot and only requires inference, you can test the model without any training

mgen test --config config.yaml

Inference Experiments

If an experiment uses predictions for a downstream task, you can run inference on new data

mgen predict --config config.yaml --config configs/examples/save_predictions.yaml

and use any associated post-processing scripts with the experiment to compile and plot the results.