Folder Configuration

Fusilli requires specific directories in the local experiment space to operate. These directories are essential for storing checkpoints, loss logs, and evaluation figures. They are passed into the Fusilli pipeline as elements of a dictionary (similar to data source files - detailed in Loading your Data). You can refer to examples in the Tutorials for usage of this dictionary.

Directory Structure Dictionary

Here is the dictionary structure for defining necessary directories:

output_paths = {
    "losses": "{path to save loss logs used for creating loss figures}",
    "figures": "{path to save evaluation figures}",
    "checkpoints": "{path to save checkpoints}",
}

Warning

Fusilli utilises predetermined file names to save files in these directories. Overwriting may occur if files with the same names exist. It’s highly recommended to maintain separate directories for each Fusilli experiment to manage files belonging to each run effectively.

Example for Creating Directory Structure

Here’s an example block to set up the necessary directory structure:

import os

# Create a timestamp for the run
run_name = "Run1"

# Create directories for this specific run
os.mkdir(f"local_figures/{run_name}") # local_figures
os.mkdir(f"checkpoints/{run_name}") # checkpoints
os.mkdir(f"loss_logs/{run_name}") # loss_log csvs

# Define paths in the dictionary
output_paths = {
    "losses": f"loss_logs/{run_name}",
    "figures": f"local_figures/{run_name}",
    "checkpoints": f"checkpoints/{run_name}",
}

You could also use timestamps to create a unique directory for each run:

from datetime import datetime

run_name = datetime.now().strftime("%Y%m%d_%H%M%S")

After this and running fusilli, the file structure should look like this:

fusilli_experiment_dir
├── fusilli_experiment.py or fusilli_experiment.ipynb
├── local_figures
│   └── Run1
│       ├── losses
│       │   └── example_loss_figure.png
│       └── example_evaluation_figure.png
├── checkpoints
│   └── Run1
│       └── example_checkpoint.pt
├── loss_logs
│   └── Run1
│       └── example_loss_log.csv
└── data
    ├── tabular1.csv
    ├── tabular2.csv
    └── image.pt