fusilli.fusionmodels.tabularimagefusion.concat_img_latent_tab_doubletrain

Concatenating the latent img space and tabular data. The latent image space is trained separately from both the tabular data and the labels, using the img_latent_subspace_method class.

Classes

`ConcatImgLatentTabDoubleTrain`(...)	Concatenating the latent img space and tabular data.
`ImgLatentSpace`(data_dims)	Pytorch lightning module: autoencoder to train the latent image space.
`concat_img_latent_tab_subspace_method`(datamodule)	Class containing the method to train the latent image space and to convert the image data to the latent image space.

class ConcatImgLatentTabDoubleTrain(prediction_task, data_dims, multiclass_dimensions)[source]

Bases: ParentFusionModel, Module

Concatenating the latent img space and tabular data. The latent image space is trained separately from both the tabular data and the labels, using the img_latent_subspace_method class.

prediction_task

Type of prediction to be performed.

Type:: str

latent_dim

Dimension of the latent image space once we encode it down. Taken from the subspace_method class and inferred from the dimensions of the input data to the model.

Type:: int

enc_img_layer

Linear layer to reduce the dimension of the latent image space. Calculated with calc_fused_layers().

Type:: nn.Linear

fused_dim

Dimension of the fused layers. Calculated with calc_fused_layers().

Type:: int

fused_layers

Sequential layer containing the fused layers. Calculated with calc_fused_layers().

Type:: nn.Sequential

final_prediction

Sequential layer containing the final prediction layers. The final prediction layers take in the number of features of the fused layers as input. Calculated with calc_fused_layers().

Type:: nn.Sequential

__init__(prediction_task, data_dims, multiclass_dimensions)[source]

Parameters:

prediction_task (str) – Type of prediction to be performed.
data_dims (list) – List containing the dimensions of the data.
multiclass_dimensions (int) – Number of classes in the multiclass classification task.

calc_fused_layers()[source]

Calculate the fused layers. If layer sizes are modified, this function will be called again to adjust the fused layers.

Return type:: None

forward(x1, x2)[source]

Forward pass of the model.

Parameters:

x1 (torch.Tensor) – Input tensor containing the tabular data.
x2 (torch.Tensor) – Input tensor containing the latent image space.

Returns:

Output tensor.

Return type:

torch.Tensor

fusion_type = 'subspace'

Type of fusion.

Type:: str

get_fused_dim()[source]

Get the number of features of the fused layers.

Return type:: None

method_name = 'Pretrained Latent Image + Tabular Data'

Name of the method.

Type:: str

modality_type = 'tabular_image'

Type of modality.

Type:: str

subspace_method

class: Class containing the method to train the latent image space.

alias of concat_img_latent_tab_subspace_method

class ImgLatentSpace(data_dims)[source]

Bases: LightningModule

Pytorch lightning module: autoencoder to train the latent image space.

data_dims

List containing the dimensions of the data.

Type:: dict

img_dim

Dimensions of the image data.

Type:: tuple

latent_dim

Dimension of the latent image space once we encode it down. Default is 64.

Type:: int

encoder

Sequential layer containing the encoder layers.

Type:: nn.Sequential

decoder

Sequential layer containing the decoder layers.

Type:: nn.Sequential

new_encoder

Sequential layer containing the encoder layers and the linear layer to reduce the dimension of the latent image space. Calculated with calc_fused_layers().

Type:: nn.Sequential

new_decoder

Sequential layer containing the decoder layers and the linear layer to increase the dimension of the latent image space. Calculated with calc_fused_layers().

Type:: nn.Sequential

__init__(data_dims)[source]

Parameters:: data_dims (list) – List containing the dimensions of the data.

calc_fused_layers()[source]

Calculate the fused layers. If layer sizes are modified, this function will be called again to adjust the fused layers.

Return type:: None

configure_optimizers()[source]

Configure the optimizers of the model.

Returns:: Adam optimizer.
Return type:: torch.optim.Adam

encode_image(x)[source]

Encode the image data. Used when the model is trained to get latent image space.

Parameters:: x (torch.Tensor) – Input data.
Returns:: Encoded image.
Return type:: torch.Tensor

forward(x)[source]

Forward pass of the model.

Parameters:: x (torch.Tensor) – Input data.
Returns:: Output of the model. Reconstruction/decoded image.
Return type:: torch.Tensor

training_step(batch, batch_idx)[source]

Training step of the model.

Parameters:

batch (torch.Tensor) – Batch of data.
batch_idx (int) – Index of the batch.

Returns:

Loss of the model.

Return type:

torch.Tensor

validation_step(batch, batch_idx)[source]

Validation step of the model.

Parameters:

batch (torch.Tensor) – Batch of data.
batch_idx (int) – Index of the batch.

Returns:

Loss of the model.

Return type:

torch.Tensor

class concat_img_latent_tab_subspace_method(datamodule, k=None, max_epochs=1000, train_subspace=True)[source]

Bases: object

Class containing the method to train the latent image space and to convert the image data to the latent image space.

datamodule

Data module containing the data.

Type:: pl.LightningDataModule

trainer

Lightning trainer.

Type:: pytorch_lightning.Trainer

autoencoder

Autoencoder to train the latent image space.

Type:: ImgLatentSpace

__init__(datamodule, k=None, max_epochs=1000, train_subspace=True)[source]

Parameters:

datamodule (class) – Data module containing the data.
k (int or None) – Number of folds for cross validation. Default is None.
max_epochs (int) – Maximum number of epochs to train the latent image space.
train_subspace (bool) – Whether to train the latent image space or not. Default is True. If False, a new trainer will not be created. Then load_ckpt() must be called to load the checkpoint of the latent image space.

convert_to_latent(test_dataset)[source]

Convert the image data to the latent image space.

Parameters:

test_dataset (Dataset) – Test dataset.

Returns:

list – List containing the raw tabular data and the latent image space.
pd.DataFrame – Dataframe containing the labels.
list – List containing the dimensions of the data.

load_ckpt(checkpoint_path)[source]

Load the checkpoint of the latent image space.

Parameters:: checkpoint_path (list) – List containing the path to the checkpoint.

train(train_dataset, val_dataset)[source]

Train the latent image space.

Parameters:

train_dataset (Dataset) – Training dataset.
val_dataset (Dataset) – Validation dataset.

Returns:

list – List containing the raw tabular data and the latent image space.
pd.DataFrame – Dataframe containing the labels.