fusilli.fusionmodels.tabularimagefusion.denoise_tab_img_maps

Denoising autoencoder for tabular data concatenated with image feature maps

Classes

`DAETabImgMaps`(prediction_task, data_dims, ...)	Using a denoising autoencoder to upsample tabular data, then concatenating with feature maps from final 3 conv layers of image data.
`DenoisingAutoencoder`(data_dims)	Denoising autoencoder for tabular data: pytorch lightning module.
`ImgUnimodalDAE`(data_dims, prediction_task, ...)	Image unimodal network to go alongside the tabular denoising autoencoder: pytorch lightning module.
`denoising_autoencoder_subspace_method`(datamodule)	Class containing the method to train the denoising autoencoder and to convert the image data to the latent image space.

class DAETabImgMaps(prediction_task, data_dims, multiclass_dimensions)[source]

Bases: ParentFusionModel, Module

Using a denoising autoencoder to upsample tabular data, then concatenating with feature maps from final 3 conv layers of image data. From Yan et al 2021: Richer fusion network for breast cancer classification on multimodal data.

prediction_task

Type of prediction.

Type:: str

subspace_method

Subspace method: denoising_autoencoder_subspace_method.

Type:: class

fusion_layers

Fusion layers combining the intermediate image maps and the tabular latent subspace.

Type:: nn.Sequential

final_prediction

Final prediction layers.

Type:: nn.Sequential

__init__(prediction_task, data_dims, multiclass_dimensions)[source]

Parameters:

prediction_task (str) – Type of prediction to be performed.
data_dims (list) – List containing the dimensions of the data.
multiclass_dimensions (int) – Number of classes in the multiclass classification task.

calc_fused_layers()[source]: Calculate the fused layers.

forward(x)[source]

Forward pass.

Parameters:: x (torch.Tensor) – Input data.
Returns:: out – Output tensor.
Return type:: torch.Tensor

fusion_type = 'subspace'

Type of fusion.

Type:: str

method_name = 'Denoising tabular autoencoder with image maps'

Name of the method.

Type:: str

modality_type = 'tabular_image'

Type of modality.

Type:: str

subspace_method: alias of denoising_autoencoder_subspace_method

class DenoisingAutoencoder(data_dims)[source]

Bases: LightningModule

Denoising autoencoder for tabular data: pytorch lightning module.

tab_dims

Dimension of the input tabular data.

Type:: int

upsampler

Upsampling layers.

Type:: nn.Sequential

downsampler

Downsampling layers.

Type:: nn.Sequential

loss

Loss function. In this case, it’s the mean squared error.

Type:: nn function

__init__(data_dims)[source]

Initialise the model.

Parameters:: data_dims (list) – List containing the dimensions of the data.

calc_fused_layers()[source]

Calculate the fused layers.

Return type:: None

configure_optimizers()[source]

Configure the optimizers of the model.

Returns:: Adam optimizer.
Return type:: torch.optim.Adam

denoise(x)[source]

Denoise the data to create the latent subspace.

Parameters:: x (torch.Tensor) – Input data.
Returns:: Latent subspace.
Return type:: torch.Tensor

forward(x)[source]

Forward pass.

Parameters:: x (torch.Tensor) – Input data.
Returns:: List containing the output.
Return type:: list

training_step(batch, batch_idx)[source]

Training step.

Parameters:

batch (torch.Tensor) – Input batch.
batch_idx (int) – Batch index.

Returns:

Loss.

Return type:

torch.Tensor

validation_step(batch, batch_idx)[source]

Validation step.

Parameters:

batch (torch.Tensor) – Input batch.
batch_idx (int) – Batch index.

Returns:

Loss.

Return type:

torch.Tensor

class ImgUnimodalDAE(data_dims, prediction_task, multiclass_dimensions)[source]

Bases: LightningModule

Image unimodal network to go alongside the tabular denoising autoencoder: pytorch lightning module.

img_dim

Dimension of the input image data.

Type:: int

multiclass_dimensions

Number of classes for multiclass classification.

Type:: int

img_layers

Image layers.

Type:: nn.ModuleDict

num_layers

Number of image layers.

Type:: int

fused_dim

Dimension of the fused layers. Final dimension of the image data after image layers.

Type:: int

prediction_task

Type of prediction.

Type:: str

loss

Loss function. Depends on the prediction type.

Type:: function

fused_layers

Fused layers.

Type:: nn.Sequential

final_prediction

Final prediction layers.

Type:: nn.Sequential

loss

Loss function. Depends on the prediction type.

Type:: function

activation

Activation function. Depends on the prediction type.

Type:: function

__init__(data_dims, prediction_task, multiclass_dimensions)[source]

Initialise the model.

Parameters:

data_dims (list) – List containing the dimensions of the data.
prediction_task (str) – Type of prediction.
multiclass_dimensions (int) – Number of classes for multiclass classification.

calc_fused_layers()[source]

Calculate the fused layers.

Return type:: None

configure_optimizers()[source]

Configure the optimizers of the model.

Returns:: Adam optimizer.
Return type:: torch.optim.Adam

forward(x)[source]

Forward pass.

Parameters:: x (torch.Tensor) – Input data.
Returns:: List containing the output.
Return type:: list

get_fused_dim()[source]: Get the dimension of the fused layers.

get_intermediate_featuremaps(x)[source]

Get the intermediate feature maps to concatenate with the tabular latent subspace.

Parameters:: x (torch.Tensor) – Input data.
Returns:: Intermediate feature maps.
Return type:: torch.Tensor

training_step(batch, batch_idx)[source]

Training step.

Parameters:

batch (torch.Tensor) – Input batch.
batch_idx (int) – Batch index.

Returns:

Loss.

Return type:

torch.Tensor

validation_step(batch, batch_idx)[source]

Validation step.

Parameters:

batch (torch.Tensor) – Input batch.
batch_idx (int) – Batch index.

Returns:

Loss.

Return type:

torch.Tensor

class denoising_autoencoder_subspace_method(datamodule, k=None, max_epochs=1000, train_subspace=True)[source]

Bases: object

Class containing the method to train the denoising autoencoder and to convert the image data to the latent image space.

datamodule

Data module containing the data.

Type:: pl.LightningDataModule

dae_trainer

Trainer for the denoising autoencoder.

Type:: pl.Trainer

img_unimodal_trainer

Trainer for the image unimodal network.

Type:: pl.Trainer

autoencoder

Tabular denoising autoencoder.

Type:: DenoisingAutoencoder

img_unimodal

Image unimodal network.

Type:: ImgUnimodalDAE

__init__(datamodule, k=None, max_epochs=1000, train_subspace=True)[source]

Parameters:

datamodule (pl.LightningDataModule) – Data module containing the data.
k (int or None) – Number of subspaces. Default is None.
max_epochs (int) – Maximum number of epochs. Default is 1000.
train_subspace (bool) – Whether to train the subspace models. Default is True.

convert_to_latent(test_dataset)[source]

Convert the image data to the latent image space.

Parameters:

test_dataset (Dataset) – Test dataset.

Returns:

list – List containing the raw tabular data and the latent image space.
pd.DataFrame – Dataframe containing the labels.
list – List containing the dimensions of the data.

load_ckpt(checkpoint_path)[source]

Load the checkpoint of the subspace models

Parameters:: checkpoint_path (list) – Paths to the checkpoints. The checkpoint list must be a list of checkpoint elements containing the state dict of the subspace models.

train(train_dataset, val_dataset)[source]

Train the latent image space.

Parameters:

train_dataset (Dataset) – Training dataset.
val_dataset (Dataset) – Validation dataset.

Returns:

list – List containing the raw tabular data and the latent image space.
pd.DataFrame – Dataframe containing the labels.