fusilli.fusionmodels.tabularimagefusion.concat_img_latent_tab_doubleloss

Concat image latent space with tabular data, trained altogether with a custom loss function: MSE + BCE.

Classes

ConcatImgLatentTabDoubleLoss(...)

Concatenating image latent space with tabular data, trained altogether with a custom loss function: MSE + BCE.

class ConcatImgLatentTabDoubleLoss(prediction_task, data_dims, multiclass_dimensions)

Bases: ParentFusionModel, Module

Concatenating image latent space with tabular data, trained altogether with a custom loss function: MSE + BCE.

prediction_task

Type of prediction to be performed. Binary, regression or multiclass.

Type:

str

fused_layers

Sequential layer containing the fused layers defined with calc_fused_layers().

Type:

nn.Sequential

final_prediction

Sequential layer containing the final prediction layers. The final prediction layers take in the number of features of the fused layers as input.

Type:

nn.Sequential

custom_loss

Additional loss function to be used for training the model. Default is MSELoss.

Type:

nn.Module

latent_dim

Size of the latent space. Default is 256.

Type:

int

encoder

Sequential layer containing the encoder layers. Default for 2D image is:

nn.Sequential(
    nn.Conv2d(1, 32, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
    nn.Conv2d(32, 64, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
    nn.Conv2d(64, 128, kernel_size=3, padding=1),
    nn.ReLU(),
    nn.MaxPool2d(kernel_size=2, stride=2),
)

Type:

nn.Sequential

decoder

Sequential layer containing the decoder layers. Default for 2D image is:

nn.Sequential(
    nn.ConvTranspose2d(128, 64, kernel_size=2, stride=2),
    nn.ReLU(),
    nn.ConvTranspose2d(64, 32, kernel_size=2, stride=2),
    nn.ReLU(),
    nn.ConvTranspose2d(32, 1, kernel_size=2, stride=2),
)

Type:

nn.Sequential

new_encoder

Sequential layer containing the encoder layers and the additional layers defined with calc_fused_layers().

Type:

nn.Sequential

new_decoder

Sequential layer containing the decoder layers and the additional layers defined with calc_fused_layers().

Type:

nn.Sequential

fused_dim

Size of the fused layers: latent dimension size + tabular data dimension size.

Type:

int

__init__(prediction_task, data_dims, multiclass_dimensions)

Parameters:

• prediction_task (str) – Type of prediction to be performed.

• data_dims (list) – List containing the dimensions of the data.

• multiclass_dimensions (int) – Number of classes in the multiclass classification task.

calc_fused_layers()

Calculate the fused layers. If layer sizes are modified, this function will be called again to adjust the fused layers.

Return type:

None

forward(x1, x2)

Forward pass of the model.

Parameters:

• x1 (torch.Tensor) – Input tensor for the tabular data.

• x2 (torch.Tensor) – Input tensor for the image data.

Returns:

list – List containing the output data: prediction and reconstructed image. [ [prediction], [reconstructed_image] ]

Return type:

list

fusion_type = 'subspace'

Type of fusion.

Type:

str

get_fused_dim()

Get the number of features of the fused layers.

Return type:

None

method_name = 'Trained Together Latent Image + Tabular Data'

Name of the method.

Type:

str

modality_type = 'tabular_image'

Type of modality.

Type:

str