fusilli.fusionmodels.tabularimagefusion.concat_img_maps_tabular_maps

Concatenating the feature maps of the first tabular modality and the feature maps of the image modality.

Classes

ConcatImageMapsTabularMaps(prediction_task, ...)

Concatenating the feature maps of the first tabular modalities and the feature maps of the image modality.

class ConcatImageMapsTabularMaps(prediction_task, data_dims, multiclass_dimensions)

Bases: ParentFusionModel, Module

Concatenating the feature maps of the first tabular modalities and the feature maps of the image modality.

prediction_task

Type of prediction to be performed.

Type:

str

mod1_layers

Dictionary containing the layers of the first modality. Calculated in the set_mod1_layers() method.

Type:

nn.ModuleDict

img_layers

Dictionary containing the layers of the image data. Calculated in the set_img_layers() method.

Type:

nn.ModuleDict

fused_dim

Number of features of the fused layers. In this method, it’s the size of the tabular layers output plus the size of the (flattened) image layers output.

Type:

int

fused_layers

Sequential layer containing the fused layers. Calculated in the calc_fused_layers() method.

Type:

nn.Sequential

final_prediction

Sequential layer containing the final prediction layers. The final prediction layers take in the number of features of the fused layers as input. Calculated in the calc_fused_layers() method.

Type:

nn.Sequential

__init__(prediction_task, data_dims, multiclass_dimensions)

Parameters:

• prediction_task (str) – Type of prediction to be performed.

• data_dims (list) – List containing the dimensions of the data.

• multiclass_dimensions (int) – Number of classes in the multiclass classification task.

calc_fused_layers()

Calculate the fused layers.

Return type:

None

forward(x1, x2)

Forward pass of the model.

Parameters:

• x1 (torch.Tensor) – Input tensor for the first tabular modality.

• x2 (torch.Tensor) – Input tensor for the image modality.

Returns:

Fused prediction.

Return type:

torch.Tensor

fusion_type = 'operation'

Type of fusion.

Type:

str

get_fused_dim()

Get the number of features of the fused layers.

Return type:

None

method_name = 'Concatenating tabular and image feature maps'

Name of the method.

Type:

str

modality_type = 'tabular_image'

Type of modality.

Type:

str