fusilli.fusionmodels.tabularimagefusion.decision

Model that fuses the first tabular data and the image data using a decision fusion approach.

Classes

ImageDecision(prediction_task, data_dims, ...)

This class implements a model that fuses the first tabular data and the image data using a decision fusion

class ImageDecision(prediction_task, data_dims, multiclass_dimensions)

Bases: ParentFusionModel, Module

This class implements a model that fuses the first tabular data and the image data using a decision fusion

approach.

mod1_layers

Dictionary containing the layers of the 1st type of tabular data.

Type:

nn.ModuleDict

img_layers

Dictionary containing the layers of the image data.

Type:

nn.ModuleDict

fused_layers

Sequential layer containing the fused layers.

Type:

nn.Sequential

final_prediction_tab1

Sequential layer containing the final prediction layers for the first tabular data.

Type:

nn.Sequential

final_prediction_img

Sequential layer containing the final prediction layers for the image data.

Type:

nn.Sequential

fusion_operation

Function that performs the fusion operation. Default is torch.mean(torch.stack([x, y]), dim=0).

Type:

function

.. warning::

fusion_operation should be done on the first dimension, i.e. the batch dimension. For example, lambda x: torch.mean(x, dim=1). The predictions of the different modalities are stacked on the first dimension before fusion_operation.

__init__(prediction_task, data_dims, multiclass_dimensions)

Parameters:

• prediction_task (str) – Type of prediction to be performed.

• data_dims (list) – List containing the dimensions of the data.

• multiclass_dimensions (int) – Number of classes in the multiclass classification task.

calc_fused_layers()

Calculates the fusion layers.

forward(x1, x2)

Forward pass of the model.

Parameters:

• x1 (torch.Tensor) – First tabular data input.

• x2 (torch.Tensor) – Image data input.

Returns:

Fused prediction.

Return type:

torch.Tensor

fusion_type = 'operation'

Type of fusion.

Type:

str

method_name = 'Image decision fusion'

Name of the method.

Type:

str

modality_type = 'tabular_image'

Type of modality.

Type:

str