fusilli.fusionmodels.tabularfusion.channelwise_att

Channel-wise multiplication fusion model for tabular data.

Classes

TabularChannelWiseMultiAttention(...)

Channel-wise multiplication fusion model for tabular data.

class TabularChannelWiseMultiAttention(prediction_task, data_dims, multiclass_dimensions)

Bases: ParentFusionModel, Module

Channel-wise multiplication fusion model for tabular data.

This class implements a model that fuses the two types of tabular data using a channel-wise multiplication approach.

If the two types of tabular data have different feature dimensions at each layer, the model will use a linear layer to make the dimensions the same. This is done to ensure that the channel-wise multiplication can be performed.

Inspired by the work of Duanmu et al. (2020) [1]: here we use two types of tabular data as the multi-modal data instead of image and non-image like in the paper.

References

Duanmu, H., Huang, P. B., Brahmavar, S., Lin, S., Ren, T., Kong, J., Wang, F., & Duong, T. Q. (2020). Prediction of Pathological Complete Response to Neoadjuvant Chemotherapy in Breast Cancer Using Deep Learning with Integrative Imaging, Molecular and Demographic Data. In A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, & L. Joskowicz (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 (pp. 242–252). Springer International Publishing. https://doi.org/10.1007/978-3-030-59713-9_24

Accompanying code: (our model is inspired by the work of Duanmu et al. (2020) [1]) https://github.com/HongyiDuanmu26/Prediction-of-pCR-with-Integrative-Deep-Learning/blob/main/CustomNet.py

mod1_layers

Dictionary containing the layers of the 1st type of tabular data.

Type:

nn.ModuleDict

mod2_layers

Dictionary containing the layers of the 2nd type of tabular data.

Type:

nn.ModuleDict

match_dim_layers

Module dictionary containing the linear layers to make the dimensions of the two types of tabular data the same. This is done to ensure that the channel-wise multiplication can be performed. This doesn’t change the mod1_layers or mod2_layers, it just makes the outputs multipliable.

Type:

nn.ModuleDict

fused_dim

Number of features of the fused layers. This is the output size of the 2nd type of tabular data’s layers.

Type:

int

fused_layers

Sequential layer containing the fused layers.

Type:

nn.Sequential

final_prediction

Sequential layer containing the final prediction layers.

Type:

nn.Sequential

__init__(prediction_task, data_dims, multiclass_dimensions)

Parameters:

• prediction_task (str) – Type of prediction to be performed.

• data_dims (list) – List containing the dimensions of the data.

• multiclass_dimensions (int) – Number of classes in the dataset.

calc_fused_layers()

Calculates the fusion layers.

Return type:

None

Raises:

• ValueError – If the number of layers in the two modalities is different.

• ValueError – If dtype of the layers is not nn.ModuleDict.

forward(x)

Forward pass of the model.

Parameters:

x (tuple) – Tuple containing the input data.

Returns:

List containing the output of the model.

Return type:

list

fusion_type = 'attention'

Type of fusion.

Type:

str

get_fused_dim()

Returns the number of features of the fused layers.

Return type:

None.

method_name = 'Channel-wise multiplication net (tabular)'

Name of the method.

Type:

str

modality_type = 'tabular_tabular'

Type of modality.

Type:

str