fusilli.fusionmodels.tabularimagefusion.channelwise_att

Image-channel-wise attention fusion model.

Classes

ImageChannelWiseMultiAttention(...)

Channel-wise multiplication net with image and tabular

class ImageChannelWiseMultiAttention(prediction_task, data_dims, multiclass_dimensions)

Bases: ParentFusionModel, Module

Channel-wise multiplication net with image and tabular

If the tabular layers and the image layers have different feature maps dimensions at each layer, the model will use a linear layer to make the tabular dimensions equal to the image layer dimensions. This is done to ensure that the channel-wise multiplication can be performed.

Inspired by the work of Duanmu et al. (2020) [1]., we use channel-wise multiplication to combine tabular data and image data.

References

Duanmu, H., Huang, P. B., Brahmavar, S., Lin, S., Ren, T., Kong, J., Wang, F., & Duong, T. Q. (2020). Prediction of Pathological Complete Response to Neoadjuvant Chemotherapy in Breast Cancer Using Deep Learning with Integrative Imaging, Molecular and Demographic Data. In A. L. Martel, P. Abolmaesumi, D. Stoyanov, D. Mateus, M. A. Zuluaga, S. K. Zhou, D. Racoceanu, & L. Joskowicz (Eds.), Medical Image Computing and Computer Assisted Intervention – MICCAI 2020 (pp. 242–252). Springer International Publishing. https://doi.org/10.1007/978-3-030-59713-9_24

Accompanying code: (our model is inspired by the work of Duanmu et al. (2020) [1]) https://github.com/HongyiDuanmu26/Prediction-of-pCR-with-Integrative-Deep-Learning/blob/main/CustomNet.py

mod1_layers

Dictionary containing the layers of the 1st type of tabular data.

Type:

nn.ModuleDict

img_layers

Dictionary containing the layers of the image data.

Type:

nn.ModuleDict

match_dim_layers

Module dictionary containing the linear layers to make the dimensions of the two types of data the same. This is done to ensure that the channel-wise multiplication can be performed. This doesn’t change the mod1_layers or img_layers, it just makes the outputs multipliable.

Type:

nn.ModuleDict

fused_dim

Number of features of the fused layers. This is the flattened output size of the image layers.

Type:

int

fused_layers

Sequential layer containing the fused layers.

Type:

nn.Sequential

final_prediction

Sequential layer containing the final prediction layers.

Type:

nn.Sequential

__init__(prediction_task, data_dims, multiclass_dimensions)

Parameters:

• prediction_task (str) – Type of prediction to be performed.

• data_dims (list) – List containing the dimensions of the data.

• multiclass_dimensions (int) – Number of classes in the multiclass classification task.

calc_fused_layers()

Calculate the fused layers.

Return type:

None

Raises:

• ValueError – If the number of layers in the two modalities is not the same.

• ValueError – If dtype of the layers is not nn.ModuleDict.

• ValueError – If the image dimensions are not valid. (Conv2D used for 3D img and vice versa)

forward(x1, x2)

Forward pass of the model.

Parameters:

• x1 (torch.Tensor) – Input tensor for the tabular data.

• x2 (torch.Tensor) – Input tensor for the image data.

Returns:

Output tensor.

Return type:

torch.Tensor

fusion_type = 'attention'

Type of fusion.

Type:

str

get_fused_dim()

Get the number of features of the fused layers.

Return type:

None

method_name = 'Channel-wise Image attention'

Name of the method.

Type:

str

modality_type = 'tabular_image'

Type of modality.

Type:

str