AarXiv2023Cited 2×

MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks

Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy, Thijs Vogels, Martin Jaggi, Tanja Käser, Mary-Anne Hartley

Abstract

'Thijs Vogels' 'Martin Jaggi' 'Tanja Käser' 'Mary-Anne Hartley'] | Vinitra Swamy | Malika Satayeva | Jibril Frej | Thierry Bossy | | --- | --- | --- | --- | | EPFL | EPFL | EPFL | EPFL | | vinitra.swamy@epfl.ch | malika.satayeva@epfl.ch | jibril.frej@epfl.ch | thierry.bossy@epfl.ch | | Thijs Vogels | Martin Jaggi | Tanja Käser | Mary-Anne Hartley | | EPFL | EPFL | EPFL | Yale, EPFL | | thijs.vogels@epfl.ch | martin.jaggi@epfl.ch | tanja.kaser@epfl.ch | mary-anne.hartley@yale.edu |

A figure from MultiModN- Multimodal, Multi-Task, Interpretable Modular Networks — fig. from the paper

§ The Valyu brief

The paper, in five minutes

Read from the full text via Valyu Contents

MultiModN sequentially fuses multimodal data, offering interpretability and robustness to missing‑not‑at‑random bias without sacrificing performance.

What’s new

i.Introduces MultiModN, a modular network that fuses any number and type of modalities sequentially.
ii.Provides interpretable‑by‑design, real‑time predictive feedback for arbitrary task combinations.
iii.Demonstrates inherent resistance to missing‑not‑at‑random (MNAR) bias, the first such approach in multimodal modeling.
iv.Shows performance comparable to parallel‑fusion baselines across four benchmark experiments.

How they did it

We construct a composable pipeline where latent representations from each modality are fused in sequence, allowing granular real‑time feedback for each task. The network is trained jointly on multiple tasks, and robustness to MNAR bias is evaluated by simulating biased missingness during inference.

What they found

Four experiments on benchmark multimodal datasets covering 10 real‑world tasks were conducted.
MultiModN's performance matches that of parallel‑fusion baselines.
Parallel‑fusion baselines suffer catastrophic failure under different MNAR patterns, while MultiModN remains robust.

§ The Valyu brief

Reading the full paper and taking notes. This takes a few seconds…

§ Ask this paper

Ask a question about this paper

Valyu reads the full text and answers from what the paper actually says.

§ Related work in other archives

Similar papers in all four archives

The same idea searched in arXiv, PubMed, bioRxiv and chemRxiv at once, so a method can lead you to where it is being applied.

AarXiv3 found

PPubMed4 found

BbioRxiv4 found

CchemRxiv2 found

Searching the other archives…