28 papers · ranked by Valyu relevance
Haonan Sun, Haoxiang Tian, Yihao Hu, Yi Cui + 4 more
'Xianfu Wang' 'Tao Zhou'] Title: Abstract Multimodal machine learning, as a prospective advancement in artificial intelligence, endeavors to emulate the brain's multimodal learning abilities with the objective to enhance interactions with humans. However, this approach requires simultaneous processing of diverse types…
Heng Luo
The rapid development of digital technologies and data-driven techniques has led to advances in multimodal learning featured by multimodality in instructional stimuli, learning spaces, behavioral pattern, and data sources (Blikstein and Worsley, [2]; Di Mitri et al., [5]; Chango et al., [3]). The combination of…
Xiuhan Li, Xiaoman Zhang, Yongle Zhao, Lu Zhang + 1 more
Play is an effective approach to engaging children in learning as an alternative to traditional lecturing. The Learning through Play (LtP) approach involves various modes of learning participation, including multi-sensory participation, interpersonal interaction, and hands-on operation, which can effectively motivate…
Lu Zhou
Human perception of the empirical world involves recognizing the diverse appearances, or 'modalities', of underlying objects. Despite the longstanding consideration of this perspective in philosophy and cognitive science, the study of multimodality remains relatively under-explored within the field of machine learning.…
Xiang He, Dongcheng Zhao, Yang Li, Qingqun Kong + 2 more
Multimodal learning enhances the perceptual ability of intelligent systems by integrating information across sensory modalities. However, most artificial intelligence approaches still rely on static fusion schemes and do not account for the dynamic nature of multisensory integration observed in the brain. In biological…
Maciej Pawłowski, Anna Wróblewska, Sylwia Sysko-Romańczuk, Ikhlas Abdel-Qader
Data processing in robotics is currently challenged by the effective building of multimodal and common representations. Tremendous volumes of raw data are available and their smart management is the core concept of multimodal learning in a new paradigm for data fusion. Although several techniques for building…
Lu Zhou
Human perception inherently operates in a multimodal manner. Similarly, as machines interpret the empirical world, their learning processes ought to be multimodal. The recent, remarkable successes in empirical multimodal learning underscore the significance of understanding this paradigm. Yet, a solid theoretical…
Shicai Wei, Chunbo Luo, Yang Luo
Multimodal learning often encounters the under-optimized problem and may have worse performance than unimodal learning. Existing methods attribute this problem to the imbalanced learning between modalities and rebalance them through gradient modulation. However, they fail to explain why the dominant modality in…
Vinitra Swamy, Malika Satayeva, Jibril Frej, Thierry Bossy + 4 more
'Thijs Vogels' 'Martin Jaggi' 'Tanja Käser' 'Mary-Anne Hartley'] | Vinitra Swamy | Malika Satayeva | Jibril Frej | Thierry Bossy | | --- | --- | --- | --- | | EPFL | EPFL | EPFL | EPFL | | vinitra.swamy@epfl.ch | malika.satayeva@epfl.ch | jibril.frej@epfl.ch | thierry.bossy@epfl.ch | | Thijs Vogels | Martin Jaggi |…
Yuanlin Huang, Zina Zhang, Jia Yu, Xiaobin Liu + 1 more
Although multimodal input has the potential to lead to more sound learning outcomes, it carries the risk of causing cognitive overload, making it difficult to determine the exact effects of multimodal input on the second language (L2) phrase learning. This study tests the efficacy of multimodal input on L2 phrase…
Qing-Yuan Jiang, Zhouyang Chi, Yang Yang
Due to the notorious modality imbalance problem, multimodal learning (MML) leads to the phenomenon of optimization imbalance, thus struggling to achieve satisfactory performance. Recently, some representative methods have been proposed to boost the performance, mainly focusing on adaptive adjusting the optimization of…
Nan Wu, Stanisław Jastrzȩbski, Kyunghyun Cho, Krzysztof J. Geras
We hypothesize that due to the greedy nature of learning in multi-modal deep neural networks, these models tend to rely on just one modality while under-fitting the other modalities. Such behavior is counter-intuitive and hurts the models' generalization, as we observe empirically. To estimate the model's dependence on…
Aedan Y. Li, Natalia Ladyka-Wojcik, Heba Qazilbash, Ali Golestani + 3 more
Combining information from multiple senses is essential to object recognition. Yet how the mind combines sensory input into coherent multimodal representations – the multimodal binding problem – remains poorly understood. Here, we applied multi-echo fMRI across a four-day paradigm, in which participants learned…
Chuang Chen, Nurullizam Jamiat
Studying China's Tang poetry is a crucially integrated part of the language curriculum in primary schools because it is an important part of its cultural heritage and classical literature. However, due to the fact that Tang poetry is written in classical Chinese, which is quite different from modern Chinese Mandarin…
Tian Xia, Xingzhong Zhao, Saiful Sheikh Muhammad Islam, Kamil Khan Mohammed + 2 more
Magnetic resonance imaging (MRI)-derived phenotypes (IDP) has enabled the discovery of numerous genomic loci associated with brain structure and function. However, most existing IDPs and learned representations are derived from a single imaging modality, missing complementary information across modalities and…
Vijay John, Yasutomo Kawanishi, Stefanos Kollias
In classification tasks, such as face recognition and emotion recognition, multimodal information is used for accurate classification. Once a multimodal classification model is trained with a set of modalities, it estimates the class label by using the entire modality set. A trained classifier is typically not…
William C. Sleeman, Rishabh Kapoor, Preetam Ghosh
Multimodal classification research has been gaining popularity in many domains that collect more data from multiple sources including satellite imagery, biometrics, and medicine. However, the lack of consistent terminology and architectural descriptions makes it difficult to compare different existing solutions. We…
Saurabh Bedi, Ella Casimiro, Gilles de Hollander, Nina Raduner + 4 more
Efficient control of behavior requires multisensory learning from information distributed across senses. However, most neurocomputational studies have focused on unisensory signals. Here, we identify distinct but interacting neurocomputational mechanisms that support learn-ing of multisensory associations. We designed…
Yao Ma, Shilin Zhao, Weixiao Wang, Yaoman Li + 1 more
Meta-learning has gained wide popularity as a training framework that is more data-efficient than traditional machine learning methods. However, its generalization ability in complex task distributions, such as multimodal tasks, has not been thoroughly studied. Recently, some studies on multimodality-based…
Authors not listed
Early prediction of drug-induced organ toxicity remains a major bottleneck in drug discovery and clinical pharmacotherapy. Most data-driven toxicity models behave as endpoint predictors: they output a label but provide limited transparency about why a compound is risky or which evidence channel dominated the decision.…
Da Song, Andrew J. Peters
Sensorimotor learning is accompanied by the emergence and strengthening of sensory responses in the prefrontal cortex (PFC) and striatum. However, it remains unclear whether learning-evoked sensory responses in these structures are modality-specific, or whether they represent a more generalized signal for…
Authors not listed
Machine olfaction—the artificial replication of the sense of smell—faces significant challenges due to the absence of large, standardized training datasets. Unlike vision, language, and audio models, which benefit from extensive corpora such as ImageNet, GLUE, and AudioSet, olfaction lacks scaled equivalents and…
Authors not listed
Artificial intelligence (AI) is poised to transform heterogeneous catalysis, ushering in a new paradigm for catalytic materials discovery. By uncovering intricate patterns in high-dimensional data, AI has been reshaping our pursuit of sustainable catalytic processes across the energy, environmental, and chemical…
Lijia Yu, Chunlei Liu, Jean Yee Hwa Yang, Pengyi Yang
Recent advances in multimodal single-cell omics technologies enable multiple modalities of molecular attributes, such as gene expression, chromatin accessibility, and protein abundance, to be profiled simultaneously at a global level in individual cells. While the increasing availability of multiple data modalities is…
Authors not listed
In recent years, the development of large language models (LLMs) has revolutionized various fields of natural science, yet their application in molecular data processing remains constrained due to the reliance on single-modality inputs and outputs. To bridge the gap between experimenters and computational tools, we…
Derek van Tilborg, Helena Brinkmann, Emanuele Criscuolo, Luke Rossen + 2 more
Deep learning is becoming increasingly relevant in drug discovery, from de novo design to protein structure prediction and synthesis planning. However, it is often challenged by the small data regimes typical of certain drug discovery tasks. In such scenarios, deep learning approaches – which are notoriously…
Thilo Womelsdorf, Marcus R. Watson, Paul Tiesinga
Flexible learning of changing reward contingencies can be realized with different strategies. A fast learning strategy involves using working memory of recently rewarded objects to guide choices. A slower learning strategy uses prediction errors to gradually update value expectations to improve choices. How the fast…
James L. McClelland, Bruce L. McNaughton, Andrew K. Lampinen
According to complementary learning systems theory, integrating new memories into the neocortex of the brain without interfering with what is already known depends on a gradual learning process, interleaving new items with previously learned items. However, empirical studies show that information consistent with prior…