24 papers · ranked by Valyu relevance
Arkaitz Zubiaga
Since their inception in the 1980s, language models (LMs) have been around for more than four decades as a means for statistically modeling the properties observed from natural language (Rosenfeld, ). Given a collection of texts as input, a language model computes statistical properties of language from those texts…
Fudong Zhang, Bo Chai, Yujie Wu, Wai Ting Siok + 1 more
Elucidating the language-brain relationship requires bridging the methodological gap between linguistics’ abstract theoretical frameworks and neuroscience’s empirical neural data. As an interdisciplinary cornerstone, computational neuroscience formalizes language’s hierarchical and dynamic structures into testable…
Michael R. Douglas
Artificial intelligence is making spectacular progress, and one of the best examples is the development of large language models (LLMs) such as OpenAI's GPT series. In these lectures, written for readers with a background in mathematics or physics, we give a brief history and survey of the state of the art, and…
Hlib Babii, Andrea Janes, Romain Robbes
When building machine learning models that operate on source code, several decisions have to be made to model source-code vocabulary. These decisions can have a large impact: some can lead to not being able to train models at all, others significantly affect performance, particularly for Neural Language Models. Yet…
Martin Schrimpf, Idan Blank, Greta Tuckute, Carina Kauf + 4 more
The neuroscience of perception has recently been revolutionized with an integrative modeling approach in which computation, brain function, and behavior are linked across many datasets and many computational models. By revealing trends across models, this approach yields novel insights into cognitive and neural…
Aisha Khatun, Anisur Rahman, Hemayet Ahmed Chowdhury, Md. Saiful Islam + 1 more
'Md. Saiful Islam' 'Ayesha Tasnim'] Abstract. Language models are at the core of natural language processing. The ability to represent natural language gives rise to its applications in numerous NLP tasks including text classification, summarization, and translation. Research in this area is very limited in Bangla due…
Alessandro Lopopolo, Evelina Fedorenko, Roger Levy, Milena Rabovsky
In this section, we provide a concise, accessible overview of key computational concepts (language models and word embeddings) and methods which are, with all the due differences and peculiarities, integral to many of the papers featured in this special issue. Word embeddings, also known as word vector representations…
Kristijan Armeni, Roel M. Willems, Stefan Frank
Cognitive neuroscientists of language comprehension study how neural computations relate to cognitive computations during comprehension. On the cognitive part of the equation, it is important that the computations and processing complexity are explicitly defined. Probabilistic language models can be used to give a…
Paul Azunre, Salomey Osei, Salomey Afua Addo, Lawrence Adu-Gyamfi + 23 more
'Stephen Moore' 'Bernard Adabankah' 'Bernard Opoku' 'Clara Asare-Nyarko' 'Samuel Nyarko' 'Cynthia Amoaba' 'Esther Dansoa Appiah' 'Felix Akwerh' 'Richard Nii Lante Lawson' 'Joel Budu' 'Emmanuel Debrah' 'Nana Boateng' 'Wisdom Ofori' 'Edwin Buabeng-Munkoh' 'Franklin Adjei' 'Isaac K. E. Ampomah' 'Joseph Otoo' 'Reindorf…
Wenzhe Yang
In this paper, we discuss how pure mathematics and theoretical physics can be applied to the study of language models. Using set theory and analysis, we formulate mathematically rigorous definitions of language models, and introduce the concept of the moduli space of distributions for a language model. We formulate a…
Raphaël Millière
This chapter critically examines the potential contributions of modern language models to theoretical linguistics. Despite their focus on engineering goals, these models' ability to acquire sophisticated linguistic knowledge from mere exposure to data warrants a careful reassessment of their relevance to linguistic…
Kenji Sagae
Recent work on the application of neural networks to language modeling has shown that models based on certain neural architectures can capture syntactic information from utterances and sentences even when not given an explicitly syntactic objective. We examine whether a fully data-driven model of language development…
Richard Futrell, Kyle Mahowald
Language models can produce fluent, grammatical text. Nonetheless, some maintain that language models don't really learn language and also that, even if they did, that would not be informative for the study of human learning and processing. On the other side, there have been claims that the success of LMs obviates the…
Adrielli Lopes Rego, Joshua Snell, Martijn Meeter
Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are yet lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear.…
Nicola Angius, Pietro Perconti, Alessio Plebe, Alessandro Acciai
This article provides an epistemological analysis of current attempts of explaining how the relatively simple algorithmic components of neural language models (NLMs) provide them with genuine linguistic competence. After introducing the Transformer architecture, at the basis of most of current NLMs, the paper firstly…
Authors not listed
Step-by-step thinking is essential in all domains of chemical sciences and engineering. While machine learning tools are broadly used, algorithms that automate reasoning are far less common. We elaborate on seven categories of human reasoning activities and connect each to applications in chemical science and…
Sanjar Adilov
Generative neural networks have shown promising results in de novo drug design. Recent studies suggest that one of the efficient ways to produce novel molecules matching target properties is to model SMILES sequences using deep learning in a way similar to language modeling in natural language processing. In this…
Shuya Nakata, Yoshiharu Mori, Shigenori Tanaka
Ultra-large virtual chemical spaces have emerged as a valuable resource for drug discovery, providing access to billions of make-on-demand compounds with high synthetic success rates. Chemical language models can potentially accelerate the exploration of these vast spaces through direct compound generation. However…
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, Berend Smit
Machine learning has revolutionized many fields and has recently found applications in chemistry and materials science. The small datasets commonly found in chemistry sparked the development of sophisticated machine-learning approaches that incorporate chemical knowledge for each application and, therefore, require…
Daniel Mitropolsky, Christos H. Papadimitriou
Despite tremendous progress in neuroscience, we do not have a compelling narrative for the precise way whereby the spiking of neurons in our brain results in high-level cognitive phenomena such as planning and language. We introduce a simple mathematical formulation of six basic and broadly accepted principles of…
Michael T. Putnam, Matthew Carlson, David Reitter
On the surface, bi- and multilingualism would seem to be an ideal context for exploring questions of typological proximity. The obvious intuition is that the more closely related two languages are, the easier it should be to implement the two languages in one mind. This is the starting point adopted here, but we…
Authors not listed
Accelerating computational materials science relies not only on hardware advances but also on software that increases the ease of working with the relevant abstractions. Creation and manipulation of crystal structures is a part of many routine materials science workflows. In this work, we demonstrate how fine tuning…
Authors not listed
Large language models (LLMs) have garnered increasing attention owing to their potential as collaborative assistants in scientific studies. However, adapting an LLM to specialized domains remains challenging because of the difficulty in incorporating domain-specific knowledge. In the present study, we propose a…
Dimitris Gkoumas, Maria Liakata
The intersection of chemistry and Artificial Intelligence (AI) is an active area of research focused on accelerating scientific discovery. While using large language models (LLMs) with scientific modalities has shown potential, there are significant challenges to address, such as improving training efficiency and…