24 papers · ranked by Valyu relevance
Michael R. Douglas
Artificial intelligence is making spectacular progress, and one of the best examples is the development of large language models (LLMs) such as OpenAI's GPT series. In these lectures, written for readers with a background in mathematics or physics, we give a brief history and survey of the state of the art, and…
Fudong Zhang, Bo Chai, Yujie Wu, Wai Ting Siok + 1 more
Elucidating the language-brain relationship requires bridging the methodological gap between linguistics’ abstract theoretical frameworks and neuroscience’s empirical neural data. As an interdisciplinary cornerstone, computational neuroscience formalizes language’s hierarchical and dynamic structures into testable…
Aisha Khatun, Anisur Rahman, Hemayet Ahmed Chowdhury, Md. Saiful Islam + 1 more
'Md. Saiful Islam' 'Ayesha Tasnim'] Abstract. Language models are at the core of natural language processing. The ability to represent natural language gives rise to its applications in numerous NLP tasks including text classification, summarization, and translation. Research in this area is very limited in Bangla due…
Kenji Sagae
Recent work on the application of neural networks to language modeling has shown that models based on certain neural architectures can capture syntactic information from utterances and sentences even when not given an explicitly syntactic objective. We examine whether a fully data-driven model of language development…
Kristijan Armeni, Roel M. Willems, Stefan Frank
Cognitive neuroscientists of language comprehension study how neural computations relate to cognitive computations during comprehension. On the cognitive part of the equation, it is important that the computations and processing complexity are explicitly defined. Probabilistic language models can be used to give a…
Alessandro Lopopolo, Evelina Fedorenko, Roger Levy, Milena Rabovsky
In this section, we provide a concise, accessible overview of key computational concepts (language models and word embeddings) and methods which are, with all the due differences and peculiarities, integral to many of the papers featured in this special issue. Word embeddings, also known as word vector representations…
Raphaël Millière
This chapter critically examines the potential contributions of modern language models to theoretical linguistics. Despite their focus on engineering goals, these models' ability to acquire sophisticated linguistic knowledge from mere exposure to data warrants a careful reassessment of their relevance to linguistic…
Wenzhe Yang
In this paper, we discuss how pure mathematics and theoretical physics can be applied to the study of language models. Using set theory and analysis, we formulate mathematically rigorous definitions of language models, and introduce the concept of the moduli space of distributions for a language model. We formulate a…
Yiding Hao, Simon Mendelsohn, Rachel Sterneck, Randi Martinez + 1 more
'Robert Frank'] By positing a relationship between naturalistic reading times and information-theoretic surprisal, surprisal theory (Hale, 2001; Levy, 2008) provides a natural interface between language models and psycholinguistic models. This paper re-evaluates a claim due to Goodkind and Bicknell (2018) that a…
Adrielli Lopes Rego, Joshua Snell, Martijn Meeter
Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are yet lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear.…
Zhuoqiao Hong, Haocheng Wang, Zaid Zada, Harshvardhan Gazula + 12 more
Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven’t kept up with the quick progress in LLM…
Sofia Serrano, Zander Brumbaugh, Noah A. Smith
| 1 | Introduction | | 2 | | --- | --- | --- | --- | | 2 | | Background: Natural language processing concepts and tools | 3 | | | 2.1 | Taskification: Defining what we want a system to do | 4 | | | | 2.1.1 Abstract vs. concrete system capabilities | 4 | | | | 2.1.2 We need data and an evaluation method for research…
Casper S. Shikali, Refuoe Mokhosi
Language modelling using neural networks requires adequate data to guarantee quality word representation which is important for natural language processing (NLP) tasks. However, African languages, Swahili in particular, have been disadvantaged and most of them are classified as low resource languages because of…
Adrielli Tina Lopes Rego, Joshua Snell, Martijn Meeter, Ronald van den Berg
'Ronald van den Berg'] Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely…
Nikola Kölbl, Stefan Rampp, Martin Kaltenhäuser, Konstantin Tziridis + 5 more
Language comprehension involves continuous prediction of upcoming words, with syntactic structure and semantic meaning intertwined in the human brain. To date, few studies have used combined magnetoencephalography (MEG) and electroen-cephalography (EEG) measurements to investigate how syntactic processing, predictive…
Authors not listed
Step-by-step thinking is essential in all domains of chemical sciences and engineering. While machine learning tools are broadly used, algorithms that automate reasoning are far less common. We elaborate on seven categories of human reasoning activities and connect each to applications in chemical science and…
Nicola Angius, Pietro Perconti, Alessio Plebe, Alessandro Acciai
This article provides an epistemological analysis of current attempts of explaining how the relatively simple algorithmic components of neural language models (NLMs) provide them with genuine linguistic competence. After introducing the Transformer architecture, at the basis of most of current NLMs, the paper firstly…
Sanjar Adilov
Generative neural networks have shown promising results in de novo drug design. Recent studies suggest that one of the efficient ways to produce novel molecules matching target properties is to model SMILES sequences using deep learning in a way similar to language modeling in natural language processing. In this…
Pieter Floris Jacobs, Robert Pollice
Scientists across domains are often challenged to master domain-specific languages (DSLs) for their research, which are merely a means to an end but are pervasive in fields like computational chemistry. Automated code generation promises to overcome this barrier, allowing researchers to focus on their core expertise.…
Kevin Maik Jablonka, Philippe Schwaller, Andres Ortega-Guerrero, Berend Smit
Machine learning has revolutionized many fields and has recently found applications in chemistry and materials science. The small datasets commonly found in chemistry sparked the development of sophisticated machine-learning approaches that incorporate chemical knowledge for each application and, therefore, require…
Authors not listed
Accelerating computational materials science relies not only on hardware advances but also on software that increases the ease of working with the relevant abstractions. Creation and manipulation of crystal structures is a part of many routine materials science workflows. In this work, we demonstrate how fine tuning…
Authors not listed
Large language models (LLMs) have garnered increasing attention owing to their potential as collaborative assistants in scientific studies. However, adapting an LLM to specialized domains remains challenging because of the difficulty in incorporating domain-specific knowledge. In the present study, we propose a…
Dimitris Gkoumas, Maria Liakata
The intersection of chemistry and Artificial Intelligence (AI) is an active area of research focused on accelerating scientific discovery. While using large language models (LLMs) with scientific modalities has shown potential, there are significant challenges to address, such as improving training efficiency and…
Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi + 3 more
Massive scale, both in terms of data availability and computation, enables significant breakthroughs in key application areas of deep learning such as natural language processing (NLP) and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of…