Search · four archives
Search · four archives
28 papers · ranked by Valyu relevance
Chengwei Wei, Yun-Cheng Wang, Bin Wang, C.‐C. Jay Kuo
Language modeling studies the probability distributions over strings of texts. It is one of the most fundamental tasks in natural language processing (NLP). It has been widely used in text generation, speech recognition, machine translation, etc. Conventional language models (CLMs) aim to predict the probability of…
Qi Liu, Matt J. Kusner, Phil Blunsom
Contextual embeddings, such as ELMo and BERT, move beyond global word representations like Word2Vec and achieve groundbreaking performance on a wide range of natural language processing tasks. Contextual embeddings assign each word a representation based on its context, thereby capturing uses of words across varied…
Zak Hussain, Marcel Binz, Rui Mata, Dirk U. Wulff
Large language models (LLMs) have the potential to revolutionize behavioral science by accelerating and improving the research cycle, from conceptualization to data analysis. Unlike closed-source solutions, open-source frameworks for LLMs can enable transparency, reproducibility, and adherence to data protection…
Christian Blüthgen
Hintergrund Große Sprachmodelle (Large Language Models, LLMs) wie ChatGPT haben die Art und Weise, wie Computer menschliche Sprache analysieren können und wie wir mit Computern interagieren können, in nur kurzer Zeit revolutioniert. Fragestellung Überblick über die Entstehung und die Grundprinzipien von…
Mohamad Ballout, Ulf Krumnack, Gunther Heidemann, Kai‐Uwe Kühnberger
Pre-trained language models have recently emerged as a powerful tool for fine-tuning a variety of language tasks. Ideally, when models are pre-trained on large amount of data, they are expected to gain implicit knowledge. In this paper, we investigate the ability of pre-trained language models to generalize to…
Micha Heilbron, Kristijan Armeni, Jan-Mathijs Schoffelen, Peter Hagoort + 1 more
Understanding spoken language requires transforming ambiguous acoustic streams into a hierarchy of representations, from phonemes to meaning. It has been suggested that the brain uses prediction to guide the interpretation of incoming input. However, the role of prediction in language processing remains disputed, with…
Tyler A. Chang, Benjamin Bergen
Transformer language models have received widespread public attention, yet their generated text is often surprising even to NLP researchers. In this survey, we discuss over 250 recent studies of English language model behavior before task-specific fine-tuning. Language models possess basic capabilities in syntax…
Dongqiu Zhang, Wenkui Li
Natural Language Understanding (NLU) and Natural Language Generation (NLG) are the general methods that support machine understanding of text content. They play a very important role in the text information processing system including recommendation and question and answer systems. There are many researches in the…
Adrielli Lopes Rego, Joshua Snell, Martijn Meeter
Although word predictability is commonly considered an important factor in reading, sophisticated accounts of predictability in theories of reading are yet lacking. Computational models of reading traditionally use cloze norming as a proxy of word predictability, but what cloze norms precisely capture remains unclear.…
Arkaitz Zubiaga
Since their inception in the 1980s, language models (LMs) have been around for more than four decades as a means for statistically modeling the properties observed from natural language (Rosenfeld, ). Given a collection of texts as input, a language model computes statistical properties of language from those texts…
Zhuoqiao Hong, Haocheng Wang, Zaid Zada, Harshvardhan Gazula + 12 more
Recent research has used large language models (LLMs) to study the neural basis of naturalistic language processing in the human brain. LLMs have rapidly grown in complexity, leading to improved language processing capabilities. However, neuroscience researchers haven’t kept up with the quick progress in LLM…
Raphaël Millière
This chapter critically examines the potential contributions of modern language models to theoretical linguistics. Despite their focus on engineering goals, these models' ability to acquire sophisticated linguistic knowledge from mere exposure to data warrants a careful reassessment of their relevance to linguistic…
Hlib Babii, Andrea Janes, Romain Robbes
When building machine learning models that operate on source code, several decisions have to be made to model source-code vocabulary. These decisions can have a large impact: some can lead to not being able to train models at all, others significantly affect performance, particularly for Neural Language Models. Yet…
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain + 28 more
Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models are trained to generate appropriate linguistic responses in a given context. We provide…
Richard Antonello, Alexander Huth
Many recent studies have shown that representations drawn from neural network language models are extremely effective at predicting brain responses to natural language. But why do these models work so well? One proposed explanation is that language models and brains are similar because they have the same objective: to…
José Alberto Benítez-Andrades, Álvaro González-Jiménez, Álvaro López-Brea, Jose Aveleira-Mata + 3 more
'Álvaro López-Brea' 'Jose Aveleira-Mata' 'José-Manuel Alija-Pérez' 'María Teresa García-Ordás' 'Carlos Fernandez-Lozano'] With the growth that social networks have experienced in recent years, it is entirely impossible to moderate content manually. Thanks to the different existing techniques in natural language…
Jack Grieve, Sara Bartl, Matteo Fuoli, Jason Grafmiller + 6 more
In this article, we introduce a sociolinguistic perspective on language modeling. We claim that language models in general are inherently modeling varieties of language, and we consider how this insight can inform the development and deployment of language models. We begin by presenting a technical definition of the…
Andrew Blanchard, Pei Zhang, Debsindhu Bhowmik, Kshitij Mehta + 4 more
Efficient methods for searching the chemical space of molecular compounds are needed to automate and accelerate the design of new functional molecules such as pharmaceuticals. Given the high cost in both resources and time for experimental efforts, computational approaches play a key role in guiding the selection of…
Sylwia Nowakowska
Two versions of Large Language ChemBERTa-2 models, pre-trained with two different methods, were fine-tuned in this work for HIV replication inhibition prediction. The best model achieved AUROC of 0.793. The changes in distributions of molecular embeddings prior to and following fine-tuning reveal models’ enhanced…
Aisha Khatun, Anisur Rahman, Hemayet Ahmed Chowdhury, Md. Saiful Islam + 1 more
'Md. Saiful Islam' 'Ayesha Tasnim'] Abstract. Language models are at the core of natural language processing. The ability to represent natural language gives rise to its applications in numerous NLP tasks including text classification, summarization, and translation. Research in this area is very limited in Bangla due…
Authors not listed
Drug Discovery is a very lengthy and resource-consuming process. However, a variety of advanced Artificial Intelligence (AI) and Deep Learning (DL) techniques are being utilized to accelerate and advance DD, such as Large Language Models (LLMs). This survey is in aim of discovering and comparing the currently available…
Julius Gelšvartas, Rimvydas Simutis, Rytis Maskeliūnas
This paper describes in detail the design of the specialized text predictor for patients with Huntington's disease. The main aim of the specialized text predictor is to improve the text input rate by limiting the phrases that the user can type in. We show that such specialized predictor can significantly improve text…
Nima Hadidi, Ebrahim Feghhi, Bryan H. Song, Idan A. Blank + 1 more
Emerging research seeks to draw neuroscientific insights from the neural predictivity of large language models (LLMs). However, as results continue to be generated at a rapid pace, there is a growing need for large-scale assessments of their robustness. Here, we analyze a wide range of models, methodological…
Nikola Kölbl, Stefan Rampp, Martin Kaltenhäuser, Konstantin Tziridis + 5 more
Language comprehension involves continuous prediction of upcoming words, with syntactic structure and semantic meaning intertwined in the human brain. To date, few studies have used combined magnetoencephalography (MEG) and electroen-cephalography (EEG) measurements to investigate how syntactic processing, predictive…
Sanjar Adilov
Generative neural networks have shown promising results in de novo drug design. Recent studies suggest that one of the efficient ways to produce novel molecules matching target properties is to model SMILES sequences using deep learning in a way similar to language modeling in natural language processing. In this…
Authors not listed
Accelerating computational materials science relies not only on hardware advances but also on software that increases the ease of working with the relevant abstractions. Creation and manipulation of crystal structures is a part of many routine materials science workflows. In this work, we demonstrate how fine tuning…
Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi + 3 more
Massive scale, both in terms of data availability and computation, enables significant breakthroughs in key application areas of deep learning such as natural language processing (NLP) and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of…
Dimitris Gkoumas, Maria Liakata
The intersection of chemistry and Artificial Intelligence (AI) is an active area of research focused on accelerating scientific discovery. While using large language models (LLMs) with scientific modalities has shown potential, there are significant challenges to address, such as improving training efficiency and…