28 papers · ranked by Valyu relevance
Zachary C. Lipton, John Berkowitz, Charles Elkan
Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that…
Ralf C. Staudemeyer, Eric Rothstein Morris
Long Short-Term Memory Recurrent Neural Networks (LSTM-RNN) are one of the most powerful dynamic classifiers publicly known. The network itself and the related learning algorithms are reasonably well documented to get an idea how it works. This paper will shed more light into understanding how LSTM-RNNs evolved and why…
Frank Emmert-Streib, Zhen Yang, Han Feng, Shailesh Tripathi + 1 more
'Matthias Dehmer'] Deep learning models stand for a new learning paradigm in artificial intelligence (AI) and machine learning. Recent breakthrough results in image analysis and speech recognition have generated a massive interest in this field because also applications in many other domains providing big data seem…
H. Francis Song, Guangyu R. Yang, Xiao-Jing Wang, Olaf Sporns
The ability to simultaneously record from large numbers of neurons in behaving animals has ushered in a new era for the study of the neural circuit mechanisms underlying cognitive functions. One promising approach to uncovering the dynamical and computational principles governing population responses is to analyze…
Jian-wei Liu, Bingrong Xu, Zhiyan Song
Recursive and recurrent neural networks are the main realization forms of sequence models based on neural networks, which have been developed rapidly in recent years. Recurrent neural networks are basically standard processing methods for machine translation, question and answer, and sequence video analysis. It is also…
Benyamin Ghojogh, Ali Ghodsi
This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in longterm dependencies. We explain close-to-identity…
Nicolas Vecoven, Damien Ernst, Guillaume Drion, Maciej Huk
Recurrent neural networks (RNNs) provide state-of-the-art performances in a wide variety of tasks that require memory. These performances can often be achieved thanks to gated recurrent cells such as gated recurrent units (GRU) and long short-term memory (LSTM). Standard gated cells share a layer internal state to…
Luis E. Herrera Rodr ́ıguez, Arif Ullah, Kennet J. Rueda Espinosa, Pavlo O. Dral + 1 more
'Pavlo O. Dral' 'Alexei A. Kananenka'] It has been recently shown that supervised machine learning (ML) algorithms can accurately and efficiently predict the long-time populations dynamics of dissipative quantum systems given only short-time population dynamics. In the present article we benchmaked 22 ML models on…
Sarah E. Marzen, Paul M. Riechers, James P. Crutchfield
Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a…
R. Devon Hjelm, Eswar Damaraju, Kyunghyun Cho, Helmut Laufs + 2 more
'Sergey M. Plis' 'Vince D. Calhoun'] We introduce a novel recurrent neural network (RNN) approach to account for temporal dynamics and dependencies in brain networks observed via functional magnetic resonance imaging (fMRI). Our approach directly parameterizes temporal dynamics through recurrent connections, which can…
Jonathan C Kao
Recurrent neural networks (RNNs) are increasingly being used to model complex cognitive and motor tasks performed by behaving animals. Here, RNNs are trained to reproduce animal behavior while also recapitulating key statistics of empirically recorded neural activity. In this manner, the RNN can be viewed as an in…
Felix Grezes
Introduction: Even before Artificial Intelligence was its own field of computational science, humanity has tried to mimic the activity of the human brain. In the early 1940s the first artificial neuron models were created as purely mathematical concepts. Over the years, ideas from neuroscience and computer science were…
Sanjar Adilov
Generative neural networks have shown promising results in de novo drug design. Recent studies suggest that one of the efficient ways to produce novel molecules matching target properties is to model SMILES sequences using deep learning in a way similar to language modeling in natural language processing. In this…
Waleed Nazih, Yasser Hifny, Wail S. Elkilani, Habib Dhahri + 1 more
'Tamer Abdelkader'] Many companies have transformed their telephone systems into Voice over IP (VoIP) systems. Although implementation is simple, VoIP is vulnerable to different types of attacks. The Session Initiation Protocol (SIP) is a widely used protocol for handling VoIP signaling functions. SIP is unprotected…
Kazunori D Yamada
Recurrent neural networks (RNNs) are among the most promising of the many artificial intelligence techniques now under development, showing great potential for memory, interaction, and linguistic understanding. Among the more sophisticated RNNs are long short-term memory (LSTM) and gated recurrent units (GRUs), which…
Chenxi Sui, Ziyang Jiang, Genesis Higueros, David Carlson + 1 more
High-performance batteries are poised for electrification of vehicles and therefore mitigate greenhouse gas emissions, which, in turn, promote a sustainable future. However, the design of optimized batteries is challenging due to the nonlinear governing physics and electrochemistry. Recent advancements have…
Pushparaja Murugan
Recurrent Networks are one of the most powerful and promising artificial neural network algorithms to processing the sequential data such as natural languages, sound, time series data. Unlike traditional feed-forward network, Recurrent Network has a inherent feed back loop that allows to store the temporal context…
Rohollah Soltani, Hui Jiang
In this paper, we study novel neural network structures to better model long term dependency in sequential data. We propose to use more memory units to keep track of more preceding states in recurrent neural networks (RNNs), which are all recurrently fed to the hidden layers as feedback through different weighted…
Yuanqi Xie, Yichen Henry Liu, Christos Constantinidis, Xin Zhou
Understanding the neural mechanisms of working memory has been a long-standing Neuroscience goal. Bump attractor models have been used to simulate persistent activity generated in the prefrontal cortex during working memory tasks and to study the relationship between activity and behavior. How realistic the assumptions…
Huzi Cheng, Joshua W. Brown
How episodic memories are formed in the brain is an outstanding puzzle for the neuroscience community. The brain areas that are critical for episodic learning (e.g., the hippocampus) are characterized by recurrent connectivity and generate frequent offline replay events. The function of the replay events is a subject…
Alexandros Goulas, Fabrizio Damicelli, Claus C Hilgetag
Biological neuronal networks (BNNs) constitute a niche for inspiration and analogy making for researchers that focus on artificial neuronal networks (ANNs). Moreover, neuroscientists increasingly use ANNs as a model for the brain. However, apart from certain similarities and analogies that can be drawn between ANNs and…
James M Murray, Peter Latham, Michael J Frank
Recurrent neural networks (RNNs) enable the production and processing of time-dependent signals such as those involved in movement or working memory. Classic gradient-based algorithms for training RNNs have been available for decades, but are inconsistent with biological features of the brain, such as causality and…
Sweta Kumari, C Vigneswaran, V. Srinivasa Chakravarthy
Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in…
Authors not listed
This research presents a novel approach to obstacle detection during navigation using a combination of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks. The primary objective is to generate accurate image captions that describe the content of images, which is crucial for applications such…
Morgan Thomas, Noel M. O'Boyle, Andreas Bender, Chris de Graaf
A plethora of AI-based techniques now exists to conduct de novo molecule generation that can devise molecules conditioned towards a particular endpoint in the context of drug design. One popular approach is using reinforcement learning to update a recurrent neural network or language-based de novo molecule generator.…
Pavel Tolmachev, Tatiana A. Engel
Trained recurrent neural networks (RNNs) have become the leading framework for modeling neural dynamics in the brain, owing to their capacity to mimic how population-level computations arise from interactions among many units with heterogeneous responses. RNN units are commonly modeled using various nonlinear…
Josep Arús-Pous, Simon Viet Johansson, Oleksii Prykhodko, Esben Jannik Bjerrum + 4 more
Recurrent Neural Networks (RNNs) trained with a set of molecules represented as unique (canonical) SMILES strings, have shown the capacity to create large chemical spaces of valid and meaningful structures. Herein we perform an extensive benchmark on models trained with subsets of GDB-13 of different sizes (1 million …
Josep Arús-Pous, Thomas Blaschke, Jean-Louis Reymond, Hongming Chen + 1 more
Recent applications of Recurrent Neural Networks enable training models that sample the chemical space. In this study we train RNN with molecular string representations (SMILES) with a subset of the enumerated database GDB-13 (975 million molecules). We show that a model trained with 1 million structures (0.1 % of the…