25 papers · ranked by Valyu relevance
Zachary C. Lipton, John Berkowitz, Charles Elkan
Countless learning tasks require dealing with sequential data. Image captioning, speech synthesis, and music generation all require that a model produce outputs that are sequences. In other domains, such as time series prediction, video analysis, and musical information retrieval, a model must learn from inputs that…
Hojjat Salehinejad, Julianne Baarbé, Sharan Sankar, Joseph Barfett + 2 more
'Errol Colak' 'Shahrokh Valaee'] Abstract—Recurrent neural networks (RNNs) are capable of learning features and long term dependencies from sequential and time-series data. The RNNs have a stack of non-linear units where at least one connection between units forms a directed cycle. A well-trained RNN can model any…
Frank Emmert-Streib, Zhen Yang, Han Feng, Shailesh Tripathi + 1 more
'Matthias Dehmer'] Deep learning models stand for a new learning paradigm in artificial intelligence (AI) and machine learning. Recent breakthrough results in image analysis and speech recognition have generated a massive interest in this field because also applications in many other domains providing big data seem…
H. Francis Song, Guangyu R. Yang, Xiao-Jing Wang, Olaf Sporns
The ability to simultaneously record from large numbers of neurons in behaving animals has ushered in a new era for the study of the neural circuit mechanisms underlying cognitive functions. One promising approach to uncovering the dynamical and computational principles governing population responses is to analyze…
Jian-wei Liu, Bingrong Xu, Zhiyan Song
Recursive and recurrent neural networks are the main realization forms of sequence models based on neural networks, which have been developed rapidly in recent years. Recurrent neural networks are basically standard processing methods for machine translation, question and answer, and sequence video analysis. It is also…
Alon Ziv
A key attribute that drives the unprecedented success of modern Recurrent Neural Networks (RNNs) on learning tasks which involve sequential data, is their ability to model intricate long-term temporal dependencies. However, a well established measure of RNNs long-term memory capacity is lacking, and thus formal…
Courtney J. Spoerer, Tim C. Kietzmann, Johannes Mehrer, Ian Charest + 2 more
'Nikolaus Kriegeskorte' 'Leyla Isik'] Deep feedforward neural network models of vision dominate in both computational neuroscience and engineering. The primate visual system, by contrast, contains abundant recurrent connections. Recurrent signal flow enables recycling of limited computational resources over time, and…
Benyamin Ghojogh, Ali Ghodsi
This is a tutorial paper on Recurrent Neural Network (RNN), Long Short-Term Memory Network (LSTM), and their variants. We start with a dynamical system and backpropagation through time for RNN. Then, we discuss the problems of gradient vanishing and explosion in longterm dependencies. We explain close-to-identity…
Nicolas Vecoven, Damien Ernst, Guillaume Drion, Maciej Huk
Recurrent neural networks (RNNs) provide state-of-the-art performances in a wide variety of tasks that require memory. These performances can often be achieved thanks to gated recurrent cells such as gated recurrent units (GRU) and long short-term memory (LSTM). Standard gated cells share a layer internal state to…
Jonathan C Kao
Recurrent neural networks (RNNs) are increasingly being used to model complex cognitive and motor tasks performed by behaving animals. Here, RNNs are trained to reproduce animal behavior while also recapitulating key statistics of empirically recorded neural activity. In this manner, the RNN can be viewed as an in…
Sarah E. Marzen, Paul M. Riechers, James P. Crutchfield
Recurrent neural networks are used to forecast time series in finance, climate, language, and from many other domains. Reservoir computers are a particularly easily trainable form of recurrent neural network. Recently, a “next-generation” reservoir computer was introduced in which the memory trace involves only a…
Kazunori D Yamada
Recurrent neural networks (RNNs) are among the most promising of the many artificial intelligence techniques now under development, showing great potential for memory, interaction, and linguistic understanding. Among the more sophisticated RNNs are long short-term memory (LSTM) and gated recurrent units (GRUs), which…
Jan Koutník, Klaus Greff, Faustino Gomez, Jürgen Schmidhuber
Sequence prediction and classification are ubiquitous and challenging problems in machine learning that can require identifying complex dependencies between temporally distant inputs. Recurrent Neural Networks (RNNs) have the ability, in theory, to cope with these temporal dependencies by virtue of the short-term…
R. Devon Hjelm, Eswar Damaraju, Kyunghyun Cho, Helmut Laufs + 2 more
'Sergey M. Plis' 'Vince D. Calhoun'] We introduce a novel recurrent neural network (RNN) approach to account for temporal dynamics and dependencies in brain networks observed via functional magnetic resonance imaging (fMRI). Our approach directly parameterizes temporal dynamics through recurrent connections, which can…
Sanjar Adilov
Generative neural networks have shown promising results in de novo drug design. Recent studies suggest that one of the efficient ways to produce novel molecules matching target properties is to model SMILES sequences using deep learning in a way similar to language modeling in natural language processing. In this…
Courtney J Spoerer, Tim C Kietzmann, Johannes Mehrer, Ian Charest + 1 more
Deep feedforward neural network models of vision dominate in both computational neuroscience and engineering. The primate visual system, by contrast, contains abundant recurrent connections. Recurrent signal flow enables recycling of limited computational resources over time, and so might boost the performance of a…
Chenxi Sui, Ziyang Jiang, Genesis Higueros, David Carlson + 1 more
High-performance batteries are poised for electrification of vehicles and therefore mitigate greenhouse gas emissions, which, in turn, promote a sustainable future. However, the design of optimized batteries is challenging due to the nonlinear governing physics and electrochemistry. Recent advancements have…
Brett W. Larsen, Shaul Druckmann
Lateral and recurrent connections are ubiquitous in biological neural circuits. Yet while the strong computational abilities of feedforward networks have been extensively studied, our understanding of the role and advantages of recurrent computations that might explain their prevalence remains an important open…
Sebastian Sanokowski
The brain cortex, which processes visual, auditory and sensory data in the brain, is known to have many recurrent connections within its layers and from higher to lower layers. But, in the case of machine learning with neural networks, it is generally assumed that strict feed-forward architectures are suitable for…
Grace W. Lindsay, Thomas D. Mrsic-Flogel, Maneesh Sahani
Behavioral studies suggest that recurrence in the visual system is important for processing degraded stimuli. There are two broad anatomical forms this recurrence can take, lateral or feedback, each with different assumed functions. Here we add four different kinds of recurrence—two of each anatomical form—to a…
Authors not listed
This research presents a novel approach to obstacle detection during navigation using a combination of Convolutional Neural Networks (CNNs) and Long Short-Term Memory (LSTM) networks. The primary objective is to generate accurate image captions that describe the content of images, which is crucial for applications such…
Qi Zhang, Chang Liu, Stephen Wu, Ryo Yoshida
In the last few years, de novo molecular design using machine learning has made great technical progress but its practical deployment has not been as successful. This is mostly owing to the cost and technical difficulty of synthesizing such computationally designed molecules. To overcome such barriers, various methods…
Sweta Kumari, C Vigneswaran, V. Srinivasa Chakravarthy
Sequential decision making tasks that require information integration over extended durations of time are challenging for several reasons including the problem of vanishing gradients, long training times and significant memory requirements. To this end we propose a neuron model fashioned after the JK flip-flops in…
Morgan Thomas, Noel M. O'Boyle, Andreas Bender, Chris de Graaf
A plethora of AI-based techniques now exists to conduct de novo molecule generation that can devise molecules conditioned towards a particular endpoint in the context of drug design. One popular approach is using reinforcement learning to update a recurrent neural network or language-based de novo molecule generator.…
Fuzhan Rahmanian, Robert M. Lee, Dominik Linzner, Kathrin Michel + 4 more
Predicting and monitoring battery life early and across chemistries is a significant challenge due to the plethora of degradation paths, form factors, and electrochemical testing protocols. Existing models typically translate poorly across different electrode, electrolyte, and additive materials, mostly require a fixed…