21 papers · ranked by Valyu relevance
Wenqi Liu, Kun Zeng
Deep neural networks have made remarkable progresses on various computer vision tasks. Recent works have shown that depth, width and shortcut connections of networks are all vital to their performances. In this paper, we introduce a method to sparsify DenseNet which can reduce connections of a L-layer DenseNet from O(L…
Andrew Ligeralde, Michael R. DeWeese
It is well known that sparse coding models trained on natural images learn basis functions whose shapes resemble the receptive fields (RFs) of simple cells in the primary visual cortex (V1). However, few studies have considered how these basis functions develop during training. In particular, it is unclear whether…
Ligeng Zhu, Ruizhi Deng, Zhiwei Deng, Greg Mori + 1 more
We explore a key architectural aspect of deep convolutional neural networks: the pattern of internal skip connections used to aggregate outputs of earlier layers for consumption by deeper layers. Such aggregation is critical to facilitate training of very deep networks in an end-to-end manner. This is a primary reason…
Weijun Luo
Neural network forms the foundation of deep learning and numerous AI applications. Classical neural networks are fully connected, expensive to train and prone to overfitting. Sparse networks tend to have convoluted structure search, suboptimal performance and limited usage. We proposed the novel uniform sparse network…
Tim Dettmers, Luke Zettlemoyer
We demonstrate the possibility of what we call sparse learning: accelerated training of deep neural networks that maintain sparse weights throughout training while achieving dense performance levels. We accomplish this by developing sparse momentum, an algorithm which uses exponentially smoothed gradients (momentum) to…
Joel Zylberberg, Jason Timothy Murphy, Michael Robert DeWeese, Olaf Sporns
'Olaf Sporns'] Sparse coding algorithms trained on natural images can accurately predict the features that excite visual cortical neurons, but it is not known whether such codes can be learned using biologically realistic plasticity rules. We have developed a biophysically motivated spiking network, relying solely on…
Joel Zylberberg, Michael Robert DeWeese, Olaf Sporns
The sparse coding hypothesis has enjoyed much success in predicting response properties of simple cells in primary visual cortex (V1) based solely on the statistics of natural scenes. In typical sparse coding models, model neuron activities and receptive fields are optimized to accurately represent input stimuli using…
Xingyu Liu, Zonglei Zhen, Jia Liu
Recently, deep convolutional neural networks (DCNNs) have attained human-level performances on challenging object recognition tasks owing to their complex internal representation. However, it remains unclear how objects are represented in DCNNs with an overwhelming number of features and non-linear operations. In…
Adrien Fois, Bernard Girau
Current representation learning methods in Spiking Neural Networks (SNNs) rely on rate-based encoding, resulting in high spike counts, increased energy consumption, and slower information transmission. In contrast, our proposed method, Weight-Temporally Coded Representation Learning (W-TCRL), utilizes temporally coded…
Kevin Lee Hunter, Lawrence Spracklen, Subutai Ahmad
In principle, sparse neural networks should be significantly more efficient than traditional dense networks. Neurons in the brain exhibit two types of sparsity; they are sparsely interconnected and sparsely active. These two types of sparsity, called weight sparsity and activation sparsity, when combined, offer the…
Xingyu Liu, Zonglei Zhen, Jia Liu
Recently, deep convolutional neural networks (DCNNs) have attained human-level performances on challenging object recognition tasks owing to their complex internal representation. However, it remains unclear how objects are represented in DCNNs with an overwhelming number of features and non-linear operations. In…
Yulong Yan, Haoming Chu, Yi Jin, Yuxiang Huan + 2 more
'Lirong Zheng'] The spiking neural network (SNN) is a possible pathway for low-power and energy-efficient processing and computing exploiting spiking-driven and sparsity features of biological systems. This article proposes a sparsity-driven SNN learning algorithm, namely backpropagation with sparsity regularization…
Denis Turcu, L. F. Abbott
Feedforward network models performing classification tasks rely on highly convergent output units that collect the information passed on by preceding layers. Although convergent output-unit like neurons may exist in some biological neural circuits, notably the cerebellar cortex, neocortical circuits do not exhibit any…
Eunwoo Kim, Chanho Ahn, Songhwai Oh
Recently, there have been increasing demands to construct compact deep architectures to remove unnecessary redundancy and to improve the inference speed. While many recent works focus on reducing the redundancy by eliminating unneeded weight parameters, it is not possible to apply a single deep network for multiple…
Sheng Zhou, Chang Gao, Tobi Delbruck, Marian Verhelst + 1 more
Artificial intelligence (AI) has made significant strides towards efficient online processing of sensory signals at the edge through the use of deep neural networks with ever-expanding size. However, this trend has brought with it escalating computational costs and energy consumption, which have become major obstacles…
David Rotermund, Klaus R. Pawelzik
While inspired by the brain, currently successful artificial neural networks lack key features of the biological original. In particular, the deep convolutional networks (DCNs) neither use pulses as signals exchanged among neurons, nor do they include recurrent connections which are both core properties of real…
David Rotermund, Klaus R. Pawelzik
Neural networks are important building blocks in artificial intelligence applications. These technical systems rely on noiseless continuous signals in stark contrast to the discrete action potentials exchanged among the neurons in real brains. A promising approach towards bridging this gap are the Spike-by-Spike (SbS)…
Yuwei Cui, Subutai Ahmad, Jeff Hawkins
Hierarchical temporal memory (HTM) provides a theoretical framework that models several key computational principles of the neocortex. In this paper we analyze an important component of HTM, the HTM spatial pooler (SP). The SP models how neurons learn feedforward connections and form efficient representations of the…
Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi + 3 more
Massive scale, both in terms of data availability and computation, enables significant breakthroughs in key application areas of deep learning such as natural language processing (NLP) and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of…
Authors not listed
Accurate prediction of redox potentials of iron (Fe) complexes, in tandem with uncertainty quantification, is essential to advance technologies related to electro-deposition and energy storage by enabling reliable modeling, guiding experimental design, and improving the efficiency of material discovery. Since…
Hongyu Yu, Yang Zhong, Junyi Ji, Xingao Gong + 1 more
This work presents Time-reversal Equivariant Neural Network (TENN) framework. With TENN, the time-reversal symmetry is considered in the equivariant neural network (ENN), which generalizes the ENN to consider physical quantities related to time-reversal symmetry such as spin and velocity of atoms. TENN-e3, as the…