19 papers · ranked by Valyu relevance
Sergey Ioffe, Christian Szegedy
Training Deep Neural Networks is complicated by the fact that the distribution of each layer's inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it no toriously hard to train…
Sergey Ioffe
Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the…
Randall Balestriero, Richard G. Baraniuk
A critically important, ubiquitous, and yet poorly understood ingredient in modern deep networks (DNs) is batch normalization (BN), which centers and normalizes the feature maps. To date, only limited progress has been made understanding why BN boosts DN learning and inference performance; work has focused exclusively…
Mark R. Saddler, Ray Gonzalez, Josh H. McDermott
Perception is thought to be shaped by the environments for which organisms are optimized. These influences are difficult to test in biological organisms but may be revealed by machine perceptual systems optimized under different conditions. We investigated environmental and physiological influences on pitch perception…
Benjamin L. Berger
In training neural networks, batch normalization has many benefits, not all of them entirely understood. But it also has some drawbacks. Foremost is arguably memory consumption, as computing the batch statistics requires all instances within the batch to be processed simultaneously, whereas without batch normalization…
Sen Yang, Xiaobao Wang, Qijuan Yang, Enzeng Dong + 2 more
'Cosimo Distante'] The single batch normalization (BN) method is commonly used in the instance segmentation algorithms. The batch size is concerned with some drawbacks. A too small sample batch size leads to a sharp drop in accuracy, but a too large batch may result in the memory overflow of graphic processing units…
Huixia Lai, Lulu Zhang, Shi Zhang, Antonio Lázaro
As a technique for accelerating and stabilizing training, the batch normalization (BN) is widely used in deep learning. However, BN cannot effectively estimate the mean and the variance of samples when training/fine-tuning with small batches of data on resource-constrained devices. It will lead to a decrease in the…
Yang Shen, Julia Wang, Saket Navlakha
A fundamental challenge at the interface of machine learning and neuroscience is to uncover computational principles that are shared between artificial and biological neural networks. In deep learning, normalization methods, such as batch normalization, weight normalization, and their many variants, help to stabilize…
Muneki Yasuda, Seishirou Ueno
—In this study, we consider classification problems based on neural networks in data-imbalanced environment. Learning from an imbalanced data set is one of the most important and practical problems in the field of machine learning. A weighted loss function based on costsensitive approach is a well-known effective…
Afifa Khaled, Chao Li, Ning Jia, Kun He
Normalization techniques have been widely used in the field of deep learning due to their capability of enabling higher learning rates and are less careful in initialization. However, the effectiveness of popular normalization technologies is typically limited to specific areas. Unlike the standard Batch Normalization…
Hyeonseong Choi, Byung Hyun Lee, Se Young Chun, Jaehwan Lee + 1 more
'Elena Loli Piccolomini'] Modern deep neural networks cannot be often trained on a single GPU due to large model size and large data size. Model parallelism splits a model for multiple GPUs, but making it scalable and seamless is challenging due to different information sharing among GPUs with communication overhead.…
Alexander Lin, Alex X. Lu
Data collected by high-throughput microscopy experiments are affected by batch effects, stemming from slight technical differences between experimental batches. Batch effects significantly impede machine learning efforts, as models learn spurious technical variation that do not generalize. We introduce batch effects…
Aaron Defazio, Léon Bottou
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set…
Michiel Bongaerts, Ramon Bonte, Serwet Demirdas, Ed H. Jacobs + 7 more
Untargeted metabolomics is an emerging technology in the laboratory diagnosis of inborn errors of metabolism (IEM). In order to judge if metabolite levels are abnormal, analysis of a large number of reference samples is crucial to correct for variations in metabolite concentrations resulting from factors such as diet…
Jungwoo Shin, HyunJin Kim, Yilun Shang
In this study, we present a novel performance-enhancing binarized neural network model called PresB-Net: Parametric Binarized Neural Network. A binarized neural network (BNN) model can achieve fast output computation with low hardware costs by using binarized weights and features. However, performance degradation is…
Baihan Lin, Shu-Tao Xia, Bin Chen
Inspired by the adaptation phenomenon of neuronal firing, we propose the regularity normalization (RN) as an unsupervised attention mechanism (UAM) which computes the statistical regularity in the implicit space of neural networks under the Minimum Description Length (MDL) principle. Treating the neural network…
Authors not listed
Machine learning models are increasingly applied to heterogeneous materials datasets spanning different synthesis routes, measurement protocols, and structural classes. Although multi-task and representation-learning approaches are commonly used to improve predictive performance, the latent representations learned by…
Authors not listed
Meteorological normalization is a key concept in studying anthropogenic effects on air pollutant concentrations and its temporal trends. While apparently successful in revealing anthropogenic effects and often used, there are downsides to the methods and limitations which should be taken into account when using it.…
Nathan Frey, Ryan Soklaski, Simon Axelrod, Siddharth Samsi + 3 more
Massive scale, both in terms of data availability and computation, enables significant breakthroughs in key application areas of deep learning such as natural language processing (NLP) and computer vision. There is emerging evidence that scale may be a key ingredient in scientific deep learning, but the importance of…