22 papers · ranked by Valyu relevance
Giorgio Taricco, Miguel Rubi
Entropy regularization is a recurring mechanism in reinforcement learning (RL), but its meaning changes across algorithmic settings. In classical online RL, entropy encourages exploration and smooths policy improvement; in inverse RL and imitation learning, maximum-entropy resolves ambiguity among expert-consistent…
Authors not listed
Electrochemical impedance spectroscopy (EIS) coupled with distribution of relaxation times (DRT) analysis is a robust framework for characterizing electrochemical systems. However, DRT deconvolution is often plagued by spurious peaks, hindering accurate process identification and quantitative parameter estimation. To…
Koffka Khan, Ercan Kuruoglu
Personalized Federated Learning (PFL) seeks to train client-specific models across distributed data silos with heterogeneous distributions. We introduce Sign-Entropy Regularization (SER), a novel entropy-based regularization technique that penalizes excessive directional variability in client-local optimization.…
Authors not listed
Inverse molecular design aims to generate novel chemical structures that satisfy multiple property constraints, yet reinforcement-learning (RL) fine-tuning can be sensitive to how objectives are converted into a scalar reward. Here, we systematically analyze how scalarization choices and stabilization mechanisms shape…
Qijun Tong, Kei Kobayashi, Steeve Zozor
The distance and divergence of the probability measures play a central role in statistics, machine learning, and many other related fields. The Wasserstein distance has received much attention in recent years because of its distinctions from other distances or divergences. Although computing the Wasserstein distance is…
Changxu Cheng, Wuheng Xu, Xiang Bai, Bin Feng + 1 more
Chinese text recognition is more challenging than Latin text due to the large amount of fine-grained Chinese characters and the great imbalance over classes, which causes a serious overfitting problem. We propose to apply Maximum Entropy Regularization to regularize the training process, which is to simply add a…
Daniele Musso
Local entropic loss functions provide a versatile framework to define architecture-aware regularization procedures. Besides the possibility of being anisotropic in the synaptic space, the local entropic smoothening of the loss function can vary during training, thus yielding a tunable model complexity. A scoping…
Abdulrahman Oladipupo Ibraheem
I introduce two novel loss functions for classification in deep learning. The two loss functions extend standard cross entropy loss by regularizing it with minimum entropy and Kullback-Leibler (K-L) divergence terms. The first of the two novel loss functions is termed mixed entropy loss (MIX-ENT for short), while the…
Michaël Blot, Thomas Robert, Nicolas Thome, Matthieu Cord
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant…
Zahra Amini Farsani, Volker J Schmid
This paper aims to solve the arterial input function (AIF) determination in dynamic contrast-enhanced MRI (DCE-MRI), an important linear ill-posed inverse problem, using the maximum entropy technique (MET) and regularization functionals. In addition, estimating the pharmacokinetic parameters from a DCE-MR image…
Hovig Tigran Bayandorian
Lp-norm regularization schemes such as L0, L1, and L2-norm regularization and Lp-norm-based regularization techniques such as weight decay, LASSO, and elastic net compute a quantity which depends on model weights considered in isolation from one another. This paper introduces a regularizer based on minimizing a novel…
Michaël Blot, Thomas Robert, Nicolas Thome, Matthieu Cord
Regularization is a big issue for training deep neural networks. In this paper, we propose a new information-theory-based regularization scheme named SHADE for SHAnnon DEcay. The originality of the approach is to define a prior based on conditional entropy, which explicitly decouples the learning of invariant…
Mackenzie J. Meni, Ryan T. White, Michael Mayo, Kevin R. Pilkiewicz
Neural networks have dramatically increased our capacity to learn from large, high-dimensional datasets across innumerable disciplines. However, their decisions are not easily interpretable, their computational costs are high, and building and training them are not straightforward processes. To add structure to these…
Antony Mizzi, David M. Walker, Michael Small, José F. F. Mendes
We derive a penalty strength criterion for ridge regression using stochastic complexity, which is a refined variant of the minimum description length principle. Since stochastic complexity does not typically account for the effect of regularization on complexity, despite its ability to simplify models, we are required…
Authors not listed
Gibbs’ paradox—the apparent discontinuity in mixing entropy for gases of varying similarity and the seeming reversibility of mixing-separation cycles—has resisted fully satisfactory resolution for 150 years. We present a solution based on categorical state theory, which posits that physical con f igurations are…
Julian Q. Kosciessa, Niels A. Kloosterman, Douglas D. Garrett
Multiscale Entropy (MSE) is used to characterize the temporal irregularity of neural time series patterns. Due to its’ presumed sensitivity to non-linear signal characteristics, MSE is typically considered a complementary measure of brain dynamics to signal variance and spectral power. However, the divergence between…
Mickael Zbili, Sylvain Rama
Calculations of entropy of a signal or mutual information between two variables are valuable analytical tools in the field of neuroscience. They can be applied to all types of data, capture nonlinear interactions and are model independent. Yet the limited size and number of recordings one can collect in a series of…
Stephanie Wankowicz, James Fraser
During protein folding, proteins transition from a disordered polymer into a globular structure, markedly decreasing their conformational degrees of freedom and consequently leading to a substantial reduction in entropy. Nonetheless, folded proteins still retain significant entropy as they fluctuate between the…
Guanghui Zhang, Xinran Wang, Steven J. Luck
Regularization has been extensively used in multivariate pattern classification (MVPA; decoding) of EEG data to mitigate the risk of overfitting. N-fold cross-validation is also used to mitigate this risk, and it is often combined with averaging across trials to improve the signal-to-noise ratio. However, the impact of…
Benjamin Kuznets-Speck, Jaekwon Jung, Pornchanan Pholraksa, Adrianne Zhong + 4 more
Classification and regression are cornerstones of computational biology and science at large, from identifying cell types to stratifying patients by disease state. Current deep learning classifiers provide accurate predictions but offer neither uncertainty estimates nor insight into which features matter most. On the…
Griffin S. Hampton, Ryan Neff, Zezheng Song, Mustapha Bouhrara + 2 more
Myelin water fraction (MWF) mapping in the central nervous system is a topic of intense research activity. One framework for this requires parameter estimation from a decaying biexponential signal. However, this is often an ill-posed nonlinear problem resulting in unreliable parameter estimates. For linear…
Authors not listed
Phase equilibrium calculations are crucial in chemical engineering design and optimization processes. The PC-SAFT equation of state (EoS) can precisely calculate phase equilibrium, but is relatively complex and computationally intensive. Surrogate models are mathematically simple models that map or regress the…