23 papers · ranked by Valyu relevance
Benyamin Ghojogh, Ali Ghodsi, Fakhri Karray, Mark Crowley
This is a tutorial and survey paper on Karush-Kuhn-Tucker (KKT) conditions, first-order and second-order numerical optimization, and distributed optimization. After a brief review of history of optimization, we start with some preliminaries on properties of sets, norms, functions, and concepts of optimization. Then, we…
Esraa Hassan, Mahmoud Y. Shams, Noha A. Hikal, Samir Elmougy
Optimization algorithms are used to improve model accuracy. The optimization process undergoes multiple cycles until convergence. A variety of optimization strategies have been developed to overcome the obstacles involved in the learning process. Some of these strategies have been considered in this study to learn more…
Esmail Abdul Fattah, Janet van Niekerk, Håvard Rue
Computing the gradient of a function provides fundamental information about its behavior. This information is essential for several applications and algorithms across various fields. One common application that require gradients are optimization techniques such as stochastic gradient descent, Newton's method and trust…
Shiliang Sun, Zehui Cao, Zhu Han, Jing Zhao
—Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization…
Pushparaja Murugan, Shanmugasundaram Durairaj
Convolution Neural Networks, known as ConvNets exceptionally perform well in many complex machine learning tasks. The architecture of ConvNets demands the huge and rich amount of data and involves with a vast number of parameters that leads the learning takes to be computationally expensive, slow convergence towards…
Mohamed A. Mokhtar, Mohamed Fathy, Yasser A. Dahab, Emad A. Sayed
In modern machine learning, optimization algorithms are crucial; they steer the training process by skillfully navigating through complex, high-dimensional loss landscapes. Among these, stochastic gradient descent with momentum (SGDM) is widely adopted for its ability to accelerate convergence in shallow regions.…
Kazunori D Yamada
In the deep learning era, a gradient descent method is the most common method to optimize parameters of neural networks. Among various mathematical optimization methods, a gradient descent method is the most naive method. Although controlling a learning rate of the method is necessary for quick convergence, the…
Md. Shoaibur Rahman
This article presents an overview of the generalized formulations of the computations, optimization, and tuning of a deep feedforward neural network. A small network has been used to systematically explain the computing steps, which were then used to establish the generalized forms of the computations in forward and…
Authors not listed
Physics-based coarse-grained (CG) models are widely used in (bio)molecular simulations, yet their parameterization remains challenging and labor-intensive. In this work, we demonstrate how recently developed gradient-based optimization methods can substantially accelerate the refinement of CG force field (FF)…
Leonard Schmiester, Daniel Weindl, Jan Hasenauer
Unknown parameters of dynamical models are commonly estimated from experimental data. However, while various efficient optimization and uncertainty analysis methods have been proposed for quantitative data, methods for qualitative data are rare and suffer from bad scaling and convergence. Here, we propose an efficient…
Akhil Shajan, Madushanka Manathunga, Andreas Goetz, Kenneth Merz
Based on a series of energy minimizations with starting structures obtained from the Baker test set of 30 organic molecules, a comparison is made between various open source geometry optimization codes that are interfaced with the open-source QUantum Interaction Computational Kernel (QUICK) program for gradient and…
Aline C. Soterroni, Roberto L. Galski, Marluce C. Scarabello, Fernando M. Ramos
'Fernando M. Ramos'] In this work, the q-Gradient (q-G) method, a q-version of the Steepest Descent method, is presented. The main idea behind the q-G method is the use of the negative of the q-gradient vector of the objective function as the search direction. The q-gradient vector, or simply the q-gradient, is a…
AKHIL SHAJAN, Madushanka Manathunga, Andreas Goetz, Kenneth Merz
Based on a series of energy minimizations with starting structures obtained from the Baker test set of 30 organic molecules, a comparison is made between various open-source geometry optimization codes that are interfaced with the open-source QUantum Interaction Computational Kernel (QUICK) program for gradient and…
Akhil Shajan, Madushanka Manathunga, Andreas Goetz, Kenneth Merz
Based on a series of energy minimizations with starting structures obtained from the Baker test set of 30 organic molecules, a comparison is made between various open- source geometry optimization codes that are interfaced with the open-source QUantum Interaction Computational Kernel (QUICK) program for gradient and…
Kaustubh Yadav
—One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value of a function we need gradient. And to update our weights we need gradient…
Simone Carlo Surace, Jean-Pascal Pfister, Wulfram Gerstner, Johanni Brea + 1 more
'Johanni Brea' 'Francis Ouellette'] This is a PLOS Computational Biology Education paper. The idea that the brain functions so as to minimize certain costs pervades theoretical neuroscience. Because a cost function by itself does not predict how the brain finds its minima, additional assumptions about the optimization…
Xuan Zuo, Hui-Yan Li, Shan Gao, Pu Zhang + 2 more
Adaptive gradient algorithms have been successfully used in deep learning. Previous work reveals that adaptive gradient algorithms mainly borrow the moving average idea of heavy ball acceleration to estimate the first- and second-order moments of the gradient for accelerating convergence. However, Nesterov acceleration…
Dariush Bahrami, Sadegh Pouriyan Zadeh
We introduce Gravity, another algorithm for gradient-based optimization. In this paper, we explain how our novel idea change parameters to reduce the deep learning model's loss. It has three intuitive hyper-parameters that the best values for them are proposed. Also, we propose an alternative to moving average. To…
Adam H. Marblestone, Greg Wayne, Konrad P. Kording
Neuroscience has focused on the detailed implementation of computation, studying neural codes, dynamics and circuits. In machine learning, however, artificial neural networks tend to eschew precisely designed codes, dynamics or circuits in favor of brute force optimization of a cost function, often using simple and…
Abduljlil Abduljlil Ali Abduljlil Habeb, Ningbo Zhu, Mundher Mohammed Taresh, Talal Ahmed Ali Ali + 1 more
'Mundher Mohammed Taresh' 'Talal Ahmed Ali Ali' 'Bilal Alatas'] While digital ocular fundus images are commonly used for diagnosing ocular tumors, interpreting these images poses challenges due to their complexity and the subtle features specific to tumors. Automated detection of ocular tumors is crucial for timely…
Ali Fozooni, Osman Kamari, Mostafa Pourtalebiyan, Masoud Gorgich + 2 more
'Mohammad Khalilzadeh' 'Amin Valizadeh'] In this study, we evaluate several nongradient (evolutionary) search strategies for minimizing mathematical function expressions. We developed and tested the genetic algorithms, particle swarm optimization, and differential evolution in order to assess their general efficacy in…
Riley Hickman, Priyansh Parakh, Austin Cheng, Qianxiang Ai + 3 more
Experiment planning algorithms are a required component of autonomous platforms for scientific discovery. Selecting a suitable optimization algorithm for a novel application is an important yet difficult choice a researcher has to make based on past empirical performance on similar tasks. To facilitate the evaluation…
Máté Mohácsi, Márk Patrik Török, Sára Sáray, Luca Tar + 1 more
Finding optimal parameters for detailed neuronal models is a ubiquitous challenge in neuroscientific research. Recently, manual model tuning has been replaced by automated parameter search using a variety of different tools and methods. However, using most of these software tools and choosing the most appropriate…