Search · four archives
Search · four archives
15 papers · ranked by Valyu relevance
Han Chen, Qiqi Wang
Many control applications can be formulated as optimization constrained by conservation laws. Such optimization can be efficiently solved by gradient-based methods, where the gradient is obtained through the adjoint method. Traditionally, the adjoint method has not been able to be implemented in "gray-box" conservation…
Saeed Asadi, Sonia Gharibzadeh, Shiva Zangeneh, Masoud Reihanifar + 2 more
Multidimensional Surface 3D Visualizations and Initial Point Sensitivity Authors: ['Saeed Asadi' 'Sonia Gharibzadeh' 'Shiva Zangeneh' 'Masoud Reihanifar' 'Mehrzad Rahimi' 'Lazim Abdullah'] This study examines several renowned gradient-based optimization techniques and focuses on their computational efficiency and…
Esmail Abdul Fattah, Janet van Niekerk, Håvard Rue
Computing the gradient of a function provides fundamental information about its behavior. This information is essential for several applications and algorithms across various fields. One common application that require gradients are optimization techniques such as stochastic gradient descent, Newton's method and trust…
Jiawei Zhang
In this paper, we aim at providing an introduction to the gradient descent based optimization algorithms for learning deep neural network models. Deep learning models involving multiple nonlinear projection layers are very challenging to train. Nowadays, most of the deep learning model training still relies on the back…
Hugo Silva, Martha White
Network? Authors: ['Hugo Silva' 'Martha White'] Oftentimes, machine learning applications using neural networks involve solving discrete optimization problems, such as in pruning, parameter-isolation-based continual learning and training of binary networks. Still, these discrete problems are combinatorial in nature and…
Zexian Liu, Wangli Chu, Hongwei Liu
A new type of stepsize, which was recently introduced by Liu et al. (Optimization 67(3):427-440, 2018), is called approximately optimal stepsize and is very efficient for gradient method. In this paper, we present an efficient gradient method with approximately optimal stepsize for large-scale unconstrained…
Kaustubh Yadav
—One of the most important parts of Artificial Neural Networks is minimizing the loss functions which tells us how good or bad our model is. To minimize these losses we need to tune the weights and biases. Also to calculate the minimum value of a function we need gradient. And to update our weights we need gradient…
Swalpa Kumar Roy, Mercedes E. Paoletti, Juan M. Haut, Shiv Ram Dubey + 3 more
'Purbayan Kar' 'Antonio Plaza' 'B.B. Chaudhuri'] Abstract—Convolutional neural networks (CNNs) are trained using stochastic gradient descent (SGD)-based optimizers. Recently, the adaptive moment estimation (Adam) optimizer has become very popular due to its adaptive momentum, which tackles the dying gradient problem of…
Feihu Han, Sida Xing, Suiyang Khoo
There introduce Particle Optimized Gradient Descent (POGD), an algorithm based on the gradient descent but integrates the particle swarm optimization (PSO) principle to achieve the iteration. From the experiments, this algorithm has adaptive learning ability. The experiments in this paper mainly focus on the training…
Kouhei Nishida, Hernán Aguirre, Shota Saito, Shinichi Shirakawa + 1 more
'Youhei Akimoto'] Black box discrete optimization (BBDO) appears in wide range of engineering tasks. Evolutionary or other BBDO approaches have been applied, aiming at automating necessary tuning of system parameters, such as hyper parameter tuning of machine learning based systems when being installed for a specific…
Aline C. Soterroni, Roberto Luiz Galski, Fernando M. Ramos
The q-gradient is an extension of the classical gradient vector based on the concept of Jackson's derivative. Here we introduce a preliminary version of the q-gradient method for unconstrained global optimization. The main idea behind our approach is the use of the negative of the q-gradient of the objective function…
Xin Xu
The Barzilai-Borwein (BB) method is an effective gradient method for solving unconstrained optimization problems. Based on the observation of two classical BB step sizes, by constructing a variational least squares model, we propose a new class of BB step sizes, each of which still has the quasi-Newton property. The…
Yaoxin Li, Jing Liu, Guozheng Lin, Yueyuan Hou + 2 more
'Jiang Zhang'] In computer science, there exist a large number of optimization problems defined on graphs, that is to find a best node state configuration or a network structure such that the designed objective function is optimized under some constraints. However, these problems are notorious for their hardness to…
Manish Kumar Sahu, S. R. Pattanaik, Santosh Kumar Panda
The modified BFGS optimization algorithm is generally used when the objective function is non-convex. In this method, one has to move in a specific direction such that the value of the objective function reduces. Therefore, the different inexact line search or exact line search plays an important role in optimization.…
Dickson Odhiambo Owuor, Thomas A. Runkler, Anne Laurent
Swarm intelligence is a discipline that studies the collective behavior that is produced by local interactions of a group of individuals with each other and with their environment. In Computer Science domain, numerous swarm intelligence techniques are applied to optimization problems that seek to efficiently find best…