15 papers · ranked by Valyu relevance
Yuxin Jiang, Wenqin Zhang, Lele Wang
Gradient coding is a distributed computing technique aiming to provide robustness against slow or non-responsive computing nodes, known as stragglers, while balancing the computational load for responsive computing nodes. Among existing gradient codes, a construction based on combinatorial designs, called BIBD gradient…
Zachary Charles, Dimitris Papailiopoulos
—Gradient descent and its many variants, including mini-batch stochastic gradient descent, form the algorithmic foundation of modern large-scale machine learning. Due to the size and scale of modern data, gradient computations are often distributed across multiple compute nodes. Unfortunately, such distributed…
Sinong Wang, Jiashang Liu, Ness B. Shroff
It has been established that when the gradient coding problem is distributed among n servers, the computation load (number of stored data partitions) of each worker is at least s + 1 in order to resists s stragglers [1]. This scheme incurs a large overhead when the number of stragglers s is large. In this paper, we…
Neophytos Charalambides, Hessam Mahdavifar, Alfred O. Hero
—A major hurdle in machine learning is scalability to massive datasets. One approach to overcoming this is to distribute the computational tasks among several workers. Gradient coding has been recently proposed in distributed optimization to compute the gradient of an objective function using multiple, possibly…
Neophytos Charalambides, Mert Pilancı, Alfred O. Hero
A major hurdle in machine learning is scalability to massive datasets. Approaches to overcome this hurdle include compression of the data matrix and distributing the computations. Leverage score sampling provides a compressed approximation of a data matrix using an importance weighted subset. Gradient coding has been…
Qi Wang, Ying Cui, Chenglin Li, Junni Zou + 1 more
—Existing gradient coding schemes introduce identical redundancy across the coordinates of gradients and hence cannot fully utilize the computation results from partial stragglers. This motivates the introduction of diverse redundancies across the coordinates of gradients. This paper considers a distributed computation…
Muhammet Balcılar, Bharath Bhushan Damodaran, Karam Naser, Franck Galpin + 1 more
'Franck Galpin' 'Pierre Hellier'] End-to-end image/video codecs are getting competitive compared to traditional compression techniques that have been developed through decades of manual engineering efforts. These trainable codecs have many advantages over traditional techniques such as easy adaptation on perceptual…
Louis-Adrien Dufrène, Quentin Lampin, Guillaume Larue
—This study investigates the problem of learning linear block codes optimized for Belief-Propagation decoders significantly improving performance compared to the state-ofthe-art. Our previous research is extended with an enhanced system design that facilitates a more effective learning process for the parity check…
Tadashi Wadayama, Lantian Wei
—This paper presents the Gradient Flow (GF) decoding for LDPC codes. GF decoding, a continuous-time methodology based on gradient flow, employs a potential energy function associated with bipolar codewords of LDPC codes. The decoding process of the GF decoding is concisely defined by an ordinary differential equation…
Jannis Clausius, Marvin Geiselhart, Stephan ten Brink
—For improving short-length codes, we demonstrate that classic decoders can also be used with real-valued, neural encoders, i.e., deep-learning based "codeword" sequence generators. Here, the classical decoder can be a valuable tool to gain insights into these neural codes and shed light on weaknesses. Specifically…
Tony Shaska
We introduce the Graded Transformer framework, embedding algebraic inductive biases via grading transformations on vector spaces. Extending Graded Neural Networks (GNNs), we propose the Linearly Graded Transformer (LGT) and Exponentially Graded Transformer (EGT), which apply parameterized scaling—via fixed or learnable…
Johannes Ballé, Philip A. Chou, David Minnen, Saurabh Singh + 4 more
'Nick Johnston' 'Eirikur Agustsson' 'Sung Jin Hwang' 'George Toderici'] Abstract—We review a class of methods that can be collected under the name nonlinear transform coding (NTC), which over the past few years have become competitive with the best linear transform codecs for images, and have superseded them in terms…
Simon Wiedemann, Heiner Kirchoffer, Stefan Matlage, Paul T. Haase + 9 more
'Arturo Marbán' 'Talmaj Marinč' 'David L. Neumann' 'Tung Thanh Nguyen' 'Ahmed Osman' 'Detlev Marpe' 'Heiko Schwarz' 'Thomas Wiegand' 'Wojciech Samek'] Abstract—The field of video compression has developed some of the most sophisticated and efficient compression algorithms known in the literature, enabling very high…
Hyunmin Cho, Jaejun Yoo, Kyong Hwan Jin
We study sinusoidal recurrence as an iterative mechanism for harmonic spectral enrichment in implicit neural representations (INRs). Our analysis reveals that sinusoidal activations induce a harmonic line spectrum, providing a spectral account of how recurrent unrolling enriches the effective spectral support. We…
Shicong Liu, Junru Shao, Hongtao Lu
Vector quantization is an essential tool for tasks involving large scale data, for example, large scale similarity search, which is crucial for content-based information retrieval and analysis. In this paper, we propose a novel vector quantization framework that iteratively minimizes quantization error. First, we…