Unified Compression Algorithm for Distributed Nonconvex Optimization: Generalized to 1-Bit, Saturation, and Bounded Noise

Abstract

In this paper, we propose a unified compression algorithm for distributed nonconvex opitmization with both the locally- and globally-bounded communication compressors, including 1-bit compressors, saturating quantizers, and the globally-bounded compressors with both relative and absolute compression errors, as well as additional arbitrary bounded noise. We provide a rigorous convergence analysis in nonconvex settings and establish linear convergence under the Polyak-Lojasiewicz (P-L) condition. Notably, we establish an $\mathcal{O}(1/\sqrt{T})$ convergence rate for the locally-bounded class in the distributed nonconvex setting, matching that achieved by the centralized algorithms with 1-bit compressors, where $T$ denotes the total number of iterations. Moreover, one initial uncompressed communication round further yields an order-wise improvement to $\mathcal{O}(1/T^{2/3})$. For the P-L setting and the globally-bounded class, we recover state-of-the-art convergence rates.