19 papers · ranked by Valyu relevance
Jonas G. Matt, Pengcheng Huang, Balz Maag
—Our increasingly digital and connected world has led to the generation of unprecedented amounts of data. This data must be efficiently managed, transmitted, and stored to preserve resources and allow scalability. Data compression has therein been a key technology for a long time, resulting in a vast landscape of…
Aviv Adler, Jennifer Tang
It is well-known in the field of lossless data compression that probabilistic nextsymbol prediction can be used to compress sequences of symbols. Deep neural networks are able to capture rich dependencies in data, offering a powerful means of estimating these probabilities and hence an avenue towards more effective…
Sören Dréano, Derek Molloy, Noel Murphy
This work introduces Llamazip, a novel lossless text compression algorithm based on the predictive capabilities of the LLaMA3 language model. Llamazip achieves significant data reduction by only storing tokens that the model fails to predict, optimizing storage efficiency without compromising data integrity. Key…
Yuriy Kim, Evgeny Belyaev
This paper is dedicated to lossless data compression with probability estimation using neural networks. First, we propose a probability estimation architecture based on a chain of neural predictors, so that each unit of the chain is defined as a neural network with the minimum possible number of weights, which is…
Yuan, Cheng, Jiawei Shao, Li + 1 more
Recent years have witnessed the rapid advancements of large language models (LLMs) and their expanding applications, leading to soaring demands for computational resources. The widespread adoption of test-time scaling further aggravates the tension between model capability and resource consumption, highlighting the…
Ya Liu, Rui Zhang, Yong Zhang, Yuwei Chen + 1 more
Large field-of-view (FOV) infrared imaging, widely utilized in applications including target detection and remote sensing, generates massive datasets that pose significant challenges for transmission and storage. To address this issue, we propose an efficient lossless compression method for large FOV infrared video.…
André Ribeiro, Rúben Garrido, Violeta Ramos, António Alberto + 27 more
Lossless data compression remains central to computer science, with direct impact on storage, communication bandwidth, computational cost, and energy consumption. It is also closely related to Algorithmic Information Theory, where compressibility provides an operational measure of structure and non-randomness. This…
Alin-Adrian Alecu, Mohammad Ali Tahouri, Adrian Munteanu, Bujor Păvăloiu + 1 more
Near-lossless coding schemes traditionally rely on uniform quantization to control the maximum absolute error ( $L_{\infty}$ norm) of residual signals, often assuming a parametric model for the source distribution. This paper introduces a novel design framework for non-uniform, entropy-aware $L_{\infty}$-oriented…
Michail Patsakis, Theodore Chronopoulos, Ioannis Mouratidis, Ilias Georgakopoulos-Soares
Genomic data repositories continue to grow as sequencing technologies improve, with the NCBI SRA alone exceeding 47 PB. General-purpose compressors treat bioinformatics files as unstructured byte streams and fail to exploit the structured nature of omics data. We present NYX, a format-aware compression system for…
Esteban-Alejandro Durán-Yáñez, Mario-Alberto Rodríguez-Díaz, Ricardo Mendoza-González, Francisco-Javier Luna-Rosas + 2 more
Voxel representations provide a simple way to represent three-dimensional objects as binary occupancy signals, but dense voxel grids and direct sparse encodings remain costly at medium and high resolutions. This paper addresses the gap between conventional dense-grid, octree, and point-cloud-codec representations and…
Neri Merhav, Jun Chen
We derive a few extended versions of the Kraft inequality for lossy compression, which pave the way to the derivation of several refinements and extensions of the well-known Shannon lower bound in a variety of instances of rate-distortion coding. These refinements and extensions include sharper bounds for one-to-one…
Alice Tor, Yuxin Wu, Stephen E Clarke, Lisa Yamada + 2 more
The complexity of neural data changes as the brain processes information during events. Universal lossless compression algorithms, which are broadly applicable and grounded in information theory, identify and exploit redundancies in data in order to compress it to essentially-optimal sizes regardless of underlying…
Shiladitya Bhattacharjee, Subha Bhattacharya, Arnab Chatterjee, Sulabh Bansal + 1 more
Conventional techniques for compression and encryption are frequently laborious and resource-intensive, rendering them inappropriate for real-time applications. A plethora of research has been presented in the current literature to address these difficulties together; yet, it fails to propose any suitable strategy.…
Marek Kokot, Amitava Roy, Travis J Wheeler, Sebastian Deorowicz
Molecular dynamics (MD) simulations model the physical movements of atoms in biomolecular systems over time, providing atomic-resolution insight into conformational changes, binding events, and dynamic behaviors that cannot be captured by static structures alone. As such, MD simulations are playing an increasingly…
Dohyeon Lee, Juyeon Park, Juheon Lee, Chungha Lee + 2 more
Holotomography (HT) is a label-free, three-dimensional quantitative phase imaging technique that captures refractive index distributions of biological samples at sub-micron resolution. As modern HT systems enable high-throughput and large-scale acquisition, they produce terabyte-scale datasets that require efficient…
Vojtech Macala, Petr Simecek
Lossless compression and probabilistic sequence modeling are two faces of the same coin: a model that assigns high probability to a sequence can encode it in few bits via arithmetic coding. We exploit this duality to evaluate genomic language models as compressors of DNA, using compression primarily as an objective…
Yanlong Gao, Haiming Xu, Wei Huang, Hao Bai + 3 more
With the extensive applications of satellite image data in environmental monitoring and geographic surveying and mapping, the amount of data has increased rapidly, which brings great challenges for transmitting and storing these images. However, when processing high-resolution and multi-spectral satellite data…
Michail Patsakis, Alexandros Tzanakakis, Ilias Georgakopoulos-Soares
Evo 2 is the largest openly available genomic foundation model, but its forty billion parameter configuration cannot be loaded onto a single 80 GB accelerator, placing genome-scale analysis beyond most laboratories. We present TurboQuant-Bio, an open toolkit that compresses Evo 2’s weights and attention cache to four…
Timothé Rouzé, Rayan Chikhi, Antoine Limasset
Petabases of sequencing data in the Sequence Read Archive (SRA) present a significant challenge for holistic reanalysis due to their sheer volume. Recent efforts have assembled this data into terabytes of unitigs, an efficient k-mer set representation that can reduce data size by an order of magnitude. However, these…