21 papers · ranked by Valyu relevance
von Laszewski, Gregor, Brewer, Wesley + 58 more
| I | | Introduction | 3 | |-----|-----------------|--------------------------------------------------------------------------------|----| | II | Definitions | | 4 | | | II-A | What is Benchmarking? | 4 | | | II-B | Lessons Learned from Traditional HPC Benchmarking | 4 | | | II-C | What is Democratization? | 4 | | | |…
Izaskun Mallona, Charlotte Soneson, Ben Carrillo, Almut Lütge + 5 more
Benchmarking, which involves collecting reference datasets and demonstrating method performance, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark…
Lukas M. Weber, Wouter Saelens, Robrecht Cannoodt, Charlotte Soneson + 5 more
'Alexander Hapfelmeier' 'Paul P. Gardner' 'Anne-Laure Boulesteix' 'Yvan Saeys' 'Mark D. Robinson'] In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the…
Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard + 4 more
'Quanming Yao' 'Huan Zhao' 'Isabelle Guyon'] Title: Summary Obtaining a standardized benchmark of computational methods is a major issue in data-science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here, we introduce Codabench, a meta-benchmark platform…
Serghei Mangul, Lana S. Martin, Brian L. Hill, Angela Ka-Mei Lam + 4 more
'Margaret G. Distler' 'Alex Zelikovsky' 'Eleazar Eskin' 'Jonathan Flint'] Computational omics methods packaged as software have become essential to modern biological research. The increasing dependence of scientists on these powerful software tools creates a need for systematic assessment of these methods, known as…
Xiaoqi Cabiria Liang, Nick Robertson, Marni Torkel, Sanghyun Kim + 3 more
The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and…
Salvador Capella-Gutierrez, Diana de la Iglesia, Juergen Haas, Analia Lourenco + 7 more
The dependence of life scientists on software has steadily grown in recent years. For many tasks, researchers have to decide which of the available bioinformatics software are more suitable for their specific needs. Additionally researchers should be able to objectively select the software that provides the highest…
A. Wind, W. H. van Harten
Background Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on benchmarking in specialty hospitals and a description of study characteristics. Methods We searched PubMed and EMBASE for articles published in…
Izaskun Mallona, Almut Luetge, Charlotte Soneson, Brian Carrillo + 4 more
'Reto Gerber' 'Daniel Incicau' 'Anthony Sonrel' 'Mark D. Robinson'] Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although…
Jianfeng Zhan
Currently, there is no consistent benchmarking across multi-disciplines. Even no previous work tries to relate different categories of benchmarks in multi-disciplines. This article investigates the origin and evolution of the benchmark term. Five categories of benchmarks are summarized, including measurement standards…
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao + 4 more
'Neil Houlsby' 'Fernando Díaz' 'Donald Metzler' 'Oriol Vinyals'] The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of a benchmark lottery that describes the overall fragility of…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Izaskun Mallona, Almut Luetge, Ben Carrillo, Daniel Incicau + 4 more
bioinformatics Authors: ['Izaskun Mallona' 'Almut Luetge' 'Ben Carrillo' 'Daniel Incicau' 'Reto Gerber' 'Anthony Sonrel' 'Charlotte Soneson' 'Mark D. Robinson'] We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts.…
Authors not listed
Validating the performance of exchange-correlation functionals is vital to ensure the reliability of DFT calculations. Typically, these validations involve benchmarking datasets. Currently, such datasets are typically assembled in an unprincipled manner, suffering from uncontrolled chemical bias, and limiting the…
Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman + 14 more
The rapid scaling of Genomic Foundation Models (GFMs) has created a critical need for standardized evaluation frameworks. Current benchmarking practices are often fragmented, relying on model-specific preprocessing and inconsistent metric implementations that hinder reproducible comparisons. We present GFMBench-API, a…
Authors not listed
Accurate benchmarks are key to assessing the accuracy and robustness of computational methods, yet most available benchmark sets focus on equilibrium geometries, limiting their utility for applications involving non-equilibrium structures such as ab initio molecular dynamics and automated reaction-path exploration. To…
Authors not listed
The construction of large benchmark sets has accelerated advancement of quantum chemistry methods, especially in density functional theory and lower-cost methods. However, these large benchmark sets can be unsuitable for cutting-edge method development, because research codes developed for fundamentally new approaches…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Jonathan Fine, Janez Konc, Ram Samudralal, Gaurav Chopra
Small molecule docking has proven to be invaluable for drug design and discovery. However, existing docking methods have several limitations, such as, ignoring interactions with essential components in the chemical environment of the binding pocket (e.g. cofactors, metal-ions, etc.), incomplete sampling of chemically…
Paul P. Gardner, James M. Paterson, Fatemeh Ashari-Ghomi, Sinan U. Umu + 2 more
Computational biology has provided widely used and powerful software tools for testing and making inferences about biological data. In the face of increasing volumes of data, heuristic methods that trade software speed for mathematical completeness must be employed. We are interested in whether trade-offs between speed…
Dimitar Georgiev, Simon Vilms Pedersen, Ruoxiao Xie, Álvaro Fernández-Galiana + 2 more
Raman spectroscopy is a non-destructive and label-free chemical analysis technique, which plays a key role in the analysis and discovery cycle of various branches of science. Nonetheless, progress in Raman spectroscopic analysis is still impeded by the lack of software, methodological and data standardisation, and the…