20 papers · ranked by Valyu relevance
Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard + 4 more
'Quanming Yao' 'Huan Zhao' 'Isabelle Guyon'] Title: Summary Obtaining a standardized benchmark of computational methods is a major issue in data-science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here, we introduce Codabench, a meta-benchmark platform…
Jianfeng Zhan
Currently, there is no consistent benchmarking across multi-disciplines. Even no previous work tries to relate different categories of benchmarks in multi-disciplines. This article investigates the origin and evolution of the benchmark term. Five categories of benchmarks are summarized, including measurement standards…
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao + 4 more
'Neil Houlsby' 'Fernando Díaz' 'Donald Metzler' 'Oriol Vinyals'] The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of a benchmark lottery that describes the overall fragility of…
Salvador Capella-Gutierrez, Diana de la Iglesia, Juergen Haas, Analia Lourenco + 7 more
The dependence of life scientists on software has steadily grown in recent years. For many tasks, researchers have to decide which of the available bioinformatics software are more suitable for their specific needs. Additionally researchers should be able to objectively select the software that provides the highest…
Nathan J. Hillson
for Benchmarking Biofoundry Capabilities Authors: ['Nathan J. Hillson'] Benchmarking compares the performance of a product or service with a competitor. In a biofoundry context, capability benchmarking enables more effective use of development resources and furthering business development efforts. Biofoundries…
Izaskun Mallona, Almut Luetge, Charlotte Soneson, Brian Carrillo + 4 more
'Reto Gerber' 'Daniel Incicau' 'Anthony Sonrel' 'Mark D. Robinson'] Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although…
Jianfeng Zhan
The measurable properties of the artifacts or objects in the computer, management, or finance disciplines are extrinsic, not inherent — dependent on their problem definitions and solution instantiations. Only after the instantiation can the solutions to the problem be measured. The processes of definition…
Izaskun Mallona, Almut Luetge, Ben Carrillo, Daniel Incicau + 4 more
bioinformatics Authors: ['Izaskun Mallona' 'Almut Luetge' 'Ben Carrillo' 'Daniel Incicau' 'Reto Gerber' 'Anthony Sonrel' 'Charlotte Soneson' 'Mark D. Robinson'] We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts.…
Xiaoqi Cabiria Liang, Nick Robertson, Marni Torkel, Sanghyun Kim + 3 more
The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and…
Serghei Mangul, Lana S. Martin, Brian L. Hill, Angela Ka-Mei Lam + 4 more
'Margaret G. Distler' 'Alex Zelikovsky' 'Eleazar Eskin' 'Jonathan Flint'] Computational omics methods packaged as software have become essential to modern biological research. The increasing dependence of scientists on these powerful software tools creates a need for systematic assessment of these methods, known as…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Authors not listed
Accurate benchmarks are key to assessing the accuracy and robustness of computational methods, yet most available benchmark sets focus on equilibrium geometries, limiting their utility for applications involving non-equilibrium structures such as ab initio molecular dynamics and automated reaction-path exploration. To…
A. Wind, W. H. van Harten
Background Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on benchmarking in specialty hospitals and a description of study characteristics. Methods We searched PubMed and EMBASE for articles published in…
Robbe Devreese, Caroline Jachmann, Bart Van Puyvelde, Holda A. Anagho-Mattanovich + 46 more
Mass spectrometry (MS)-based proteomics is a well-established strategy for analyzing complex biological mixtures. Many MS instruments and data acquisition strategies are available, and the data they acquire differ substantially, thus requiring tailored analysis algorithms. Hence, many dedicated bioinformatics workflows…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Ariel Larey, Elay Dahan, Amit Bleiweiss, Raizy Kellerman + 14 more
The rapid scaling of Genomic Foundation Models (GFMs) has created a critical need for standardized evaluation frameworks. Current benchmarking practices are often fragmented, relying on model-specific preprocessing and inconsistent metric implementations that hinder reproducible comparisons. We present GFMBench-API, a…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Authors not listed
The construction of large benchmark sets has accelerated advancement of quantum chemistry methods, especially in density functional theory and lower-cost methods. However, these large benchmark sets can be unsuitable for cutting-edge method development, because research codes developed for fundamentally new approaches…
Sterling G. Baird, Taylor D. Sparks
In scientific disciplines, benchmarks play a vital role in driving progress forward. For a benchmark to be effective, it must closely resemble real-world tasks. If the level of difficulty or relevance is inadequate, it can impede progress in the field. Moreover, benchmarks should have low computational overhead to…
Authors not listed
The incredible advances in deep learning architectures have not only led to successful protein structure prediction methods, but also to many interesting new methods related to protein-ligand docking and co-folding. The most recent biomolecular foundation model, Boltz-2, even claimed to approach the performance of…