22 papers · ranked by Valyu relevance
Simon Ott, Adriano Barbosa-Silva, Kathrin Blagec, Jan Brauner + 1 more
'Matthias Samwald'] Benchmarks are crucial to measuring and steering progress in artificial intelligence (AI). However, recent studies raised concerns over the state of AI benchmarking, reporting issues such as benchmark overfitting, benchmark saturation and increasing centralization of benchmark dataset creation. To…
Mostafa Dehghani, Yi Tay, Alexey A. Gritsenko, Zhe Zhao + 4 more
'Neil Houlsby' 'Fernando Díaz' 'Donald Metzler' 'Oriol Vinyals'] The world of empirical machine learning (ML) strongly relies on benchmarks in order to determine the relative effectiveness of different algorithms and methods. This paper proposes the notion of a benchmark lottery that describes the overall fragility of…
Yi Lin, Yuzhe Yang, George Shih, Yifan Peng
Public benchmarks have become a cornerstone for advancing artificial intelligence (AI) in radiology. In this context, a benchmark refers to a publicly available dataset paired with a standardized task, reference labels (ground truth), and an agreed-on evaluation protocol and metrics. Open imaging datasets create a…
Jianfeng Zhan
Currently, there is no consistent benchmarking across multi-disciplines. Even no previous work tries to relate different categories of benchmarks in multi-disciplines. This article investigates the origin and evolution of the benchmark term. Five categories of benchmarks are summarized, including measurement standards…
Salvador Capella-Gutierrez, Diana de la Iglesia, Juergen Haas, Analia Lourenco + 7 more
The dependence of life scientists on software has steadily grown in recent years. For many tasks, researchers have to decide which of the available bioinformatics software are more suitable for their specific needs. Additionally researchers should be able to objectively select the software that provides the highest…
Xiaoqi Cabiria Liang, Nick Robertson, Marni Torkel, Sanghyun Kim + 3 more
The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and…
David L. Mobley, Michael K. Gilson
Binding free energy calculations based on molecular simulations provide predicted affinities for biomolecular complexes. These calculations begin with a detailed description of a system, including its chemical composition and the interactions between its components. Simulations of the system are then used to compute…
Izaskun Mallona, Almut Luetge, Charlotte Soneson, Brian Carrillo + 4 more
'Reto Gerber' 'Daniel Incicau' 'Anthony Sonrel' 'Mark D. Robinson'] Benchmarking, which involves collecting reference datasets and demonstrating method performances, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although…
Authors not listed
Accurate benchmarks are key to assessing the accuracy and robustness of computational methods, yet most available benchmark sets focus on equilibrium geometries, limiting their utility for applications involving non-equilibrium structures such as ab initio molecular dynamics and automated reaction-path exploration. To…
Jianfeng Zhan
The measurable properties of the artifacts or objects in the computer, management, or finance disciplines are extrinsic, not inherent — dependent on their problem definitions and solution instantiations. Only after the instantiation can the solutions to the problem be measured. The processes of definition…
Amelia Hardy, Anka Reuel, Kiana Jafari Meimandi, Lisa Soder + 5 more
Practitioners Authors: ['Amelia Hardy' 'Anka Reuel' 'Kiana Jafari Meimandi' 'Lisa Soder' 'Allie Griffith' 'Dylan M. Asmar' 'Sanmi Koyejo' 'Michael S. Bernstein' 'Mykel J. Kochenderfer'] AMELIA HARDY∗ and ANKA REUEL∗ , Stanford University, USA KIANA JAFARI MEIMANDI, Stanford University, USA LISA SODER, interface – Tech…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Izaskun Mallona, Almut Luetge, Ben Carrillo, Daniel Incicau + 4 more
bioinformatics Authors: ['Izaskun Mallona' 'Almut Luetge' 'Ben Carrillo' 'Daniel Incicau' 'Reto Gerber' 'Anthony Sonrel' 'Charlotte Soneson' 'Mark D. Robinson'] We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts.…
Moritz B. Milde, Saeed Afshar, Ying Xu, Alexandre Marcireau + 8 more
'Damien Joubert' 'Bharath Ramesh' 'Yeshwanth Bethi' 'Nicholas O. Ralph' 'Sami El Arja' 'Nik Dennler' 'André van Schaik' 'Gregory Cohen'] Neuromorphic engineering aims to build (autonomous) systems by mimicking biological systems. It is motivated by the observation that biological organisms-from algae to primates-excel…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Jochen O. Mierau, Brigit C.A. Toebes
Legally mandated public health benchmarks are grounded in the human right to health, as set out in a range of international human rights treaties to which most countries worldwide are bound. The most frequently cited right to health provision is Article 12 of the International Covenant on Economic, Social and Cultural…
Wouter T. Gude, Benjamin Brown, Sabine N. van der Veer, Heather L. Colquhoun + 6 more
'Heather L. Colquhoun' 'Noah M. Ivers' 'Jamie C. Brehaut' 'Zach Landis-Lewis' 'Christopher J. Armitage' 'Nicolette F. de Keizer' 'Niels Peek'] Background Audit and feedback (A&F) is a common quality improvement strategy with highly variable effects on patient care. It is unclear how A&F effectiveness can be maximised.…
Authors not listed
The construction of large benchmark sets has accelerated advancement of quantum chemistry methods, especially in density functional theory and lower-cost methods. However, these large benchmark sets can be unsuitable for cutting-edge method development, because research codes developed for fundamentally new approaches…
A. Wind, W. H. van Harten
Background Although benchmarking may improve hospital processes, research on this subject is limited. The aim of this study was to provide an overview of publications on benchmarking in specialty hospitals and a description of study characteristics. Methods We searched PubMed and EMBASE for articles published in…
Samuel Genheden, Esben Bjerrum
We introduce a framework for benchmarking multi-step retrosynthesis methods, i.e. route predictions, called PaRoutes. The framework consists of two sets of 10,000 synthetic routes extracted from the patent literature, a list of stock compounds, and a curated set of reactions on which one-step retrosynthesis models can…
Montgomery Gray, John Herbert
Over the past 25 years there has been remarkable progress towards accurate description of nonbonded interactions within the context of density functional theory (DFT). Various methods have been devised to capture London dispersion, which is the most exacting contribution to noncovalent interactions; these strategies…