25 papers · ranked by Valyu relevance
Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz + 1 more
'Ryan J. Urbanowicz' 'Jason H. Moore'] Background The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different…
Barnard, Amanda S
Benchmark data sets are a cornerstone of machine learning development and applications, ensuring new methods are robust, reliable and competitive. The relative rarity of benchmark sets in computational science, due to the uniqueness of the problems and the pace of change in the associated domains, makes evaluating new…
Anasua Sarkar, Yang Yang, Mauno Vihinen
Development of new computational methods and testing their performance has to be done on experimental data. Only in comparison to existing knowledge can method performance be assessed. For that purpose, benchmark datasets with known and verified outcome are needed. High-quality benchmark datasets are valuable and may…
Bo Wen, William Noble
Training machine learning models for tasks such as de novo sequencing or spectral clustering requires large collections of confidently identified spectra. Here we describe a dataset of 2.8 million high-confidence peptide-spectrum matches derived from nine different species. The dataset is based on a previously…
Han Xing, Yongjian Zhao
Predicting drug-diseases associations provides hints in developing new drugs. Various computational methods have been developed. To develop better models for predicting drug-disease associations, two types of key resources must be obtained, the benchmarking dataset and the baseline performances. Collecting these…
Sterling G. Baird, Taylor D. Sparks
In scientific disciplines, benchmarks play a vital role in driving progress forward. For a benchmark to be effective, it must closely resemble real-world tasks. If the level of difficulty or relevance is inadequate, it can impede progress in the field. Moreover, benchmarks should have low computational overhead to…
Randal S. Olson, William La Cava, Patryk Orzechowski, Ryan J. Urbanowicz + 1 more
'Ryan J. Urbanowicz' 'Jason H. Moore'] The selection, development, or comparison of machine learning methods in data mining can be a difficult task based on the target problem and goals of a particular study. Numerous publicly available real-world and simulated benchmark datasets have emerged from different sources…
Roberta Rocca, Tal Yarkoni
Consensus on standards for evaluating models and theories is an integral part of every science. Nonetheless, in psychology, relatively little focus has been placed on defining reliable communal metrics to assess model performance. Evaluation practices are often idiosyncratic and are affected by a number of shortcomings…
Kohulan Rajan, Henning Otto Brinkhaus, M. Isabel Agea, Achim Zielesny + 1 more
The number of publications describing chemical structures has increased steadily over the last decades. However, the majority of published chemical information is currently not available in machine-readable form in public databases. It remains a challenge to automate the process of information extraction in a way that…
Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard + 4 more
'Quanming Yao' 'Huan Zhao' 'Isabelle Guyon'] Title: Summary Obtaining a standardized benchmark of computational methods is a major issue in data-science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here, we introduce Codabench, a meta-benchmark platform…
Xiaoqi Cabiria Liang, Nick Robertson, Marni Torkel, Sanghyun Kim + 3 more
The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and…
Authors not listed
The incredible advances in deep learning architectures have not only led to successful protein structure prediction methods, but also to many interesting new methods related to protein-ligand docking and co-folding. The most recent biomolecular foundation model, Boltz-2, even claimed to approach the performance of…
Izaskun Mallona, Charlotte Soneson, Ben Carrillo, Almut Lütge + 5 more
Benchmarking, which involves collecting reference datasets and demonstrating method performance, is a requirement for the development of new computational tools, but also becomes a domain of its own to achieve neutral comparisons of methods. Although a lot has been written about how to design and conduct benchmark…
Salvador Capella-Gutierrez, Diana de la Iglesia, Juergen Haas, Analia Lourenco + 7 more
The dependence of life scientists on software has steadily grown in recent years. For many tasks, researchers have to decide which of the available bioinformatics software are more suitable for their specific needs. Additionally researchers should be able to objectively select the software that provides the highest…
Amandalynne Paullada, Inioluwa Deborah Raji, Emily M. Bender, Emily Denton + 1 more
'Emily Denton' 'Alex Hanna'] Title: Summary In this work, we survey a breadth of literature that has revealed the limitations of predominant practices for dataset collection and use in the field of machine learning. We cover studies that critically review the design and development of datasets with a focus on negative…
Yi Lyu, Pei-Chieh Lo, Natan Lidukhover
The problem that we are trying to solve is to build a new benchmark for the data lakes. Although there is an increasing need for data lakes these days, there are not many standardized benchmarks for data lakes. By doing that, we are hoping to get a more objective and comparative understanding of different data lake…
Rui Han, Xiaoyi Lü
Big data systems address the challenges of capturing, storing, managing, analyzing, and visualizing big data. Within this context, developing benchmarks to evaluate and compare big data systems has become an active topic for both research and industry communities. To date, most of the state-of-the-art big data…
Riley Hickman, Priyansh Parakh, Austin Cheng, Qianxiang Ai + 3 more
Experiment planning algorithms are a required component of autonomous platforms for scientific discovery. Selecting a suitable optimization algorithm for a novel application is an important yet difficult choice a researcher has to make based on past empirical performance on similar tasks. To facilitate the evaluation…
Ulysse Rubens, Romain Mormont, Lassi Paavolainen, Volker Bäcker + 15 more
Automated image analysis has become key to extract quantitative information from scientific microscopy bioimages, but the methods involved are now often so refined that they can no longer be unambiguously described using written protocols. We introduce BIAFLOWS, a software tool with web services and a user interface…
Authors not listed
The construction of large benchmark sets has accelerated advancement of quantum chemistry methods, especially in density functional theory and lower-cost methods. However, these large benchmark sets can be unsuitable for cutting-edge method development, because research codes developed for fundamentally new approaches…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Izaskun Mallona, Almut Luetge, Ben Carrillo, Daniel Incicau + 4 more
bioinformatics Authors: ['Izaskun Mallona' 'Almut Luetge' 'Ben Carrillo' 'Daniel Incicau' 'Reto Gerber' 'Anthony Sonrel' 'Charlotte Soneson' 'Mark D. Robinson'] We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts.…
Ali Shirali, Rediet Abebe, Moritz Hardt
Dynamic benchmarks interweave model fitting and data collection in an attempt to mitigate the limitations of static benchmarks. In contrast to an extensive theoretical and empirical study of the static setting, the dynamic counterpart lags behind due to limited empirical studies and no apparent theoretical foundation…
Jianfeng Zhan
The measurable properties of the artifacts or objects in the computer, management, or finance disciplines are extrinsic, not inherent — dependent on their problem definitions and solution instantiations. Only after the instantiation can the solutions to the problem be measured. The processes of definition…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…