23 papers · ranked by Valyu relevance
Michael Pradel, Cristian Cadar, Islem Bouzenia
Numerous software analysis tools exist today, yet applying them to diverse open-source projects remains challenging due to environment setup, dependency resolution, and tool configuration. LLM-based agents offer a potential solution, yet no prior work has systematically studied their effectiveness on the specific task…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Tobias Jumel, Andrej Shevchenko
We present an instrument-independent benchmarking procedure and software (LFQ_bout) for validation and comparative evaluation of the performance of LC-MS/MS and data processing workflows in bottom-up proteomics. It enables back-to-back comparison of common and emerging workflows, e.g. diaPASEF or ScanningSWATH, and…
Xiaoqi Cabiria Liang, Nick Robertson, Marni Torkel, Sanghyun Kim + 3 more
The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Ahmet Kapkiç, Pratanu Mandal, Shu Wan, Paras Sheth + 4 more
Analysis and Machine Learning Authors: ['Ahmet Kapkiç' 'Pratanu Mandal' 'Shu Wan' 'Paras Sheth' 'Abhinav Gorantla' 'Yoonhyuk Choi' 'Huan Liu' 'K. Selçuk Candan'] While witnessing the exceptional success of machine learning (ML) technologies in many applications, users are starting to notice a critical shortcoming of…
Maxwell Venetos, Masha Elkin, Connor Delaney, John Hartwig + 1 more
NMR spectroscopy is an important analytical technique in synthetic organic chemistry, but its integration into high-throughput experimentation workflows has been limited by the necessity to manually analyze NMR spectra of new chemical entities. Current efforts to automate the analysis of NMR spectra rely on comparisons…
Yue Cao, Lijia Yu, Marni Torkel, Sanghyun Kim + 5 more
With the rapid development of computational methods for single-cell sequencing data, benchmarking serves as a valuable resource. As the number of benchmarking studies surges, it is timely to assess the current state of the field. We conducted a systematic literature search and assessed 282 papers, including all 130…
Yue Cao, Lijia Yu, Marni Torkel, Sanghyun Kim + 5 more
With the rapid development of computational methods for single-cell sequencing data, benchmarking serves a valuation resource. As the number of benchmarking studies surges, it is timely to assess the current state of the field. We conducted a systematic literature search and assessed 245 papers, including all 95…
Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan + 4 more
'Siddharth Narayanan' 'Geemi P Wellawatte' 'Andrew White' 'Lorenzo Sani' 'Samuel G Rodriques'] Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and…
Rafael Salles, Ricardo Farias
The Internet of Things (IoT) envisions billions of everyday objects sharing information. As new devices, applications and communication protocols are proposed for the IoT context, their evaluation, comparison, tuning and optimization become crucial and raise the need for a proper benchmark. While edge computing aims to…
Nils Japke, Christoph Witzko, Martin Grambow, David Bermbach
In this paper, we use a testbed microservice application which includes three performance issues to study the detection capabilities of both approaches. In extensive benchmarking experiments, we increase the severity of each performance issue stepwise, run both an application benchmark and the microbenchmark suite, and…
Zhen Xu, Sergio Escalera, Adrien Pavão, Magali Richard + 4 more
'Quanming Yao' 'Huan Zhao' 'Isabelle Guyon'] Title: Summary Obtaining a standardized benchmark of computational methods is a major issue in data-science communities. Dedicated frameworks enabling fair benchmarking in a unified environment are yet to be developed. Here, we introduce Codabench, a meta-benchmark platform…
Ian Knight, Khanh Tang, Olivier Mailhot, John Irwin
Molecular docking is a widely used technique for leveraging protein structure in ligand discovery, but as a method, it remains difficult to utilize due to limitations that have not been adequately addressed. Despite some progress towards automation, docking still requires expert guidance, hindering its adoption by a…
Authors not listed
The analysis of molecular dynamics (MD) simulations is a critical but fragmented process, often requiring researchers to chain together multiple software tools and write bespoke scripts for routine structural and dynamic analyses. This workflow complexity creates a significant barrier to efficiency, standardization…
Ren Jing, Xin Chang, Hua Wang
The evaluation or metrology of commercial bank management efficiency is the core of its effective management. The existing management efficiency of commercial banks adopts evaluation scheme rather than metrology. There are four shortcomings with the evaluation scheme, the evaluation is a ranking of advantages and/or…
Izaskun Mallona, Almut Luetge, Ben Carrillo, Daniel Incicau + 4 more
bioinformatics Authors: ['Izaskun Mallona' 'Almut Luetge' 'Ben Carrillo' 'Daniel Incicau' 'Reto Gerber' 'Anthony Sonrel' 'Charlotte Soneson' 'Mark D. Robinson'] We describe an alpha version of a new benchmarking system, Omnibenchmark, to facilitate benchmark formalization and execution in solo and community efforts.…
Martin Grambow, Tobias Pfandzelter, David Bermbach
Due to the popularity of the FaaS programming model, there is now a wide variety of commercial and open-source FaaS systems. Hence, for comparison of different FaaS systems and their configuration options, FaaS application developers rely on FaaS benchmarking frameworks. Existing frameworks, however, tend to evaluate…
Filippo Schiavio, Lubomír Bulej, Walter Binder
Developers often use microbenchmarks to choose the most performant implementation of a method or a class. On the Java Virtual Machine (JVM), this is commonly done using the Java Microbenchmark Harness (JMH) which addresses common pitfalls of measuring code performance on the JVM. However, even using JMH guidelines…
Authors not listed
The construction of large benchmark sets has accelerated advancement of quantum chemistry methods, especially in density functional theory and lower-cost methods. However, these large benchmark sets can be unsuitable for cutting-edge method development, because research codes developed for fundamentally new approaches…
Authors not listed
Investigating the molecular structure of soil organic matter (SOM), along with its intramolecular interactions and interactions with other soil components and xenobiotics, is essential due to its ecological importance. However, the complexity and heterogeneity of SOM present significant challenges for systematic…
Authors not listed
Alchemical Free Energy methods are gaining traction in computer-aided drug discovery. An expanding array of methodologies is available for the setup, execution and analysis of relative binding free energy (RBFE) calculations. However, the sharing of algorithms and protocols developed by different organisations is often…
Dimitar Georgiev, Simon Vilms Pedersen, Ruoxiao Xie, Álvaro Fernández-Galiana + 2 more
Raman spectroscopy is a non-destructive and label-free chemical analysis technique, which plays a key role in the analysis and discovery cycle of various branches of science. Nonetheless, progress in Raman spectroscopic analysis is still impeded by the lack of software, methodological and data standardisation, and the…