22 papers · ranked by Valyu relevance
Michael Pradel, Cristian Cadar, Islem Bouzenia
Numerous software analysis tools exist today, yet applying them to diverse open-source projects remains challenging due to environment setup, dependency resolution, and tool configuration. LLM-based agents offer a potential solution, yet no prior work has systematically studied their effectiveness on the specific task…
Lukas M. Weber, Wouter Saelens, Robrecht Cannoodt, Charlotte Soneson + 5 more
'Alexander Hapfelmeier' 'Paul P. Gardner' 'Anne‐Laure Boulesteix' 'Yvan Saeys' 'Mark D. Robinson'] In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the…
Anthony Sonrel, Almut Luetge, Charlotte Soneson, Izaskun Mallona + 16 more
Computational methods represent the lifeblood of modern molecular biology. Benchmarking is important for all methods, but with a focus here on computational methods, benchmarking is critical to dissect important steps of analysis pipelines, formally assess performance across common situations as well as edge cases, and…
Lukas M. Weber, Wouter Saelens, Robrecht Cannoodt, Charlotte Soneson + 5 more
'Alexander Hapfelmeier' 'Paul P. Gardner' 'Anne-Laure Boulesteix' 'Yvan Saeys' 'Mark D. Robinson'] In computational biology and other sciences, researchers are frequently faced with a choice between several computational methods for performing data analyses. Benchmarking studies aim to rigorously compare the…
Elisa Drelie Gelasca, Boguslaw Obara, Dmitry Fedorov, Kristian Kvilekval + 1 more
Background We present a biosegmentation benchmark that includes infrastructure, datasets with associated ground truth, and validation methods for biological image analysis. The primary motivation for creating this resource comes from the fact that it is very difficult, if not impossible, for an end-user to choose from…
Xiaoqi Cabiria Liang, Nick Robertson, Marni Torkel, Sanghyun Kim + 3 more
The rapid growth of computational methods for the computational biology field highlights the critical role of benchmarking in guiding method selection. However, there is no standardised data structure that effectively links and stores datasets, performance metrics and available ground truth. Without such a unified and…
Martin Grambow, Christoph Laaber, Philipp Leitner, David Bermbach + 1 more
'Muhammad Aleem'] Performance problems in applications should ideally be detected as soon as they occur, i.e., directly when the causing code modification is added to the code repository. To this end, complex and cost-intensive application benchmarks or lightweight but less relevant microbenchmarks can be added to…
Maxwell Venetos, Masha Elkin, Connor Delaney, John Hartwig + 1 more
NMR spectroscopy is an important analytical technique in synthetic organic chemistry, but its integration into high-throughput experimentation workflows has been limited by the necessity to manually analyze NMR spectra of new chemical entities. Current efforts to automate the analysis of NMR spectra rely on comparisons…
Yue Cao, Lijia Yu, Marni Torkel, Sanghyun Kim + 5 more
With the rapid development of computational methods for single-cell sequencing data, benchmarking serves as a valuable resource. As the number of benchmarking studies surges, it is timely to assess the current state of the field. We conducted a systematic literature search and assessed 282 papers, including all 130…
Yue Cao, Lijia Yu, Marni Torkel, Sanghyun Kim + 5 more
With the rapid development of computational methods for single-cell sequencing data, benchmarking serves a valuation resource. As the number of benchmarking studies surges, it is timely to assess the current state of the field. We conducted a systematic literature search and assessed 245 papers, including all 95…
Ludovico Mitchener, Jon M Laurent, Benjamin Tenmann, Siddharth Narayanan + 4 more
'Siddharth Narayanan' 'Geemi P Wellawatte' 'Andrew White' 'Lorenzo Sani' 'Samuel G Rodriques'] Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research. Existing benchmarks for measuring this potential and guiding future development continue to evolve from pure recall and…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Nils Japke, Christoph Witzko, Martin Grambow, David Bermbach
In this paper, we use a testbed microservice application which includes three performance issues to study the detection capabilities of both approaches. In extensive benchmarking experiments, we increase the severity of each performance issue stepwise, run both an application benchmark and the microbenchmark suite, and…
Alexander L. R. Lubbock, Carlos F. Lopez
Computational systems biology analyses typically make use of multiple software and their dependencies, which often run across heterogeneous compute environments. This can introduce differences in performance and reproducibility. Capturing metadata (e.g. package versions, GPU model) currently requires repetitious code…
Rafael Salles, Ricardo Farias
The Internet of Things (IoT) envisions billions of everyday objects sharing information. As new devices, applications and communication protocols are proposed for the IoT context, their evaluation, comparison, tuning and optimization become crucial and raise the need for a proper benchmark. While edge computing aims to…
Christoph Laaber, Harald C. Gall, Philipp Leitner
Regression testing comprises techniques which are applied during software evolution to uncover faults effectively and efficiently. While regression testing is widely studied for functional tests, performance regression testing, e.g., with software microbenchmarks, is hardly investigated. Applying test case…
Martin Grambow, Tobias Pfandzelter, Luk Burchard, Carsten J. Schubert + 2 more
'Carsten J. Schubert' 'Max Zhao' 'David Bermbach'] In this paper, we design and present BeFaaS, an extensible application-centric benchmarking framework for FaaS environments that focuses on the evaluation of FaaS platforms through realistic and typical examples of FaaS applications. BeFaaS includes a built-in…
Kobi Felton, Jan Rittig, Alexei Lapkin
In the fine chemicals industry, reaction screening and optimisation are essential to development of new products. However, this screening can be extremely time and labor intensive, especially when intuition is used. Machine learning offers a solution through iterative suggestions of new experiments based on past…
Authors not listed
The construction of large benchmark sets has accelerated advancement of quantum chemistry methods, especially in density functional theory and lower-cost methods. However, these large benchmark sets can be unsuitable for cutting-edge method development, because research codes developed for fundamentally new approaches…
Roland Pellegrini, Igor Ivkić, Markus Tauber
Cloud Service Providers deliver their products in form of "as-a-Service", which are typically categorized by the level of abstraction. This approach hides the implementation details and shows only functionality to the user. However, the problem is that it is hard to measure the performance of Cloud services, because…
Authors not listed
Alchemical Free Energy methods are gaining traction in computer-aided drug discovery. An expanding array of methodologies is available for the setup, execution and analysis of relative binding free energy (RBFE) calculations. However, the sharing of algorithms and protocols developed by different organisations is often…
Dimitar Georgiev, Simon Vilms Pedersen, Ruoxiao Xie, Álvaro Fernández-Galiana + 2 more
Raman spectroscopy is a non-destructive and label-free chemical analysis technique, which plays a key role in the analysis and discovery cycle of various branches of science. Nonetheless, progress in Raman spectroscopic analysis is still impeded by the lack of software, methodological and data standardisation, and the…