Reframing ontology fact acquisition during large language model fine-tuning as a time-to-event process

Abstract

Introduction Large language models can be fine-tuned to improve retrieval of biomedical facts that are not reliably accessible after pretraining. We used ontology term-identifier mappings from the Human Phenotype Ontology (HPO) and Gene Ontology (GO) as structured biomedical facts to study fact acquisition during fine-tuning. Methods Each ontology fact consisted of a term and its corresponding machine-readable identifier. We reframed fact acquisition as a discrete time-to-event process indexed by training epoch. Llama-3.1-8B Instruct was fine-tuned on HPO and GO ontology facts. Deterministic retrieval accuracy was assessed at baseline and after each fine-tuning epoch. Repeated stochastic decoding at baseline was used to probe for latent parametric support. Baseline-incorrect facts that were recovered at least once under stochastic decoding were classified as latent-knowledge-positive; facts not recovered under stochastic decoding were classified as latent-knowledge-negative. We distinguished trained-fact acquisition, defined for facts included in the fine-tuning set, from untrained-fact acquisition, defined for facts withheld from training. Fact loss was defined as the first transition from correct to incorrect retrieval among facts that were correct in the base model. Kaplan-Meier estimators were used to construct fact-acquisition and fact-loss curves over training epochs, and Cox proportional hazards models were used to identify predictors of acquisition rate. Results At baseline, the model correctly retrieved 1.1% (9/800) of HPO facts and 5.6% (44/802) of GO facts. After 20 epochs of supervised fine-tuning, correct retrieval increased to 71.9% (575/800) for HPO-trained facts, 61.8% (248/401) for GO-trained facts, and 11.2% (45/401) for GO-untrained facts. Latent-knowledge-positive facts were acquired more efficiently than latent-knowledge-negative facts. Corpus-based fact support in the biomedical literature had smaller positive effects on fact acquisition. Untrained-fact acquisition for withheld GO facts was uncommon, occurring in 5.8% (22/378) of baseline-incorrect withheld facts, but was more likely for latent-knowledge-positive facts.