Rare Genomic Subtype Discovery from RNA-seq via Autoencoder Embeddings and Stability-Aware Clustering

Abstract

Unsupervised learning on high-dimensional RNA-seq data can reveal molecular subtypes beyond standard labels. We combine an autoencoder-based representation with clustering and stability analysis to search for rare but reproducible genomic subtypes. On the UCI "Gene Expression Cancer RNA-Seq" dataset (801 samples, 20,531 genes; BRCA, COAD, KIRC, LUAD, PRAD), a pan-cancer analysis shows clusters aligning almost perfectly with tissue-of-origin (Cram´er's V = 0.887), serving as a negative control. We therefore reframe the problem within KIRC (n = 146): we select the top 2,000 highly variable genes, standardize them, train a feed-forward autoencoder (128-dimensional latent space), and run k-means for k = 2 . . . 10. While global indices favor small k, scanning k with a pre-specified discovery rule (rare < 10% and stable with Jaccard ≥ 0.60 across 20 seeds after Hungarian alignment) yields a simple solution at k = 5 (silhouette = 0.129, DBI = 2.045) with a rare cluster C0 (6.85% of patients) that is highly stable (Jaccard = 0.787). Cluster-vsrest differential expression (Welch's t-test, Benjamini–Hochberg FDR) identifies coherent markers, including strong down-regulation of gene 11713, gene 13678, gene 16402, gene 3777 and up-regulation of gene 751, gene 17397, gene 2760. Overall, pan-cancer clustering is dominated by tissue-of-origin, whereas a stabilityaware within-cancer approach reveals a rare, reproducible KIRC subtype.