MAM: Multinomial Attention Masking for Foundation Models on Sparse Single-Cell RNA-seq Data
Published in Proceedings of the 7th Conference on Health, Inference, and Learning (CHIL), PMLR 333, 2026
Multinomial Attention Masking (MAM) uses learned attention maps to select informative gene positions for masking during foundation model pretraining. The study evaluates the approach on sparse single-cell RNA-seq data and reports improvements over uniform masking for cell-type classification.
Recommended citation: Naziri, A., Asgari, A., An, A., Sachlos, E., & Seyyed-Kalantari, L. (2026). MAM: Multinomial Attention Masking for Foundation Models on Sparse Single-Cell RNA-seq Data. Proceedings of the 7th Conference on Health, Inference, and Learning, PMLR 333, 295–311.
Download Paper