Nvidia releases JEPA-DNA, a genomic foundation model with latent-space prediction

Nvidia's New Genomic Model

Nvidia has introduced JEPA-DNA, a new genomic foundation model, now available on Hugging Face. This release signifies a shift in AI approaches for structured fields like genomics, moving beyond the limitations of purely language-based models that primarily rely on token prediction.

Hybrid Architecture for DNA Understanding

JEPA-DNA incorporates a hybrid architecture that combines standard token-level DNA language modeling with a Joint Embedding Predictive Architecture (JEPA). This dual learning objective allows the model to not only reconstruct missing tokens but also to supervise the model's global sequence embedding in a latent space. This approach is similar to predictive architectures championed by Yann LeCun, Executive Chairman of AMI Labs.

Beyond Token Prediction

Traditional genomic models have often mirrored natural language processing (NLP) models, using masked language modeling (MLM) to predict missing literal tokens in DNA sequences. While effective for local syntax, this method often struggles with the broader functional meaning of a sequence. JEPA-DNA addresses this by predicting the functional representation of masked genomic segments rather than their character-by-character makeup.

Upgraded DNABERT-2

The newly released checkpoint, JEPA-DNA-DNABERT2, functions as a model-agnostic continual pre-training framework. It builds upon DNABERT-2, a 117 million-parameter model. By integrating JEPA, the model learns the basic structure of the data, producing representations that are more effective for biological tasks while retaining generative capabilities.