Skip to content

Pretraining

Definition
AI-generated

Pretraining is the first, large-scale training phase where a model learns general representations from broad data—next-token prediction for text, masked modeling, contrastive image objectives, or denoising autoencoding—before fine-tuning or prompting on a downstream task.

Topics
LLM and Agents Machine learning concepts
Related
BERT (Bidirectional Encoder Representations from Transformers) Deep Learning Embedding

Why it matters in GWAS

DNA, protein, and single-cell foundation models rely on pretraining corpora and tokenization choices that shape inductive bias; downstream GWAS interpretation still requires association study design independent of the model’s pretraining domain.

Example usage

"The methods explicitly include Pretraining to support interpretation of the main findings."

BERT (Bidirectional Encoder Representations from Transformers) Deep Learning Embedding Fine-Tuning Foundation Model Genome-Wide Association Study (GWAS) Single-Cell Foundation Model Tokenization Transformer Transfer Learning
Related-term network

Click any node to open that term page.

Node order. The current term is first. Next come every page linked from Related terms, in the same top-to-bottom order as that list. Then new terms appear in two-hop layers: for each of those direct neighbors (in that same order), their related terms are appended if not already listed. A third pass does the same for two-hop nodes. The walk stops when a fixed maximum number of nodes is reached.

References

  • Devlin J, et al. (2019). BERT: pre-training of deep bidirectional transformers for language understanding. NAACL.
  • Brown T, et al. (2020). Language models are few-shot learners. NeurIPS.
← Preregistration Prevalence →

Last updated (UTC · Git history)