Skip to content

Masked Attention

Definition
AI-generated

Masked attention is self-attention (or a head within multi-head attention) where an attention mask prevents some positions from attending to others—most famously a causal mask in autoregressive decoders so token *i* cannot attend to future tokens *j > i*, and masks in bidirectional encoders that hide held-out spans for pretraining objectives (e.g. masked language modeling).

Topics
LLM and Agents Machine learning concepts
Related
Attention Mechanism Multi-Head Attention Pretraining

Why it matters in GWAS

Encoder–decoder setups (sequence to phenotype labels, protein structure heads) use masks to separate modalities or padding; mis-specified padding masks can leak batch structure into predictions. Literature on DNA transformers should state whether training used causal (GPT-style) or bidirectional (BERT-style) masking because transfer behavior differs.

Example usage

"The decoder used causal masked attention during fine-tuning so the model only conditioned on upstream sequence when scoring the variant site."

Attention Mechanism Multi-Head Attention Pretraining Self-Attention Transformer
Related-term network

Click any node to open that term page.

Node order. The current term is first. Next come every page linked from Related terms, in the same top-to-bottom order as that list. Then new terms appear in two-hop layers: for each of those direct neighbors (in that same order), their related terms are appended if not already listed. A third pass does the same for two-hop nodes. The walk stops when a fixed maximum number of nodes is reached.

References

  • Vaswani A, et al. (2017). Attention is all you need. NeurIPS.
← Markov chain Monte Carlo (MCMC) Maternally expressed gene (MEG) →

Last updated (UTC · Git history)