Skip to content

Attention Mechanism

Definition
AI-generated

An attention mechanism reweights inputs (tokens, positions, or features) based on learned compatibility scores, producing context-dependent representations; self-attention relates all positions to each other and is the core of transformer architectures.

Topics
LLM and Agents
Related
Cross-Attention Large Language Model (LLM) Masked Attention

Why it matters in GWAS

Attention layers appear in sequence models for DNA and RNA, protein language models, and multimodal biomedical foundation models used for annotation or literature mining. Attention weights are sometimes interpreted as biological importance but are not guaranteed causal maps without experiments.

Example usage

"In our variant-prioritization model, an attention mechanism upweighted enhancer features near immune genes, improving held-out prediction compared with uniform feature weighting."

Cross-Attention Large Language Model (LLM) Masked Attention Multi-Head Attention Neural Network Self-Attention Transformer
Related-term network

Click any node to open that term page.

Node order. The current term is first. Next come every page linked from Related terms, in the same top-to-bottom order as that list. Then new terms appear in two-hop layers: for each of those direct neighbors (in that same order), their related terms are appended if not already listed. A third pass does the same for two-hop nodes. The walk stops when a fixed maximum number of nodes is reached.

References

  • Vaswani A, et al. (2017). Attention is all you need. NeurIPS.
← Anatomical Therapeutic Chemical Code (ATC) Autoencoder →

Last updated (UTC · Git history)