Skip to content

Multi-Head Attention

Definition
AI-generated

Multi-head attention runs several self-attention (or cross-attention) mechanisms in parallel with distinct learned projections, concatenating or mixing their outputs so the model can attend to different relationship patterns—syntax, long-range pairing, motif structure—within one layer.

Topics
LLM and Agents
Related
Attention Mechanism Genome-Wide Association Study (GWAS) Cross-Attention

Why it matters in GWAS

Sequence transformers for regulatory or protein modeling rely on multiple heads to capture diverse k-mer or contact patterns; ablation studies sometimes tie specific heads to motifs but claims need orthogonal validation.

Example usage

"A replication step checks whether Multi-Head Attention assumptions remain stable across cohorts."

Attention Mechanism Genome-Wide Association Study (GWAS) Cross-Attention Large Language Model (LLM) Masked Attention Self-Attention Transformer
Related-term network

Click any node to open that term page.

Node order. The current term is first. Next come every page linked from Related terms, in the same top-to-bottom order as that list. Then new terms appear in two-hop layers: for each of those direct neighbors (in that same order), their related terms are appended if not already listed. A third pass does the same for two-hop nodes. The walk stops when a fixed maximum number of nodes is reached.

References

  • Vaswani A, et al. (2017). Attention is all you need. NeurIPS.
← Multi-Ancestry Transcriptome-Wide Association Study Multi-Omic Single-Cell Integration →

Last updated (UTC · Git history)