Skip to content

Token

Definition
AI-generated

A token is the atomic unit an LLM’s tokenizer produces from text—often a subword piece rather than a whole word.

Topics
LLM and Agents
Synonyms
Subword token (when applicable)
Plural
Tokens
Related
Context Window Genome-Wide Association Study (GWAS) Large Language Model (LLM)

Why it matters in GWAS

Long methods sections, full GWAS appendices, or pasted summary-statistics tables can exhaust context or budget; knowing token boundaries helps decide what to RAG-retrieve versus paste, and explains truncation artifacts in variant lists or citations.

Example usage

"We chunked the supplementary table so each chunk stayed under the provider’s per-request token cap."

Context Window Genome-Wide Association Study (GWAS) Large Language Model (LLM) Prompt Retrieval-Augmented Generation (RAG) Tokenization
Related-term network

Click any node to open that term page.

Node order. The current term is first. Next come every page linked from Related terms, in the same top-to-bottom order as that list. Then new terms appear in two-hop layers: for each of those direct neighbors (in that same order), their related terms are appended if not already listed. A third pass does the same for two-hop nodes. The walk stops when a fixed maximum number of nodes is reached.

References

  • Sennrich R, Haddow B, Birch A. (2016). Neural machine translation of rare words with subword units. ACL.
← Time-to-Event Tokenization →

Last updated (UTC · Git history)