Skip to content

Tokenization

Definition
AI-generated

Tokenization is the process of splitting raw input (text, sequence, or other discrete stream) into tokens—units a model consumes—using a fixed vocabulary, learned subword merges (BPE, SentencePiece), k-mers, codons, or domain-specific rules.

Topics
LLM and Agents Machine learning concepts
Related
Embedding Genome-Wide Association Study (GWAS) Large Language Model (LLM)

Why it matters in GWAS

Genomic and protein transformers inherit biases from k-mer or residue tokenization (stride, ambiguity codes, uppercase masking); text LLM workflows inherit tokenizer and vocabulary choices that affect rare allele symbols, rsIDs, and chemical names. Methods should name the tokenizer and vocabulary version alongside model checkpoints for reproducibility.

Example usage

"We used byte-pair tokenization for the trait descriptions but fixed 6-mer tokenization for the reference window fed to the DNA encoder."

Embedding Genome-Wide Association Study (GWAS) Large Language Model (LLM) Pretraining Token Transformer
Related-term network

Click any node to open that term page.

Node order. The current term is first. Next come every page linked from Related terms, in the same top-to-bottom order as that list. Then new terms appear in two-hop layers: for each of those direct neighbors (in that same order), their related terms are appended if not already listed. A third pass does the same for two-hop nodes. The walk stops when a fixed maximum number of nodes is reached.

References

  • Sennrich R, Haddow B, Birch A. (2016). Neural machine translation of rare words with subword units. ACL.
← Token Tool Calling →

Last updated (UTC · Git history)