Skip to main content
A Semantic ID is a short sequence of integer codes standing in for one thing. Similar things share a prefix, so the vocabulary carries meaning rather than assigning arbitrary indices.
Eight semantic levels and one collision suffix. The tokenizer card reports the actual layout. Keep your original item_id as the application key.

​
What you can tokenize

The tokenizer takes text and returns codes. It does not care what the text describes, which is the point. For a recommender this is a catalog. For a simulation it is the action space of a population. Both are the same call.

​
Fit one

  1. Import what you want encoded with text for every entry, or use a dataset already in your workspace. The shape is on Data format.
  2. Fit a tokenizer and poll its job.
  3. Read the codes, or export the tokenizer bundle to run it yourself.
  4. Use the codes in a Jean model, or train your own on that vocabulary.
The beta uses text embeddings and residual quantization. A bundle contains the entry mapping, encoder identity, codebooks, and token layout. Export is subject to artifact access terms.

​
Compatibility

Codes have meaning only within one immutable tokenizer version. Independently fitted codebooks are not interchangeable. Adding or changing entries creates a new version; existing mappings stay fixed. A generative model must explicitly support the tokenizer it decodes into. IDs identify things, not people. They do not establish identity links across datasets.