> ## Documentation Index
> Fetch the complete documentation index at: https://docs.jeanmemory.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Tokenize anything

> Turn items, behaviors, or contexts into short integer codes a model can learn over.

A Semantic ID is a short sequence of integer codes standing in for one thing. Similar things
share a prefix, so the vocabulary carries meaning rather than assigning arbitrary indices.

```json theme={"dark"}
{"item_id":"sku_42","tokenizer_id":"tok_demo_v1","codes":[17,204,8,91,3,12,44,6,0]}
```

Eight semantic levels and one collision suffix. The tokenizer card reports the actual layout.
Keep your original `item_id` as the application key.

## What you can tokenize

The tokenizer takes text and returns codes. It does not care what the text describes, which is
the point.

| You have                         | You tokenize                  | So that                                                                          |
| -------------------------------- | ----------------------------- | -------------------------------------------------------------------------------- |
| A product or content catalog     | Item titles and descriptions  | A recommender predicts over a vocabulary where neighbors are related             |
| Logged actions from real people  | Descriptions of what was done | A simulator emits behavior as tokens instead of free text it cannot be scored on |
| Situations, prompts, or contexts | The context description       | A model conditions on where a person was, not just what they touched             |
| Survey or instrument responses   | The response text             | Open-ended answers join the same sequence as clicks and purchases                |

For a recommender this is a catalog. For a simulation it is the action space of a population.
Both are the same call.

## Fit one

1. [Import what you want encoded](/api/create-dataset) with text for every entry, or use a
   dataset already in your workspace. The shape is on [Data format](/data-format).
2. [Fit a tokenizer](/api/fit-tokenizer) and poll its job.
3. [Read the codes](/api/assign-semantic-ids), or export the
   [tokenizer bundle](/api/export-tokenizer) to run it yourself.
4. Use the codes in a [Jean model](/models), or train your own on that vocabulary.

The beta uses text embeddings and residual quantization. A bundle contains the entry mapping,
encoder identity, codebooks, and token layout. Export is subject to artifact access terms.

## Compatibility

Codes have meaning only within one immutable tokenizer version. Independently fitted codebooks
are not interchangeable. Adding or changing entries creates a new version; existing mappings stay
fixed. A generative model must explicitly support the tokenizer it decodes into.

IDs identify things, not people. They do not establish identity links across datasets.