> ## Documentation Index
> Fetch the complete documentation index at: https://docs.jeanmemory.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Semantic IDs
> Turn a catalog into discrete semantic tokens that a generative recommender can actually generate.
**Spec-only preview.** The endpoints on this page describe what we are building. They are not live yet. [Reach out](mailto:jonathan@jeantechnologies.com) to shape the surface or join the closed beta.
Classical recommenders give every item a random integer ID and learn an embedding row for it. That works, and it has three costs that get worse with scale: the embedding table grows with the catalog, a new item is meaningless until it has interactions, and nothing learned about one catalog transfers to another.
Generative recommenders replace the random ID with a **semantic ID**: a short sequence of discrete codes derived from what the item actually is. Recommendation then becomes next-token prediction over a vocabulary where the tokens mean something.
```
item_8f3a2b →
coarse finer finest
```
The model is no longer retrieving from an index. It is *generating* an identifier, and every prefix of that identifier is a real region of item space.
## Why the tokenizer is the bottleneck
Most of the interesting variance in generative recommendation sits in how you build the codes, not in the model that consumes them. A tokenizer that collapses distinct items into the same code caps your ceiling before training starts. One that ignores behavior produces codes that are semantically tidy and commercially useless.
This is the part of the stack we think is underbuilt, and it is where this product sits.
## How the codes are built
Each item's text and images go through frozen encoders, one per modality, concatenated into a single content vector. Nothing about interactions yet.
An RQ-VAE assigns a code at level 1, takes the residual, assigns a code at level 2, and so on. Level 1 lands the item in a coarse neighborhood, later levels refine within it.
Co-occurrence in real user sequences pulls on the quantizer, so items that get consumed together land near each other in code space. See below.
Items that quantize identically get a disambiguating suffix, so a semantic ID always maps to exactly one item.
### Collaborative signal, and why not to fuse it directly
The obvious move is to concatenate a collaborative filtering embedding onto the content embedding before quantizing. It does not hold up. CF embeddings drift with popularity, so an item's semantic ID would change as it trends, and a code that means something different this month is not an identifier.
[PLUM](https://arxiv.org/abs/2510.07784) makes the better argument: use behavior as a **training objective** rather than an input feature. A co-occurrence contrastive loss on the quantizer pushes items that appear together in user sequences toward nearby codes, while the code itself stays a function of stable content. You get the collaborative structure without inheriting the drift.
[UTGRec](https://arxiv.org/abs/2504.04405) arrives at a similar place from the transfer direction, using co-occurrence alignment and reconstruction so a single tokenizer generalizes across domains rather than being refit per catalog.
| Approach | Code depends on | Drifts with popularity | Transfers |
| ----------------------------- | ------------------------- | ---------------------- | ------------------------- |
| Content only | Item content | No | Yes, but ignores behavior |
| CF embedding fused into input | Content plus interactions | Yes | Poorly |
| **Co-occurrence contrastive** | Item content | No | Yes |
### Multi-resolution codebooks
Uniform codebooks give every level the same cardinality, which wastes capacity: by level 4 there is very little residual entropy left to encode, and a wide codebook there mostly sits empty.
PLUM sizes the codebook as a function of depth, `2048 / 2^(level-1)`:
| Level | Codebook size | Encodes |
| ----- | ------------- | ---------------------- |
| 1 | 2048 | Coarse semantic region |
| 2 | 1024 | Sub-region |
| 3 | 512 | Fine distinction |
| 4 | 256 | Residual detail |
Paired with **progressive masking**, where training randomly truncates to the first `r` levels, this forces an actual hierarchy: the level-1 code has to be meaningful on its own, not just meaningful in combination with the levels below it. That property is what makes constrained beam search work well at decode time, because pruning on a prefix prunes a coherent region.
## The API
Four endpoints. Fit a tokenizer, check on it, assign IDs, generate.
```bash Fit a tokenizer theme={"dark"}
curl https://api.jeantechnologies.com/v1/tokenizers \
-H "Authorization: Bearer $JEAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "catalog-v1",
"catalog": { "uri": "s3://acme/catalog.jsonl" },
"codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
"collaborative_signal": {
"mode": "co_occurrence_contrastive",
"interactions": { "uri": "s3://acme/sequences.jsonl" }
}
}'
```
```bash Assign semantic IDs theme={"dark"}
curl https://api.jeantechnologies.com/v1/semantic-ids \
-H "Authorization: Bearer $JEAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"tokenizer_id": "tok_9k2m",
"items": [
{ "item_id": "sku_771", "title": "Merino crew socks",
"description": "Mid-weight, charcoal.",
"image_url": "https://acme.com/771.jpg" }
]
}'
```
```bash Generate theme={"dark"}
curl https://api.jeantechnologies.com/v1/generate \
-H "Authorization: Bearer $JEAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"tokenizer_id": "tok_9k2m",
"history": ["sku_310", "sku_884", "sku_771"],
"limit": 10
}'
```
A generation response returns real items, because beam search is constrained to prefixes that exist in the codebook:
```json theme={"dark"}
{
"recommendations": [
{
"item_id": "sku_402",
"codes": [1487, 302, 91, 12],
"tokens": "",
"score": 0.31
}
],
"beams_explored": 40,
"beams_pruned_invalid": 0
}
```
Full reference: [Fit tokenizer](/api-reference/semantic-ids/fit-tokenizer), [Get tokenizer](/api-reference/semantic-ids/get-tokenizer), [Assign semantic IDs](/api-reference/semantic-ids/assign-semantic-ids), [Generate](/api-reference/semantic-ids/generate).
## Cold start
This is the property that tends to matter most in practice. A semantic ID is a function of content, so an item that went live sixty seconds ago and has zero interactions still gets a code in the right neighborhood, and the model can recommend it immediately.
Pass a brand-new item to `POST /semantic-ids` and it is assigned against the existing codebooks without refitting. The response flags it:
```json theme={"dark"}
{ "item_id": "sku_new", "codes": [1487, 88, 405, 3], "cold_start": true }
```
Refit when the catalog's *distribution* shifts, not when it grows.
## What we are still deciding
Writing this page is partly how we are working out the product. Open questions, and we would rather hear from you than guess:
UTGRec argues for one tokenizer transferring across domains. A per-tenant fit is likely better on your catalog in isolation. The tradeoff is cold start quality on day one versus ceiling at maturity.
Some teams want `POST /generate` and nothing else. Others want to export semantic IDs and feed them into a ranker they already own. These imply fairly different products.
The whole approach assumes items carry real text or images. Catalogs where half the items are a bare SKU string change the design.
## References
* [PLUM: Adapting Pre-trained Language Models for Industrial-scale Generative Recommendations](https://arxiv.org/abs/2510.07784). SID-v2, co-occurrence contrastive collaborative signal, multi-resolution codebooks, progressive masking.
* [Universal Item Tokenization for Transferable Generative Recommendation](https://arxiv.org/abs/2504.04405) (UTGRec). Tree-structured codebooks, dual content decoders, cross-domain transfer.
* [MiniOneRec](https://github.com/AkaliKong/MiniOneRec). Open-source end to end pipeline: SID construction, supervised fine-tuning, then GRPO reinforcement learning with constrained beam search. Also hosts TS-Rec, implementing fine-grained semantics integration.
[jonathan@jeantechnologies.com](mailto:jonathan@jeantechnologies.com)