> ## Documentation Index
> Fetch the complete documentation index at: https://docs.jeanmemory.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Fit tokenizer
> Fit a semantic ID tokenizer on your catalog.
The one call that matters. Everything downstream inherits the quality of this fit.
Item content is encoded per modality and residually quantized into `levels` codebooks. Level 1 places the item in a coarse semantic region; each subsequent level refines within it. See [Semantic IDs](/products/semantic-ids) for the reasoning behind the defaults.
```bash curl theme={"dark"}
curl https://api.jeantechnologies.com/v1/tokenizers \
-H "Authorization: Bearer $JEAN_API_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "catalog-v1",
"catalog": { "uri": "s3://acme/catalog.jsonl" },
"modalities": ["text", "image"],
"codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
"collaborative_signal": {
"mode": "co_occurrence_contrastive",
"interactions": { "uri": "s3://acme/sequences.jsonl" },
"weight": 0.3
},
"progressive_masking": true
}'
```
```python Python theme={"dark"}
job = requests.post(
"https://api.jeantechnologies.com/v1/tokenizers",
headers={"Authorization": f"Bearer {os.environ['JEAN_API_KEY']}"},
json={
"name": "catalog-v1",
"catalog": {"uri": "s3://acme/catalog.jsonl"},
"codebook": {"levels": 4, "resolution": "multi", "base_size": 2048},
"collaborative_signal": {
"mode": "co_occurrence_contrastive",
"interactions": {"uri": "s3://acme/sequences.jsonl"},
},
},
).json()
```
```ts TypeScript theme={"dark"}
const res = await fetch("https://api.jeantechnologies.com/v1/tokenizers", {
method: "POST",
headers: {
Authorization: `Bearer ${process.env.JEAN_API_KEY}`,
"Content-Type": "application/json",
},
body: JSON.stringify({
name: "catalog-v1",
catalog: { uri: "s3://acme/catalog.jsonl" },
codebook: { levels: 4, resolution: "multi", base_size: 2048 },
collaborative_signal: {
mode: "co_occurrence_contrastive",
interactions: { uri: "s3://acme/sequences.jsonl" },
},
}),
});
const tokenizer = await res.json();
```
## Input format
`catalog` is JSONL or Parquet, one item per record:
```json theme={"dark"}
{"item_id": "sku_771", "title": "Merino crew socks", "description": "Mid-weight, charcoal.", "image_url": "https://acme.com/771.jpg", "attributes": {"brand": "Acme"}}
```
`collaborative_signal.interactions` is one user sequence per record, oldest first:
```json theme={"dark"}
{"user_id": "u_18", "items": ["sku_310", "sku_884", "sku_771"], "timestamps": [1735689600, 1736294400, 1736899200]}
```
## Choosing a codebook
`resolution: "multi"` sizes each level as `base_size / 2^(level-1)`. With the defaults:
| Level | Size | Encodes |
| ----- | ---- | ---------------------- |
| 1 | 2048 | Coarse semantic region |
| 2 | 1024 | Sub-region |
| 3 | 512 | Fine distinction |
| 4 | 256 | Residual detail |
Total addressable space is the product of the level sizes, so the defaults cover roughly 275 billion distinct IDs. The binding constraint is almost never address space. It is whether level 1 has enough distinct semantic regions in your catalog to fill 2048 codes.
Rule of thumb: start with `base_size` near the number of genuinely distinct categories in your catalog, rounded up to a power of two. Check `codebook_utilization` on the fitted tokenizer and adjust. Utilization below about 0.5 at level 1 means you oversized it.
`resolution: "uniform"` is available for parity with older RQ-VAE setups but is not recommended. By level 4 there is little residual entropy left to encode, and a full-width codebook there mostly sits unused.
## Response
Fitting is asynchronous. The call returns `202` immediately:
```json theme={"dark"}
{
"tokenizer_id": "tok_9k2m",
"name": "catalog-v1",
"status": "queued",
"codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
"created_at": "2026-08-31T18:04:11Z",
"ready_at": null
}
```
Poll [Get tokenizer](/api-reference/semantic-ids/get-tokenizer) until `status` is `ready`.
## OpenAPI
````yaml POST /tokenizers
openapi: 3.1.0
info:
title: Jean Technologies API
version: 1.0.0
description: >-
Foundation models of human behavior. Fit a semantic ID tokenizer on your
catalog, generate recommendations over it, and send outcomes back.
servers:
- url: https://api.jeantechnologies.com/v1
description: Production
security:
- bearerAuth: []
paths:
/tokenizers:
post:
tags:
- Semantic IDs
summary: Fit tokenizer
description: >-
Fit a semantic ID tokenizer on your catalog. Item content is encoded per
modality, then residually quantized into `codebook.levels` codebooks. If
`collaborative_signal` is enabled, co-occurrence in your interaction
sequences is applied as a contrastive objective on the quantizer, so
items consumed together land in nearby code regions while the code
itself stays a function of stable content.
Fitting is asynchronous. Poll `GET /tokenizers/{tokenizer_id}` for
status.
operationId: fitTokenizer
requestBody:
required: true
content:
application/json:
schema:
$ref: '#/components/schemas/FitTokenizerRequest'
example:
name: catalog-v1
catalog:
uri: s3://acme/catalog.jsonl
format: jsonl
modalities:
- text
- image
codebook:
levels: 4
resolution: multi
base_size: 2048
collaborative_signal:
mode: co_occurrence_contrastive
interactions:
uri: s3://acme/sequences.jsonl
format: jsonl
weight: 0.3
progressive_masking: true
responses:
'202':
description: Fit queued
content:
application/json:
schema:
$ref: '#/components/schemas/Tokenizer'
example:
tokenizer_id: tok_9k2m
name: catalog-v1
status: queued
codebook:
levels: 4
resolution: multi
base_size: 2048
created_at: '2026-08-31T18:04:11Z'
ready_at: null
'400':
$ref: '#/components/responses/BadRequest'
'401':
$ref: '#/components/responses/Unauthorized'
components:
schemas:
FitTokenizerRequest:
type: object
required:
- name
- catalog
properties:
name:
type: string
example: catalog-v1
catalog:
$ref: '#/components/schemas/DatasetRef'
modalities:
type: array
items:
type: string
enum:
- text
- image
default:
- text
- image
description: >-
Which item content to encode. Each modality gets its own frozen
encoder; outputs are concatenated before quantization.
codebook:
$ref: '#/components/schemas/CodebookConfig'
collaborative_signal:
$ref: '#/components/schemas/CollaborativeSignalConfig'
progressive_masking:
type: boolean
default: true
description: >-
Randomly truncate to the first r levels during training so each
level has to be meaningful on its own rather than only in
combination with deeper levels. This is what makes
prefix-constrained beam search prune a coherent region at decode
time.
Tokenizer:
type: object
properties:
tokenizer_id:
type: string
example: tok_9k2m
name:
type: string
example: catalog-v1
status:
type: string
enum:
- queued
- fitting
- ready
- failed
example: ready
codebook:
$ref: '#/components/schemas/CodebookConfig'
items_tokenized:
type: integer
example: 1840221
codebook_utilization:
type: array
items:
type: number
description: Fraction of each level's codebook in use, ordered by level.
example:
- 0.97
- 0.91
- 0.78
- 0.61
collision_rate:
type: number
description: >-
Fraction of items that required a disambiguating suffix to stay
uniquely addressable.
example: 0.004
created_at:
type: string
format: date-time
example: '2026-08-31T18:04:11Z'
ready_at:
type: string
format: date-time
nullable: true
example: '2026-08-31T19:22:47Z'
DatasetRef:
type: object
required:
- uri
properties:
uri:
type: string
description: >-
Location of the dataset. An `s3://` or `gs://` path, or an https URL
we have been granted read access to.
example: s3://acme/catalog.jsonl
format:
type: string
enum:
- jsonl
- parquet
default: jsonl
CodebookConfig:
type: object
description: Residual quantization layout.
properties:
levels:
type: integer
minimum: 1
maximum: 8
default: 4
description: >-
Number of residual quantization levels. Every semantic ID is this
many codes long.
resolution:
type: string
enum:
- multi
- uniform
default: multi
description: >-
`multi` halves codebook cardinality at each level (`base_size /
2^(level-1)`), matching the residual entropy actually left to encode
at depth. `uniform` gives every level `base_size` and tends to leave
deep codebooks mostly empty.
base_size:
type: integer
default: 2048
description: Cardinality of the level-1 codebook.
CollaborativeSignalConfig:
type: object
description: How behavioral signal shapes the quantizer.
properties:
mode:
type: string
enum:
- co_occurrence_contrastive
- none
default: co_occurrence_contrastive
description: >-
`co_occurrence_contrastive` applies co-occurrence as a contrastive
loss on the quantizer while the code stays a function of stable
content. Collaborative filtering embeddings are deliberately not
fused into the quantizer input: they drift with item popularity, and
an identifier that changes as an item trends is not an identifier.
interactions:
$ref: '#/components/schemas/DatasetRef'
weight:
type: number
default: 0.3
description: Weight of the contrastive term relative to content reconstruction.
Error:
type: object
properties:
error:
type: object
properties:
code:
type: string
example: invalid_request
message:
type: string
example: Field 'domain' is required.
details:
type: object
additionalProperties: true
responses:
BadRequest:
description: Invalid request
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
Unauthorized:
description: Missing or invalid API key
content:
application/json:
schema:
$ref: '#/components/schemas/Error'
securitySchemes:
bearerAuth:
type: http
scheme: bearer
description: API key issued by Jean Technologies. Contact the team for access.
````