> ## Documentation Index
> Fetch the complete documentation index at: https://docs.jeanmemory.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Get tokenizer

> Status and fit diagnostics for a tokenizer.

Poll this after [fitting](/api-reference/semantic-ids/fit-tokenizer). It is also where you find out whether the fit was any good.

```bash theme={"dark"}
curl https://api.jeantechnologies.com/v1/tokenizers/tok_9k2m \
  -H "Authorization: Bearer $JEAN_API_KEY"
```

```json theme={"dark"}
{
  "tokenizer_id": "tok_9k2m",
  "name": "catalog-v1",
  "status": "ready",
  "codebook": { "levels": 4, "resolution": "multi", "base_size": 2048 },
  "items_tokenized": 1840221,
  "codebook_utilization": [0.97, 0.91, 0.78, 0.61],
  "collision_rate": 0.004,
  "created_at": "2026-08-31T18:04:11Z",
  "ready_at": "2026-08-31T19:22:47Z"
}
```

## Reading the diagnostics

<AccordionGroup>
  <Accordion title="codebook_utilization" icon="chart-bar">
    Fraction of each level's codebook in use, ordered by level. Healthy fits are high at level 1 and taper with depth.

    Low utilization at **level 1** means the codebook is wider than your catalog is diverse. Reduce `base_size` and refit.

    Low utilization at the **deepest level** is normal and not worth chasing. There is little residual entropy left down there by design.
  </Accordion>

  <Accordion title="collision_rate" icon="git-merge">
    Fraction of items that quantized to identical codes and needed a disambiguating suffix. Suffixed items are still uniquely addressable, but they are effectively invisible to the semantics of the code, so a high rate erodes the whole premise.

    Under 0.01 is fine. Above roughly 0.05, add a level or widen `base_size`.
  </Accordion>

  <Accordion title="status: failed" icon="triangle-alert">
    Most common causes are an unreadable `catalog.uri`, item records missing both `title` and `description`, or an interactions file whose `item_id` values do not join to the catalog. The error body names which.
  </Accordion>
</AccordionGroup>

<Note>
  Fit time scales with catalog size. Roughly 30 to 90 minutes for a few million items, longer when `modalities` includes `image`.
</Note>


## OpenAPI

````yaml GET /tokenizers/{tokenizer_id}
openapi: 3.1.0
info:
  title: Jean Technologies API
  version: 1.0.0
  description: >-
    Foundation models of human behavior. Fit a semantic ID tokenizer on your
    catalog, generate recommendations over it, and send outcomes back.
servers:
  - url: https://api.jeantechnologies.com/v1
    description: Production
security:
  - bearerAuth: []
paths:
  /tokenizers/{tokenizer_id}:
    get:
      tags:
        - Semantic IDs
      summary: Get tokenizer
      description: >-
        Fetch a tokenizer's status and fit diagnostics. `codebook_utilization`
        reports the fraction of each level's codebook actually in use. Low
        utilization at level 1 usually means the catalog is less semantically
        diverse than the codebook is sized for, and `base_size` should come
        down.
      operationId: getTokenizer
      parameters:
        - name: tokenizer_id
          in: path
          required: true
          schema:
            type: string
            example: tok_9k2m
      responses:
        '200':
          description: Tokenizer
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/Tokenizer'
              example:
                tokenizer_id: tok_9k2m
                name: catalog-v1
                status: ready
                codebook:
                  levels: 4
                  resolution: multi
                  base_size: 2048
                items_tokenized: 1840221
                codebook_utilization:
                  - 0.97
                  - 0.91
                  - 0.78
                  - 0.61
                collision_rate: 0.004
                created_at: '2026-08-31T18:04:11Z'
                ready_at: '2026-08-31T19:22:47Z'
        '404':
          $ref: '#/components/responses/NotFound'
components:
  schemas:
    Tokenizer:
      type: object
      properties:
        tokenizer_id:
          type: string
          example: tok_9k2m
        name:
          type: string
          example: catalog-v1
        status:
          type: string
          enum:
            - queued
            - fitting
            - ready
            - failed
          example: ready
        codebook:
          $ref: '#/components/schemas/CodebookConfig'
        items_tokenized:
          type: integer
          example: 1840221
        codebook_utilization:
          type: array
          items:
            type: number
          description: Fraction of each level's codebook in use, ordered by level.
          example:
            - 0.97
            - 0.91
            - 0.78
            - 0.61
        collision_rate:
          type: number
          description: >-
            Fraction of items that required a disambiguating suffix to stay
            uniquely addressable.
          example: 0.004
        created_at:
          type: string
          format: date-time
          example: '2026-08-31T18:04:11Z'
        ready_at:
          type: string
          format: date-time
          nullable: true
          example: '2026-08-31T19:22:47Z'
    CodebookConfig:
      type: object
      description: Residual quantization layout.
      properties:
        levels:
          type: integer
          minimum: 1
          maximum: 8
          default: 4
          description: >-
            Number of residual quantization levels. Every semantic ID is this
            many codes long.
        resolution:
          type: string
          enum:
            - multi
            - uniform
          default: multi
          description: >-
            `multi` halves codebook cardinality at each level (`base_size /
            2^(level-1)`), matching the residual entropy actually left to encode
            at depth. `uniform` gives every level `base_size` and tends to leave
            deep codebooks mostly empty.
        base_size:
          type: integer
          default: 2048
          description: Cardinality of the level-1 codebook.
    Error:
      type: object
      properties:
        error:
          type: object
          properties:
            code:
              type: string
              example: invalid_request
            message:
              type: string
              example: Field 'domain' is required.
            details:
              type: object
              additionalProperties: true
  responses:
    NotFound:
      description: Resource not found
      content:
        application/json:
          schema:
            $ref: '#/components/schemas/Error'
  securitySchemes:
    bearerAuth:
      type: http
      scheme: bearer
      description: API key issued by Jean Technologies. Contact the team for access.

````