Auto Chunking

Under AI → Auto Chunking every file bucket of the service is listed, with its chunking rules. A rule says which files it takes, the collection their chunks go to, and how big a chunk is. From then on every file uploaded to the bucket, by your app or by the console, is split by the first rule it matches. A bucket without rules is never touched.

The chunks get no vector from the rule. Their collection embeds them with its own rule, set under Auto Embeddings, like any other collection: one place for embedding, the collection where the vectors live and where you search.

The rule

Open a bucket and Add chunking rule:

Name: written on every chunk as rule.
Condition: which files the rule takes, below. Empty: every file.
Chunks into: the collection of the chunks, <bucket>_chunks by default, created empty on the first file if missing.
Advanced: chunk size in characters, 1000 by default; the overlap taken from the chunk before, 200; the splitter, auto to cut source code at function and class boundaries and the rest in windows, text for windows always, code for boundaries whenever the language is known.

Save writes the whole list on the bucket, as chunking in its metadata. The rules are tried in the order of the list, and the first one a file matches is applied, as in an ACL: the arrows move a rule up or down.

The condition

A JSON object with three keys, each optional; when there are more, all must match.

Key Matches

contentType

the type detected from the file’s content, a string or a list; a trailing matches any suffix: "application/vnd.openxmlformats-officedocument.".

extension

the extension of the file’s name, a string or a list, with or without the dot.

metadata

a MongoDB filter on the file’s metadata, the document your app sends with the upload. Every query operator works but those that run code.

The quick buttons under the field write the common ones: PDF, Office, Text, Markdown, HTML, Source code, By folder, By metadata.

By folder. A bucket has no folders: a folder is part of the name, which your app sends in the file’s metadata.filename. Two rules on a bucket docs send the contracts and the design documents to two collections:

{ "chunking": [
    { "name": "legal",
      "filter": { "metadata": { "filename": { "$regex": "^legals/" } } },
      "target-collection": "legal_chunks" },
    { "name": "tech",
      "filter": { "extension": ".pdf", "metadata": { "filename": { "$regex": "^design/" } } },
      "target-collection": "tech_chunks" } ] }

legals/contratto.pdf goes to legal_chunks, design/project.pdf to tech_chunks. The condition sees only what is in metadata: upload with the name there.

curl -X POST https://<service>/docs.files \
  -H "Authorization: Bearer <token>" \
  -F file=@contratto.pdf \
  -F 'metadata={"filename": "legals/contratto.pdf"}'

By metadata. A field of your own does the same without a convention on the name: {"metadata": {"area": "legal"}}, uploading with metadata={"filename": "contratto.pdf", "area": "legal"}.

Under the rule

Chunks: how many the rule wrote.
Embedding: the rule of the chunks collection, if it has one, with a link to Auto Embeddings. Without one the chunks have no vector: Add embedding opens Auto Embeddings on that collection with text into vector filled in and, for a contextual model, the chunks of a file grouped by fileId, so each vector knows the rest of its document.

Each chunk carries text, chunkIndex, fileId, source, filename, contentType, the file’s metadata and rule. Replacing or deleting a file replaces or deletes its chunks.

Upload a file to try

Under the rules, choose a file and, if you want, a folder: the file is uploaded to the bucket with metadata.filename set to the folder and the name, as your app would. The page says which rule it matched, shows its chunks and whether they have a vector, and repeats the warnings of the upload. The file stays in the bucket: Delete the file and its chunks removes both.

Limits

The chunks of a file are written in one request, with the limits of any request. On a free service a file is up to 2 MB, and so are its chunks together: a file whose chunks exceed it is stored, its chunks are not, and the warning of the upload says so. Free is for trying; for a real archive, a shared or dedicated service. The chunks and their vectors count toward the storage of the plan.

What it costs

Splitting is free. Embedding the chunks is billed by your provider to your account, one embedding per chunk, or one call per file with a contextual model grouped by fileId.