= Auto Chunking :nav-title: Auto Chunking :description: Rules on a file bucket that split the files uploaded to it into text chunks, in a collection you choose; that collection embeds them, and you search it like any other. :keywords: chunking, RAG, GridFS, file upload, PDF search, document chunks, contextual embeddings, voyage-context-4 :group: AI :order: 27 Under **AI → Auto Chunking** every file bucket of the service is listed, with its chunking rules. A rule says which files it takes, the collection their chunks go to, and how big a chunk is. From then on every file uploaded to the bucket, by your app or by the console, is split by the first rule it matches. A bucket without rules is never touched. The chunks get no vector from the rule. Their collection embeds them with its own rule, set under xref:auto-embeddings.adoc[Auto Embeddings], like any other collection: one place for embedding, the collection where the vectors live and where you search. == The rule Open a bucket and **Add chunking rule**: * **Name**: written on every chunk as `rule`. * **Condition**: which files the rule takes, below. Empty: every file. * **Chunks into**: the collection of the chunks, `_chunks` by default, created empty on the first file if missing. * **Advanced**: chunk size in characters, 1000 by default; the overlap taken from the chunk before, 200; the splitter, `auto` to cut source code at function and class boundaries and the rest in windows, `text` for windows always, `code` for boundaries whenever the language is known. **Save** writes the whole list on the bucket, as `chunking` in its metadata. The rules are tried in the order of the list, and the first one a file matches is applied, as in an ACL: the arrows move a rule up or down. [[the-condition]] == The condition A JSON object with three keys, each optional; when there are more, all must match. [cols="1,3"] |=== | Key | Matches | `contentType` | the type detected from the file's content, a string or a list; a trailing `*` matches any suffix: `"application/vnd.openxmlformats-officedocument.*"`. | `extension` | the extension of the file's name, a string or a list, with or without the dot. | `metadata` | a MongoDB filter on the file's `metadata`, the document your app sends with the upload. Every query operator works but those that run code. |=== The quick buttons under the field write the common ones: PDF, Office, Text, Markdown, HTML, Source code, By folder, By metadata. **By folder.** A bucket has no folders: a folder is part of the name, which your app sends in the file's `metadata.filename`. Two rules on a bucket `docs` send the contracts and the design documents to two collections: [source,json] ---- { "chunking": [ { "name": "legal", "filter": { "metadata": { "filename": { "$regex": "^legals/" } } }, "target-collection": "legal_chunks" }, { "name": "tech", "filter": { "extension": ".pdf", "metadata": { "filename": { "$regex": "^design/" } } }, "target-collection": "tech_chunks" } ] } ---- `legals/contratto.pdf` goes to `legal_chunks`, `design/project.pdf` to `tech_chunks`. The condition sees only what is in `metadata`: upload with the name there. [source] ---- curl -X POST https:///docs.files \ -H "Authorization: Bearer " \ -F file=@contratto.pdf \ -F 'metadata={"filename": "legals/contratto.pdf"}' ---- **By metadata.** A field of your own does the same without a convention on the name: `{"metadata": {"area": "legal"}}`, uploading with `metadata={"filename": "contratto.pdf", "area": "legal"}`. == Under the rule * **Chunks**: how many the rule wrote. * **Embedding**: the rule of the chunks collection, if it has one, with a link to Auto Embeddings. Without one the chunks have no vector: **Add embedding** opens Auto Embeddings on that collection with `text` into `vector` filled in and, for a contextual model, the chunks of a file grouped by `fileId`, so each vector knows the rest of its document. Each chunk carries `text`, `chunkIndex`, `fileId`, `source`, `filename`, `contentType`, the file's `metadata` and `rule`. Replacing or deleting a file replaces or deletes its chunks. == Upload a file to try Under the rules, choose a file and, if you want, a folder: the file is uploaded to the bucket with `metadata.filename` set to the folder and the name, as your app would. The page says which rule it matched, shows its chunks and whether they have a vector, and repeats the warnings of the upload. The file stays in the bucket: **Delete the file and its chunks** removes both. == Limits The chunks of a file are written in one request, with the limits of any request. On a free service a file is up to 2 MB, and so are its chunks together: a file whose chunks exceed it is stored, its chunks are not, and the warning of the upload says so. Free is for trying; for a real archive, a shared or dedicated service. The chunks and their vectors count toward the storage of the plan. == What it costs Splitting is free. Embedding the chunks is billed by your provider to your account, one embedding per chunk, or one call per file with a contextual model grouped by `fileId`.