# chunk-wizard

Token-budget chunking library

- Role: Creator & Solo Developer
- Period: Jun 2026
- Status: released
- Domain: ai
- Type: case-study
- Open source: yes
- Source code: https://github.com/kilgor/chunk-wizard
- Human page: https://alicem.dev/projects/chunk-wizard

## Summary

A single-file Python utility that splits text into token-budget-aware chunks before it goes to an LLM. It detects the text's natural structure and picks the splitting strategy that keeps chunks as full as possible without exceeding the budget.

## Case study

### Problem

LLMs have a fixed context window. Feed them too much and they truncate silently or fail; split every N characters and you cut sentences in half and destroy structure. In my own pipelines, local models with deliberately limited context windows could not process the large documents going into a vector database.

### Approach

Measure first: the character budget comes from the model's context window, a share reserved for the reply, and an estimate of characters per token. Then detect the text's natural structure and try four strategies: Markdown headings, paragraphs, lines and sentences. Each produces a set of chunks scored by utilization, and the best-filled strategy wins.

### Architecture

One Python file with no dependencies. An in-memory function returns the chunks for a text; file-based functions split a file into numbered chunk files and reassemble them later; a third function only calculates chunk boundaries. Every operation returns the same result structure with success, data, error and metadata.

### Result

Released as open source under the MIT license and built for local LLM vector database pipelines. Its limits are documented: a single unit larger than the budget passes through whole, the token estimate is approximate, and only text files are supported.

## Highlights

- Token budget calculated from the model's context window, not arbitrary sizes
- Tries four strategies (heading, paragraph, line, sentence) and keeps the one with the best utilization
- No dependencies (Python standard library only); MIT licensed

## Technologies

Python, NLP, Local LLMs, Vector Databases