GitHub

Getting Started

Install tokf and run your first filtered command in minutes.

Installation

Homebrew (macOS and Linux)

brew install mpecan/tokf/tokf

cargo

cargo install tokf

Build from source

git clone https://github.com/mpecan/tokf
cd tokf
cargo build --release
# binary at target/release/tokf

How it works

tokf run git push origin main

tokf looks up a filter for git push, runs the command, and applies the filter. The filter logic lives in plain TOML files — no recompilation required. Anyone can author, share, or override a filter.

Compressing output is not the only thing being in the middle is good for. A pipeline reports its last stage’s exit code, so just check 2>&1 | tail -8 reports tail’s success even when the tests failed — and the shortcuts an agent reaches for to keep output small are exactly the ones that discard the failure signal. Opt into pipeline capture and tokf runs the first stage itself, so it can tell you when a pipeline hid a command’s verdict.

Set up automatic filtering

If you use an AI coding tool, install the hook so every command is filtered automatically — no tokf run prefix needed:

# Claude Code (recommended: --global so it works in every project)
tokf hook install --global

# OpenCode
tokf hook install --tool opencode --global

# OpenAI Codex CLI
tokf hook install --tool codex --global

Drop --global to install for the current project only. See Claude Code hook for details on each tool, the --path flag, and optional extras like the filter-authoring skill.

Usage

Run a command with filtering

tokf run git push origin main
tokf run cargo test
tokf run docker build .

Apply a filter to a fixture

tokf apply filters/git/push.toml tests/fixtures/git_push_success.txt --exit-code 0

Verify filter test suites

tokf verify                    # run all test suites
tokf verify git/push           # run a specific suite
tokf verify --list             # list available suites and case counts
tokf verify --json             # output results as JSON
tokf verify --require-all      # fail if any filter has no test suite
tokf verify --list --require-all  # show coverage per filter
tokf verify --scope project    # only project-local filters (.tokf/filters/)
tokf verify --scope global     # only user-level filters (~/.config/tokf/filters/)
tokf verify --scope stdlib     # only built-in stdlib (filters/ in CWD)
tokf verify --safety           # run safety checks (prompt injection, shell injection, hidden unicode)
tokf verify git/push --safety  # safety check a specific filter

Task runner filtering

tokf automatically wraps make and just so that each recipe line is individually filtered:

make check    # each recipe line (cargo test, cargo clippy, ...) is filtered
just test     # same — each recipe runs through tokf

See Rewrite configuration for details and customization.

Explore available filters

tokf ls                    # list all filters
tokf which "cargo test"    # which filter would match
tokf show git/push         # print the TOML source

Customize a built-in filter

tokf eject cargo/build            # copy to .tokf/filters/ (project-local)
tokf eject cargo/build --global   # copy to ~/.config/tokf/filters/ (user-level)

This copies the filter TOML and its test suite to your config directory, where it shadows the built-in. Edit the ejected copy freely — tokf’s priority system ensures your version is used instead of the original.

Flags

FlagDescription
--timingPrint how long filtering took
--verboseShow which filter was matched (also explains skipped rewrites)
--no-filterPass output through without filtering
--no-cacheBypass the filter discovery cache
--no-mask-exit-codeDisable exit-code masking. By default tokf exits 0 and prepends Error: Exit code N on failure. Also propagates into hook-emitted tokf run rewrites (tokf hook --no-mask-exit-code handle), including each segment of compound &&/;/|| commands
--preserve-colorPreserve ANSI color codes in filtered output (env: TOKF_PRESERVE_COLOR=1). See Color passthrough below
--baseline-pipePipe command for fair baseline accounting (injected by rewrite)
--prefer-lessCompare filtered vs piped output and use whichever is smaller (requires --baseline-pipe)

Color passthrough

By default, filters with strip_ansi = true permanently remove ANSI escape codes. The --preserve-color flag changes this: tokf strips ANSI internally for pattern matching (skip, keep, dedup) but restores the original colored lines in the final output. When --preserve-color is active it overrides strip_ansi = true in the filter config.

tokf does not force commands to emit color — you must ensure the child command outputs ANSI codes (e.g. via FORCE_COLOR=1 or --color=always):

# Node.js / Vitest / Jest
FORCE_COLOR=1 tokf run --preserve-color npm test

# Cargo
tokf run --preserve-color cargo test -- --color=always

# Or set the env var once for all invocations
export TOKF_PRESERVE_COLOR=1
FORCE_COLOR=1 tokf run npm test

Limitations: color passthrough applies to the skip/keep/dedup pipeline (stages 2–2.5). The match_output, parse, and lua_script stages operate on clean text and are unaffected by this flag. [[replace]] rules run on the raw text before the color split, so when --preserve-color is enabled their patterns may need to account for ANSI escape codes, similar to branch-level skip patterns, which also match against the restored colored text.

Built-in filter library

FilterCommand
git/addgit add
git/commitgit commit
git/diffgit diff — runs the real git diff and summarises it as one line per file (src/main.rs | +4 -3) plus a totals line, so the full patch stays recoverable with tokf raw. Pass -p/--patch/--stat/-U<n>/--name-only/--name-status/--numstat/--shortstat/--raw to skip the filter and get the requested format instead
git/loggit log — runs the real git log and renders one line per commit (<short-sha> <subject>), capped at 20; the full history stays recoverable with tokf raw. Pass -p/--patch/--format/--pretty/--graph/--stat/--shortstat/--dirstat/--oneline/--name-only/--name-status/-L to skip the filter. Empty results emit a one-line hint pointing at common causes (untracked pathspec, missing --all, missing --follow) instead of nothing — this stops agents looping through flag variations trying to escape a non-existent filter
git/pushgit push
git/showgit show — runs the real git show and renders the commit’s sha, subject, author and date plus one line per file with change counts; the full patch stays recoverable with tokf raw. Pass -p/--patch/--stat/--format/--pretty/--numstat/--shortstat/--raw to skip the filter
git/statusgit status — runs git status --porcelain=v1 -b -uall --find-renames; shows branch + upstream sync state ([synced], [ahead N], [behind N], (no upstream)) and one porcelain line per changed file (M src/main.rs, ?? scratch.rs, R old.rs -> new.rs). -uall lists every untracked file individually instead of collapsing newly-created directories. When 3+ files share a directory prefix the listing is restructured into a directory tree (see [tree]), writing each shared prefix once. Measured 24.4% averaged token reduction across the bundled test fixtures
cargo/buildcargo build
cargo/checkcargo check
cargo/clippycargo clippy
cargo/fmtcargo fmt
cargo/installcargo install *
cargo/testcargo test
docker/*docker build, docker compose, docker images, docker ps
npm/runnpm run *
npm/testnpm test, pnpm test, yarn test (with vitest/jest variants)
pnpm/*pnpm add, pnpm install
go/*go build, go vet
gradle/*gradle build, gradle test, gradle dependencies
gh/*gh pr list, gh pr view, gh pr checks, gh issue list, gh issue view
kubectl/*kubectl get pods
next/*next build
prisma/*prisma generate
pytestPython test runner — runs pytest as typed and keeps the failing assertion lines (> / E) plus the pass/fail summary; the full tracebacks stay recoverable with tokf raw. Pass -q/--tb/-v/-x/--collect-only/--pdb to skip the filter
tscTypeScript compiler
lsls