Rewrite Configuration

Rewrite configuration (`rewrites.toml`)

tokf looks for a rewrites.toml file in two locations (first found wins):

Project-local: .tokf/rewrites.toml — scoped to the current repository
User-level: ~/.config/tokf/rewrites.toml — applies to all projects

This file controls custom rewrite rules, skip patterns, and pipe handling. All [pipe], [skip], and [[rewrite]] sections documented below go in this file.

Task runner integration (make, just)

Task runners like make and just execute recipe lines via a shell ($SHELL -c 'recipe_line'). By default, only the outer make/just command is visible to tokf — child commands (cargo test, uv run mypy, etc.) pass through unfiltered.

tokf solves this with built-in wrapper rules that inject tokf as the task runner’s shell. Each recipe line is then individually matched against installed filters:

# What you type:
make check

# What tokf rewrites it to:
make SHELL=tokf check

# What make then does for each recipe line:
tokf -c 'cargo test'          → filter matches → filtered output
tokf -c 'cargo clippy'        → filter matches → filtered output
tokf -c 'echo done'           → no filter → delegates to sh

For just, the --shell flag is used instead:

just test  →  just --shell tokf --shell-arg -cu test

Exit code preservation

Shell mode (tokf -c '...') always propagates the real exit code — no masking, no “Error: Exit code N” prefix. This means make sees the actual exit code from each recipe line and stops on failure as expected.

Shell mode (`tokf -c`)

When invoked as tokf -c 'command' (or with combined flags like -cu, -ec), tokf enters string mode. The command string is passed through the rewrite system, which rewrites matching commands to tokf run --no-mask-exit-code .... The rewritten command is then delegated to sh -c for execution. If no filter matches, the command is delegated to sh unchanged.

When invoked with multiple arguments after -c (e.g. tokf -c git status), tokf enters argv mode. Each argument is shell-escaped and joined into a command string, which is then processed the same way as string mode. This form is used by PATH shims.

Shell mode is not typically invoked directly; it is called by task runners (make, just) and PATH shims.

Compound and complex recipe lines

Compound commands (&&, ||, ;) are split at chain operators and each segment is individually rewritten. This means both halves of git add . && cargo test can be filtered. Pipes, redirections, and other shell constructs within each segment are handled by the rewrite system’s pipe stripping logic (see Piped commands) or passed through to sh unchanged.

Debugging task runner rewrites

Use tokf rewrite --verbose "make check" to confirm the wrapper rewrite is active and see which rule fired.

Shell mode also respects environment variables for diagnostics (since it has no access to CLI flags like --verbose):

TOKF_VERBOSE=1 make check     # print filter resolution details for each recipe line
TOKF_NO_FILTER=1 make check   # bypass filtering entirely, delegate all recipe lines to sh

Overriding or disabling wrappers

The built-in wrappers for make and just can be overridden or disabled via [[rewrite]] or [skip] entries in .tokf/rewrites.toml:

# Override the make wrapper with a custom one:
# "make check" → "make SHELL=tokf .SHELLFLAGS=-ec check"
# Note: use (?:[^\\s]*/)? prefix to also match full paths like /usr/bin/make
[[rewrite]]
match = "^(?:[^\\s]*/)?make(\\s.*)?$"
replace = "make SHELL=tokf .SHELLFLAGS=-ec{1}"

# Or disable it entirely:
[skip]
patterns = ["^make"]

Adding wrappers for other task runners

You can add wrappers for other task runners via [[rewrite]]. The exact mechanism depends on how the task runner invokes recipe lines — check its documentation for shell override options:

# Example: if your task runner respects $SHELL for recipe execution
[[rewrite]]
match = "^(?:[^\\s]*/)?mise run(\\s.*)?$"
replace = "SHELL=tokf mise run{1}"

Routing to generic commands

For commands that don’t have a dedicated filter, you can route them through generic commands (tokf err, tokf test, tokf summary) via rewrite rules:

# .tokf/rewrites.toml

# Build commands → error extraction
[[rewrite]]
match = "^mix compile"
replace = "tokf err {0}"

# Test runners → failure extraction
[[rewrite]]
match = "^mix test"
replace = "tokf test {0}"

# Long-running commands → heuristic summary
[[rewrite]]
match = "^terraform plan"
replace = "tokf summary {0}"

Note: User rewrite rules fire before filter matching. Only add these for commands that don’t already have a filter — check with tokf which "<command>". Commands with dedicated filters (e.g. cargo build, git status) produce better output through tokf run.

Piped commands

When a command is piped to a simple output-shaping tool (grep, tail, or head), tokf strips the pipe automatically and uses its own structured filter output instead. The original pipe suffix is passed to --baseline-pipe so token savings are still calculated accurately.

# These ARE rewritten — pipe is stripped, tokf applies its filter:
cargo test | grep FAILED
cargo test | tail -20
git diff HEAD | head -5

Multi-pipe chains, pipes to other commands, or pipe targets with unsupported flags are left unchanged:

# These are NOT rewritten — tokf leaves them alone:
kubectl get pods | grep Running | wc -l   # multi-pipe chain
cargo test | wc -l                        # wc not supported
cargo test | tail -f                      # -f (follow) not supported

If you want tokf to wrap a piped command that wouldn’t normally be rewritten, add an explicit rule to .tokf/rewrites.toml:

[[rewrite]]
match = "^cargo test \\| tee"
replace = "tokf run {0}"

Use tokf rewrite --verbose "cargo test | grep FAILED" to see how a command is being rewritten.

Disabling pipe stripping

If you prefer tokf to never strip pipes (leaving piped commands unchanged), add a [pipe] section to .tokf/rewrites.toml:

[pipe]
strip = false   # default: true

When strip = false, commands like cargo test | tail -5 pass through the shell unchanged. Non-piped commands are still rewritten normally.

Prefer less context mode

Sometimes the piped output (e.g. tail -5) is actually smaller than the filtered output. The prefer_less option tells tokf to compare both at runtime and use whichever is smaller:

[pipe]
prefer_less = true   # default: false

When a pipe is stripped, tokf injects --prefer-less alongside --baseline-pipe. At runtime:

The filter runs normally
The original pipe command also runs on the raw output
tokf prints whichever result is smaller

When the pipe output wins, the event is recorded with pipe_override = 1 in the tracking DB. The tokf gain command shows how many times this happened:

tokf gain summary
  total runs:     42
  input tokens:   12,500 est.
  output tokens:  3,200 est.
  tokens saved:   9,300 est. (74.4%)
  pipe preferred: 5 runs (pipe output was smaller than filter)

Note: strip = false takes priority — if pipe stripping is disabled, prefer_less has no effect.

Environment variable prefixes

Leading KEY=VALUE assignments are automatically stripped before matching, so env-prefixed commands are rewritten correctly:

# These ARE rewritten — env vars are preserved, the command is wrapped:
DEBUG=1 git status              → DEBUG=1 tokf run git status
RUST_LOG=debug cargo test       → RUST_LOG=debug tokf run cargo test
A=1 B=2 cargo test | tail -5   → A=1 B=2 tokf run --baseline-pipe 'tail -5' cargo test

The env vars are passed through verbatim to the underlying command; tokf only rewrites the executable portion.

Skip patterns and env var prefixes

User-defined skip patterns in .tokf/rewrites.toml match against the full shell segment, including any leading env vars. A pattern ^cargo will not skip RUST_LOG=debug cargo test because the segment doesn’t start with cargo:

[skip]
patterns = ["^cargo"]   # skips "cargo test" but NOT "RUST_LOG=debug cargo test"

To skip a command regardless of any env prefix, use a pattern that accounts for it:

[skip]
patterns = ["(?:^|\\s)cargo\\s"]   # matches "cargo" anywhere after start or whitespace