Rewrite Configuration

Rewrite configuration (`rewrites.toml`)

tokf looks for a rewrites.toml file in two locations (first found wins):

Project-local: .tokf/rewrites.toml — scoped to the current repository
User-level: ~/.config/tokf/rewrites.toml — applies to all projects

This file controls custom rewrite rules, skip patterns, and pipe handling. All [pipe], [skip], and [[rewrite]] sections documented below go in this file.

Task runner integration (make, just)

Task runners like make and just execute recipe lines via a shell ($SHELL -c 'recipe_line'). By default, only the outer make/just command is visible to tokf — child commands (cargo test, uv run mypy, etc.) pass through unfiltered.

tokf solves this with built-in wrapper rules that inject tokf as the task runner’s shell. Each recipe line is then individually matched against installed filters:

# What you type:
make check

# What tokf rewrites it to:
make SHELL=tokf check

# What make then does for each recipe line:
tokf -c 'cargo test'          → filter matches → filtered output
tokf -c 'cargo clippy'        → filter matches → filtered output
tokf -c 'echo done'           → no filter → delegates to sh

For just, the --shell flag is used instead:

just test  →  just --shell tokf --shell-arg -cu test

Exit code preservation

Shell mode (tokf -c '...') always propagates the real exit code — no masking, no “Error: Exit code N” prefix. This means make sees the actual exit code from each recipe line and stops on failure as expected.

Shell mode (`tokf -c`)

When invoked as tokf -c 'command' (or with combined flags like -cu, -ec), tokf enters string mode. The command string is passed through the rewrite system, which rewrites matching commands to tokf run --no-mask-exit-code .... The rewritten command is then delegated to sh -c for execution. If no filter matches, the command is delegated to sh unchanged.

When invoked with multiple arguments after -c (e.g. tokf -c git status), tokf enters argv mode. Each argument is shell-escaped and joined into a command string, which is then processed the same way as string mode. This form is used by PATH shims.

Shell mode is not typically invoked directly; it is called by task runners (make, just) and PATH shims.

Compound and complex recipe lines

Compound commands (&&, ||, ;) are split at chain operators and each segment is individually rewritten. This means both halves of git add . && cargo test can be filtered. Pipes, redirections, and other shell constructs within each segment are handled by the rewrite system’s pipe stripping logic (see Piped commands) or passed through to sh unchanged.

Debugging task runner rewrites

Use tokf rewrite --verbose "make check" to confirm the wrapper rewrite is active and see which rule fired.

Shell mode also respects environment variables for diagnostics (since it has no access to CLI flags like --verbose):

TOKF_VERBOSE=1 make check     # print filter resolution details for each recipe line
TOKF_NO_FILTER=1 make check   # bypass filtering entirely, delegate all recipe lines to sh

Overriding or disabling wrappers

The built-in wrappers for make and just can be overridden or disabled via [[rewrite]] or [skip] entries in .tokf/rewrites.toml:

# Override the make wrapper with a custom one:
# "make check" → "make SHELL=tokf .SHELLFLAGS=-ec check"
# Note: use (?:[^\\s]*/)? prefix to also match full paths like /usr/bin/make
[[rewrite]]
match = "^(?:[^\\s]*/)?make(\\s.*)?$"
replace = "make SHELL=tokf .SHELLFLAGS=-ec{1}"

# Or disable it entirely:
[skip]
patterns = ["^make"]

Adding wrappers for other task runners

You can add wrappers for other task runners via [[rewrite]]. The exact mechanism depends on how the task runner invokes recipe lines — check its documentation for shell override options:

# Example: if your task runner respects $SHELL for recipe execution
[[rewrite]]
match = "^(?:[^\\s]*/)?mise run(\\s.*)?$"
replace = "SHELL=tokf mise run{1}"

Routing to generic commands

For commands that don’t have a dedicated filter, you can route them through generic commands (tokf err, tokf test, tokf summary) via rewrite rules:

# .tokf/rewrites.toml

# Build commands → error extraction
[[rewrite]]
match = "^mix compile"
replace = "tokf err {0}"

# Test runners → failure extraction
[[rewrite]]
match = "^mix test"
replace = "tokf test {0}"

# Long-running commands → heuristic summary
[[rewrite]]
match = "^terraform plan"
replace = "tokf summary {0}"

Note: User rewrite rules fire before filter matching. Only add these for commands that don’t already have a filter — check with tokf which "<command>". Commands with dedicated filters (e.g. cargo build, git status) produce better output through tokf run.

Piped commands

When a command is piped to a simple output-shaping tool (grep, tail, or head), tokf strips the pipe automatically and uses its own structured filter output instead. The original pipe suffix is passed to --baseline-pipe so token savings are still calculated accurately.

# These ARE rewritten — pipe is stripped, tokf applies its filter:
cargo test | grep FAILED
cargo test | tail -20
git diff HEAD | head -5

Multi-pipe chains, pipes to other commands, or pipe targets with unsupported flags are left unchanged:

# These are NOT rewritten — tokf leaves them alone:
kubectl get pods | grep Running | wc -l   # multi-pipe chain
cargo test | wc -l                        # wc not supported
cargo test | tail -f                      # -f (follow) not supported

If you want tokf to wrap a piped command that wouldn’t normally be rewritten, add an explicit rule to .tokf/rewrites.toml:

[[rewrite]]
match = "^cargo test \\| tee"
replace = "tokf run {0}"

Use tokf rewrite --verbose "cargo test | grep FAILED" to see how a command is being rewritten.

Disabling pipe stripping

If you prefer tokf to never strip pipes (leaving piped commands unchanged), add a [pipe] section to .tokf/rewrites.toml:

[pipe]
strip = false   # default: true

When strip = false, commands like cargo test | tail -5 pass through the shell unchanged. Non-piped commands are still rewritten normally.

Prefer less context mode

Sometimes the piped output (e.g. tail -5) is actually smaller than the filtered output. The prefer_less option tells tokf to compare both at runtime and use whichever is smaller:

[pipe]
prefer_less = true   # default: false

When a pipe is stripped, tokf injects --prefer-less alongside --baseline-pipe. At runtime:

The filter runs normally
The original pipe command also runs on the raw output
tokf prints whichever result is smaller

When the pipe output wins, the event is recorded with pipe_override = 1 in the tracking DB. The tokf gain command shows how many times this happened:

tokf gain summary
  total runs:     42
  input tokens:   12,500 est.
  output tokens:  3,200 est.
  tokens saved:   9,300 est. (74.4%)
  pipe preferred: 5 runs (pipe output was smaller than filter)

Note: strip = false takes priority — if pipe stripping is disabled, prefer_less has no effect.

External permission engine

By default, tokf does no permission checking — the AI tool (Claude Code, Gemini, Cursor) handles its own deny/ask rules natively. tokf only rewrites commands that match a filter and auto-allows them.

You can optionally delegate permission decisions to an external process — a “sub-hook” that performs deeper semantic analysis of commands. When configured, the engine is consulted on every command, not just ones tokf has a filter for. This lets the engine auto-approve safe commands without the user being prompted.

Configuration

Add a [permissions] section to .tokf/rewrites.toml:

[permissions]
engine = "external"

[permissions.external]
command = "dippy"
args = ["hook", "handle", "--mode", "{format}"]
timeout_ms = 3000    # default: 5000
on_error = "builtin" # what to do if the engine fails

Tool format (`{format}`)

The {format} placeholder in args is replaced with the AI tool identifier before spawning:

Tool	Default value
Claude Code	`claude-code`
Gemini CLI	`gemini`
Cursor	`cursor`

If the engine expects different names, add a format_map:

[permissions.external]
command = "my-engine"
args = ["check", "--tool", "{format}"]
format_map = { "claude-code" = "claude", "gemini" = "google" }

Protocol

tokf spawns the engine process (command + args, with {format} resolved)
The original hook JSON (from the AI tool) is written to the engine’s stdin
The engine returns a standard hook response JSON on stdout — the same format the AI tool expects
tokf extracts the permission decision from the response and applies it to its own rewritten command

The engine sees the original command (not the rewritten one). tokf controls the rewrite; the engine controls the permission decision. Engines that exit with a non-zero status but still produce valid JSON will have their JSON verdict honoured — this supports engines like Dippy that use exit codes for signalling alongside a valid response.

Hook JSON reference (for engine developers)

The engine receives the AI tool’s hook JSON verbatim on stdin. The format depends on the tool:

Claude Code (--mode claude-code):

{"tool_name": "Bash", "tool_input": {"command": "git push --force"}}

Expected response — set permissionDecision to "allow", "deny", or "ask" (or omit for ask):

{
  "hookSpecificOutput": {
    "hookEventName": "PreToolUse",
    "permissionDecision": "allow",
    "updatedInput": {"command": "git push --force"}
  }
}

Gemini CLI (--mode gemini):

{"tool_name": "run_shell_command", "tool_input": {"command": "git push --force"}}

Expected response — set decision to "allow", "deny", or "ask":

{
  "decision": "allow",
  "hookSpecificOutput": {
    "tool_input": {"command": "git push --force"}
  }
}

Cursor (--mode cursor):

{"command": "git push --force"}

Expected response — set permission to "allow", "deny", or "ask":

{
  "permission": "allow",
  "updated_input": {"command": "git push --force"}
}

Error handling (`on_error`)

When the engine fails (crash, timeout, invalid output), the on_error field determines the fallback:

Value	Behaviour
`"ask"` (default)	Fail closed — prompt user for permission
`"allow"`	Fail open — auto-allow the command
`"builtin"`	Fall back to built-in deny/ask rule matching

Example: Dippy integration

Dippy is a permission engine that performs deep semantic analysis of bash commands — auto-approving safe commands while blocking dangerous ones.

[permissions]
engine = "external"

[permissions.external]
command = "dippy"
args = ["hook", "handle", "--mode", "{format}"]
timeout_ms = 3000
on_error = "builtin"

Environment variable prefixes

Leading KEY=VALUE assignments are automatically stripped before matching, so env-prefixed commands are rewritten correctly:

# These ARE rewritten — env vars are preserved, the command is wrapped:
DEBUG=1 git status              → DEBUG=1 tokf run git status
RUST_LOG=debug cargo test       → RUST_LOG=debug tokf run cargo test
A=1 B=2 cargo test | tail -5   → A=1 B=2 tokf run --baseline-pipe 'tail -5' cargo test

The env vars are passed through verbatim to the underlying command; tokf only rewrites the executable portion.

Skip patterns and env var prefixes

User-defined skip patterns in .tokf/rewrites.toml match against the full shell segment, including any leading env vars. A pattern ^cargo will not skip RUST_LOG=debug cargo test because the segment doesn’t start with cargo:

[skip]
patterns = ["^cargo"]   # skips "cargo test" but NOT "RUST_LOG=debug cargo test"

To skip a command regardless of any env prefix, use a pattern that accounts for it:

[skip]
patterns = ["(?:^|\\s)cargo\\s"]   # matches "cargo" anywhere after start or whitespace

Implicit skip rules

Three implicit skip rules are always active and can’t be disabled. They cover cases where rewriting would silently corrupt the agent’s data, so there’s no plausible reading under which the rewrite would be correct.

Heredocs

Commands that contain a top-level heredoc (<<EOF, <<-EOF) are passed through unchanged. Wrapping them with tokf run would break the lexical binding between the command and its heredoc body.

# Not rewritten — heredoc body would be cut off:
cat <<EOF > /tmp/cfg.yaml
key: value
EOF

Heredocs inside command substitution

Commands that contain a heredoc anywhere inside a $(...) (or backtick) command substitution are also skipped — even when the substitution is buried deep inside an argument like git commit -m "$(cat <<'EOF' … EOF)". The heredoc body lives in a logically-separate region of the source from the surrounding command, and downstream re-tokenization (clap argv parsing in tokf run, second-pass shell parsers, byte-offset pipe stripping) can slice through it. The canonical failure mode this prevents is git commit -m "$(cat <<'EOF' … EOF)" 2>&1 | tail -10 mangling -m’s value into git: error: switch 'm' requires a value.

# Not rewritten — multi-line commit messages, gh PR bodies, etc.:
git add foo && git commit -m "$(cat <<'EOF'
feat: a thing

with multi-line body
EOF
)" && git push

gh pr create -b "$(cat <<EOF
## Summary
…
EOF
)"

Unlike the output-redirect skip below, this rule fires for the whole compound: if any segment contains a substitution-nested heredoc, sibling git add / git push segments are also passed through. This is intentionally conservative — re-emitting any part of a compound that contains this construct risks downstream byte-offset slicing into the heredoc body.

Output redirection

Commands that redirect output to a file (>, >>, &>, &>>, >|, 1>, 2>, <>, etc.) are also passed through unchanged. The agent explicitly redirected to a file because they want the raw output for downstream processing — interposing tokf’s filter would write filtered bytes into the file and silently corrupt what the agent reads back.

# These are NOT rewritten — tokf leaves them alone:
git diff > /tmp/diff.txt          # explicit output redirect — agent wants raw form
git log --all > history.txt       # for grep/awk processing on the file
cargo test > test.log 2>&1        # combined output to file
git status > /dev/null            # output discarded; nothing to filter
exec 3<> /tmp/sock                # read+write file open

# These ARE still rewritten — fd remap only, no file involved:
git diff 2>&1                     # merges stderr into stdout, both still feed the agent
git diff 1>&2                     # redirects stdout to stderr — still no file
git diff >&-                      # closes a file descriptor

# Compound commands are handled per-segment — only the segment with the
# redirect is skipped, the other segments are still rewritten:
git diff > foo.txt; git status    # → "git diff > foo.txt; tokf run git status"
git status && git diff > foo.txt  # → "tokf run git status && git diff > foo.txt"

tee in a pipeline (git diff | tee log.txt) is not currently treated as an output redirect because tee is a command argument, not a redirect operator. The current pipe-handling behaviour is preserved. This is a known follow-up.

Transparent-arg commands

Some commands take a shell-code payload that runs in a different environment — ssh HOST 'cmd' runs cmd on the remote host, where tokf is not installed and shouldn’t be referenced. For these commands tokf must be especially conservative: it can still wrap the local invocation with tokf run (which preserves the argv byte-for-byte), but user [[rewrite]] regex rules are not applied because a sufficiently broad pattern can splice text into the opaque payload and break the remote call.

The built-in list — always active, can’t be disabled — is ssh, mosh, slogin. Basename matching applies, so /usr/bin/ssh is treated identically to ssh.

You can extend the list via [transparent] for tools like kubectl exec, docker exec, etc.:

[transparent]
commands = ["kubectl", "doctl"]

Names are matched against the basename of the command’s first word. commands extends — not replaces — the built-in list, so kubectl is added on top of ssh/mosh/slogin.

What still happens for transparent commands:

The standard tokf run <cmd> wrap from a matching filter (argv-preserving prefix only).
Pipe stripping with --baseline-pipe '<suffix>' (flags inserted between tokf run and <cmd>; the inner argv is untouched).
The built-in ^tokf skip and the heredoc / output-redirect skips above.

What is gated:

User [[rewrite]] regex rules in rewrites.toml. They run on the full command string and could splice text into the inner argv, so they’re skipped when the first command word’s basename is in the transparent list.

If you genuinely need a regex rewrite for an ssh-class command, invoke it explicitly: tokf run ssh … is preserved by the ^tokf skip rule, so it won’t be re-rewritten.

This was added to address #338, where a long-output ssh HOST 'cmd' would land tokf on the remote bash and exit 127.

Debug settings

The [debug] section enables diagnostic logging for the rewrite system. All settings are off by default.

[debug]
log_parse_failures = true

Field	Default	Description
`log_parse_failures`	`false`	Log to stderr when the bash parser (rable) fails to parse a command, causing the rewrite system to fall back to simple string matching. Helps diagnose unexpected “unmatched quote” errors or commands that should have been skipped but weren’t.