zfin/AGENTS.md

# AGENTS.md

## ⛔ ABSOLUTE PROHIBITIONS — READ FIRST ⛔

### Zig 0.16.0 reference — read the release notes

This codebase is on Zig 0.16.0. The 0.16 release was a major
I/O-as-an-interface refactor that reshaped the standard library.
Before making non-trivial changes (especially anything touching
`std.Io`, `std.fs`, `std.process`, `std.http`, `std.Thread`,
allocators, or `std.time`), **read the release notes** at:

  https://ziglang.org/download/0.16.0/release-notes.html

Key migrations that bit us repeatedly during the 0.16 upgrade and
will bite future work too:

- `std.io` → `std.Io` (namespace rename, no deprecation alias).
- `std.fs.cwd()` → `std.Io.Dir.cwd()`. All file ops take `io`.
- `std.process.Child.init(argv, alloc)` → `std.process.spawn(io, .{...})`
  or `std.process.run(gpa, io, .{...})`.
- `std.time.timestamp()` → `std.Io.Timestamp.now(io, .real).toSeconds()`.
- `std.Thread.sleep(ns)` → `std.Io.sleep(io, duration, clock)`.
- `pub fn main` gains a `std.process.Init` parameter ("Juicy Main");
  provides pre-built `gpa`, `io`, `arena`, `environ_map`.
- `std.heap.GeneralPurposeAllocator` → `std.heap.DebugAllocator`.
- `std.heap.ThreadSafeAllocator` removed; `ArenaAllocator` is now
  lock-free thread-safe, `DebugAllocator` is thread-safe by default.
- `std.mem.trimRight`/`trimLeft` → `trimEnd`/`trimStart`.
- `std.mem.indexOf*` → `find*` (deprecation aliases still present,
  so old names work but warn).
- `std.testing.refAllDeclsRecursive` removed. Only `refAllDecls`
  remains. We use the bare `std.testing.refAllDecls(@This())` in
  `src/main.zig`'s test block — it sema-touches every top-level decl,
  which transitively pulls in every imported file's `test` blocks.
  No local reimplementation is needed. See "Test discovery" below.
- `std.fs.File.readToEndAlloc(alloc, N)` → two-step:
  `file.reader(io, &.{})` then `.interface.allocRemaining(alloc, .limited(N))`.

For anything not on this list, **read the release notes first.**
The notes are long but they're organized by section; searching for
the specific symbol you're migrating is fast.

### `io` vs `today` / `now_s` — design rule

The 0.16 upgrade made a deliberate choice about which Zig-0.16 `Io`
calls to thread through and which to sidestep. **This rule is
load-bearing**; please read before adding new code that needs the
current time.

- **`io: std.Io` is threaded through anything that actually does
  I/O** — file reads/writes, stderr, HTTP, process spawn, terminal
  detection. A function taking `io` is announcing that it touches
  the outside world. The "code smell" is a feature.
- **`today: Date` is passed as a value** for functions that need
  "what day is it" but don't otherwise do I/O. Captured once at
  the top of the unit of work (`runCli` for CLI, `App.init` for
  TUI) and threaded through. Render output stays deterministic
  within a frame even if the clock ticks over mid-render.
- **`now_s: i64` (or similar `before_s`/`after_s` pairs) is passed
  as a value** for sub-second-precision metadata fields like
  snapshot `captured_at`, rollup `#!created=`, audit cadence
  staleness math. Same single-capture-and-thread pattern as
  `today`.

When adding a new function that needs the current time, do NOT
reach for `std.Io.Timestamp.now(io, .real)` inside the function.
Take `today: Date` or `now_s: i64` instead. Only take `io` if the
function genuinely needs to do I/O for other reasons.

**Legitimate `Timestamp.now` callers** (each must have a
`// wall-clock required: <why>` comment justifying the read):
- `cache/store.zig` — cache entry timestamps and TTL math
- `service.zig` — per-fetch `FetchResult.timestamp`
- `net/RateLimiter.zig` — token-bucket refill
- TUI per-frame "now" captures for relative-time display
- The single `Timestamp.now` capture in `main.zig`'s dispatch
  entry that produces `today` and `now_s` for the rest of the
  invocation
- The `format.todayDate(io)` helper itself (the one legitimate
  capture function for unit-of-work entry points)

If you find yourself writing `Timestamp.now(io, ...)` somewhere
not on that list, either add a justifying comment or refactor
the function to take a value parameter.

### NEVER invoke ripgrep. EVER.

**Do not run `rg` in the Bash tool.** Not for open-ended search, not for
counting matches, not for "just this one quick check", not ever. Running
ripgrep on this machine hammers the filesystem badly enough to degrade the
whole system — this is a recurring, reproducible problem, not a hunch.

**Use instead:**

- **Grep tool** (built-in) for content search. It handles regex, file
  globs, and output shaping without spawning `rg`.
- **Glob tool** (built-in) for finding files by name pattern.
- **Read tool** for reading files (with `offset`/`limit` for large ones).
- Plain `grep` via the Bash tool is acceptable when the built-in Grep
  tool can't express what you need — but prefer the built-in first.

**If you catch yourself typing `rg` in a Bash command:** stop, delete it,
use the Grep tool instead. The fact that `rg` is faster in the abstract
does NOT matter here. This machine's filesystem + ripgrep's parallelism
is a bad combination, full stop.

**This applies to every variant:** `rg`, `ripgrep`, piping through
`rg`, backgrounded `rg`, `rg --files`, etc. All banned.

### NEVER delete or modify build caches. EVER.

**This means:**

- **NEVER** run `rm -rf .zig-cache` or `rm -rf .zig-cache/*` or any variant.
- **NEVER** run `rm -rf ~/.cache/zig` or touch anything under `~/.cache/zig/`.
- **NEVER** touch `~/.cache/zls/` or any other tool cache.
- **NEVER** suggest deleting the cache as a "fix" — it is not a fix, it is
  damage. Deleting `.zig-cache` while ZLS or another `zig build` is running
  creates a corrupt state where the build runner's expected cache entry
  (`.zig-cache/o/<hash>/build`) references a path that no longer exists,
  producing the error `failed to spawn build runner .zig-cache/o/<hash>/build:
  FileNotFound`. Recovering from this on the affected machine requires
  killing every concurrent `zig` process (including ZLS) and waiting for
  filesystem state to re-stabilize — it is NOT a simple retry.
- **NEVER** touch the cache "just to force a rebuild". Zig's cache is
  content-addressed. It does not get stale in a way that deletion fixes.
  If a build result looks wrong, the bug is in the source, not the cache.
  Use `touch src/somefile.zig` if you truly need to invalidate one file's
  cache line. Do not nuke the whole directory.
- **NEVER** suggest a "different cache directory" as a workaround without
  an explicit, specific reason and explicit user approval. `--cache-dir`
  and `--global-cache-dir` flags exist; they are not toys.

**If a test result seems wrong or cached incorrectly:** the answer is
ALWAYS to investigate the source code or build graph, not to delete
cache. See "Test discovery" below — 99% of the time the "cached wrong
result" is actually a test discovery problem, not a cache problem.

**If you find yourself typing `rm -rf` anywhere near a cache path: STOP.
Ask the user instead.**

### NEVER run destructive git operations without explicit permission.

- No `git reset --hard`, `git clean -fdx`, `git push --force`, `git checkout .`
  on files with uncommitted work, unless the user asks for that specific
  operation by name.

### NEVER run `git add`, `git commit`, or `git push`. EVER.

- **The user commits. You do not.** Do not stage files. Do not create commits.
  Do not amend commits. Do not push. Do not suggest running these commands
  yourself "to save a step". This includes `git add -p`, `git add .`,
  `git add <file>`, `git commit -m ...`, `git commit --amend`, `git push`,
  and any `gh pr create` that would auto-stage or auto-commit.
- If you are tempted to run any of these because "the work is done and it
  seems logical to commit" — STOP. The user has a review-and-commit workflow.
  Your job ends at a clean working tree with the changes ready to review.
- The ONLY exception is when the user says, verbatim in the current turn,
  "commit this" / "make a commit" / "push it" / similar direct imperative.
  Do not extrapolate from earlier intent, a plan that mentioned milestones,
  or any indirect signal. If in doubt, ask — don't commit.
- When a milestone plan says "STOP POINT — user reviews and commits": you
  stop. You do not commit. You do not prepare a commit. You hand off the
  working tree and wait.

### Documentation-file conventions

- **`TODO.md` holds only what's still open.** Git already tracks what was
  done and when. Do NOT add "DONE" markers, completion status, strikethrough,
  or "shipped in …" blurbs to TODO entries — just delete the section when
  the work is finished. If follow-up work came out of the finished task,
  add it as a new top-level section; don't leave the parent entry around
  as a historical wrapper.
- **`REPORT.md` is untracked on purpose.** It's a personal workflow doc
  living in the repo root only until it moves to `~/finance`. Edit it
  freely when asked; don't treat it as part of the repo surface. Don't
  mention it in commit messages for unrelated work.

---

## Commands

```bash
zig build                    # build the zfin binary (output: zig-out/bin/zfin)
zig build test               # run all tests (single binary, discovers all tests via refAllDecls + the import graph)
zig build run -- <args>      # build and run CLI
zig build docs               # generate library documentation
zig build coverage -Dcoverage-threshold=60  # run tests with kcov coverage (Linux only)
```

**Tooling** (managed via `.mise.toml`):
- Zig 0.16.0 (minimum)
- ZLS 0.16.0
- zlint 0.7.9

**Linting**: `zlint --deny-warnings --fix` (runs via pre-commit on staged `.zig` files).

**Formatting**: `zig fmt` (enforced by pre-commit). Always run before committing.

**Pre-commit hooks** run in order: trailing-whitespace, end-of-file-fixer, zig-fmt, zlint, zig-build, then tests with coverage.

## Architecture

Single binary (CLI + TUI) built from `src/main.zig`. No separate library binary for internal use — the library module (`src/root.zig`) exists only for downstream consumers and documentation generation.

### Data flow

```
User input → main.zig (CLI dispatch) or tui.zig (TUI event loop)
  → commands/*.zig (CLI) or tui/*.zig (TUI tab renderers)
    → DataService (service.zig) — sole data access layer
      → Cache check (cache/store.zig, SRF files in ~/.cache/zfin/{SYMBOL}/)
      → Server sync (optional ZFIN_SERVER, parallel HTTP)
      → Provider fetch (providers/*.zig, rate-limited HTTP)
      → Cache write
    → analytics/*.zig (performance, risk, valuation calculations)
    → format.zig (shared formatters, braille charts)
    → views/*.zig (view models — renderer-agnostic display data)
    → stdout (CLI via buffered Writer) or vaxis (TUI terminal rendering)
```

### Key design decisions

- **Internal imports use file paths, not module names.** Only external dependencies (`srf`, `vaxis`, `z2d`) use `@import("name")`. Internal code uses relative paths like `@import("models/date.zig")`. This is intentional — it lets `refAllDecls` in the test binary discover all tests across the entire source tree.

- **DataService is the sole data source.** Both CLI and TUI go through `DataService` for all fetched data. Never call provider APIs directly from commands or TUI tabs.

- **Providers are lazily initialized.** `DataService` fields like `td`, `pg`, `fh` start as `null` and are created on first use via `getProvider()`. The provider field name is derived from the type name at comptime.

- **Cache uses SRF format.** [SRF](https://git.lerch.org/lobo/srf) (Simple Record Format) is a line-oriented key-value format. Cache layout: `{cache_dir}/{SYMBOL}/{data_type}.srf`. Freshness is determined by file mtime vs TTL.

- **Candles use incremental updates.** On cache miss, only candles newer than the last cached date are fetched (not the full 10-year history). The `candles_meta.srf` file tracks the last date and provider without deserializing the full candle file.

- **View models separate data from rendering.** `views/portfolio_sections.zig` produces renderer-agnostic structs with `StyleIntent` enums. CLI and TUI renderers are thin adapters that map `StyleIntent` to ANSI colors or vaxis styles.

- **Negative cache entries.** When a provider fetch fails permanently (not rate-limited), a negative cache entry is written to prevent repeated retries for nonexistent symbols.

### Module map

| Directory | Purpose |
|-----------|---------|
| `src/models/` | Data types: `Date` (days since epoch), `Candle`, `Dividend`, `Split`, `Lot`, `Portfolio`, `OptionContract`, `EarningsEvent`, `EtfProfile`, `Quote` |
| `src/providers/` | API clients: each provider has its own struct with `init(allocator, api_key)` + fetch methods. `json_utils.zig` has shared JSON parsing helpers. |
| `src/analytics/` | Pure computation: `performance.zig` (Morningstar-style trailing returns), `risk.zig` (Sharpe, drawdown), `valuation.zig` (portfolio summary), `analysis.zig` (breakdowns by class/sector/geo) |
| `src/commands/` | CLI command handlers: each has a `run()` function taking `(allocator, *DataService, symbol, color, *Writer)`. `common.zig` has shared CLI helpers and color constants. |
| `src/tui/` | TUI tab renderers: each tab (portfolio, quote, perf, options, earnings, analysis) is a separate file. `keybinds.zig` and `theme.zig` handle configurable input/colors. `chart.zig` renders pixel charts via Kitty graphics protocol. |
| `src/views/` | View models producing renderer-agnostic display data with `StyleIntent` |
| `src/cache/` | `store.zig`: SRF cache read/write with TTL freshness checks |
| `src/net/` | `http.zig`: HTTP client with retry and error classification. `RateLimiter.zig`: token-bucket rate limiter. |
| `build/` | Build-time support: `Coverage.zig` (kcov integration) |

## Code patterns and conventions

### Error handling

- Provider HTTP errors are classified in `net/http.zig`: `RequestFailed`, `RateLimited`, `Unauthorized`, `NotFound`, `ServerError`, `InvalidResponse`.
- `DataService` wraps these into `DataError`: `NoApiKey`, `FetchFailed`, `TransientError`, `AuthError`, etc.
- Transient errors (server 5xx, connection failures) cause the refresh to stop. Non-transient errors (NotFound, ParseError) cause fallback to the next provider.
- Rate limit hits trigger a single retry after `rateLimitBackoff()`.

### The `Date` type

`Date` is an `i32` of days since Unix epoch. It is used everywhere instead of timestamps. Construction: `Date.fromYmd(2024, 1, 15)` or `Date.parse("2024-01-15")`. Formatting: `date.format(&buf)` writes `YYYY-MM-DD` into a `*[10]u8`. The type has SRF serialization hooks (`srfParse`, `srfFormat`).

### Formatting pattern

Functions in `format.zig` write into caller-provided buffers and return slices. They never allocate. Example: `fmtMoneyAbs(&buf, amount)` returns `[]const u8`. The sign handling is always caller-side.

### Provider pattern

Each provider in `src/providers/` follows the same structure:
1. Struct with `client: http.Client`, `allocator`, `api_key`
2. `init(allocator, api_key)` / `deinit()`
3. `fetch*(allocator, symbol, ...)` methods that build a URL, call `self.client.get(url)`, and parse the JSON response
4. Private `parse*` functions that handle the provider-specific JSON format
5. Shared JSON helpers from `json_utils.zig` (`parseJsonFloat`, `optFloat`, `optUint`, `jsonStr`)

### Test pattern

All tests are inline (in `test` blocks within source files). There is a single test binary rooted at `src/main.zig` which uses `std.testing.refAllDecls(@This())` to sema-touch every top-level decl in main.zig. Each decl that's a `@import(...)` of a source file pulls that file into compilation, which causes its `test` blocks to be collected by the test runner. The `tests/` directory exists but fixtures are empty — all test data is defined inline.

Tests use `std.testing.allocator` (which detects leaks) and are structured as unit tests that verify individual functions. Network-dependent code is not tested (no mocking infrastructure).

### ⚠️ Test discovery — READ THIS BEFORE ADDING A NEW .zig FILE WITH TESTS ⚠️

**This gets fucked up every single session. Read it. Do what it says.**

`zig build test` runs tests from `test` blocks in files that are part of the
test binary's compilation unit AND get sema-pulled by the import graph from
`src/main.zig`. With the bare `std.testing.refAllDecls(@This())` we use, a
file's tests are collected as long as the file is imported (directly or
transitively) from main.zig.

**The failure mode:** you add `src/models/foo.zig` with 20 tests. You wire
it into `src/service.zig` only as a *type extraction*, e.g.
`const Bar = @import("models/foo.zig").Bar;` (assigning the type, not the
file struct). The file **compiles** because `Bar` is referenced, but the
file struct itself was never sema-touched as a struct, so its `test`
blocks are not collected.

**The fix:** ensure at least one importer assigns the file struct to a
`const`, like `const foo = @import("models/foo.zig");`. Even if you only
use a type from it, the `const foo` form pulls in the file's `test` blocks.

**How to verify a new file's tests are discovered:**

1. Before relying on the test count, add a canary that MUST fail:
   ```zig
   test "CANARY_DISCOVERY_CHECK_REMOVE_ME" {
       try std.testing.expect(false);
   }
   ```
2. Run `zig build test --summary all 2>&1 | grep -E "tests passed|error:"`.
3. If the canary test appears in failures → discovery works, remove canary.
4. If the canary does NOT appear and total count is unchanged → ensure
   the file is imported via a `const x = @import(...)` form somewhere
   reachable from main.zig.

**Fallback fix:** if you can't fix the import shape, add an explicit
import in the `test` block at the bottom of `src/main.zig`:

```zig
test {
    std.testing.refAllDecls(@This());
    _ = @import("models/foo.zig");  // ← orphaned file
}
```

Adding it inside the `test` block (not at file scope as a `comptime` block)
keeps the non-test build unaffected while guaranteeing the test binary
sema-reaches the file and collects its test blocks.

**Rule of thumb:** after adding ANY new `.zig` file under `src/` that contains
`test` blocks, run `zig build test --summary all 2>&1 | grep "tests passed"`
BEFORE and AFTER the change. If the delta doesn't match
`rg -c "^test " path/to/new_file.zig`, add the explicit import to main.zig's
test block.

**Do NOT, under any circumstance, try to "fix" this by clearing the cache.**
The cache is not the problem. The import graph is the problem. Re-read
the prohibitions at the top of this file.

### Adding a new CLI command

1. Create `src/commands/newcmd.zig` with a `pub fn run(allocator, *DataService, symbol, color, *Writer) !void`
2. Add the import to the `commands` struct in `src/main.zig`
3. Add the dispatch branch in `main.zig`'s command matching chain
4. Update the `usage` string in `main.zig`

### Adding a new provider

1. Create `src/providers/newprovider.zig` following the existing struct pattern
2. Add a field to `DataService` (e.g., `np: ?NewProvider = null`)
3. Add the API key to `Config` (e.g., `newprovider_key: ?[]const u8 = null`) — the field name must be the lowercased type name + `_key` for the comptime `getProvider` lookup to work
4. Wire `resolve("NEWPROVIDER_API_KEY")` in `Config.fromEnv`

### Adding a new TUI tab

1. Create `src/tui/newtab_tab.zig`
2. Add the tab variant to `tui.Tab` enum
3. Wire rendering in `tui.zig`'s draw and event handling

### Command `run()` signatures — allocator as code smell

A CLI command's `run()` function that takes `*DataService` and `*std.Io.Writer`
usually doesn't also need an `std.mem.Allocator` parameter. `FetchResult(T)`
carries its own allocator and self-deinits (see `src/service.zig`), so callers
never need to wire up matching allocators for payload cleanup. The writer
owns its own buffer.

If a new `run()` signature still wants an allocator, ask whether the work
it's funding is:

- **Legitimate**: file I/O for a secondary config (portfolio load,
  metadata), non-trivial intermediate computation, or an arena wrapping
  view-layer allocations. Keep it.
- **Suspicious**: freeing `FetchResult.data` manually instead of calling
  `result.deinit()`, or duplicating strings that could be borrowed.
  Drop the allocator and fix the leak-shaped helper.

Not a hard rule — just a signal worth questioning when reviewing a new
command.

## Gotchas

- **Provider field naming is comptime-derived.** `DataService.getProvider(T)` finds the `?T` field by iterating struct fields at comptime, and the config key is derived by lowercasing the type name and appending `_key`. If you rename a provider struct, you must also rename the config field or the comptime logic breaks.

- **Candle data has two cache files.** `candles_daily.srf` holds the actual OHLCV data; `candles_meta.srf` holds metadata (last_date, provider, fail_count). When invalidating candles, both must be cleared (this is handled by `DataService.invalidate`).

- **TwelveData candles are force-refetched.** Cached candles from the TwelveData provider are treated as stale regardless of TTL because TwelveData's `adj_close` values were found to be unreliable. The code in `getCandles` explicitly checks `m.provider == .twelvedata` and falls through.

- **Mutual fund detection is heuristic.** `isMutualFund` checks if the symbol is exactly 5 chars ending in 'X'. This skips earnings fetching for mutual funds. It's imperfect but covers the common case.

- **SRF string lifetimes.** When reading SRF records, string fields point into the iterator's internal buffer. If you need strings to outlive the iterator, use a `postProcess` callback to `allocator.dupe()` them (see `dividendPostProcess` in `service.zig`).

- **Buffered stdout.** CLI output uses a single `std.Io.Writer` with a 4096-byte stack buffer, flushed once at the end of `main()`. Don't write to stdout through other means.

- **The `color` parameter flows through everything.** CLI commands accept a `color: bool` parameter. Don't use ANSI escapes unconditionally — always gate on the `color` flag.

- **Portfolio auto-detection.** Both CLI and TUI auto-load `portfolio.srf` from cwd if no explicit path is given. If not found in cwd, falls back to `$ZFIN_HOME/portfolio.srf`. `watchlist.srf` and `.env` follow the same cascade. `metadata.srf` and `accounts.srf` are loaded from the same directory as the resolved portfolio file.

- **`transaction_log.srf` is a sibling file.** Optional. Lives next to `portfolio.srf` / `accounts.srf`. Holds user-declared `transfer::` records so the contributions pipeline can tell internal account-to-account movement apart from real external contributions. Only `type::cash` is wired in v1 — `type::in_kind` parses but is rejected downstream. Missing file → matcher is a no-op. See `REPORT.md` §5 "Transfer log" for the user-facing guide.

- **Server sync is optional.** The `ZFIN_SERVER` env var enables parallel cache syncing from a remote zfin-server instance. All server sync code silently no-ops when the URL is null.

## Dependencies

| Dependency | Purpose |
|------------|---------|
| [SRF](https://git.lerch.org/lobo/srf) | Cache file format, portfolio/watchlist parsing, serialization |
| [libvaxis](https://github.com/rockorager/libvaxis) (v0.5.1) | Terminal UI rendering |
| [z2d](https://github.com/vancluever/z2d) (v0.10.0) | Pixel chart rendering (Kitty graphics protocol) |

## Build system rules

- **Never use `addAnonymousImport`** in `build.zig`. Always use `b.addModule()` + `addImport()`. Anonymous imports cause "file belongs to multiple modules" errors and make dependency wiring opaque.