lobo/zfin
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions
zfin/AGENTS.md

27 KiB
Raw Blame History

AGENTS.md

⛔ ABSOLUTE PROHIBITIONS — READ FIRST ⛔

Zig 0.16.0 reference — read the release notes

This codebase is on Zig 0.16.0. The 0.16 release was a major I/O-as-an-interface refactor that reshaped the standard library. Before making non-trivial changes (especially anything touching std.Io, std.fs, std.process, std.http, std.Thread, allocators, or std.time), read the release notes at:

https://ziglang.org/download/0.16.0/release-notes.html

Key migrations that bit us repeatedly during the 0.16 upgrade and will bite future work too:

  • std.io → std.Io (namespace rename, no deprecation alias).
  • std.fs.cwd() → std.Io.Dir.cwd(). All file ops take io.
  • std.process.Child.init(argv, alloc) → std.process.spawn(io, .{...}) or std.process.run(gpa, io, .{...}).
  • std.time.timestamp() → std.Io.Timestamp.now(io, .real).toSeconds().
  • std.Thread.sleep(ns) → std.Io.sleep(io, duration, clock).
  • pub fn main gains a std.process.Init parameter ("Juicy Main"); provides pre-built gpa, io, arena, environ_map.
  • std.heap.GeneralPurposeAllocator → std.heap.DebugAllocator.
  • std.heap.ThreadSafeAllocator removed; ArenaAllocator is now lock-free thread-safe, DebugAllocator is thread-safe by default.
  • std.mem.trimRight/trimLeft → trimEnd/trimStart.
  • std.mem.indexOf* → find* (deprecation aliases still present, so old names work but warn).
  • std.testing.refAllDeclsRecursive removed. Only refAllDecls remains. We use the bare std.testing.refAllDecls(@This()) in src/main.zig's test block — it sema-touches every top-level decl, which transitively pulls in every imported file's test blocks. No local reimplementation is needed. See "Test discovery" below.
  • std.fs.File.readToEndAlloc(alloc, N) → two-step: file.reader(io, &.{}) then .interface.allocRemaining(alloc, .limited(N)).

For anything not on this list, read the release notes first. The notes are long but they're organized by section; searching for the specific symbol you're migrating is fast.

io vs today / now_s — design rule

The 0.16 upgrade made a deliberate choice about which Zig-0.16 Io calls to thread through and which to sidestep. This rule is load-bearing; please read before adding new code that needs the current time.

  • io: std.Io is threaded through anything that actually does I/O — file reads/writes, stderr, HTTP, process spawn, terminal detection. A function taking io is announcing that it touches the outside world. The "code smell" is a feature.

  • today: Date is passed as a value for functions that need "what day is it" but don't otherwise do I/O. Captured once at the top of the unit of work (runCli for CLI, App.init for TUI) and threaded through. Render output stays deterministic within a frame even if the clock ticks over mid-render.

    • Use the name today only when the parameter genuinely means "the current calendar day." That's the App-level app.today field and the per-command today arg threaded from runCli. The caller can't reasonably pass anything other than the actual current day to these.
    • Use as_of: Date when the parameter is the reference date for a computation, and the caller might legitimately pass any value. Examples: rolling-windows blocks (the --as-of flag back-dates the view), CAGR / annualization helpers, snapshot-resolution helpers. The function should work correctly whether the caller passes today's date, last week's date, or 2014. Calling such a parameter today invites bugs because future maintainers will assume the function uses it as "now" rather than "the reference date you asked about."
    • When in doubt: if the function's behavior would be nonsensical with a date that isn't today, use today. Otherwise use as_of. Most analytics functions are as_of.

  • now_s: i64 (or similar before_s/after_s pairs) is passed as a value for sub-second-precision metadata fields like snapshot captured_at, rollup #!created=, audit cadence staleness math. Same single-capture-and-thread pattern as today.

When adding a new function that needs the current time, do NOT reach for std.Io.Timestamp.now(io, .real) inside the function. Take today: Date or now_s: i64 instead. Only take io if the function genuinely needs to do I/O for other reasons.

Legitimate Timestamp.now callers (each must have a // wall-clock required: <why> comment justifying the read):

  • cache/store.zig — cache entry timestamps and TTL math
  • service.zig — per-fetch FetchResult.timestamp
  • net/RateLimiter.zig — token-bucket refill
  • TUI per-frame "now" captures for relative-time display
  • The single Timestamp.now capture in main.zig's dispatch entry that produces today and now_s for the rest of the invocation
  • The format.todayDate(io) helper itself (the one legitimate capture function for unit-of-work entry points)

If you find yourself writing Timestamp.now(io, ...) somewhere not on that list, either add a justifying comment or refactor the function to take a value parameter.

NEVER invoke ripgrep. EVER.

Do not run rg in the Bash tool. Not for open-ended search, not for counting matches, not for "just this one quick check", not ever. Running ripgrep on this machine hammers the filesystem badly enough to degrade the whole system — this is a recurring, reproducible problem, not a hunch.

Use instead:

  • Grep tool (built-in) for content search. It handles regex, file globs, and output shaping without spawning rg.
  • Glob tool (built-in) for finding files by name pattern.
  • Read tool for reading files (with offset/limit for large ones).
  • Plain grep via the Bash tool is acceptable when the built-in Grep tool can't express what you need — but prefer the built-in first.

If you catch yourself typing rg in a Bash command: stop, delete it, use the Grep tool instead. The fact that rg is faster in the abstract does NOT matter here. This machine's filesystem + ripgrep's parallelism is a bad combination, full stop.

This applies to every variant: rg, ripgrep, piping through rg, backgrounded rg, rg --files, etc. All banned.

NEVER delete or modify build caches. EVER.

This means:

  • NEVER run rm -rf .zig-cache or rm -rf .zig-cache/* or any variant.
  • NEVER run rm -rf ~/.cache/zig or touch anything under ~/.cache/zig/.
  • NEVER touch ~/.cache/zls/ or any other tool cache.
  • NEVER suggest deleting the cache as a "fix" — it is not a fix, it is damage. Deleting .zig-cache while ZLS or another zig build is running creates a corrupt state where the build runner's expected cache entry (.zig-cache/o/<hash>/build) references a path that no longer exists, producing the error failed to spawn build runner .zig-cache/o/<hash>/build: FileNotFound. Recovering from this on the affected machine requires killing every concurrent zig process (including ZLS) and waiting for filesystem state to re-stabilize — it is NOT a simple retry.
  • NEVER touch the cache "just to force a rebuild". Zig's cache is content-addressed. It does not get stale in a way that deletion fixes. If a build result looks wrong, the bug is in the source, not the cache. Use touch src/somefile.zig if you truly need to invalidate one file's cache line. Do not nuke the whole directory.
  • NEVER suggest a "different cache directory" as a workaround without an explicit, specific reason and explicit user approval. --cache-dir and --global-cache-dir flags exist; they are not toys.

If a test result seems wrong or cached incorrectly: the answer is ALWAYS to investigate the source code or build graph, not to delete cache. See "Test discovery" below — 99% of the time the "cached wrong result" is actually a test discovery problem, not a cache problem.

If you find yourself typing rm -rf anywhere near a cache path: STOP. Ask the user instead.

NEVER run destructive git operations without explicit permission.

  • No git reset --hard, git clean -fdx, git push --force, git checkout . on files with uncommitted work, unless the user asks for that specific operation by name.

NEVER run git add, git commit, or git push. EVER.

  • The user commits. You do not. Do not stage files. Do not create commits. Do not amend commits. Do not push. Do not suggest running these commands yourself "to save a step". This includes git add -p, git add ., git add <file>, git commit -m ..., git commit --amend, git push, and any gh pr create that would auto-stage or auto-commit.
  • If you are tempted to run any of these because "the work is done and it seems logical to commit" — STOP. The user has a review-and-commit workflow. Your job ends at a clean working tree with the changes ready to review.
  • The ONLY exception is when the user says, verbatim in the current turn, "commit this" / "make a commit" / "push it" / similar direct imperative. Do not extrapolate from earlier intent, a plan that mentioned milestones, or any indirect signal. If in doubt, ask — don't commit.
  • When a milestone plan says "STOP POINT — user reviews and commits": you stop. You do not commit. You do not prepare a commit. You hand off the working tree and wait.

Documentation-file conventions

  • TODO.md holds only what's still open. Git already tracks what was done and when. Do NOT add "DONE" markers, completion status, strikethrough, or "shipped in …" blurbs to TODO entries — just delete the section when the work is finished. If follow-up work came out of the finished task, add it as a new top-level section; don't leave the parent entry around as a historical wrapper.
  • REPORT.md is untracked on purpose. It's a personal workflow doc living in the repo root only until it moves to ~/finance. Edit it freely when asked; don't treat it as part of the repo surface. Don't mention it in commit messages for unrelated work.

Commands

zig build                    # build the zfin binary (output: zig-out/bin/zfin)
zig build test               # run all tests (single binary, discovers all tests via refAllDecls + the import graph)
zig build run -- <args>      # build and run CLI
zig build docs               # generate library documentation
zig build coverage           # run tests with kcov coverage (Linux only). See "Coverage" section.
zig build coverage -Dcoverage-threshold=65  # fail build if coverage < N% (pre-commit uses 65)

Tooling (managed via .mise.toml):

  • Zig 0.16.0 (minimum)
  • ZLS 0.16.0
  • zlint 0.7.9

Linting: zlint --deny-warnings --fix (runs via pre-commit on staged .zig files).

Formatting: zig fmt (enforced by pre-commit). Always run before committing.

Pre-commit hooks run in order: trailing-whitespace, end-of-file-fixer, zig-fmt, zlint, zig-build, then tests with coverage.

Architecture

Single binary (CLI + TUI) built from src/main.zig. No separate library binary for internal use — the library module (src/root.zig) exists only for downstream consumers and documentation generation.

Data flow

User input → main.zig (CLI dispatch) or tui.zig (TUI event loop)
  → commands/*.zig (CLI) or tui/*.zig (TUI tab renderers)
    → DataService (service.zig) — sole data access layer
      → Cache check (cache/store.zig, SRF files in ~/.cache/zfin/{SYMBOL}/)
      → Server sync (optional ZFIN_SERVER, parallel HTTP)
      → Provider fetch (providers/*.zig, rate-limited HTTP)
      → Cache write
    → analytics/*.zig (performance, risk, valuation calculations)
    → format.zig (shared formatters, braille charts)
    → views/*.zig (view models — renderer-agnostic display data)
    → stdout (CLI via buffered Writer) or vaxis (TUI terminal rendering)

Key design decisions

  • Internal imports use file paths, not module names. Only external dependencies (srf, vaxis, z2d) use @import("name"). Internal code uses relative paths like @import("models/date.zig"). This is intentional — it lets refAllDecls in the test binary discover all tests across the entire source tree.

  • DataService is the sole data source. Both CLI and TUI go through DataService for all fetched data. Never call provider APIs directly from commands or TUI tabs.

  • Providers are lazily initialized. DataService fields like td, pg, fh start as null and are created on first use via getProvider(). The provider field name is derived from the type name at comptime.

  • Cache uses SRF format. SRF (Simple Record Format) is a line-oriented key-value format. Cache layout: {cache_dir}/{SYMBOL}/{data_type}.srf. Freshness is determined by file mtime vs TTL.

  • Candles use incremental updates. On cache miss, only candles newer than the last cached date are fetched (not the full 10-year history). The candles_meta.srf file tracks the last date and provider without deserializing the full candle file.

  • View models separate data from rendering. views/portfolio_sections.zig produces renderer-agnostic structs with StyleIntent enums. CLI and TUI renderers are thin adapters that map StyleIntent to ANSI colors or vaxis styles.

  • Negative cache entries. When a provider fetch fails permanently (not rate-limited), a negative cache entry is written to prevent repeated retries for nonexistent symbols.

Module map

Directory Purpose
src/models/ Data types: Date (days since epoch), Candle, Dividend, Split, Lot, Portfolio, OptionContract, EarningsEvent, EtfProfile, Quote
src/providers/ API clients: each provider has its own struct with init(allocator, api_key) + fetch methods. json_utils.zig has shared JSON parsing helpers.
src/analytics/ Pure computation: performance.zig (Morningstar-style trailing returns), risk.zig (Sharpe, drawdown), valuation.zig (portfolio summary), analysis.zig (breakdowns by class/sector/geo)
src/commands/ CLI command handlers: each has a run() function taking (allocator, *DataService, symbol, color, *Writer). common.zig has shared CLI helpers and color constants.
src/tui/ TUI tab renderers: each tab (portfolio, quote, perf, options, earnings, analysis) is a separate file. keybinds.zig and theme.zig handle configurable input/colors. chart.zig renders pixel charts via Kitty graphics protocol.
src/views/ View models producing renderer-agnostic display data with StyleIntent
src/cache/ store.zig: SRF cache read/write with TTL freshness checks
src/net/ http.zig: HTTP client with retry and error classification. RateLimiter.zig: token-bucket rate limiter.
build/ Build-time support: Coverage.zig (kcov integration)

Code patterns and conventions

Error handling

  • Provider HTTP errors are classified in net/http.zig: RequestFailed, RateLimited, Unauthorized, NotFound, ServerError, InvalidResponse.
  • DataService wraps these into DataError: NoApiKey, FetchFailed, TransientError, AuthError, etc.
  • Transient errors (server 5xx, connection failures) cause the refresh to stop. Non-transient errors (NotFound, ParseError) cause fallback to the next provider.
  • Rate limit hits trigger a single retry after rateLimitBackoff().

The Date type

Date is an i32 of days since Unix epoch. It is used everywhere instead of timestamps. Construction: Date.fromYmd(2024, 1, 15) or Date.parse("2024-01-15"). Formatting: date.format(&buf) writes YYYY-MM-DD into a *[10]u8. The type has SRF serialization hooks (srfParse, srfFormat).

Formatting pattern

Functions in format.zig write into caller-provided buffers and return slices. They never allocate. Example: fmtMoneyAbs(&buf, amount) returns []const u8. The sign handling is always caller-side.

Provider pattern

Each provider in src/providers/ follows the same structure:

  1. Struct with client: http.Client, allocator, api_key
  2. init(allocator, api_key) / deinit()
  3. fetch*(allocator, symbol, ...) methods that build a URL, call self.client.get(url), and parse the JSON response
  4. Private parse* functions that handle the provider-specific JSON format
  5. Shared JSON helpers from json_utils.zig (parseJsonFloat, optFloat, optUint, jsonStr)

Test pattern

All tests are inline (in test blocks within source files). There is a single test binary rooted at src/main.zig which uses std.testing.refAllDecls(@This()) to sema-touch every top-level decl in main.zig. Each decl that's a @import(...) of a source file pulls that file into compilation, which causes its test blocks to be collected by the test runner. The tests/ directory exists but fixtures are empty — all test data is defined inline.

Tests use std.testing.allocator (which detects leaks) and are structured as unit tests that verify individual functions. Network-dependent code is not tested (no mocking infrastructure).

Test discovery

zig build test runs every test block in every file reachable from src/main.zig via the import graph, courtesy of std.testing.refAllDecls(@This()) in main.zig's bottom test {} block. "Reachable" means: somewhere in the chain there's a const foo = @import("foo.zig"); (or any other binding to the imported file struct, including pub const, struct field, etc).

The verification workflow is dead simple:

zig build test --summary all 2>&1 | grep "tests passed"

Run it before and after a change. If the count moved the way you expected, you're done. If it didn't, fix it. There is no further analysis required — no manual graph walking, no canary tests, no dependency archaeology.

The one gotcha: if you import a file purely as a type extraction — const T = @import("foo.zig").T; — the test blocks in foo.zig are NOT collected. The fix is to bind the file struct itself somewhere: const foo = @import("foo.zig");. You'll know this happened because the test count won't go up after adding tests to a new file. The _ = @import(...) escape hatch in main.zig's test block is also fine if reshaping imports is inconvenient — refAllDecls will sema-touch it.

Do NOT clear the cache "to be sure." Cache is content-addressed; it isn't the problem. See the prohibitions at the top of this file.

Coverage

Test coverage is measured by zig build coverage, which runs the test binary under kcov and emits an HTML report under coverage/ plus a one-line summary on stdout:

Total test coverage: 65.15% (15399/23638)

The pre-commit hook enforces a coverage floor. The current floor is 65% (set in .pre-commit-config.yaml). The hook runs zig build coverage -Dcoverage-threshold=65 and fails the commit if coverage drops below that threshold. Bumping the floor over time is encouraged — every time we push the actual coverage materially higher, raise the floor in the pre-commit config in the same commit so the gain is locked in.

Coverage expectations for new work:

  1. For most features, coverage should go UP, or you should be able to explain why not. New analytics modules, parsers, loaders, formatters, and pure-domain transforms are easy to cover and should be — they're the load-bearing logic. New tests on existing files also nudge the percentage up by exercising more lines of the same code.

  2. It's fine for some additions to push coverage flat or slightly down. Examples:

    • New TUI rendering paths. TUI code that needs an interactive vaxis context (event handlers, draw callbacks, mouse handling) is hard to test without a vaxis-aware test harness, which we don't have. Adding a new tab will add uncovered lines unless you can extract a pure render function and test it in isolation.
    • New provider HTTP code. We don't mock providers; live network calls aren't run in tests. Provider request/response parsers ARE testable (and SHOULD be tested) — extract them from the HTTP-bound code so they can be exercised with fixture bytes.
    • CLI command dispatch glue in src/main.zig. The command match chain itself doesn't need tests; the command's run() function and helpers should.

  3. If coverage drops, document why in the commit message. A single sentence — "Adds TUI tab; pure render fn covered; event handlers and mouse handlers uncovered, no test harness" — is enough. Future-you will thank present-you.

How to investigate uncovered lines:

zig build coverage
# Open coverage/index.html in a browser, or:
ls coverage/                # per-file HTML reports

Each file's report shows red lines (uncovered) and green lines (covered). For a quick numeric breakdown by file, the kcov JSON output under coverage/kcov-merged/coverage.json is greppable.

Common reasons coverage looks lower than expected:

  • A new .zig file's tests aren't being discovered. Check the Test discovery section above. The tests pass-or-fail report will say "0 tests" for that module if it's orphaned.
  • A function has many branches but the tests only exercise the happy path. Add error-path tests with expectError.
  • A switch over many enum variants only tests one. Loop the test over all variants.
  • Code is dead. Either start using it or delete it.

Don't game the metric. If you find yourself adding tests that don't actually verify behavior just to pump the percentage — try expect(true) calls, tests that only construct types and check field defaults, etc. — stop. The gate exists to catch real regressions in test discipline; gaming it produces tests that will fail to catch real bugs later.

Adding a new CLI command

  1. Create src/commands/newcmd.zig with a pub fn run(allocator, *DataService, symbol, color, *Writer) !void
  2. Add the import to the commands struct in src/main.zig
  3. Add the dispatch branch in main.zig's command matching chain
  4. Update the usage string in main.zig

Adding a new provider

  1. Create src/providers/newprovider.zig following the existing struct pattern
  2. Add a field to DataService (e.g., np: ?NewProvider = null)
  3. Add the API key to Config (e.g., newprovider_key: ?[]const u8 = null) — the field name must be the lowercased type name + _key for the comptime getProvider lookup to work
  4. Wire resolve("NEWPROVIDER_API_KEY") in Config.fromEnv

Adding a new TUI tab

  1. Create src/tui/newtab_tab.zig
  2. Add the tab variant to tui.Tab enum
  3. Wire rendering in tui.zig's draw and event handling

Command run() signatures — allocator as code smell

A CLI command's run() function that takes *DataService and *std.Io.Writer usually doesn't also need an std.mem.Allocator parameter. FetchResult(T) carries its own allocator and self-deinits (see src/service.zig), so callers never need to wire up matching allocators for payload cleanup. The writer owns its own buffer.

If a new run() signature still wants an allocator, ask whether the work it's funding is:

  • Legitimate: file I/O for a secondary config (portfolio load, metadata), non-trivial intermediate computation, or an arena wrapping view-layer allocations. Keep it.
  • Suspicious: freeing FetchResult.data manually instead of calling result.deinit(), or duplicating strings that could be borrowed. Drop the allocator and fix the leak-shaped helper.

Not a hard rule — just a signal worth questioning when reviewing a new command.

Gotchas

  • Provider field naming is comptime-derived. DataService.getProvider(T) finds the ?T field by iterating struct fields at comptime, and the config key is derived by lowercasing the type name and appending _key. If you rename a provider struct, you must also rename the config field or the comptime logic breaks.

  • Candle data has two cache files. candles_daily.srf holds the actual OHLCV data; candles_meta.srf holds metadata (last_date, provider, fail_count). When invalidating candles, both must be cleared (this is handled by DataService.invalidate).

  • TwelveData candles are force-refetched. Cached candles from the TwelveData provider are treated as stale regardless of TTL because TwelveData's adj_close values were found to be unreliable. The code in getCandles explicitly checks m.provider == .twelvedata and falls through.

  • Mutual fund detection is heuristic. isMutualFund checks if the symbol is exactly 5 chars ending in 'X'. This skips earnings fetching for mutual funds. It's imperfect but covers the common case.

  • SRF string lifetimes. When reading SRF records, string fields point into the iterator's internal buffer. If you need strings to outlive the iterator, use a postProcess callback to allocator.dupe() them (see dividendPostProcess in service.zig).

  • Buffered stdout. CLI output uses a single std.Io.Writer with a 4096-byte stack buffer, flushed once at the end of main(). Don't write to stdout through other means.

  • The color parameter flows through everything. CLI commands accept a color: bool parameter. Don't use ANSI escapes unconditionally — always gate on the color flag.

  • Portfolio auto-detection. Both CLI and TUI auto-load portfolio.srf from cwd if no explicit path is given. If not found in cwd, falls back to $ZFIN_HOME/portfolio.srf. watchlist.srf and .env follow the same cascade. metadata.srf and accounts.srf are loaded from the same directory as the resolved portfolio file.

  • transaction_log.srf is a sibling file. Optional. Lives next to portfolio.srf / accounts.srf. Holds user-declared transfer:: records so the contributions pipeline can tell internal account-to-account movement apart from real external contributions. Only type::cash is wired in v1 — type::in_kind parses but is rejected downstream. Missing file → matcher is a no-op. See REPORT.md §5 "Transfer log" for the user-facing guide.

  • Server sync is optional. The ZFIN_SERVER env var enables parallel cache syncing from a remote zfin-server instance. All server sync code silently no-ops when the URL is null.

Dependencies

Dependency Purpose
SRF Cache file format, portfolio/watchlist parsing, serialization
libvaxis (v0.5.1) Terminal UI rendering
z2d (v0.10.0) Pixel chart rendering (Kitty graphics protocol)

Build system rules

  • Never use addAnonymousImport in build.zig. Always use b.addModule() + addImport(). Anonymous imports cause "file belongs to multiple modules" errors and make dependency wiring opaque.