Compare commits
4 commits
0e1a4862db
...
8de28d73bc
| Author | SHA1 | Date | |
|---|---|---|---|
| 8de28d73bc | |||
| 143395e7ce | |||
| 1bea4d621a | |||
| dcd5885fcc |
6 changed files with 142 additions and 99 deletions
68
AGENTS.md
68
AGENTS.md
|
|
@ -372,10 +372,46 @@ local_pii_tokens=$(awk -F: '...' "$ZFIN_HOME/accounts.srf" ...)
|
|||
grep -rn -E "$local_pii_tokens" src/ | grep -v ie_data.csv
|
||||
```
|
||||
|
||||
The grep should always return zero non-`ie_data.csv` hits before
|
||||
committing. The `ie_data.csv` exclusion is because the Shiller
|
||||
dataset contains coincidental numeric matches in historical-year
|
||||
fields that aren't PII.
|
||||
The grep is a **starting point, not a pass/fail gate** - it produces
|
||||
false positives that must be triaged rather than dismissed wholesale.
|
||||
Two known classes:
|
||||
|
||||
- **Short numeric tokens.** A 3- or 4-digit account number matches
|
||||
coincidentally inside Unix timestamps, trade volumes, hex color
|
||||
literals, fake git SHAs, dollar amounts in CSV fixtures, and even
|
||||
file line numbers. The `ie_data.csv` exclusion exists for the same
|
||||
reason (Shiller's historical-year fields).
|
||||
- **Generic account-type names.** An account literally named for an
|
||||
IRS category (`Roth IRA`, `Inherited IRA`, `HSA`) matches inside the
|
||||
approved placeholder vocabulary above - e.g. the real name
|
||||
`Inherited IRA` is a substring of the sanctioned fixture value
|
||||
`Sample Inherited IRA`.
|
||||
|
||||
So the rule is: **inspect the context of every hit**, and confirm each
|
||||
is either coincidental or a generic category before dismissing it. A
|
||||
hit whose surrounding context is a real-looking account name, a
|
||||
composite identifier, or a value that could only have come from the
|
||||
user's data is PII and must be fixed in the same change.
|
||||
|
||||
When reporting the result, redact: print the matching lines with the
|
||||
sensitive token substituted out, or report only `file:line`. Do not
|
||||
paste raw matches into a transcript.
|
||||
|
||||
**`ZFIN_HOME` does NOT isolate you for smoke tests.** Pointing
|
||||
`ZFIN_HOME` at a scratch directory does not sandbox the whole tool:
|
||||
`ZFIN_AUDIT_FILES` is read independently (`audit/hygiene.zig`) and
|
||||
points at wherever real brokerage exports get downloaded. A flagless
|
||||
`zfin audit` will discover, parse, and print **real account numbers and
|
||||
balances** from that directory regardless of `ZFIN_HOME`. This has
|
||||
already caused one leak into a session transcript.
|
||||
|
||||
So when smoke-testing against scratch data:
|
||||
|
||||
- Run under `env -u ZFIN_AUDIT_FILES` (and unset any other
|
||||
path-carrying `ZFIN_*` var the command consumes).
|
||||
- Bound the output. Never pipe a whole command's output through a broad
|
||||
`grep`/`head` - `sed -n '/Section Header/,/^$/p'` the one section
|
||||
under test, so an unexpected section can't spill into the transcript.
|
||||
|
||||
If you're uncertain whether something is PII, **ask before
|
||||
committing.** PII can be surgically removed from a working
|
||||
|
|
@ -592,6 +628,28 @@ Each file's report shows red lines (uncovered) and green lines
|
|||
(covered). For a quick numeric breakdown by file, the kcov JSON
|
||||
output under `coverage/kcov-merged/coverage.json` is greppable.
|
||||
|
||||
**Two traps if you script against the coverage output.** Both have
|
||||
already produced a confidently-wrong "fully covered" claim:
|
||||
|
||||
1. **Per-line data lives in the `.js` files, not the `.html`.** Each
|
||||
`coverage/<binary>/<file>.<hash>.html` has only a handful of
|
||||
`class="lineNum"` nodes; the real per-line records are in the
|
||||
sibling `.js` as
|
||||
`{"lineNum":" 42","line":"...","class":"lineCov|lineNoCov|linePartCov"}`.
|
||||
A scraper pointed at the HTML matches **zero** lines, so
|
||||
"0 uncovered" silently means "0 tracked." Always assert that the
|
||||
tracked-line count is non-zero before trusting a coverage verdict,
|
||||
and sanity-check the scraper against a line you know is uncovered.
|
||||
2. **`coverage/` accumulates stale per-binary directories across
|
||||
runs.** Old reports for a file you just edited stay on disk with the
|
||||
previous source text. Filter to reports whose embedded `"line"`
|
||||
content matches the current file (grep for a string you just added)
|
||||
before unioning them.
|
||||
|
||||
Note that per-file numbers are a union across binaries: the same source
|
||||
file appears in several `coverage/<binary>/` directories, and a line
|
||||
covered by any of them counts.
|
||||
|
||||
**Common reasons coverage looks lower than expected:**
|
||||
|
||||
- A new `.zig` file's tests aren't being discovered. Check the
|
||||
|
|
@ -777,7 +835,7 @@ command.
|
|||
|
||||
- **Portfolio auto-detection.** Both CLI and TUI resolve `portfolio*.srf` the same way, and it is ZFIN_HOME-exclusive when set: if `$ZFIN_HOME` is set, only that directory is searched (cwd is NOT a fallback - a project directory that incidentally ships a `portfolio.srf` must not shadow the user's canonical data); if `$ZFIN_HOME` is unset, cwd is searched. An explicit `-p` path resolves the same exclusive way. `watchlist.srf` and the `.env` file follow the same rule (`.env` adds process environment variables as a higher-priority tier on top). `metadata.srf` and `accounts.srf` are loaded from the same directory as the resolved portfolio file.
|
||||
|
||||
- **`transaction_log.srf` is a sibling file.** Optional. Lives next to `portfolio.srf` / `accounts.srf`. Holds user-declared `transfer::` records so the contributions pipeline can tell internal account-to-account movement apart from real external contributions. Only `type::cash` is wired in v1 - `type::in_kind` parses but is rejected downstream. Missing file -> matcher is a no-op. See `REPORT.md` section 5 "Transfer log" for the user-facing guide.
|
||||
- **`transaction_log.srf` is a sibling file.** Optional. Lives next to `portfolio.srf` / `accounts.srf`. Holds user-declared `transfer::` records so the contributions pipeline can tell internal account-to-account movement apart from real external contributions. Both `type::cash` (verified against the destination's dollar value, $1 tolerance) and `type::in_kind` (verified against share counts, 1%/0.01-share tolerance) are wired into the classifier. Missing file -> matcher is a no-op. See `docs/reference/config/transaction-log-srf.md` for the user-facing guide.
|
||||
|
||||
- **Server sync is optional.** The `ZFIN_SERVER` env var enables parallel cache syncing from a remote zfin-server instance. All server sync code silently no-ops when the URL is null.
|
||||
|
||||
|
|
|
|||
|
|
@ -89,8 +89,21 @@ transfer::2026-05-02,type::cash,amount:num:50000,from::Joint taxable,to::Pat Rot
|
|||
```
|
||||
|
||||
zfin matches the transfer against the diff and removes it from the
|
||||
attribution total. Only `type::cash` is wired up today; `in_kind`
|
||||
parses but isn't yet supported.
|
||||
attribution total.
|
||||
|
||||
If the **securities** moved rather than cash -- an ACAT transfer or an
|
||||
in-kind rollover -- use `type::in_kind` and point `dest_lot` at the lot
|
||||
that arrived:
|
||||
|
||||
```srf
|
||||
transfer::2026-05-02,type::in_kind,amount:num:87000,from::Old 401k,to::Pat Roth,dest_lot::VTI@2024-01-15
|
||||
```
|
||||
|
||||
In-kind records are verified by share count rather than by `amount`
|
||||
(which is informational there), so the shares that left have to match
|
||||
the shares that arrived. See the
|
||||
[reference](../reference/config/transaction-log-srf.md#typecash-vs-typein_kind)
|
||||
for the exact tolerances and what happens when they don't line up.
|
||||
|
||||
## Cash that *is* a contribution
|
||||
|
||||
|
|
|
|||
|
|
@ -2,8 +2,8 @@
|
|||
|
||||
`transaction_log.srf` is an optional sibling of `portfolio.srf` that
|
||||
declares real-world transactions which change how zfin interprets the
|
||||
portfolio diff. In v1 it holds exactly one kind of record:
|
||||
**transfers** -- money you moved between accounts you own.
|
||||
portfolio diff. It holds exactly one kind of record: **transfers** --
|
||||
money or securities you moved between accounts you own.
|
||||
|
||||
## Why it exists
|
||||
|
||||
|
|
@ -42,21 +42,63 @@ transfer::2026-05-02,type::cash,amount:num:4700,from::Sample IRA,to::Sample Brok
|
|||
| Field | Type | Required | Description |
|
||||
|------------|--------|----------|---------------------------------------------------------------------|
|
||||
| `transfer` | date | Yes | Transfer date (`YYYY-MM-DD`); the record key. |
|
||||
| `type` | string | Yes | `cash` or `in_kind` (see v1 scope below). |
|
||||
| `type` | string | Yes | `cash` (money moved) or `in_kind` (securities moved). See below. |
|
||||
| `amount` | num | Yes | Dollar amount transferred to this destination. |
|
||||
| `from` | string | Yes | Source account name (matches an `account::` in your portfolio). |
|
||||
| `to` | string | Yes | Destination account name. |
|
||||
| `dest_lot` | string | Yes | Where it landed: `cash`, or `SYMBOL@YYYY-MM-DD` for a specific lot. |
|
||||
|
||||
## v1 scope and limits
|
||||
## `type::cash` vs `type::in_kind`
|
||||
|
||||
Both are matched by the contributions classifier; they differ in what
|
||||
moved and therefore in how the record is verified.
|
||||
|
||||
**`type::cash`** -- dollars left one account and arrived at another,
|
||||
possibly getting invested on arrival. The `amount` is load-bearing: it's
|
||||
matched against the destination's value, and the three outcomes are
|
||||
|
||||
- **within $1** -- fully a transfer; contributes $0 to attribution.
|
||||
- **destination worth more than `amount`** -- a *partial* transfer. Only
|
||||
`amount` is cancelled out; the excess is still counted as new money.
|
||||
That's the "I moved $5k in and also added $2k of my own" case.
|
||||
- **`amount` exceeds the destination's value by more than $1** --
|
||||
rejected, and reported under **Flagged for review**. You can't have
|
||||
moved more into a lot than the lot is worth, so the record is
|
||||
presumed wrong rather than trusted.
|
||||
|
||||
**`type::in_kind`** -- the securities themselves moved (an ACAT
|
||||
transfer, an in-kind rollover); no cash changed hands. Requires
|
||||
`dest_lot::SYMBOL@YYYY-MM-DD` -- `dest_lot::cash` is rejected, since
|
||||
nothing can land as cash in a transfer where no cash moved.
|
||||
|
||||
```srf
|
||||
#!srfv1
|
||||
# 300 shares of VTI moved from one account to the other, no cash involved
|
||||
transfer::2026-05-02,type::in_kind,amount:num:87000,from::Sample IRA,to::Sample Brokerage,dest_lot::VTI@2024-01-15
|
||||
```
|
||||
|
||||
Here `amount` is informational only. The moved value comes from the
|
||||
destination lot itself, and verification is by **share count**: the
|
||||
shares that left `from` must match the shares that arrived at `to`,
|
||||
within 1% or 0.01 shares (whichever is larger). A mismatch is reported
|
||||
under **Flagged for review** rather than silently absorbing what might
|
||||
be a real contribution.
|
||||
|
||||
A missing source side is not an error for either type -- the sending
|
||||
account may be untracked (an external rollover origin, a 401k you don't
|
||||
model). Only the destination is required.
|
||||
|
||||
Because no cash funded an in-kind move, there is no partial outcome: a
|
||||
destination is either fully a transfer or not one at all.
|
||||
|
||||
## Scope and limits
|
||||
|
||||
- Only `transfer::` records. Buys, sells, and dividends stay inferred
|
||||
from the portfolio diff.
|
||||
- Only `type::cash` is wired into the contributions classifier.
|
||||
`type::in_kind` parses (the format is forward-compatible) but the
|
||||
matcher rejects it with an "in-kind transfers not yet supported"
|
||||
message.
|
||||
- Forward-looking only -- there is no historical reconstruction.
|
||||
- Account names are matched byte-exactly, so a
|
||||
[renamed account](../../guides/set-up-accounts.md#renaming-an-account)
|
||||
breaks records that reference the old name.
|
||||
|
||||
## See also
|
||||
|
||||
|
|
|
|||
|
|
@ -254,18 +254,6 @@ pub const AccountMap = struct {
|
|||
return null;
|
||||
}
|
||||
|
||||
/// Return all entries matching a given institution.
|
||||
pub fn entriesForInstitution(self: AccountMap, institution: []const u8) []const AccountTaxEntry {
|
||||
var count: usize = 0;
|
||||
for (self.entries) |e| {
|
||||
if (e.institution) |inst| {
|
||||
if (std.mem.eql(u8, inst, institution)) count += 1;
|
||||
}
|
||||
}
|
||||
if (count == 0) return &.{};
|
||||
return self.entries;
|
||||
}
|
||||
|
||||
/// Is cash-balance movement on `account` treated as a real
|
||||
/// contribution (vs. internal noise) for the attribution total?
|
||||
/// Defaults to false when the account isn't in the map.
|
||||
|
|
|
|||
|
|
@ -1633,9 +1633,11 @@ pub fn renderKeyComparisonRows(
|
|||
now: KeyMetrics,
|
||||
events_enabled: bool,
|
||||
) !void {
|
||||
// `then` and `now` are computed against the same projections.srf
|
||||
// (REPORT.md §4 - the "then" side reuses today's config), so
|
||||
// their horizons agree. Use whichever side is convenient.
|
||||
// `then` and `now` are computed against the same projections.srf:
|
||||
// the historical side re-runs today's config against the older
|
||||
// portfolio snapshot rather than trying to recover the config as it
|
||||
// was then (which isn't recorded anywhere). So both sides share a
|
||||
// horizon, and either one can supply the label.
|
||||
const events_label: []const u8 = if (events_enabled) "included" else "excluded";
|
||||
try cli.printFg(out, color, cli.CLR_MUTED, " ({d}-year horizon, lifecycle events {s})\n", .{ now.horizon_years, events_label });
|
||||
|
||||
|
|
|
|||
|
|
@ -53,8 +53,9 @@
|
|||
//! destination share-addition, per-symbol.
|
||||
//! - No historical reconstruction - forward-looking only.
|
||||
//!
|
||||
//! See `REPORT.md` §5 for the full usage guide and
|
||||
//! `src/commands/contributions.zig` for the classifier integration.
|
||||
//! See `docs/reference/config/transaction-log-srf.md` for the full
|
||||
//! usage guide and `src/commands/contributions.zig` for the classifier
|
||||
//! integration.
|
||||
|
||||
const std = @import("std");
|
||||
const builtin = @import("builtin");
|
||||
|
|
@ -201,31 +202,6 @@ pub const TransactionLog = struct {
|
|||
}
|
||||
self.allocator.free(self.transfers);
|
||||
}
|
||||
|
||||
/// Return transfers whose `transfer` date falls within `[start, end]`
|
||||
/// inclusive. The returned slice is allocator-owned - caller must
|
||||
/// free it.
|
||||
///
|
||||
/// Works only because `parseTransactionLogFile` preserves file
|
||||
/// order. If callers ever need chronological ordering regardless
|
||||
/// of file layout, sort on the way out instead of on the way in -
|
||||
/// file order is sometimes meaningful for a human reviewer
|
||||
/// (grouping related records together).
|
||||
pub fn transfersInWindow(
|
||||
self: *const TransactionLog,
|
||||
allocator: std.mem.Allocator,
|
||||
start: Date,
|
||||
end: Date,
|
||||
) ![]const TransferRecord {
|
||||
var out: std.ArrayList(TransferRecord) = .empty;
|
||||
errdefer out.deinit(allocator);
|
||||
for (self.transfers) |r| {
|
||||
if (r.transfer.days < start.days) continue;
|
||||
if (r.transfer.days > end.days) continue;
|
||||
try out.append(allocator, r);
|
||||
}
|
||||
return try out.toOwnedSlice(allocator);
|
||||
}
|
||||
};
|
||||
|
||||
/// Parse `data` (the contents of a `transaction_log.srf` file) into a
|
||||
|
|
@ -238,6 +214,11 @@ pub const TransactionLog = struct {
|
|||
/// only hard errors are allocator failures and SRF-level parse errors
|
||||
/// that prevent the iterator from starting at all.
|
||||
///
|
||||
/// Records come out in **file order**, deliberately not sorted by date.
|
||||
/// File order is often meaningful to a human reviewer - related records
|
||||
/// get grouped together - and any consumer that needs chronological
|
||||
/// ordering can sort on the way out. Pinned by a test below.
|
||||
///
|
||||
/// The SRF record layout is `transfer::<date>,type::<t>,amount:num:<n>,
|
||||
/// from::<a>,to::<b>,dest_lot::<dl>[,note::<n>]`. SRF's
|
||||
/// `fields.to(TransferRecord)` does the coercion: each key matches a
|
||||
|
|
@ -677,47 +658,6 @@ test "parseTransactionLogFile: malformed record skipped, subsequent record survi
|
|||
try testing.expectEqual(@as(f64, 3000), log.transfers[0].amount);
|
||||
}
|
||||
|
||||
test "transfersInWindow: inclusive on both ends" {
|
||||
var log = try parseTransactionLogFile(testing.allocator,
|
||||
\\#!srfv1
|
||||
\\transfer::2026-04-30,type::cash,amount:num:100,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\transfer::2026-05-01,type::cash,amount:num:200,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\transfer::2026-05-15,type::cash,amount:num:300,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\transfer::2026-05-31,type::cash,amount:num:400,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\transfer::2026-06-01,type::cash,amount:num:500,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\
|
||||
);
|
||||
defer log.deinit();
|
||||
|
||||
const slice = try log.transfersInWindow(
|
||||
testing.allocator,
|
||||
Date.fromYmd(2026, 5, 1),
|
||||
Date.fromYmd(2026, 5, 31),
|
||||
);
|
||||
defer testing.allocator.free(slice);
|
||||
try testing.expectEqual(@as(usize, 3), slice.len);
|
||||
try testing.expectEqual(@as(f64, 200), slice[0].amount);
|
||||
try testing.expectEqual(@as(f64, 300), slice[1].amount);
|
||||
try testing.expectEqual(@as(f64, 400), slice[2].amount);
|
||||
}
|
||||
|
||||
test "transfersInWindow: empty window returns empty slice" {
|
||||
var log = try parseTransactionLogFile(testing.allocator,
|
||||
\\#!srfv1
|
||||
\\transfer::2026-05-01,type::cash,amount:num:100,from::Acct A,to::Acct B,dest_lot::cash
|
||||
\\
|
||||
);
|
||||
defer log.deinit();
|
||||
|
||||
const slice = try log.transfersInWindow(
|
||||
testing.allocator,
|
||||
Date.fromYmd(2027, 1, 1),
|
||||
Date.fromYmd(2027, 12, 31),
|
||||
);
|
||||
defer testing.allocator.free(slice);
|
||||
try testing.expectEqual(@as(usize, 0), slice.len);
|
||||
}
|
||||
|
||||
test "parseTransactionLogFile: preserves file order (not sorted)" {
|
||||
var log = try parseTransactionLogFile(testing.allocator,
|
||||
\\#!srfv1
|
||||
|
|
@ -728,7 +668,7 @@ test "parseTransactionLogFile: preserves file order (not sorted)" {
|
|||
);
|
||||
defer log.deinit();
|
||||
try testing.expectEqual(@as(usize, 3), log.transfers.len);
|
||||
// File order preserved - see `transfersInWindow` docstring for why.
|
||||
// File order preserved, NOT date-sorted - see the parser's docstring.
|
||||
try testing.expectEqual(@as(f64, 3), log.transfers[0].amount);
|
||||
try testing.expectEqual(@as(f64, 1), log.transfers[1].amount);
|
||||
try testing.expectEqual(@as(f64, 2), log.transfers[2].amount);
|
||||
|
|
|
|||
Loading…
Add table
Reference in a new issue