lobo/zfin-server
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages 1 Wiki Activity Actions

Compare commits

base: lobo:2e233fe3b6f10f712d97f10d565182041cb68758
Branches Tags
lobo:master
..
compare: lobo:e96ce91ac33fd97390316def5e3f3e79b5e64553
Branches Tags
lobo:master

No commits in common. "2e233fe3b6f10f712d97f10d565182041cb68758" and "e96ce91ac33fd97390316def5e3f3e79b5e64553" have entirely different histories.
2e233fe3b6 ... e96ce91ac3

3 changed files with 51 additions and 467 deletions
Show stats Expand all files Collapse all files

48
README.md
View file

@ -30,7 +30,6 @@ curl http://localhost:8080/AAPL/returns?fmt=xml
| `GET /symbols` | `application/json` | List of tracked symbols | | `GET /symbols` | `application/json` | List of tracked symbols |
| `GET /:symbol/returns` | `application/json` | Trailing 1/3/5/10yr returns + volatility | | `GET /:symbol/returns` | `application/json` | Trailing 1/3/5/10yr returns + volatility |
| `GET /:symbol/returns?fmt=xml` | `application/xml` | Same, XML for LibreCalc | | `GET /:symbol/returns?fmt=xml` | `application/xml` | Same, XML for LibreCalc |
| `GET /:symbol/watch` | `application/json` | Add the symbol to the watchlist (authenticated) |
| `GET /:symbol/quote` | `application/json` | Latest quote | | `GET /:symbol/quote` | `application/json` | Latest quote |
| `GET /:symbol/candles` | `application/x-srf` | Raw SRF cache file | | `GET /:symbol/candles` | `application/x-srf` | Raw SRF cache file |
| `GET /:symbol/dividends` | `application/x-srf` | Raw SRF cache file | | `GET /:symbol/dividends` | `application/x-srf` | Raw SRF cache file |
@ -42,52 +41,6 @@ curl http://localhost:8080/AAPL/returns?fmt=xml
| `GET /_edgar/tickers_funds` | `application/x-srf` | EDGAR mutual-fund ticker map (~3 MB) | | `GET /_edgar/tickers_funds` | `application/x-srf` | EDGAR mutual-fund ticker map (~3 MB) |
| `GET /_edgar/tickers_companies` | `application/x-srf` | EDGAR company ticker map (~5 MB) | | `GET /_edgar/tickers_companies` | `application/x-srf` | EDGAR company ticker map (~5 MB) |
The SRF endpoints behave as an L2 cache. A present cache file is served
as-is (even if stale -- the `refresh` cron is the freshness authority).
On a miss the server fetches once from the upstream provider, fills the
cache, and serves the result; it returns 404 only if that fetch also
fails. A miss never adds the symbol to the tracked set, so an ad-hoc
lookup doesn't create recurring `refresh` load.
## Authentication
`/`, `/help`, and `/:symbol/returns` (the LibreCalc endpoint) are always
public. Every other endpoint -- the raw SRF cache files, `/quote`,
`/symbols`, and the EDGAR ticker maps -- is gated behind a shared key
when `ZFIN_SERVER_API_KEY` is set:
```sh
export ZFIN_SERVER_API_KEY=your-secret
# Supply the key as a header (preferred)...
curl -H "X-API-Key: your-secret" http://localhost:8080/AAPL/candles
# ...or as a query parameter (curl convenience)
curl "http://localhost:8080/AAPL/candles?api_key=your-secret"
```
If `ZFIN_SERVER_API_KEY` is unset or empty, the server is fully open --
a soft cutover so the key can roll out to clients before enforcement is
switched on.
## Watchlist
Adding a symbol to the watchlist enrolls it in the `refresh` cron, so it
gets fetched on every run from then on -- the one operation that creates
recurring provider load. There are two ways in, gated to fit each
consumer:
- **`GET /:symbol/watch`** -- authenticated (API key), for your own
tooling deliberately growing the tracked set.
- **`GET /:symbol/returns?watch=true`** -- public, but the add only
happens when the request carries LibreOffice's `WEBSERVICE`
User-Agent. This lets the non-technical user add symbols from a
spreadsheet while keeping random callers from growing the cron set.
(Obscurity-grade: a User-Agent is trivially spoofable, which is
acceptable under the casual-traffic threat model.)
Adds are serialized and written atomically, and implausible symbols are
rejected, so a stray request can't corrupt or balloon `portfolio.srf`.
## LibreCalc usage ## LibreCalc usage
``` ```
@ -147,7 +100,6 @@ All configuration is via environment variables:
| `ZFIN_USER_EMAIL` | Yes | Contact email for SEC EDGAR User-Agent header | | `ZFIN_USER_EMAIL` | Yes | Contact email for SEC EDGAR User-Agent header |
| `ZFIN_PORTFOLIO` | No | Path to portfolio SRF (default: `portfolio.srf`) | | `ZFIN_PORTFOLIO` | No | Path to portfolio SRF (default: `portfolio.srf`) |
| `ZFIN_CACHE_DIR` | No | Cache directory (default: `~/.cache/zfin`) | | `ZFIN_CACHE_DIR` | No | Cache directory (default: `~/.cache/zfin`) |
| `ZFIN_SERVER_API_KEY` | No | If set, require this key on every endpoint except `/`, `/help`, and `/:symbol/returns` |
## License ## License

4
build.zig.zon
View file

@ -14,8 +14,8 @@
.hash = "httpz-0.0.0-PNVzrLjJCAD37S0CcrXpsjSqr86hVjK0rsALTDJ98AAJ", .hash = "httpz-0.0.0-PNVzrLjJCAD37S0CcrXpsjSqr86hVjK0rsALTDJ98AAJ",
}, },
.zfin = .{ .zfin = .{
.url = "git+https://git.lerch.org/lobo/zfin#020fb2db77a224458fedd8aa28dc5bb085a5a5c3", .url = "git+https://git.lerch.org/lobo/zfin#0cd01dd452338ddabdededb7acf4f34662d81434",
.hash = "zfin-0.0.0-J-B21qYiTgAoITt2wXuVtKxZdgsvltZl7Ye2wZqErIGw", .hash = "zfin-0.0.0-J-B21lviSwDjydtbQ1fAPHEHFeA7dE_0Uxd-mvbFOF3w",
}, },
}, },
} }

456
src/main.zig
View file

@ -27,19 +27,6 @@ const App = struct {
/// to 500ms. Captured once at App.init so the dispatch hot path /// to 500ms. Captured once at App.init so the dispatch hot path
/// doesn't re-parse on every request. /// doesn't re-parse on every request.
slow_threshold_ms: u64, slow_threshold_ms: u64,
/// Optional shared API key. When set (via `ZFIN_SERVER_API_KEY`),
/// every endpoint except the public surface (`/`, `/help`, and
/// `/:symbol/returns`) requires a matching key, supplied either as
/// an `X-API-Key` header or an `api_key` query parameter. When null
/// (env var unset or empty) the server is fully open - this
/// soft cutover lets the key roll out to clients before enforcement
/// is switched on. Captured once at init.
api_key: ?[]const u8,
/// Serializes the portfolio read-modify-write across concurrent
/// watchlist adds (httpz dispatches requests on multiple threads).
/// Without it two simultaneous adds could both read the old file and
/// the second writer would clobber the first's new symbol.
watch_mutex: std.Io.Mutex = .init,
fn init(io: std.Io, allocator: std.mem.Allocator, environ: *const std.process.Environ.Map) App { fn init(io: std.Io, allocator: std.mem.Allocator, environ: *const std.process.Environ.Map) App {
const config = zfin.Config.fromEnv(io, allocator, environ); const config = zfin.Config.fromEnv(io, allocator, environ);
@ -48,12 +35,6 @@ const App = struct {
std.fmt.parseInt(u64, s, 10) catch 500 std.fmt.parseInt(u64, s, 10) catch 500
else else
500; 500;
// Treat an empty value as unset so `ZFIN_SERVER_API_KEY=` can't
// accidentally enable enforcement with an empty (never-matching) key.
const api_key: ?[]const u8 = if (environ.get("ZFIN_SERVER_API_KEY")) |v|
(if (v.len == 0) null else v)
else
null;
return .{ return .{
.io = io, .io = io,
.environ = environ, .environ = environ,
@ -61,7 +42,6 @@ const App = struct {
.config = config, .config = config,
.svc = svc, .svc = svc,
.slow_threshold_ms = slow_threshold_ms, .slow_threshold_ms = slow_threshold_ms,
.api_key = api_key,
}; };
} }
@ -80,10 +60,9 @@ const App = struct {
// logging. `.awake` (monotonic) avoids spurious negatives // logging. `.awake` (monotonic) avoids spurious negatives
// on system clock skew. // on system clock skew.
const start_ns = std.Io.Timestamp.now(self.io, .awake).nanoseconds; const start_ns = std.Io.Timestamp.now(self.io, .awake).nanoseconds;
try action(self, req, res);
// Registered before the gate and the action so every exit path - // using defer here so we execute unconditionally
// a 401 from the gate, an error from the action, or a normal
// response - emits the slow/error log line.
defer { defer {
const elapsed_ns = std.Io.Timestamp.now(self.io, .awake).nanoseconds - start_ns; const elapsed_ns = std.Io.Timestamp.now(self.io, .awake).nanoseconds - start_ns;
const elapsed_ms: u64 = @intCast(@divTrunc(elapsed_ns, std.time.ns_per_ms)); const elapsed_ms: u64 = @intCast(@divTrunc(elapsed_ns, std.time.ns_per_ms));
@ -101,20 +80,6 @@ const App = struct {
}); });
} }
} }
// API-key gate. Soft cutover: only enforced when a key is
// configured. The public surface (`/`, `/help`, and the
// LibreOffice `/:symbol/returns` endpoint) is never gated.
if (self.api_key) |expected| {
if (!pathIsPublic(req.url.path) and !providedKeyMatches(req, expected)) {
res.status = 401;
res.content_type = httpz.ContentType.TEXT;
res.body = "Unauthorized: missing or invalid API key\n";
return;
}
}
try action(self, req, res);
} }
}; };
@ -124,93 +89,6 @@ fn shouldLogRequest(elapsed_ms: u64, status: u16, threshold_ms: u64) bool {
return elapsed_ms > threshold_ms or status >= 400; return elapsed_ms > threshold_ms or status >= 400;
} }
/// Pure predicate: is this request path part of the public surface that
/// never requires an API key? Public = the landing page, the help text,
/// and the LibreOffice returns endpoint (`/:symbol/returns`, including
/// its `?fmt=xml` and `?watch=true` variants - the query string is not
/// part of the path). Everything else (raw SRF cache files, quotes, the
/// symbol list, EDGAR maps, entity facts) is gated.
fn pathIsPublic(path: []const u8) bool {
if (std.mem.eql(u8, path, "/")) return true;
if (std.mem.eql(u8, path, "/help")) return true;
return isSymbolReturnsPath(path);
}
/// True only for paths shaped exactly like `/<symbol>/returns`: a single
/// non-empty symbol segment followed by the literal `returns`. Guards
/// against `/returns` (no symbol) and multi-segment look-alikes such as
/// `/a/b/returns` or `/AAPL/returns/extra`.
fn isSymbolReturnsPath(path: []const u8) bool {
const suffix = "/returns";
if (path.len <= suffix.len) return false;
if (!std.mem.endsWith(u8, path, suffix)) return false;
if (path[0] != '/') return false;
const symbol = path[1 .. path.len - suffix.len];
if (symbol.len == 0) return false;
if (std.mem.indexOfScalar(u8, symbol, '/') != null) return false;
return true;
}
/// Length-checked equality of a caller-supplied key against the
/// configured one. The threat model is casual/incidental traffic, not a
/// timing-attack adversary, so plain `mem.eql` is sufficient. A null
/// (absent header/param) never matches.
fn keyMatches(provided: ?[]const u8, expected: []const u8) bool {
const p = provided orelse return false;
return std.mem.eql(u8, p, expected);
}
/// Read the caller-supplied API key from the `X-API-Key` header
/// (preferred) or an `api_key` query parameter (curl convenience) and
/// compare against the configured key. httpz wants the header name in
/// lowercase. A malformed query string is treated as "no key".
fn providedKeyMatches(req: *httpz.Request, expected: []const u8) bool {
if (keyMatches(req.header("x-api-key"), expected)) return true;
const q = req.query() catch return false;
return keyMatches(q.get("api_key"), expected);
}
/// Case-insensitive User-Agent substrings permitted to add to the
/// watchlist via the public `/:symbol/returns?watch=true` path. This is
/// obscurity-grade (a UA is trivially spoofable) and matches the
/// casual-traffic threat model: it keeps crawlers and stray browsers
/// from growing the tracked set - and thus the recurring cron-refresh
/// load - while letting the non-technical user's LibreOffice WEBSERVICE
/// calls through. The authenticated `/:symbol/watch` route bypasses this
/// (a valid API key is a stronger signal than any UA).
///
/// Confirmed empirically - LibreOffice's WEBSERVICE sends e.g.
/// "LibreOffice 24.2.7.2 denylistedbackend/8.5.0 OpenSSL/3.0.13"
/// (it also fires a WebDAV OPTIONS preflight that 404s harmlessly; the
/// real GET carries the same User-Agent). Matching the version-agnostic
/// "LibreOffice" token keeps this robust across releases.
const watch_user_agents = [_][]const u8{"LibreOffice"};
/// True if `ua` matches one of `watch_user_agents` (case-insensitive
/// substring). A null/absent User-Agent never matches.
fn userAgentMayWatch(ua: ?[]const u8) bool {
const agent = ua orelse return false;
for (watch_user_agents) |needle| {
if (std.ascii.indexOfIgnoreCase(agent, needle) != null) return true;
}
return false;
}
/// Sanity gate for symbols entering the tracked set (and thus recurring
/// cron load): non-empty, <=16 chars, and only the characters real
/// tickers use - uppercase letters, digits, and `.`/`-` for class
/// shares. Not a real ticker validator; just enough to keep junk like an
/// over-long or path-shaped segment out of the portfolio file. Symbols
/// reach here already upper-cased by `upperDupe`.
fn isPlausibleSymbol(sym: []const u8) bool {
if (sym.len == 0 or sym.len > 16) return false;
for (sym) |c| {
const ok = (c >= 'A' and c <= 'Z') or (c >= '0' and c <= '9') or c == '.' or c == '-';
if (!ok) return false;
}
return true;
}
// ── Route handlers ─────────────────────────────────────────── // ── Route handlers ───────────────────────────────────────────
fn handleIndex(_: *App, _: *httpz.Request, res: *httpz.Response) !void { fn handleIndex(_: *App, _: *httpz.Request, res: *httpz.Response) !void {
@ -234,7 +112,6 @@ fn handleHelp(_: *App, _: *httpz.Request, res: *httpz.Response) !void {
\\Endpoints: \\Endpoints:
\\ GET /{SYMBOL}/returns Trailing 1/3/5/10yr returns (JSON) \\ GET /{SYMBOL}/returns Trailing 1/3/5/10yr returns (JSON)
\\ GET /{SYMBOL}/returns?fmt=xml Trailing returns (XML, for LibreCalc) \\ GET /{SYMBOL}/returns?fmt=xml Trailing returns (XML, for LibreCalc)
\\ GET /{SYMBOL}/watch Add SYMBOL to the watchlist (authenticated)
\\ GET /{SYMBOL}/quote Latest quote (JSON) \\ GET /{SYMBOL}/quote Latest quote (JSON)
\\ GET /{SYMBOL}/candles Raw SRF cache file \\ GET /{SYMBOL}/candles Raw SRF cache file
\\ GET /{SYMBOL}/candles_meta Candle freshness metadata (SRF) \\ GET /{SYMBOL}/candles_meta Candle freshness metadata (SRF)
@ -249,21 +126,6 @@ fn handleHelp(_: *App, _: *httpz.Request, res: *httpz.Response) !void {
\\ GET /_edgar/tickers_companies EDGAR company ticker map (SRF) \\ GET /_edgar/tickers_companies EDGAR company ticker map (SRF)
\\ GET /symbols List of tracked symbols \\ GET /symbols List of tracked symbols
\\ \\
\\Auth:
\\ All endpoints except /, /help, and /{SYMBOL}/returns require an
\\ API key when ZFIN_SERVER_API_KEY is set (X-API-Key header or
\\ ?api_key= query parameter).
\\
\\Caching:
\\ SRF endpoints serve from the local cache; on a miss the server
\\ fetches once from the provider, fills the cache, then serves
\\ (404 only if that fetch also fails).
\\
\\Watchlist (add a symbol to the cron refresh set):
\\ GET /{SYMBOL}/watch authenticated; for your own tooling
\\ GET /{SYMBOL}/returns?watch=true public, but only LibreOffice's
\\ WEBSERVICE User-Agent is honored
\\
\\Returns fields: \\Returns fields:
\\ lastClose Last closing price \\ lastClose Last closing price
\\ trailing{1,3,5,10}YearReturn Total return with dividend reinvestment \\ trailing{1,3,5,10}YearReturn Total return with dividend reinvestment
@ -327,20 +189,13 @@ fn handleReturns(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
const arena = res.arena; const arena = res.arena;
const symbol = try upperDupe(arena, raw_symbol); const symbol = try upperDupe(arena, raw_symbol);
// Auto-add to watchlist if requested. UA-gated (obscurity) so only // Auto-add to watchlist if requested
// LibreOffice WEBSERVICE calls - not random browsers/crawlers - can
// grow the tracked set via this public endpoint. Best-effort: the
// returns response below is served regardless of whether the add ran.
const q = try req.query(); const q = try req.query();
if (q.get("watch")) |w| { if (q.get("watch")) |w| {
if (std.ascii.eqlIgnoreCase(w, "true")) { if (std.ascii.eqlIgnoreCase(w, "true")) {
if (userAgentMayWatch(req.header("user-agent"))) {
appendWatchSymbol(app, symbol) catch |err| { appendWatchSymbol(app, symbol) catch |err| {
log.warn("failed to append watch symbol {s}: {t}", .{ symbol, err }); log.warn("failed to append watch symbol {s}: {}", .{ symbol, err });
}; };
} else {
log.debug("watch add for {s} skipped: User-Agent not allowlisted", .{symbol});
}
} }
} }
@ -468,36 +323,6 @@ fn handleReturns(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
}); });
} }
/// Authenticated explicit watchlist add. Not on the public allowlist, so
/// `dispatch` requires the API key - unlike the UA-gated `?watch=true`
/// path, this is for the operator's own tooling deliberately growing the
/// tracked set (and accepting the recurring cron-refresh cost).
fn handleWatch(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
const raw_symbol = req.param("symbol") orelse {
res.status = 400;
res.body = "Missing symbol";
return;
};
const arena = res.arena;
const symbol = try upperDupe(arena, raw_symbol);
appendWatchSymbol(app, symbol) catch |err| switch (err) {
error.InvalidSymbol => {
res.status = 400;
res.body = "Invalid symbol";
return;
},
else => {
res.status = 500;
res.body = try std.fmt.allocPrint(arena, "Failed to add watch symbol: {t}", .{err});
return;
},
};
res.content_type = httpz.ContentType.JSON;
res.body = try std.fmt.allocPrint(arena, "{{\"symbol\":\"{s}\",\"watched\":true}}", .{symbol});
}
fn handleQuote(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleQuote(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
const raw_symbol = req.param("symbol") orelse { const raw_symbol = req.param("symbol") orelse {
res.status = 400; res.status = 400;
@ -519,25 +344,8 @@ fn handleQuote(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
, .{ symbol, q.close, q.open, q.high, q.low, q.volume, q.previous_close }); , .{ symbol, q.close, q.open, q.high, q.low, q.volume, q.previous_close });
} }
/// Identifies which `DataService` fetch to run when a served SRF file is fn handleSrfFile(app: *App, req: *httpz.Request, res: *httpz.Response, filename: []const u8) !void {
/// absent (see `fetchOnMiss`). Kept separate from `zfin.cache.DataType` return handleSrfFileByKey(app, req, res, "symbol", filename);
/// because the mapping isn't 1:1 - both `candles_daily.srf` and
/// `candles_meta.srf` are populated by a single `getCandles` call.
const SrfKind = enum {
candles,
dividends,
splits,
earnings,
options,
classification,
etf_metrics,
entity_facts,
tickers_funds,
tickers_companies,
};
fn handleSrfFile(app: *App, req: *httpz.Request, res: *httpz.Response, filename: []const u8, kind: SrfKind) !void {
return handleSrfFileByKey(app, req, res, "symbol", filename, kind);
} }
/// Generalized SRF cache-file passthrough: reads /// Generalized SRF cache-file passthrough: reads
@ -546,7 +354,7 @@ fn handleSrfFile(app: *App, req: *httpz.Request, res: *httpz.Response, filename:
/// uses `"symbol"`; CIK-keyed routes (e.g. `/:cik/entity_facts`) /// uses `"symbol"`; CIK-keyed routes (e.g. `/:cik/entity_facts`)
/// pass `"cik"` instead. The cache-key segment is uppercased /// pass `"cik"` instead. The cache-key segment is uppercased
/// (safe for both symbols and zero-padded CIK digit strings). /// (safe for both symbols and zero-padded CIK digit strings).
fn handleSrfFileByKey(app: *App, req: *httpz.Request, res: *httpz.Response, key_param: []const u8, filename: []const u8, kind: SrfKind) !void { fn handleSrfFileByKey(app: *App, req: *httpz.Request, res: *httpz.Response, key_param: []const u8, filename: []const u8) !void {
const raw_key = req.param(key_param) orelse { const raw_key = req.param(key_param) orelse {
res.status = 400; res.status = 400;
res.body = "Missing key"; res.body = "Missing key";
@ -554,7 +362,7 @@ fn handleSrfFileByKey(app: *App, req: *httpz.Request, res: *httpz.Response, key_
}; };
const arena = res.arena; const arena = res.arena;
const key = try upperDupe(arena, raw_key); const key = try upperDupe(arena, raw_key);
return serveSrfFile(app, res, key, filename, kind); return serveSrfFile(app, res, key, filename);
} }
/// Static-key SRF cache-file passthrough for routes that don't /// Static-key SRF cache-file passthrough for routes that don't
@ -562,30 +370,21 @@ fn handleSrfFileByKey(app: *App, req: *httpz.Request, res: *httpz.Response, key_
/// `<cache_dir>/_edgar/tickers_funds.srf` directly). The `key` /// `<cache_dir>/_edgar/tickers_funds.srf` directly). The `key`
/// is a literal directory name; not uppercased because the /// is a literal directory name; not uppercased because the
/// cache uses `_edgar` as-is. /// cache uses `_edgar` as-is.
fn handleStaticSrfFile(app: *App, res: *httpz.Response, key: []const u8, filename: []const u8, kind: SrfKind) !void { fn handleStaticSrfFile(app: *App, res: *httpz.Response, key: []const u8, filename: []const u8) !void {
return serveSrfFile(app, res, key, filename, kind); return serveSrfFile(app, res, key, filename);
} }
/// Inner shared helper. Serves `<cache_dir>/<key>/<filename>` as raw SRF /// Inner shared helper. Reads the file, computes etag, sets
/// with a sha256 ETag. L2-cache contract: a *present* file is served /// headers, sends. Caller has already resolved the cache-key
/// as-is even when stale - cron is the freshness authority, so reads /// segment (per-request param or static literal).
/// never trigger a refetch - while an *absent* file triggers a one-shot fn serveSrfFile(app: *App, res: *httpz.Response, key: []const u8, filename: []const u8) !void {
/// provider fetch (`fetchOnMiss`) to populate it, after which we re-read
/// and serve. If the fetch still can't produce the file, we fall back to
/// the original 404.
fn serveSrfFile(app: *App, res: *httpz.Response, key: []const u8, filename: []const u8, kind: SrfKind) !void {
const arena = res.arena; const arena = res.arena;
const path = try std.fs.path.join(arena, &.{ app.config.cache_dir, key, filename }); const path = try std.fs.path.join(arena, &.{ app.config.cache_dir, key, filename });
const content = std.Io.Dir.cwd().readFileAlloc(app.io, path, arena, .limited(10 * 1024 * 1024)) catch {
const content = readCacheFile(app, arena, path) orelse blk: {
// Cache miss -> fetch from the provider, fill the cache, re-read.
fetchOnMiss(app, key, kind);
break :blk readCacheFile(app, arena, path) orelse {
res.status = 404; res.status = 404;
res.body = "Cache file not found"; res.body = "Cache file not found";
return; return;
}; };
};
// Body integrity header: sha256 of the bytes we're about to send. // Body integrity header: sha256 of the bytes we're about to send.
// Clients can use this to detect mid-stream truncation that Zig's // Clients can use this to detect mid-stream truncation that Zig's
@ -607,129 +406,54 @@ fn serveSrfFile(app: *App, res: *httpz.Response, key: []const u8, filename: []co
res.body = content; res.body = content;
} }
/// Read a cache file into the request arena, or null when it can't be
/// read. Absent is the common case; any read error is treated as a miss
/// (matches the pre-fetch-on-miss behavior of falling back to 404).
fn readCacheFile(app: *App, arena: std.mem.Allocator, path: []const u8) ?[]u8 {
return std.Io.Dir.cwd().readFileAlloc(app.io, path, arena, .limited(10 * 1024 * 1024)) catch null;
}
/// Populate the cache for `key` by running the matching `DataService`
/// fetch, then discard the parsed result - the caller re-reads the
/// canonical bytes off disk. Best-effort and synchronous: any provider
/// error (NotFound, rate limit, auth, transient, parse) is logged and
/// swallowed so the caller falls back to a 404.
///
/// Backpressure caveat: these fetches share zfin's rate limiter, so when
/// the token bucket is drained (e.g. mid-cron) this blocks the httpz
/// worker until a token frees. Accepted for now; a 202 + poll path is
/// the planned escape hatch if blocking becomes a problem.
fn fetchOnMiss(app: *App, key: []const u8, kind: SrfKind) void {
const svc = &app.svc;
switch (kind) {
// getCandles writes both candles_daily.srf and candles_meta.srf.
.candles => {
const r = svc.getCandles(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.dividends => {
const r = svc.getDividends(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.splits => {
const r = svc.getSplits(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.earnings => {
const r = svc.getEarnings(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.options => {
const r = svc.getOptions(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.classification => {
const r = svc.getClassification(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.etf_metrics => {
const r = svc.getEtfMetrics(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
// `key` is the CIK here (resolved from the :cik route param).
.entity_facts => {
const r = svc.getEntityFacts(key, .{}) catch |err| return logFetchMiss(key, kind, err);
r.deinit();
},
.tickers_funds => {
var m = svc.loadMutualFundTickerMap(.{}) catch |err| return logFetchMiss(key, kind, err);
m.deinit();
},
.tickers_companies => {
var m = svc.loadCompanyTickerMap(.{}) catch |err| return logFetchMiss(key, kind, err);
m.deinit();
},
}
}
/// Log a failed populate. NotFound is the normal "no such symbol / no
/// data" outcome (debug); everything else is operator-relevant (warn).
fn logFetchMiss(key: []const u8, kind: SrfKind, err: anyerror) void {
if (err == error.NotFound) {
log.info("fetch-on-miss {s} {s}: {s}", .{ key, @tagName(kind), @errorName(err) });
} else {
log.warn("fetch-on-miss {s} {s}: {s}", .{ key, @tagName(kind), @errorName(err) });
}
}
fn handleCandles(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleCandles(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "candles_daily.srf", .candles); return handleSrfFile(app, req, res, "candles_daily.srf");
} }
fn handleCandlesMeta(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleCandlesMeta(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "candles_meta.srf", .candles); return handleSrfFile(app, req, res, "candles_meta.srf");
} }
fn handleDividends(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleDividends(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "dividends.srf", .dividends); return handleSrfFile(app, req, res, "dividends.srf");
} }
fn handleSplits(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleSplits(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "splits.srf", .splits); return handleSrfFile(app, req, res, "splits.srf");
} }
fn handleEarnings(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleEarnings(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "earnings.srf", .earnings); return handleSrfFile(app, req, res, "earnings.srf");
} }
fn handleOptions(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleOptions(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "options.srf", .options); return handleSrfFile(app, req, res, "options.srf");
} }
fn handleClassification(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleClassification(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "classification.srf", .classification); return handleSrfFile(app, req, res, "classification.srf");
} }
fn handleEtfMetrics(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleEtfMetrics(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
return handleSrfFile(app, req, res, "etf_metrics.srf", .etf_metrics); return handleSrfFile(app, req, res, "etf_metrics.srf");
} }
fn handleEntityFacts(app: *App, req: *httpz.Request, res: *httpz.Response) !void { fn handleEntityFacts(app: *App, req: *httpz.Request, res: *httpz.Response) !void {
// CIK-keyed route: cache layout is // CIK-keyed route: cache layout is
// `<cache_dir>/<CIK>/entity_facts.srf` (the CIK is the // `<cache_dir>/<CIK>/entity_facts.srf` (the CIK is the
// zero-padded 10-digit string Wikidata's P5531 emits). // zero-padded 10-digit string Wikidata's P5531 emits).
return handleSrfFileByKey(app, req, res, "cik", "entity_facts.srf", .entity_facts); return handleSrfFileByKey(app, req, res, "cik", "entity_facts.srf");
} }
fn handleTickersFunds(app: *App, _: *httpz.Request, res: *httpz.Response) !void { fn handleTickersFunds(app: *App, _: *httpz.Request, res: *httpz.Response) !void {
// Static-key route: `<cache_dir>/_edgar/tickers_funds.srf` // Static-key route: `<cache_dir>/_edgar/tickers_funds.srf`
// is a single file shared across all symbol lookups, not a // is a single file shared across all symbol lookups, not a
// per-symbol cache. // per-symbol cache.
return handleStaticSrfFile(app, res, "_edgar", "tickers_funds.srf", .tickers_funds); return handleStaticSrfFile(app, res, "_edgar", "tickers_funds.srf");
} }
fn handleTickersCompanies(app: *App, _: *httpz.Request, res: *httpz.Response) !void { fn handleTickersCompanies(app: *App, _: *httpz.Request, res: *httpz.Response) !void {
return handleStaticSrfFile(app, res, "_edgar", "tickers_companies.srf", .tickers_companies); return handleStaticSrfFile(app, res, "_edgar", "tickers_companies.srf");
} }
// ── Helpers ────────────────────────────────────────────────── // ── Helpers ──────────────────────────────────────────────────
@ -851,24 +575,13 @@ fn fmtInt(arena: std.mem.Allocator, value: ?u8) []const u8 {
return "null"; return "null";
} }
/// Append a watch lot for `symbol` to the portfolio SRF file, unless it /// Append a watch lot for the given symbol to the portfolio SRF file,
/// is already tracked. Serialized across requests via `app.watch_mutex` /// unless it already exists. Best-effort — errors are logged, not fatal.
/// and written atomically, so a concurrent add or a mid-write crash can't
/// clobber or truncate the portfolio file. Returns `error.InvalidSymbol`
/// for implausible symbols; callers decide how loud to be.
fn appendWatchSymbol(app: *App, symbol: []const u8) !void { fn appendWatchSymbol(app: *App, symbol: []const u8) !void {
if (!isPlausibleSymbol(symbol)) return error.InvalidSymbol;
const portfolio_path = app.environ.get("ZFIN_PORTFOLIO") orelse "portfolio.srf"; const portfolio_path = app.environ.get("ZFIN_PORTFOLIO") orelse "portfolio.srf";
const allocator = app.allocator; const allocator = app.allocator;
const io = app.io; const io = app.io;
// Serialize the whole read-modify-write so concurrent adds don't lose
// updates (a last-writer-wins race would otherwise drop a symbol).
// Uncancelable so a canceled request can't abandon a half-done write.
app.watch_mutex.lockUncancelable(io);
defer app.watch_mutex.unlock(io);
// Read and deserialize existing portfolio (or start empty) // Read and deserialize existing portfolio (or start empty)
const file_data = std.Io.Dir.cwd().readFileAlloc(io, portfolio_path, allocator, .limited(10 * 1024 * 1024)) catch |err| { const file_data = std.Io.Dir.cwd().readFileAlloc(io, portfolio_path, allocator, .limited(10 * 1024 * 1024)) catch |err| {
if (err == error.FileNotFound) return writeNewPortfolio(io, allocator, portfolio_path, symbol); if (err == error.FileNotFound) return writeNewPortfolio(io, allocator, portfolio_path, symbol);
@ -885,7 +598,7 @@ fn appendWatchSymbol(app: *App, symbol: []const u8) !void {
} }
// Build new lot list with the watch entry appended // Build new lot list with the watch entry appended
const new_lots = try allocator.alloc(zfin.Lot, portfolio.lots.len + 1); var new_lots = try allocator.alloc(zfin.Lot, portfolio.lots.len + 1);
defer allocator.free(new_lots); defer allocator.free(new_lots);
@memcpy(new_lots[0..portfolio.lots.len], portfolio.lots); @memcpy(new_lots[0..portfolio.lots.len], portfolio.lots);
new_lots[portfolio.lots.len] = .{ new_lots[portfolio.lots.len] = .{
@ -896,10 +609,16 @@ fn appendWatchSymbol(app: *App, symbol: []const u8) !void {
.security_type = .watch, .security_type = .watch,
}; };
// Serialize and write atomically. // Serialize and write
const output = try zfin.cache.serializePortfolio(allocator, new_lots); const output = try zfin.cache.serializePortfolio(allocator, new_lots);
defer allocator.free(output); defer allocator.free(output);
try writeFileAtomic(io, allocator, portfolio_path, output);
const file = try std.Io.Dir.cwd().createFile(io, portfolio_path, .{});
defer file.close(io);
var write_buf: [4096]u8 = undefined;
var fw = file.writer(io, &write_buf);
try fw.interface.writeAll(output);
try fw.interface.flush();
log.info("added watch symbol {s} to {s}", .{ symbol, portfolio_path }); log.info("added watch symbol {s} to {s}", .{ symbol, portfolio_path });
} }
@ -914,41 +633,17 @@ fn writeNewPortfolio(io: std.Io, allocator: std.mem.Allocator, path: []const u8,
}}; }};
const output = try zfin.cache.serializePortfolio(allocator, &lot); const output = try zfin.cache.serializePortfolio(allocator, &lot);
defer allocator.free(output); defer allocator.free(output);
try writeFileAtomic(io, allocator, path, output);
const file = try std.Io.Dir.cwd().createFile(io, path, .{});
defer file.close(io);
var write_buf: [4096]u8 = undefined;
var fw = file.writer(io, &write_buf);
try fw.interface.writeAll(output);
try fw.interface.flush();
log.info("created {s} with watch symbol {s}", .{ path, symbol }); log.info("created {s} with watch symbol {s}", .{ path, symbol });
} }
/// Crash-safe file write: write to `<path>.tmp`, fsync, then rename over
/// `path`. A mid-write crash leaves the prior file intact rather than a
/// truncated portfolio. (zfin's internal `atomic.writeFileAtomic` isn't
/// part of its public module, so we keep a small local copy.)
fn writeFileAtomic(io: std.Io, allocator: std.mem.Allocator, path: []const u8, bytes: []const u8) !void {
const tmp_path = try std.fmt.allocPrint(allocator, "{s}.tmp", .{path});
defer allocator.free(tmp_path);
{
var tmp_file = try std.Io.Dir.cwd().createFile(io, tmp_path, .{ .truncate = true, .exclusive = false });
errdefer {
tmp_file.close(io);
std.Io.Dir.cwd().deleteFile(io, tmp_path) catch |err| {
log.debug("atomic write cleanup deleteFile({s}): {t}", .{ tmp_path, err });
};
}
try tmp_file.writeStreamingAll(io, bytes);
// fsync so the data is durable before the rename appears.
try tmp_file.sync(io);
tmp_file.close(io);
}
std.Io.Dir.cwd().rename(tmp_path, std.Io.Dir.cwd(), path, io) catch |err| {
std.Io.Dir.cwd().deleteFile(io, tmp_path) catch |del_err| {
log.debug("atomic write cleanup deleteFile({s}): {t}", .{ tmp_path, del_err });
};
return err;
};
}
// ── Refresh command ────────────────────────────────────────── // ── Refresh command ──────────────────────────────────────────
fn refresh(io: std.Io, allocator: std.mem.Allocator, environ: *const std.process.Environ.Map) !u8 { fn refresh(io: std.Io, allocator: std.mem.Allocator, environ: *const std.process.Environ.Map) !u8 {
@ -1322,10 +1017,6 @@ pub fn main(init: std.process.Init) !u8 {
// Symbol routes // Symbol routes
router.get("/:symbol/returns", handleReturns, .{}); router.get("/:symbol/returns", handleReturns, .{});
// Authenticated explicit watchlist add (API key required - not on
// the public allowlist). Distinct from the UA-gated
// /:symbol/returns?watch=true path used by LibreOffice.
router.get("/:symbol/watch", handleWatch, .{});
router.get("/:symbol/quote", handleQuote, .{}); router.get("/:symbol/quote", handleQuote, .{});
router.get("/:symbol/candles", handleCandles, .{}); router.get("/:symbol/candles", handleCandles, .{});
router.get("/:symbol/candles_meta", handleCandlesMeta, .{}); router.get("/:symbol/candles_meta", handleCandlesMeta, .{});
@ -1370,9 +1061,6 @@ fn printUsage(io: std.Io) !void {
\\ \\
\\Environment: \\Environment:
\\ ZFIN_PORTFOLIO Path to portfolio SRF file (default: portfolio.srf) \\ ZFIN_PORTFOLIO Path to portfolio SRF file (default: portfolio.srf)
\\ ZFIN_SERVER_API_KEY If set, require this key on every endpoint
\\ except /, /help, and /:symbol/returns. Supply it
\\ as an X-API-Key header or ?api_key= query param.
\\ TWELVEDATA_API_KEY TwelveData API key \\ TWELVEDATA_API_KEY TwelveData API key
\\ POLYGON_API_KEY Polygon API key \\ POLYGON_API_KEY Polygon API key
\\ FINNHUB_API_KEY Finnhub API key \\ FINNHUB_API_KEY Finnhub API key
@ -1436,62 +1124,6 @@ test "shouldLogRequest: custom threshold respected" {
try std.testing.expect(shouldLogRequest(2500, 200, 2000)); try std.testing.expect(shouldLogRequest(2500, 200, 2000));
} }
test "pathIsPublic: public surface (no key required)" {
try std.testing.expect(pathIsPublic("/"));
try std.testing.expect(pathIsPublic("/help"));
try std.testing.expect(pathIsPublic("/AAPL/returns"));
// Symbols with dots (class shares) still match.
try std.testing.expect(pathIsPublic("/BRK.B/returns"));
}
test "pathIsPublic: gated surface (key required)" {
try std.testing.expect(!pathIsPublic("/AAPL/candles"));
try std.testing.expect(!pathIsPublic("/AAPL/quote"));
try std.testing.expect(!pathIsPublic("/symbols"));
try std.testing.expect(!pathIsPublic("/_edgar/tickers_funds"));
try std.testing.expect(!pathIsPublic("/0000320193/entity_facts"));
}
test "pathIsPublic: returns look-alikes do not slip through" {
try std.testing.expect(!pathIsPublic("/returns")); // no symbol
try std.testing.expect(!pathIsPublic("/a/b/returns")); // extra segment
try std.testing.expect(!pathIsPublic("/AAPL/returns/extra")); // suffix, not exact
try std.testing.expect(!pathIsPublic("/help/secret")); // help prefix only
try std.testing.expect(!pathIsPublic("//returns")); // empty symbol
}
test "keyMatches" {
try std.testing.expect(keyMatches("s3cret", "s3cret"));
try std.testing.expect(!keyMatches("s3cret", "other"));
try std.testing.expect(!keyMatches(null, "s3cret"));
try std.testing.expect(!keyMatches("", "s3cret"));
// Length-checked: neither a prefix nor an extension matches.
try std.testing.expect(!keyMatches("s3cre", "s3cret"));
try std.testing.expect(!keyMatches("s3cretX", "s3cret"));
}
test "userAgentMayWatch" {
try std.testing.expect(userAgentMayWatch("LibreOffice 24.8"));
try std.testing.expect(userAgentMayWatch("libreoffice")); // case-insensitive
try std.testing.expect(userAgentMayWatch("Mozilla/5.0 LibreOffice/7.6"));
try std.testing.expect(!userAgentMayWatch("Mozilla/5.0 (X11; Linux x86_64)"));
try std.testing.expect(!userAgentMayWatch("curl/8.14.1"));
try std.testing.expect(!userAgentMayWatch(null));
try std.testing.expect(!userAgentMayWatch(""));
}
test "isPlausibleSymbol" {
try std.testing.expect(isPlausibleSymbol("AAPL"));
try std.testing.expect(isPlausibleSymbol("BRK.B"));
try std.testing.expect(isPlausibleSymbol("BRK-B"));
try std.testing.expect(isPlausibleSymbol("X"));
try std.testing.expect(!isPlausibleSymbol("")); // empty
try std.testing.expect(!isPlausibleSymbol("aapl")); // lowercase (upper-cased before this)
try std.testing.expect(!isPlausibleSymbol("AB CD")); // space
try std.testing.expect(!isPlausibleSymbol("../etc/passwd")); // path-shaped junk
try std.testing.expect(!isPlausibleSymbol("ABCDEFGHIJKLMNOPQ")); // 17 chars, too long
}
test "refreshExit: hard failure dominates, then lag, else clean" { test "refreshExit: hard failure dominates, then lag, else clean" {
try std.testing.expectEqual(@as(u8, 0), refreshExit(0, 0)); try std.testing.expectEqual(@as(u8, 0), refreshExit(0, 0));
try std.testing.expectEqual(@as(u8, 75), refreshExit(0, 3)); try std.testing.expectEqual(@as(u8, 75), refreshExit(0, 3));