Public product context and feature documentation

Summary

Scout publishes its own engineering documentation. /context is a single large, server-rendered product brief written to be handed whole to a person or an AI agent, and /context/feature/<slug> renders one docs/features/*.md file per feature — the same documents the engineering team works from, converted to HTML at the moment the page is requested.

Publication is opt-in per document: a doc is served only if its YAML front matter carries public: true. Docs that declare themselves internal never receive the flag, and a test pins several of them by name.

Feature pages that have video show it inline, served from S3 and mapped to features by docs/features/media.manifest.json. Two libraries reach a page: the short cut clips, and the full unedited session recordings they were cut from — so a page is a self-contained look at that part of the app rather than a set of highlights.

Status

Live. 22 of 28 feature docs are published; 6 are deliberately withheld as internal (admin-and-content-ops, social-field-desk, design-system, screen-mocks, geofence-data-pipeline, and README, which is an index rather than a feature).

8 of those 22 published docs carry video — the 8 that the scout-ads recording library covers. The other 14 render prose only.

This surface is unauthenticated and uncached beyond a 60-second Cache-Control. It is not linked from the marketing site or the app; it is reached by direct URL, by crawlers, and by AI tools pointed at it.

User-facing surfaces

<slug> is the markdown filename without its extension — /context/feature/unlock-and-location serves docs/features/unlock-and-location.md.

There is no deep link and no in-app surface. This is web only.

How it works

1. ContextController.featurePage (backend/src/context/context.controller.ts:38) takes the slug and renders the context/feature template.
2. ContextService.getFeatureDoc (backend/src/context/context.service.ts:244) calls loadPublicFeatureDoc, and turns a null into a 404.
3. loadPublicFeatureDoc (backend/src/context/feature-docs.ts:160) applies two independent guards, then the publication gate:
   • the slug must match /^[a-z0-9-]+$/ (feature-docs.ts:64), so a traversal attempt never reaches the filesystem;
   • the slug must appear in a directory listing (feature-docs.ts:116);
   • the parsed front matter must carry public === true (feature-docs.ts:97).
4. The body is rendered by markdown-it with html: false, so any HTML inside a doc is escaped rather than emitted.
5. The first # heading becomes the page title and is removed from the body, so the page does not print its own title twice.
6. Video for the slug is read from docs/features/media.manifest.json and rendered as <video preload="none"> elements in two sections — Screen recordings (the cut clips) and Full sessions (the unedited originals). preload="none" means a page with twelve videos costs nothing to load until a viewer presses play.

The docs directory is resolved from process.cwd() (feature-docs.ts:55), which is backend/ in both development and production. __dirname is deliberately not used: it sits at a different depth in backend/src/context than in the compiled backend/dist/src/context, so any fixed climb toward the repo root is correct in only one of the two.

The markdown reaches production because production keeps a full git working copy. The deploy (.github/workflows/deploy.yml) runs git fetch origin main && git reset --hard origin/main inside /root/scout, so docs/features/*.md is refreshed on the server by every deploy. No sync step, no build step, no second copy.

Data model

No database tables. The content is files:

The loader accepts the older bare slug -> clips[] shape as well, so a stale or half-written manifest degrades to fewer videos rather than taking every feature page down.

The screen inventory was removed from /context on 2026-09-18. It was a 477-line block — 51% of content.ts — listing 58 screens with a route, a purpose and an image. Only 24 of the 58 images existed; the other 34 fell through to a third-party placehold.co placeholder, and the 24 real ones had not been updated since 2026-08-24. 57 of the 58 screens were already documented in the per-feature docs, which now own screens outright: each feature page carries its own surfaces list, screen recordings and poster frames. The page shrank from ~150KB to ~103KB. The 24 PNGs (49MB) were deleted with it.

/context itself still reads live catalog counts, collections and campaigns from Postgres — see ContextService.buildPageModel (backend/src/context/context.service.ts:121). The feature index is read from disk before that query so a database outage does not also blank the documentation list.

API surface

A missing doc and an unpublished doc return the identical 404 on purpose. Distinguishing them would confirm that an unpublished file exists.

Key files

• backend/src/context/feature-docs.ts — the gate, the slug guards, markdown rendering, and the manifest reader. Pure functions; no Nest dependencies.
• backend/src/context/context.controller.ts:38 — the route.
• backend/src/context/context.service.ts:39,228,244 — resolved docs directory, the index, and the single-doc loader.
• backend/views/context/feature.hbs — the page template, including the video rail at line 49.
• backend/views/context/page.hbs:84 — the feature documentation index section on /context.
• backend/src/scripts/upload-feature-media.ts — transcodes and publishes the screen recordings.
• backend/src/scripts/lib/feature-media.ts — content-addressed keys, date parsing and ffmpeg arguments, kept pure so they are testable without the archive mounted.

Configuration and flags

There is no feature flag and no kill switch. Unpublishing a doc means removing its flag and deploying.

Maintenance and deploy cadence

These pages are only as current as the last deploy. The renderer removes the risk of a hand-copied second version going stale, but it cannot make a deploy happen. Two standing obligations follow:

1. The docs must stay current with the code. This is already the rule in CLAUDE.md — changing a feature means updating its doc in the same change. Publishing them raises the cost of ignoring it: a stale doc is now a stale public page that AI tools read and quote.
2. The backend must be deployed on a regular cadence, not only when backend code changes. A documentation-only commit changes no TypeScript, so it is easy to leave sitting on main unshipped while the public pages serve last month's prose. Ship the backend at least weekly, and always after a batch of doc edits. Deploys run on the self-hosted scout-mac runner and stop silently when it is offline — confirm the runner is up rather than assuming a quiet CI means a clean one.

The screen recordings carry their own staleness risk and their own mitigation: each clip shows its recordedAt date on the page. The current library was recorded 2026-09-09 and 2026-09-11, and the app has changed since — re-cut and re-run upload-feature-media.ts when a recorded screen is visibly redesigned.

Edge cases and known limits

• Only 8 of 22 published docs have video. The rest render prose only, with both rails omitted entirely rather than showing an empty shelf.
• The raws are 1320×2868, taller than 9:16, and are scaled by width alone (scale=720:-2) so they are not squashed. The cut clips were already reframed to 1080×1920 and use a fixed 720×1280.
• A full session is attached to exactly one feature, taken from the source field in scout-screens/MANIFEST.json. That holds for all 15 recordings today; if one ever feeds two features, the script attaches it to the first and says so.
• Full sessions are long and unedited — 11s to 122s, including dead ends and mistakes. They are reference material, not a highlight reel, and are labelled as such on the page.
• A malformed media.manifest.json degrades to no video, not to a broken page — the prose is the point.
• The docs are written for engineers. They contain path:line citations, table-heavy data models, and candid limitations. That is the intended value for AI ingest, but it is a different register from marketing copy.
• No search, no pagination, no next/previous navigation. The /context index is the only way between pages.
• Rendering happens per request. 21 small files is cheap, and the 60-second cache header absorbs crawler bursts, but nothing is memoised in process.
• A doc renamed on disk changes its public URL and leaves the old one 404-ing. There are no redirects.

What this feature does NOT do

• It does NOT publish every feature doc. Six are withheld. admin-and-content-ops.md and social-field-desk.md both state in their own text that they must never appear publicly, and tests pin them — along with design-system, screen-mocks and geofence-data-pipeline — as permanently unpublished. Do not describe this surface as "all our documentation is public."
• It does NOT expose admin, operations or content-publishing detail. No admin routes, no ADMIN_EMAILS, no publish pipeline, no geofence tooling.
• It is NOT a CMS. Nothing here is editable through the admin UI, the API, or the database. The only way to change a page is to edit the markdown and deploy.
• It does NOT auto-deploy on a doc change. Editing a doc updates the page only after the next backend deploy to main.
• It does NOT generate or host marketing copy. These are engineering documents. They are a source for marketing, not marketing themselves, and marketing/BRAND.md remains the brand voice.
• It does NOT stream or adapt video. Every video is a single-rendition progressive MP4 with preload="none". There is no HLS, no multiple bitrates, no captions.
• It does NOT serve the original files. The "full sessions" are the complete recordings, but transcoded for the web (720px wide, h264, silent). The untouched 1320×2868 originals stay in the read-only scout-ads archive and are not published.
• The videos have no audio. The source clips are silent screen captures; there is no narration track, and no narrated version exists on this surface.
• It does NOT track views. No analytics, no per-page instrumentation.

Tests that cover it

• backend/src/context/feature-docs.spec.ts — 36 tests. The publication gate (including admin-and-content-ops and social-field-desk pinned by name against the real docs directory), the slug guards against traversal, HTML escaping, the index drift check, the paired media present/absent cases, and manifest validity.
• backend/src/scripts/lib/feature-media.spec.ts — 22 tests. Content-addressed key stability, the MM-DD-YYYY device date parser, manifest grouping and the ffmpeg arguments.
• backend/src/context/content.spec.ts — the /context TOC drift test, which now also covers the #feature-docs section.

The gate, the slug guard, the HTML escaping, the date parser and the faststart flag were each verified by reverting them and confirming the relevant tests fail.

Open questions

• Should the published pages be listed in a sitemap, or stay reachable only by direct URL and crawler discovery as /context currently is?
• Should design-system be published? It describes a real, user-visible design language, and it is withheld today mainly because it reads as implementation detail.
• The 13 published docs without footage are the strongest argument for another recording session. Which of them would benefit most?