sync-and-catalog-delivery

Summary

Scout's mobile app does not ship its catalog (categories, collections, campaigns, patches, patch types, patch↔collection memberships) inside the app bundle. It pulls the whole thing from one unauthenticated backend endpoint, GET /api/sync/content, on first launch, then keeps it warm with periodic deltas. The response is cached client-side in a TanStack Query cache (persisted to its own MMKV store), not in the app's Zustand store, which itself only persists a different MMKV key (scout-app-store) holding purely local/user data. The app is explicitly not offline-first: reading the already-synced catalog works offline, but the two things a "collect" actually needs — turning up new/changed content and confirming you're standing inside a polygon-based patch's boundary — both require a live network round trip. Publishing new content on the backend does not push to devices; a device discovers it by polling, gated by a 5-minute staleTime and refetch-on-foreground/refetch-on-reconnect, so a freshly published patch typically reaches an already-open app within 5 minutes and a backgrounded/relaunched app immediately (subject to that same staleness window).

Status

Live in production. This is the only mechanism the mobile app has for discovering catalog content — there is no bundled seed data and no CDN-fronted static catalog file.

Surfaces

• GET /api/sync/content (backend/src/sync/sync.controller.ts:17) — full/delta catalog pull. Unauthenticated.
• GET /api/sync/patch/:id (backend/src/sync/sync.controller.ts:29) — single-patch fetch for deep links to a patch not present in the synced set (e.g. an admin_only/store-exclusive patch). Unauthenticated, and deliberately does not apply visibility filtering.
• Mobile: mobile/src/query/queries/content.ts (contentQuery) is the client-side query that calls the endpoint, merges deltas, and is the thing every screen ultimately reads from.
• Mobile: mobile/src/hooks/useContentSync.ts — the loading/error/retry surface screens actually mount.
• Mobile: mobile/src/providers/SyncProvider.tsx — owns the background refetch triggers, the syncNow() escape hatch, and (unrelated to catalog delivery but living in the same provider) flushing locally-queued user-data writes.

How it works

Server side. SyncService.getContent(since?, isAdmin) (backend/src/sync/sync.service.ts:194) is a single method that serves both the full sync (no since) and the delta sync (since = an ISO timestamp) — the query shape barely changes between the two:

• categories, patchTypes: filtered by updatedAt >= since when a cursor is given, nothing else.
• collections: same since filter, plus adminOnly: false for non-admin callers (sync.service.ts:203-205).
• patches: same since filter, plus VISIBLE_PATCH_WHERE for non-admins — a patch is visible only if it belongs to at least one non-admin_only collection (backend/src/common/patch-visibility.ts:31-33, applied at sync.service.ts:207-209).
• patchCollections (the patch↔collection membership rows): filtered to collection.adminOnly = false for non-admins, but not scoped by since at all (sync.service.ts:211-213) — every content sync, delta or full, returns the entire current membership table for visible collections, not just rows touched since the cursor.
• campaigns: same since filter, plus a requirement that at least one member collection is non-admin_only, so a campaign whose collections are all store-exclusive stays invisible rather than appearing as an empty card (sync.service.ts:215-226).

isAdmin is derived per-request from an optional bearer token or admin_token cookie via isAdminEmailFromToken (sync.controller.ts:19-21); admins get the unfiltered catalog (for in-app QA of unpublished/store-exclusive content), everyone else gets the filtered one. The filter is not gated on NODE_ENV — the prod droplet runs with NODE_ENV unset, and an env-based gate was the actual production bug this code fixed (sync.service.ts:196-201, regression-tested in backend/src/sync/__tests__/sync.service.admin-only.spec.ts).

Everything comes back as one JSON object, snake_case, built by a single $transaction of six findMany calls plus one raw-SQL pass computing two derived polygon metrics (bounding-box "reach" in meters and true polygon area) that get attached to each patch (sync.service.ts:232-271). The whole call is wrapped in withDbRetry for the transient "connection recycled by the pooler" case (sync.service.ts:228-251).

Client side. contentQuery (mobile/src/query/queries/content.ts:49) is a TanStack Query queryOptions object with staleTime: 5 min and gcTime: Infinity. Its queryFn deliberately does not just return what the server sent: it reads the previous cached value via queryClient.getQueryData, calls api.content.sync(prev?.syncedAt), transforms the response (snake_case → camelCase, per-entity transformX functions from mobile/src/domain/store.ts:150-444), and merges the incoming delta into the previous cache by id (mergeById, store.ts:445-452; mergePatchCollections, store.ts:453-464) before returning the merged whole as the new cache value. The delta cursor (syncedAt) is stored inside the cached value itself, not beside it, specifically so that losing the cache also loses the cursor and forces a correct full resync (comment at content.ts:28-36).

Refetching is driven by ordinary TanStack Query mechanics wired to React Native via mobile/src/query/bindings.ts:19-35: onlineManager bound to NetInfo, focusManager bound to AppState. That gives the query client's defaults — refetchOnWindowFocus: true, refetchOnReconnect: true (mobile/src/query/client.ts:25-33) — real triggers: content refetches when the app is foregrounded or the network reconnects, on top of the plain 5-minute staleness re-fetch on mount. SyncProvider no longer runs its own AppState/NetInfo listeners for this (it used to); a comment there enumerates exactly which responsibilities moved to contentQuery's staleTime and the bindings file (mobile/src/providers/SyncProvider.tsx:29-41). The one thing SyncProvider still adds is syncNow(force), which bypasses staleTime via queryClient.refetchQueries (used by the post-camera-roll-import screen to force a real content pull) — wrapped in refetchIfOnline so an offline caller doesn't hang forever waiting on a paused, never-resolving refetch (SyncProvider.tsx:100-122).

Persistence of the query cache (catalog data) is entirely separate from the Zustand "AppStore" persistence (user/local data): PersistQueryClientProvider (mounted in mobile/app/_layout.tsx:543) uses queryPersistOptions (mobile/src/query/persistOptions.ts:13) with maxAge: Infinity — explicitly chosen because the library's own default (24h) throws away the entire persisted query client, cursor included, past that age, which would force a ~6.4 MB full re-download (the number is from a comment measuring the real catalog, content.ts:44) for anyone who skips opening the app for a day.

Content-sync visibility in the UI

The user essentially never sees an explicit "syncing catalog" indicator. Two screens mount useContentSync() and gate on it differently:

• FieldGuideHome (mobile/src/components/home-v3/FieldGuideHome.tsx:263,551 — the view model folds it into a single status discriminant, :114,631) and Near Me (mobile/app/(drawer)/near-me.tsx:157,261 — the view model folds it into a single loading field the layout reads) both fold contentLoading (TanStack's isPending — true only when there is no data yet, i.e. the very first fetch of a cold install) into their overall loading gate alongside font-loading and location resolution, and render the shared page-loader skeleton (per the house style in CLAUDE.md) while it's true.
• FieldGuideHome additionally surfaces contentError as a blocking full-screen error with a "Retry" button wired to useContentSync().retry (status === 'error', FieldGuideHome.tsx:757-781) — but only when there is no cached data to fall back on (useContentSync.ts:20: error: data ? null : error). Once any content has ever synced successfully, a later background refetch failure (stale wifi, 5xx, offline) is swallowed silently from the user's point of view — isFetching toggles SyncProvider's status to 'syncing' internally, but nothing in the two screens above renders that status as UI chrome.
• There is no "last synced" timestamp, progress bar, or toast shown anywhere for a background delta — the whole mechanism is invisible once the app has any content cached at all.

The other sync/ surfaces (context, not catalog delivery)

backend/src/sync/ also hosts the JWT-guarded user-data endpoints — getUserData, push, merge (guest→account migration), reconcileDevice, resetProgress (sync.service.ts:329-875). These move the opposite direction (device → server: collected patches, trips, purchases) and are a separate concern from catalog delivery, but they share the controller, share SyncPushDto's caps (MAX_TRIPS_PER_PUSH = 250, MAX_STOPS_PER_TRIP = 500, MAX_TOTAL_STOPS_PER_PUSH = 5000; backend/src/sync/dto/sync.dto.ts:77-160), and share the flushPendingPatches client-side flush that SyncProvider also owns (SyncProvider.tsx:70-83). push/migrateGuest are deliberately tolerant of stale/unknown patch ids on the device (apply the valid subset, name the rest as rejected, never fail the whole request) — see backend/src/sync/__tests__/sync.service.unknown-patches.spec.ts's header comment for the incident this fixed.

How long until a freshly published patch reaches an installed app

There is no push path, so the answer is entirely a function of when the client's next contentQuery fetch fires and whether it lands within the 5-minute staleTime window:

• App already open and foregrounded, past the 5-minute mark since its last fetch: the next AppState→active focus event or NetInfo reconnect event triggers a refetch (bindings.ts); if the app was never backgrounded, nothing re-triggers it until something remounts the query or staleTime is checked again on a re-render that re-subscribes — in practice, a session left open and idle on one screen for a long time will not automatically re-poll on a timer, only on focus/reconnect/remount.
• App backgrounded and resumed: the resume is itself an AppState active event, so refetchOnWindowFocus's RN binding fires — if staleTime (5 min) has elapsed since the last successful fetch, this issues a real request.
• Cold start (app was fully closed): the persisted query cache hydrates instantly from MMKV (persister.ts) so the UI paints immediately with whatever was last cached, then useQuery evaluates staleness against the hydrated dataUpdatedAt — almost any cold start after more than 5 minutes away triggers an immediate background refetch on mount.
• Forced/guaranteed pull: only syncNow(true), called today after a camera-roll import completes (per SyncProvider.tsx:100-117's comment), bypasses staleTime unconditionally via refetchQueries.

So the practical answer is "within 5 minutes for an app in active use, and on the next foreground/cold-start otherwise" — never instantaneous, and never guaranteed at all without connectivity at one of those trigger points.

Data model

Relevant backend/prisma/schema.prisma models (all @@schema("public")):

• Category (schema.prisma:18) — id, name, displayName, displayOrder, iconName, imageUrl/imageBlurhash, color.
• Collection (schema.prisma:36) — id, name, description, categoryId, campaignId, iconUrl/iconBlurhash, patchCount, productId/productHandle/priceUsd, storeVisible, adminOnly (the visibility switch this whole feature filters on), publicPurchase, sortOrder.
• Campaign (schema.prisma:70) — id, name, tagline, description, heroImageUrl/heroBlurhash, color, sortOrder. Carries no adminOnly of its own — visibility is derived from its member collections (see above).
• Patch (schema.prisma:88) — the largest model in the schema: identity/media fields, latitude/longitude (Decimal(10,7)), collectionType, protectedAreaId/geofenceId (mutually-exclusive links to the two polygon tables, neither of which is modeled in Prisma — they're PostGIS-only, read via raw SQL), Wikipedia/Wikidata fields, historical-context fields, and ~38 *Data JSON columns (battlefieldData, parkData, zooData, …), one per patch type, listed at schema.prisma:169-207. mapSyncPatch (sync.service.ts:84-161) serializes essentially the entire row to the wire.
• PatchCollection (schema.prisma:247) — the join table: patchId, collectionId, unique on the pair, cascade-deletes with either parent.
• PatchType (schema.prisma:260) — id, slug, label, icon, bgColor/iconColor, sortOrder. The catalog of patch "kinds" the *Data columns key off of.
• UserPatch (schema.prisma:785) — per-user collected-patch records; not part of the catalog, served by the separate, JWT-guarded GET /api/sync/user-data.

Visibility predicate: VISIBLE_PATCH_WHERE/ORPHANED_PATCH_WHERE/visiblePatchSql in backend/src/common/patch-visibility.ts:32-63 — "visible" means "belongs to at least one collection with admin_only = false". This is the app's only definition of a shippable patch; the doc comment there notes it is also enforced outside sync (camera-roll import, unlock) so a hidden patch can't be earned from a photo either.

API surface

No application-level rate limiting or throttling was found anywhere in backend/src (no @nestjs/throttler, no express-rate-limit) — /api/sync/content and /api/location/check-polygons (the unlock endpoint, also unauthenticated: backend/src/location/location.controller.ts:28-33) are both open to unlimited unauthenticated calls at the application layer. Whatever protection exists (if any) would have to live at the proxy/nginx layer, which is outside this repo.

Request body/response sizes are uncapped by any compression: backend/src/configure-app.ts sets up JSON body parsing (10 MB default limit, configure-app.ts:76-78) and CORS (configure-app.ts:80-83) but no compression() middleware anywhere in the backend — responses (including this one) are served uncompressed JSON.

Payload shape. Every patch alone serializes ~60 fields (mapSyncPatch, sync.service.ts:84-161) — identity/media, coordinates, two derived polygon metrics, historical-context fields, and ~38 nullable *Data JSON blobs (only one of which is ever non-null per patch, since it's keyed by patch type, but all ~38 keys are present on every object). Multiplied across a catalog on the order of ~2,000 patches (2,171 rows in the repo's own PATCH_COLLECTION_INVENTORY.md, flagged elsewhere in this repo as a source that can go stale) plus every collection/campaign/category/patchType/patchCollection row, this is what the content.ts:44 comment's ~6.37 MB figure describes for a full sync. A delta sync is only smaller on the categories/collections/patches/patchTypes/campaigns tables (each scoped by updated_at >= since); the patchCollections table, as noted above, is sent in full on every call regardless of since, so it puts a floor under how small any delta response can be.

Key files (annotated path:line list)

• backend/src/sync/sync.controller.ts:17-31 — the two unauthenticated GETs.
• backend/src/sync/sync.service.ts:194-327 — getContent, the whole catalog query + visibility filtering + serialization.
• backend/src/sync/sync.service.ts:84-161 — mapSyncPatch, the wire shape of one patch.
• backend/src/common/patch-visibility.ts — the one shared "is this patch reachable" predicate (Prisma + raw-SQL spellings).
• backend/src/sync/dto/sync.dto.ts:19-26 — SyncContentResponseDto (loosely typed — every field is any[], so the wire contract is enforced by hand-written tests, not by the DTO).
• mobile/src/query/queries/content.ts — contentQuery: the delta-merge queryFn, ContentCache shape, and setPatchLocationImage (a targeted cache patch used by admin-in-app QA).
• mobile/src/query/persister.ts — dedicated scout-query-cache MMKV instance backing the query client's persistence, deliberately separate from the Zustand store's MMKV id.
• mobile/src/query/persistOptions.ts — maxAge: Infinity and why.
• mobile/src/query/client.ts — staleTime, gcTime, retry policy (shouldRetry, terminal on any 4xx, two retries on 5xx/network).
• mobile/src/query/bindings.ts — RN AppState/NetInfo wired into TanStack's focus/online managers.
• mobile/src/hooks/useContentSync.ts — the { isLoading, hasCache, error, retry } surface screens consume.
• mobile/src/providers/SyncProvider.tsx — syncNow, useSync(), and the (unrelated) pending-patch-queue flush living in the same provider.
• mobile/src/domain/store.ts:150-464 — transformX functions, mergeById, mergePatchCollections (shared by the store file and content.ts).
• mobile/src/domain/store.ts:497-510 — the AppState interface's own comment documenting that content moved off Zustand entirely in "Task 14".
• mobile/src/domain/store.ts:2791-2876 — partialize, the literal list of what persists to the scout-app-store MMKV key (catalog is conspicuously absent, called out by comment).
• mobile/src/domain/persistStorage.ts — native MMKV adapter for the Zustand store (scout-app-store MMKV id).
• mobile/src/domain/persistStorage.web.ts — web fallback via IndexedDB, because MMKV-web/localStorage's ~5 MB quota overflows once the (formerly-Zustand-resident) catalog is added — kept even though the catalog itself has since moved to the query persister, per its own comment.
• mobile/src/domain/api.ts:125-129 — the one HTTP call, api.content.sync(since), unauthenticated by omission of requireAuth.
• mobile/src/services/location/tracker.ts:380-527 — where unlock actually happens; see "How it works" for the network dependency this creates.
• backend/src/location/location.controller.ts:28-33 and location.service.ts — the polygon-containment endpoint the unlock path calls.

Configuration and flags

• No feature flag gates content sync itself — it is unconditional core plumbing.
• FIVE_MIN staleness constant is duplicated in two places with the same value: mobile/src/query/queries/content.ts:19 and mobile/src/query/client.ts:4 (the latter is the global default every query, including this one, inherits).
• maxAge: Infinity in mobile/src/query/persistOptions.ts:35 is a deliberate override of the TanStack persist-client library default (24h) — changing it back would silently reintroduce a full-catalog re-download after any day-long gap in app opens.
• No environment variable controls the admin-only filter; the isAdmin bypass is entirely token-derived (isAdminEmailFromToken), and the CLAUDE.md-documented history here is that this filter was previously and incorrectly gated on NODE_ENV, which leaked store-only collections into production because the droplet leaves NODE_ENV unset.

Edge cases and known limits

• Delta merges never delete. Both mergeById (store.ts:445-452) and mergePatchCollections (store.ts:453-464) only add/overwrite entries present in the incoming payload; neither removes a client-cached entity whose id is simply absent from a later response. The backend has no explicit tombstone/delete signal in this DTO at all (SyncContentResponseDto has no deletedIds field of any kind). Net effect: if a patch, collection, campaign, or patch↔collection membership is removed from the visible catalog (unpublished, orphaned, or its parent collection flipped to admin_only) without the patch's own updated_at moving in a way the client's next delta window would re-fetch it as absent-and-thus-caught, the client can keep showing/allowing-navigation-to stale content indefinitely, until a full resync (cleared cache) happens. This is consistent with the deliberate design of getPatchById (sync.controller.ts:25-31) staying unfiltered specifically so an already-collected/cached patch can still render — but it means "hidden" and "actually gone from the client" are not the same event.
• patchCollections is never delta-scoped. Every getContent call — full or incremental — returns the entire current visible membership table (sync.service.ts:211-213 has no sinceFilter spread in). Combined with the merge-never-deletes behavior above, a patch detached from a collection stops appearing in new patchCollections responses but the old membership row is never explicitly removed client-side by this mechanism.
• Unlock requires a live round trip for essentially every patch. Per mobile/src/services/location/tracker.ts:499-514, any patch linked to protectedAreaId or geofenceId (which per CLAUDE.md now includes city and state patches, and buffered-point fallbacks — i.e., nearly the whole catalog) is checked via POST /api/location/check-polygons, a server-side PostGIS containment query. If that call fails (including "offline"), the code logs a warning and skips those patches for that pass (tracker.ts:504-505) — there is no local queue, no cached polygon geometry on device, and no retry beyond "try again on the next location update." Only the small remainder of non-polygon patches (shouldUnlockPatch, tracker.ts:434-457) unlock purely on-device via a radius check against patch.latitude/longitude.
• Collecting a patch, once unlock is decided, is fully local and offline-tolerant. collectPatch (store.ts:1543) is a synchronous, local Zustand action that appends to pendingSync (store.ts:1571) — no network call. The actual server write happens later via flushPendingPatches, invoked by syncNow()/on sign-in/at the next successful location-check cycle. So the write path degrades gracefully; the decision path (for polygon patches) does not.
• No schema-version/cache-buster on the persisted query cache. Nothing forces a hard invalidation of the persisted ContentCache if the shape the server sends changes; new fields simply arrive as part of the next merged fetch, and any pre-existing cached entity not touched by a later delta keeps whatever shape it had when last written.
• First-fetch failure blocks the whole app content-wise (see "Content-sync visibility" below): useContentSync().error is only non-null when there is no cached data at all (error: data ? null : error, useContentSync.ts:20), so a background refetch failure with an existing cache is silent by design, but a first-ever fetch failure with no cache shows a blocking error screen with Retry.

What this does NOT do

• Does not push content to devices. There is no websocket/SSE/push-notification mechanism tied to publishing; discovery is exclusively client-initiated polling (searched backend/src/content-publish and backend/src/sync for any such wiring — none found).
• Does not compress responses (no compression() middleware anywhere in backend/src).
• Does not rate-limit or authenticate the catalog-read endpoints at the application layer.
• Does not delete client-cached entities on delta sync (see Edge cases).
• Is not offline-first: it caches what has already synced for offline reading, but does not queue or retry the unlock-decision network call, and cannot discover new/changed content without connectivity.
• Content is not part of the Zustand useAppStore/scout-app-store MMKV persistence — that store is explicitly documented (store.ts:497-510) as no longer holding categories/collections/campaigns/patches/patchTypes/patchCollections.

Tests that cover it

• backend/src/sync/__tests__/sync.service.admin-only.spec.ts — asserts the adminOnly/visibility filter is applied to collection, patch, and patchCollection where-clauses for non-admins, is bypassed for admins, and — critically — is applied identically whether NODE_ENV is production, development, or unset (the exact production regression this guards).
• backend/src/sync/__tests__/sync.service.campaigns.spec.ts / sync.service.empty-campaigns.spec.ts — campaign visibility derived from member-collection visibility.
• backend/src/sync/__tests__/sync.service.unknown-patches.spec.ts — push-side resilience to catalog ids the client holds that the server no longer knows about (not catalog delivery itself, but the mirror-image failure mode).
• backend/src/sync/__tests__/sync.service.get-patch-by-id.spec.ts — the deep-link single-patch fetch, unfiltered.
• mobile/src/query/__tests__/content.test.ts — full-vs-delta call shape, delta-merge-not-replace behavior, and cursor advancement, against the real contentQuery.
• mobile/src/hooks/__tests__/campaignsPersisted.test.ts (referenced from persistOptions.ts:11) — round-trips the real queryPersistOptions (dehydrate → persist → restore → hydrate) rather than a hand-copied approximation.
• mobile/src/providers/__tests__/SyncProvider.test.tsx — syncNow, force-refetch-bypasses-staleTime, offline-safe mutation behavior.
• No test found that specifically exercises "a patch/collection is removed from the visible set after a client already has it cached" through the delta path end-to-end — consistent with the "delta merges never delete" limitation noted above being unverified-by-test, not just unimplemented.

Open questions

• Whether nginx (or another reverse proxy) applies rate limiting or response compression in front of this endpoint in production — that layer is not in this repository, so it could not be verified here.
• The true current catalog size/payload weight was not independently measured in this pass; the ~6.37 MB figure cited in "How it works" comes from a code comment (mobile/src/query/queries/content.ts:44) describing the full catalog at the time it was written, not a fresh measurement, and PATCH_COLLECTION_INVENTORY.md (2,171 patch rows as of this pass) is flagged elsewhere in this repo's own conventions as potentially stale.
• Whether a patch/collection/campaign's updated_at is reliably bumped by every operation that changes its visibility (e.g., detaching a collection's last visible link, or flipping admin_only) was not traced through the admin/content-publish write paths (out of scope for this document) — that trace would determine how often the "delta merges never delete" limitation is actually hit in practice versus purely theoretical.
• Whether there is any server-side or client-side telemetry that would surface a stuck/stale device (one that has silently stopped getting deltas, e.g. due to a persistently failing background refetch) was not found and was not exhaustively searched for.