Manifest Pipeline & Deployment

A module that wants to appear in whisker-host has to do two things: get its manifest into DynamoDB, and (for federation modules) get its bundle onto a CDN-fronted S3 bucket the host can fetch from. Both are fully automated from a merged PR. Neither requires a host PR.

End-to-end picture

flowchart TB
  subgraph repo["module repo"]
    src["src/moduleManifest.ts"]
    ci["Jenkinsfile"]
    gha[".github/workflows/notify-host.yml"]
  end

  subgraph build["CI on merge"]
    gen["whisker generate-manifest"]
    val["whisker validate-manifest"]
    bld["vite build (federation only)"]
  end

  subgraph s3["AWS S3"]
    bucketTest["whisker-manifest-test/\n  <moduleName>/module-manifest.json"]
    bucketProd["whisker-manifest-prod/\n  <moduleName>/module-manifest.json"]
    bucketMod["whisker-modules-{env}/\n  <moduleName>/\n    remoteEntry.js\n    assets/*"]
  end

  subgraph publisher["whisker-manifest-publisher Lambda"]
    parse["parse + ModuleManifestSchema.parse()"]
    merge["mergeManifestForUpsert(existing, incoming)"]
    write["DynamoDB PutItem with optimistic concurrency"]
    dlq["SQS DLQ on failure"]
  end

  subgraph runtime["At runtime in the user's browser"]
    bff["sim-bff /manifests/v1\n(whisker-bff planned)"]
    ddb["DynamoDB: per-env module registry"]
    host["whisker-host"]
    remote["Vite loadRemoteModule(name) → remoteEntry.js"]
  end

  subgraph hostsync["build-time host handshake"]
    dispatch["repository_dispatch: module-updated"]
    hostRepo["whisker-host repo"]
    syncscript["sync-deployed-registry.js\nfetches bff/manifests/v1"]
    testpr["auto-PR → develop (auto-merged)"]
    prodpr["promotion PR develop → main\n(manual approval)"]
  end

  src --> gen
  gen --> val
  val --> ci
  ci --> bucketTest
  ci --> bucketProd
  bld --> bucketMod

  bucketTest -->|S3 event| parse
  bucketProd -->|S3 event| parse
  parse --> merge
  merge --> write
  write --> ddb
  parse -.fails.-> dlq
  merge -.exhausted retries.-> dlq

  gha --> dispatch
  dispatch --> hostRepo
  hostRepo --> syncscript
  syncscript -->|target_env=test| testpr
  syncscript -->|target_env=prod| prodpr

  ddb --> bff
  bff --> host
  bff --> syncscript
  bucketMod --> remote
  remote --> host

What’s built today

Track 1 — runtime manifest (S3 → Lambda → DynamoDB)

CI writes to a different bucket depending on the branch:

Branch	Bucket	First-publish enabled
Feature / test	whisker-manifest-test	Written as-is — visible the moment CI completes
main	whisker-manifest-prod	Written as-is on first publish; subsequent redeploys preserve the admin-controlled enabled state

The Lambda code path is identical in both environments; only the DynamoDB table and merge anchor differ. S3 key shape is <bucket>/<moduleName>/module-manifest.json; the Lambda cross-checks manifest.identity.name against that prefix and sends a mismatch straight to the DLQ.

What the Lambda does:

1. Skip if key doesn't end in /module-manifest.json
2. Reject non-flat <moduleName>/module-manifest.json layouts
3. GetObject from S3 → JSON.parse → ModuleManifestSchema.parse()
4. Cross-check manifest.identity.name === expectedModuleName
5. GetItem from DynamoDB (current row)
6. merged = mergeManifestForUpsert(existing, incoming)
7. PutItem with ConditionExpression: revision = <previous>
8. On conflict: re-read + re-merge + retry up to 3×
9. On exhaustion or validation failure: send to SQS DLQ

Promise.allSettled over the S3 event’s records means one bad manifest in a batch never blocks the others. See Host ↔ module contract for the full merge rules.

CloudFormation (whisker-manifest-publisher/cloudformation/manifest-publisher.yaml) provisions, per env: a Node 24 arm64 Lambda, an IAM role scoped to the manifest bucket + table + DLQ, a KMS-encrypted SQS DLQ, and optional CloudWatch alarms. The S3 → Lambda notification itself lives in the platform stack (S3 only allows one notification config per bucket).

Track 2 — build-time host handshake (GitHub Actions → host)

Track 1 tells the host what to render; track 2 tells it where the bundle lives so Vite Module Federation can resolve remoteEntry URLs at host build time. create-whisker scaffolds .github/workflows/notify-host.yml into every new module, which fires a repository_dispatch: module-updated at the host repo on push/release.

The host’s .github/workflows/sync-deployed-registry.yml listens for that dispatch (or a manual workflow_dispatch):

Target env	What runs	Outcome
test	Regenerates module-registry.json, remote-loader.ts, federated-modules.d.ts from the live registry. Auto-PR opened and auto-merged to develop.	New module visible in the next develop build — zero module-team PRs against the host.
prod	Refuses to proceed if a test-sync PR is still pending. Opens a manual develop → main promotion PR.	Human approval gate before prod ships.

The host’s sync script hits the public manifest endpoint directly — no Nike credentials needed for the GitHub Actions runner, since the endpoint is already designed to be readable pre-login.

Iframe consumers skip track 2 entirely — there’s no bundle to resolve.

Co-location of module bundles in S3

Federation modules ship to a per-module prefix:

s3://whisker-modules-test/<moduleName>/{index.html, remoteEntry.js, assets/}
s3://whisker-modules-prod/<moduleName>/{...same shape...}

These URLs are baked into the manifest’s integration.remoteAssetUrls (federation only). There’s no s3DeployUris field in config/whisker.config.json — it would just be this same bucket convention persisted a second time with nothing to keep it in sync, so whisker validate-manifest derives it on the fly from the manifest’s own identity.name instead. One bucket per env means one CloudFront distribution and one IAM policy template cover the whole platform. Iframe consumers deploy wherever they already deployed — the protocol is URL-driven, not bundle-driven.

DynamoDB schema (effective)

TABLE: whisker-module-manifests-{env}
PK:    moduleName (string)
ATTRS: manifest, revision (optimistic concurrency),
       createdAt, updatedAt, lastModifiedBy

The Lambda is the only writer — there is no admin UI yet (see Risks & improvements).

The runtime fetch path

1. whisker-host calls sim-bff /manifests/v1 — a single GET.
2. The BFF reads DynamoDB and returns the full array.
3. The host caches the array (sessionStorage + in-memory) and renders nav/widgets/pages from it. A background revalidate keeps it fresh.
4. If the call fails entirely, the host renders from sessionStorage and logs to NR instead of showing a blank page.
5. For federation modules the host then calls loadRemoteModule(moduleName, componentPath); for iframe consumers it mounts IframeRenderer with the URL from integration.urls.

Local development

whisker dev regenerates config/generated/local/module-registry.json so the host sees the locally-running module alongside whatever DynamoDB returns, without touching the deployed registry.

What’s intentionally split for safety

Mechanism	Why it’s split
Test vs prod buckets/tables	A test misconfiguration can never bleed into production — same Lambda code, different env binding
Manifest vs bundle	A manifest update doesn’t redeploy code; a code redeploy doesn’t change the manifest
Runtime ingest (track 1) vs build-time handshake (track 2)	Runtime is dynamic and self-service; build-time updates trigger host rebuilds — slower but safer for federation entry-point changes
Source-controlled fields vs admin-editable fields in the merge	Source can iterate freely; admin rollout decisions are sacred

Planned / proposed

whisker-bff extraction (planned)

Today track 1 reads from sim-bff (/sim-bff/manifests/v1). No whisker-bff code exists yet anywhere in the platform. The plan is to extract a platform-owned whisker-bff that owns:

• GET /manifests/v1 (the runtime registry)
• POST /legacy-berm/v1 + /refresh (legacy login + refresh)
• Future component-provider endpoints WCL needs at runtime

sim-bff keeps serving inventory routes unchanged until the migration is intentional — only TODO comments mark the platform routes it serves on borrowed time. GET /manifests/v1 will stay unauthenticated on the new service too: neither local devs nor CI/GitHub Actions need credentials to call it, and the host’s existing BERM flow already covers everything after login. Module teams won’t see any change at the call site — the host swaps its base URL, CI repoints at the new endpoint.

See Risks & improvements for priority and the whisker-config-panel admin UI this pipeline is still missing.