# Deployment

> Release commands in order: [RELEASE.md](RELEASE.md). This file explains how
> the website deploy and its update system work.
>
> Tools this file assumes (bash, `curl`, an upload method): see
> [RELEASE.md](RELEASE.md#tools-this-file-assumes) or the machine-wide list in
> [README](README.md#requirements-on-a-new-machine).

> Steps 1-4 below (build, backup, merge) are automated by
> `npm run deploy:prepare -- --version X.Y.Z [--grace 3]`
> (see `scripts/deploy-prepare.js` -- plain Node, works the same on
> Windows, Linux, and CI runners). It also bumps `NEXT_PUBLIC_APP_VERSION`
> in `.env.local` for you (pass `--skip-env` in CI if the version is
> already set another way) and refuses to run if the new version matches
> what's already in `./out`, since the in-app update check
> (`src/lib/update/`) only fires on a string difference. The
> manual steps below remain as reference / fallback.

This project uses Next.js with `output: 'export'`.

The production build is generated in:

    out/

## Why preserve old assets?

Next.js uses hashed filenames:

    layout-OLD_HASH.js
    layout-NEW_HASH.js

When code/text changes, the hash can change.

Some users may still have old HTML cached, which references the old JS hash. Therefore, old `_next/static` files must remain available after deployment.

## Deployment Steps

### 1. Build

    npm run build

This creates:

    out/

### 2. Backup the new build

    Copy-Item ".\out" ".\out-new-backup" -Recurse

### 3. Merge old static assets

Set the path to the previous release:

    $old = "C:\Deployments\previous-release\out\_next\static"
    $new = ".\out\_next\static"

Run:

    Get-ChildItem -Path $old -Recurse -File | ForEach-Object {

        $relative = $_.FullName.Substring($old.Length).TrimStart('\')
        $destination = Join-Path $new $relative

        if (-not (Test-Path $destination)) {

            $dir = Split-Path $destination -Parent

            if (-not (Test-Path $dir)) {
                New-Item -ItemType Directory -Path $dir -Force | Out-Null
            }

            Copy-Item $_.FullName $destination
            Write-Host "COPIED: $relative"
        }
    }

This means:

    NEW files → keep
    OLD files that don't exist → copy
    Same filename → NEW wins
    Different hashed filename → both exist

### 4. IMPORTANT: Do NOT merge the entire old `out`

Only merge:

    old/out/_next/static/

Do NOT replace:

    index.html
    pages
    other HTML files

The final output should be:

    out/
    ├── NEW HTML
    └── _next/
        └── static/
            ├── OLD assets
            └── NEW assets

### 5. Test locally

    npx serve ".\out"

Open the localhost URL and check DevTools → Network.

`_next/static/*` requests should return `200` and JavaScript/CSS, not HTML.

### 6. Deploy

Deploy the final `out/` directory.

**Upload `.htaccess` explicitly, then reload OpenLiteSpeed. Both halves of
this step fail silently.**

OLS parses `.htaccess` into memory and serves those rules until it is
reloaded -- it does **not** re-read the file per request the way Apache does.
So a correctly uploaded `.htaccess` can sit on disk having no effect at all,
indefinitely, with the server happily serving the previous version's rules.
After uploading, do a **Graceful Restart** (aaPanel), or:

    /usr/local/lsws/bin/lswsctrl restart

`scripts/verify-deploy.sh` asserts this via `/__htaccess-probe`, which is the
only check that can tell the current file from an older one -- every other
rule in `.htaccess` shipped in some earlier release and passes either way.

`.htaccess` is a dotfile. FTP/SFTP clients hide dotfiles by default, and
`scp out/* user@host:/path` drops them because the shell glob never matches
them. Production ran the 2.2.3 `.htaccess` through two later releases
because of this -- which meant no `Cache-Control` header on anything, so
browsers fell back to heuristic freshness and held individual routes'
HTML for days. That is what stranded users on old code showing a Buy
button for courses they already owned.

If you use rsync, this uploads dotfiles correctly:

    rsync -av --delete-after ./out/ user@host:/path/to/webroot/

(the trailing slash on `./out/` matters). With an FTP client, enable
"show hidden files" and confirm `.htaccess` transferred.

`out/download/` holds the Android APK ([RELEASE.md](RELEASE.md)).
`deploy:prepare` carries it forward like the old assets, so build the website
with `deploy:prepare`, not a bare `npm run build`, or the upload deletes the APK.

### 7. Verify the deploy actually landed

    bash scripts/verify-deploy.sh

Exits non-zero if anything is wrong. It checks that the served
`version.json` buildId matches the one you just built, and that the
current `.htaccess` is live (HTML `no-cache`, missing `_next` assets
returning a real 404, `version.json` `no-store`, `.txt` RSC payloads
`no-cache`, hashed assets `immutable`).

A release is not deployed until this passes. `scripts/cache-probe.sh`
prints the same headers unasserted, for before/after comparison when
changing server or CDN settings.

Both scripts take the route they probe from `PROBE_ROUTE`. **Update it when
you rename a route.** It was hardcoded to `/course/view/` when 2.3.2 renamed
that route away, and the probes kept "passing" for a whole release: the
catch-all serves the 404 page at status 200 with `text/html`, so the
Cache-Control assertions were checking headers on an error page. Check `[0]`
now proves the probed route is real (the 404 page carries
`data-page="not-found"`) before anything downstream trusts it.

## How a user ends up on the latest version, and stays there

All of it lives in `src/lib/update/`, wired by `initUpdateSystem()` from
`src/lib/provider/client-providers.tsx`. Seven layers, each covering a case
the one before it cannot.

| # | layer | file | covers |
|---|---|---|---|
| 1 | service worker | `service-worker.ts`, `public/sw.js` | decides document freshness in code we ship; the only layer that fixes a PWA cold start properly |
| 2 | `_r` stamping | `identity.ts` | gives every build its own cache key for every RSC payload |
| 3 | version check | `version-check.ts` | `BUILD_ID` vs `version.json`, on mount / route change / focus / visibility / 5-min timer |
| 4 | build integrity | `version-check.ts` | a chunk from another deployment inside a current shell, which (3) cannot see |
| 5 | chunk recovery | `chunk-errors.ts` | assets that fail outright |
| 6 | entry-point heal | `entry-points.ts` | repairs the cached copy of `start_url` so a fix survives closing the browser |
| 7 | legacy re-arm | `legacy.ts` | un-disarms the pre-2.2.8 check for clients still running it |

Supporting: `budget.ts` (attempt limits) and `reload.ts` (cache drop +
URL-busting reload). Layers 6 and 7 run from inside layer 3, at the one moment
we can prove we are on the current build.

Every layer degrades independently. If the service worker cannot register,
2-7 still work exactly as before; the worker makes them cheaper and more
reliable, it is never their precondition.

### The service worker

`public/sw.js`, registered as `/sw.js?v=<BUILD_ID>`.

    navigations + .txt payloads  ->  network first, cache as fallback
    /_next/static/*              ->  cache first (content-hashed, immutable)
    version.json                 ->  never cached

Network-first on documents is the point: it takes the freshness decision away
from the browser's HTTP cache, which is where every version of this bug has
lived. A PWA cold start at `start_url` now asks the network instead of
replaying something stored months ago under heuristic freshness.

Three things about it are load-bearing, and all three have burned other teams:

- **`?v=<BUILD_ID>` in the script URL.** The vhost caches `text/javascript`
  for a year, and `sw.js` is `text/javascript`. A worker frozen in the HTTP
  cache is the classic PWA disaster -- the browser never fetches a new one and
  every user is pinned to the build that installed it, a failure far stickier
  than a stale document because a bad worker also controls its own
  replacement. `?v=` makes each build's worker a URL the cache has never seen.
  `updateViaCache: "none"` is the backup, and the spec's 24h cap the backstop.
- **`skipWaiting()` + `clients.claim()`.** By default a new worker waits until
  every tab of the old one closes. An installed PWA is often never closed, so
  the default would leave a new worker waiting indefinitely -- reintroducing
  the exact "user never receives the update" failure.
- **Missing chunks must still 404.** `cacheFirst` only stores `ok` responses
  and returns the network result as-is, so a missing asset still reaches
  layer 5 as a real failure instead of being masked.

**Kill switch:** if the worker ever misbehaves, replace `public/sw.js` with
the self-unregistering version documented at the top of that file and deploy.
`unregisterServiceWorker()` in `service-worker.ts` is the client-side
equivalent, deliberately not called anywhere.

### Why a fix used to come undone overnight

The HTTP cache is keyed by **full URL**. Sending a stuck user a link with a
junk query (`/?v=2.2.3-1`) works because that URL has never been requested:
cache miss, fresh fetch, correct app, user reports it fixed.

The entry for `/` is never touched. It still holds whatever was stored when
the server sent no `Cache-Control`, and the browser fixed its freshness at
store time -- so it will not revalidate for days, and correcting the headers
server-side cannot reach it. **Nothing server-side can fix a client that never
asks.** Then the browser closes, and the PWA relaunches at `start_url` -- the
one entry nobody repaired.

Layer 1 removes this for anyone with the worker installed. Layer 6 is the
bridge for everyone else, because installing the worker requires loading
current code once, which is the same bootstrapping problem.

### The pre-2.2.8 clients

The original check (commit `e0dc7f3`) writes `localStorage["app-version"]`
*before* verifying the reload landed. localStorage survives closing the
browser, so a client whose reload came back stale ends up permanently
convinced it is current: one rescue attempt per release, then silence.

Layer 7 writes a sentinel that can never equal a real version, which re-arms
that check. An old bundle then bounces itself to `?version=X` -- a fresh URL,
so current code -- which re-arms it again. A stuck client rescues itself on
every cold start instead of once per release. It terminates in every
direction; see the reasoning in `legacy.ts`.

## Renaming a route

In a static export a rename is a delete plus a create, and clients hold
cached copies of older pages that still link to the old path. Both halves
need handling:

1. **Server side** -- add a `RewriteRule` to `public/.htaccess` under
   RETIRED ROUTES. Covers hard navigation: refresh, typed URL, PWA cold
   start. Use `[R=301,L,QSA]`; without `QSA` the `?id=` is dropped and the
   redirect lands on a different error.
2. **Client side** -- add the pair to `RETIRED_ROUTES` in
   `src/app/not-found.tsx`. A client-side navigation from a cached older
   page never reaches the server, so no rewrite can fire; Next resolves it
   in the browser and renders the 404 component, which redirects from there.
3. **Verifier** -- add the pair to `RETIRED_ROUTES` in
   `scripts/verify-deploy.sh`, and move `PROBE_ROUTE` to the new path.

Skipping any of these strands a cohort of real users on a dead end they
reached by tapping an ordinary link.

## Old `_next/static` assets: a bounded window

`deploy:prepare` carries old chunks forward so a client holding a cached
copy of an older page can still load the chunks that page names -- otherwise
the chunk 404s and the screen breaks.

This used to merge from the previous `./out`, which was itself a merged
build, so every release re-inherited everything and assets lived **forever**.
That is what let a months-old chunk comparing `purchase_status` against the
retired `"complete"` value keep rendering inside a current shell: the profile
screen showed the new version while the course screen showed last season's
behaviour.

Silent wrong data is worse than a loud broken screen. A missing chunk raises
`ChunkLoadError`, and `src/lib/chunk-error-recovery.ts` turns that into a
reload onto the current build. So the window is now finite:

    npm run deploy:prepare -- --version X.Y.Z --grace 3

`--grace 3` = this build plus the previous 2 releases. Each run saves a
**pristine** snapshot of `_next/static` to `deploy-backups/static-<ts>-<ver>/`
*before* merging (a post-merge snapshot has no age -- it contains
everything), and merges the window forward from those.

Nothing is ever deleted by the script. Releases that fall outside the window
are listed and left on disk; they simply stop being uploaded. Do not delete
`deploy-backups/static-*` by hand while it is still inside the window --
those exact files are what old clients are loading.

## Server: OpenLiteSpeed (aaPanel)

Production runs **OpenLiteSpeed 1.8.5** under aaPanel. Two consequences
that are easy to get wrong:

1. OLS reads **rewrite rules** from `.htaccess`, but has **no mod_headers**.
   `Header set` directives there are silently discarded. All Cache-Control
   configuration lives in `deploy/openlitespeed-vhost.conf`, which goes in
   `/usr/local/lsws/conf/vhosts/<domain>/vhost.conf` (aaPanel: Website ->
   Config), followed by a Graceful Restart.

2. aaPanel can rewrite `vhost.conf` when site settings change in the panel.
   Treat `deploy/openlitespeed-vhost.conf` as the source of truth and
   re-run `scripts/verify-deploy.sh` after any panel-side change.

Verify the server edition with:

    /usr/local/lsws/bin/lshttpd -v

`... Open` means OpenLiteSpeed (headers via vhost.conf). `... Enterprise`
would mean `.htaccess` headers work and the vhost contexts are unnecessary.

## Important Server Rule

If a `_next/static/*.js` file does not exist, the server/CDN should return:

    404

NOT:

    <!DOCTYPE html>

Returning `index.html` for a missing JS file causes:

    SyntaxError: Unexpected token '<'