1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220

# Deploying the gitweb browser

How to build and deploy the gitweb service to the upspin VM (`instance-2`). It is
a single Go binary (page templates + static assets embedded via `go:embed`) that
runs as a docker-compose service alongside the existing upspin stack, exactly
like [vault](../vault/DEPLOY.md) and [markdown](../markdown/DEPLOY.md).

## Architecture at a glance

- **Image:** `gcr.io/cosmocr-at-upspin/gitweb:latest`
- **Host:** `instance-2` (GCP, `us-central1-a`, COS 121), static IP `34.28.234.94`
- **TLS host:** `git.cosmocr.at` — the container terminates its own TLS with a
  certbot cert; nginx SNI-routes `git.cosmocr.at → gitweb:8446`.
  (DNS is a `*.cosmocr.at` wildcard → `34.28.234.94`, so no per-host record is
  needed.)
- **Runs as upspin user** `git@cosmocr.at`. It reads any bare repo whose
  `Access` file grants it `read,list`; push write-back additionally needs
  `create,write,delete` for `git@cosmocr.at` on the repo. The demo repo is
  `git@cosmocr.at/demo.git`; third parties expose a repo by granting
  `read,list: git@cosmocr.at`.
- **Config & keys** live on the persistent data disk (not in the image):
  - `/mnt/disks/data/gitweb/config` — upspin config (`secrets: /gitweb/secrets`)
  - `/mnt/disks/data/gitweb/secrets/{public,secret}.upspinkey`
  - `/mnt/disks/data/letsencrypt/live/git.cosmocr.at/` — TLS cert
- **Authoritative infra sources** (in the `control` repo):
  - `upspin/vm/cloud-init-cos121.yaml` — the VM's cloud-init (embeds the
    docker-compose + nginx.conf used on boot). **Edit here first, and it is only
    live once pushed to the instance metadata `user-data` key** (see "If you
    changed config" — cloud-init re-applies the metadata on every boot and
    **overwrites** the `/var/upspin` config files, so anything not in the
    metadata is reverted on the next reboot).
  - `upspin/vm/docker-compose.vault.yml` — rendered from the cloud-init for local
    reference; the live `/var/upspin/docker-compose.yml` is written from the
    metadata on boot (all of vault, markdown, and gitweb live in it).
  - `upspin/nginx/nginx.conf` — canonical nginx config (the SNI map).
  - `upspin/vm/deploy-git.sh` — first-time end-to-end deploy script.
  - `upspin/secrets/git/` — the browser's upspin config + keys (git-ignored).
    Note the workstation-side secrets are under `secrets/git/`, but the on-VM
    service + data dir are named `gitweb` (matching the compose service).

## Prerequisites

- `docker` locally (for `--platform linux/amd64` builds) and `gcloud` configured
  for project `cosmocr-at-upspin`. (The Dockerfile is COPY-only over a
  cross-compiled `CGO_ENABLED=0 GOOS=linux GOARCH=amd64` binary, so the amd64
  image builds on an arm workstation with no qemu.)
- SSH to the VM. The deploy script defaults to `~/.ssh/google_compute_engine`;
  override with `DEPLOY_SSH_KEY` to use a passphrase-less key:
  ```
  DEPLOY_SSH_KEY=$HOME/.ssh/nopass_claude_code ./deploy-git.sh
  ```
- The `git@cosmocr.at` upspin user must exist on the keyserver and be in the
  keyserver's Writers. (One-time; already done — see
  `upspin/doc/provision_user.md`.)

## Routine redeploy (code change)

This is the common case. Because the templates + assets are embedded, **any**
change needs a rebuilt image.

1. **Build + push** the image from the gitweb repo:
   ```
   make push
   ```
   (= `CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build`, `docker build
   --platform linux/amd64`, tag, `docker push`.)

2. **Pull + recreate** the container on the VM:
   ```
   # authenticate the Docker daemon to gcr.io
   sudo env HOME=/var/upspin /bin/sh /var/upspin/auth.sh

   # pull the new image
   sudo docker run --rm \
     -v /var/run/docker.sock:/var/run/docker.sock \
     -v /var/upspin/.docker:/root/.docker \
     -v /var/upspin:/var/upspin -w /var/upspin \
     docker/compose:1.29.2 -p patrick pull gitweb

   # recreate just the gitweb container (rm + create; see "compose recreate" note)
   sudo docker rm -f patrick_gitweb_1
   sudo docker run --rm \
     -v /var/run/docker.sock:/var/run/docker.sock \
     -v /var/upspin/.docker:/root/.docker \
     -v /var/upspin:/var/upspin -w /var/upspin \
     docker/compose:1.29.2 -p patrick up -d --no-recreate gitweb
   ```
   (Re-running `deploy-git.sh` also works and is idempotent; the cert step is a
   no-op once the cert exists — but see the certbot gotcha below.)

3. **Verify:**
   ```
   sudo docker logs --tail 5 patrick_gitweb_1        # "gitweb: serving on :8446"
   curl -sI https://git.cosmocr.at/git@cosmocr.at/demo.git/   # 200
   ```

### If you changed config, the compose service, or nginx

This covers any change to the **compose services** (incl. `restart:` policies),
the **nginx routing/SNI map**, or the upspin config — i.e. anything that lives in
`cloud-init-cos121.yaml`.

> **⚠️ The instance metadata is the source of truth, not the files on the VM.**
> On every boot, cloud-init re-applies the VM's `user-data` metadata and
> **overwrites** `/var/upspin/docker-compose.yml` and
> `/var/upspin/nginx/nginx.conf`. A change you only `sudo cp` onto the VM (or
> `nginx -s reload`) works **until the next reboot**, then silently reverts. This
> is exactly how the vault service once vanished on its first reboot: it was added
> to the running VM and to the repo, but never pushed to the metadata. **Always
> push the metadata.**

**1. Edit + push the metadata (authoritative — survives reboots):**

```
# edit upspin/vm/cloud-init-cos121.yaml in the control repo, then from that repo:
gcloud compute instances add-metadata instance-2 --zone us-central1-a \
  --metadata-from-file user-data=upspin/vm/cloud-init-cos121.yaml
```

Mirror the same edit into `upspin/vm/docker-compose.vault.yml` and
`upspin/nginx/nginx.conf` so the local reference copies stay identical. The push
takes effect on the **next reboot**; verify it landed (run on the VM):

```
curl -s -H "Metadata-Flavor: Google" \
  http://metadata.google.internal/computeMetadata/v1/instance/attributes/user-data \
  | grep -c 'gitweb:'      # expect a non-zero count
```

**2. Apply now without waiting for a reboot (optional, for a live change):**
copy the rendered files up and reload/recreate only the affected pieces:

```
# control/upspin/vm/docker-compose.vault.yml -> /var/upspin/docker-compose.yml
# control/upspin/nginx/nginx.conf            -> /var/upspin/nginx/nginx.conf
sudo docker exec patrick_nginx_1 nginx -t && sudo docker exec patrick_nginx_1 nginx -s reload
# recreate any container whose service/config changed (rm + up; see note below)
```

These hot-apply the change immediately; step 1 is what makes it durable. **Do
both** — they are not alternatives.

## First-time setup (new VM / from scratch)

Use `control/upspin/vm/deploy-git.sh`, which does it all idempotently: stages the
config + keys onto the data disk, issues the TLS cert (certbot webroot via the
running nginx), updates nginx + compose, pulls the image, recreates nginx (for the
SNI route) and starts gitweb, then verifies. Manual outline:

1. **DNS:** covered by the `*.cosmocr.at` wildcard → `34.28.234.94` (no per-host
   record needed).
2. **Config + keys** onto the data disk: `/mnt/disks/data/gitweb/config` +
   `/mnt/disks/data/gitweb/secrets/*.upspinkey` (keys `0600`, root-owned).
3. **TLS cert** (nginx must be serving `:80` for the ACME challenge):
   ```
   sudo docker run --rm \
     -v /mnt/disks/data/certbot-www:/var/www/certbot:rw \
     -v /mnt/disks/data/letsencrypt:/etc/letsencrypt:rw \
     certbot/certbot certonly --webroot --webroot-path=/var/www/certbot \
       --register-unsafely-without-email --agree-tos --no-eff-email \
       -d git.cosmocr.at
   ```
4. **nginx route** and the **gitweb compose service** (already in
   `cloud-init-cos121.yaml`): SNI route `git.cosmocr.at gitweb:8446;`.
5. **Recreate nginx**, then **start gitweb** (commands as in "Routine redeploy").

Before the first run, make sure the infra edits are in `cloud-init-cos121.yaml`
**and pushed to the metadata** (see "If you changed config"): the `gitweb:`
compose service and the `git.cosmocr.at gitweb:8446;` line in the nginx `map`.

## Gotchas / troubleshooting

- **Config on the VM reverts on reboot if it isn't in the instance metadata.**
  cloud-init re-applies `user-data` on every boot and overwrites the
  `/var/upspin` config files. A `sudo cp` / `nginx -s reload` only holds until
  the next reboot. Push every infra change with `gcloud compute instances
  add-metadata … --metadata-from-file user-data=cloud-init-cos121.yaml` (see "If
  you changed config"). Symptom of a miss: after a reboot the service is gone
  from `docker-compose.yml` (compose logs `Found orphan containers …`) or nginx
  returns `no host in upstream ""`.

- **certbot step must stay non-interactive.** The `certonly` invocation runs with
  `--keep-until-expiring -n`, so on a re-run where the cert already exists it's a
  silent no-op instead of prompting `1: Keep / 2: Renew`. Without those flags it
  would block on stdin and die with `EOFError`, aborting the script (`set -e`)
  before the pull/recreate steps — don't remove them. (`deploy-markdown.sh` and
  `deploy-vault.sh` carry the same flags.)

- **Compose `command:` is flags only.** The image `ENTRYPOINT` is already
  `/bin/gitweb`; do **not** repeat `/bin/gitweb` in the compose `command:`. If you
  do, Go's flag parser stops at that leading positional and silently ignores all
  flags (including `-tls_cert`), so it serves **plain HTTP** → clients get a TLS
  "wrong version number" through nginx.

- **Compose recreate `KeyError: ContainerConfig`.** docker-compose 1.29.2 on
  COS's Docker errors on in-place recreate. Always `docker rm -f <container>`
  first, then `up -d --no-recreate`. (This is also why a redeploy with a new
  image must `docker rm -f patrick_gitweb_1` — `up` alone with `--no-recreate`
  keeps the old container/image.)

- **nginx network aliases (hairpin NAT).** GCP does not let a VM reach its own
  external IP, so the gitweb container can't talk to `upspin.cosmocr.at` /
  `key.cosmocr.at` via the public IP. The shared nginx service carries those as
  network `aliases`, so the gitweb upspin client resolves them to nginx and
  routes back in via SNI. Don't remove them.

- **Reading other users' `ee`-encrypted files.** A `read,list` grant lets gitweb
  *list and fetch* a repo, but an `ee`-packed (encrypted) object is only
  *readable* if a key was wrapped for `git@cosmocr.at` — the owner must run
  `upspin share -fix -r <repo>` after granting. `plain`/`eeintegrity` objects
  render with just the grant. Objects gitweb can't decrypt surface as `404`.

- **TLS cert.** nginx must be serving `:80` for the certbot webroot ACME
  challenge; the cert lives at `/mnt/disks/data/letsencrypt/live/git.cosmocr.at/`
  and the container hot-reloads it via the `certReloader` (no restart on renewal).

- **Restart policy is `unless-stopped`** (like every service in the stack). A
  `docker.service` drop-in gates the daemon on the data-disk mount, so it can't
  start against an unmounted disk. The store makes no upspin calls until the first
  request, so there's no startup readiness race.