1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294

# FR: serve protocol v2 for upload-pack

Status: open — proposed, not started
Filed: 2026-09-21
Affects: `internal/githttp` (upload-pack path), `internal/server/githttp.go` (routing)
Builds on: `3bed207`, `7f24700` — the v0 negotiation fixes, which made fetch
work but left it transferring far more than the delta
Measured against: stock `git-http-backend` (git 2.47.3) in a local lab, recipe
at the end

## Summary

gitweb speaks only protocol v0, and a restricted dialect of it: the
advertisement offers no `multi_ack`, no `multi_ack_detailed`, no `side-band`.
That combination makes efficient negotiation impossible over stateless HTTP, so
an incremental fetch ships most of the history even when the client is one
commit behind. It also means every connection re-advertises every ref.

A client that sends `Git-Protocol: version=2` gets no v2 response today and
falls back to v0 (`internal/githttp/githttp.go:11`); gitweb never reads the
header at all. Since git 2.26 v2 is the client default, so this fallback is the
normal path, not an edge case.

## Symptom

Measured on `git.cosmocr.at` against `rpg137@gmail.com/projects/control.git`
(57 commits, 304 objects), with a client one commit behind:

```
$ git fetch origin
Receiving objects: 100% (175/175), 51.44 KiB
```

**175 objects to deliver a one-commit delta.** The client already held every
object but three. Before `3bed207` this fetch did not merely transfer too much,
it failed outright with `protocol error: bad line length character: PACK`; the
transfer volume is what is left after that fix.

## Why v0 cannot be pushed further here

The two fixes so far took the v0 path as far as it goes. The remaining loss is
structural, and the trace shows exactly where.

Over stateless HTTP git offers its haves in batches — the first holds 16 — and
**rebuilds its request buffer each round, sending only the haves it has not
sent yet.** This is directly observable: after a round in which gitweb replied
`ACK`, the next request on the wire was

```
want b36f8ff… ofs-delta agent=git/2.47.3
0000
done
```

with no `have` lines at all.

That leaves a server advertising no `multi_ack` with two moves in a
negotiation round, and neither works:

| Reply to a `done`-less round | Result |
|---|---|
| `NAK` | Negotiation continues, but the client's *next* request carries only the haves it has not yet sent. The `done` round therefore presents an older slice of history, the pack is cut against that, and we get the 175 objects above. |
| `ACK <oid>` | Without `multi_ack` a plain ACK means *negotiation over, pack follows*. git acts on it at once, sends a bare `done` request with no haves, and reads that response as a raw packfile — the fetch dies with `protocol error: bad pack header`. Verified: this is what `7f24700` had to back out. |

A real git server escapes because it can answer the first batch meaningfully:
`ACK <oid> ready`. gitweb has no way to say "ready" in the dialect it
advertises. Fixing that means adding a capability, and the two candidates are
`multi_ack_detailed` (v0) or v2.

## Measurements

Same 24-commit repo, same client, one commit behind, one commit to fetch. The
lab server is stock `git-http-backend`; the recipe is at the end.

| Server | Negotiation | Objects transferred |
|---|---|---|
| gitweb, v0 single-ack (today) | ≥2 POSTs, no convergence signal | **175** of 304 |
| stock git, v0 + `multi_ack_detailed` | 1 POST, `ACK … ready` | **3** |
| stock git, protocol v2 | 1 `command=fetch`, 16 haves, `ready` | **3** |

Ref advertisement, same repo grown to 201 refs, bytes returned by `info/refs`
on **every** connection:

| | bytes |
|---|---|
| v0 (all refs, always) | 13,458 |
| v2 (`ls-refs` on demand) | 131 |

The v2 advertisement carries no refs at all:

```
version 2
agent=git/2.47.3
ls-refs=unborn
fetch=shallow wait-for-done
server-option
object-format=sha1
```

The client then issues `command=ls-refs` with `ref-prefix refs/heads/` and
friends, so it asks for the refs it actually wants.

## The honest comparison: v2 vs `multi_ack_detailed`

**On transfer volume the two are equivalent — both measured 3 objects.** If the
only goal is to stop shipping the whole history, `multi_ack_detailed` on v0 is
the smaller change and gets there. This FR should not pretend otherwise.

v2 earns its cost on the things `multi_ack_detailed` does not touch:

- **Ref advertisement.** 13,458 → 131 bytes at 201 refs, on every connection.
  This is the win that grows with the repo rather than with the fetch.
- **It is what clients already ask for.** v2 has been the client default since
  git 2.26. Every fetch today pays a fallback.
- **Room to grow.** `server-option`, partial-clone filters, and `wait-for-done`
  are v2-only. Shallow clone — currently unsupported, see
  `docs/ARCHITECTURE.md` "Failure modes and limitations" — is negotiated inside
  the v2 `fetch` command rather than bolted onto the want list.
- **One command surface.** v2 is a request/response command protocol
  (`ls-refs`, `fetch`) rather than a stateful conversation squeezed through
  stateless POSTs, which is a better fit for how this server is built.

Against that, v2 is substantially more code (see below), and every line of it
is ours because go-git supplies none of it.

A defensible sequencing: **land `multi_ack_detailed` first** as a contained fix
that captures the transfer win, then do v2 as a separate change for the
advertisement and the feature surface. That keeps a large protocol rewrite off
the critical path of a bug that is costing bytes today.

## What implementing v2 requires

go-git v5.12.0 has **no** v2 support: `plumbing/protocol/` contains only
`packp`, and nothing in `plumbing/` mentions `command=ls-refs` or `version=2`.
Every piece below is new code in `internal/githttp`.

1. **Version selection.** Read the `Git-Protocol` request header and route on
   it. Nothing reads it today; it must reach both the `info/refs` GET
   (`internal/server/githttp.go:34`) and the `git-upload-pack` POST
   (`internal/server/githttp.go:43`).
2. **Capability advertisement.** Replace the ref list from
   `AdvertisedReferencesContext` (`internal/githttp/githttp.go:65`) with the v2
   capability list when v2 is requested. v0 must keep working unchanged.
3. **`ls-refs`.** Parse `ref-prefix`, `symrefs`, `peel`, `unborn`; emit the
   matching refs. This replaces the always-everything advertisement.
4. **`fetch`.** Parse the command's arguments (`want`, `have`, `done`,
   `thin-pack`, `ofs-delta`, `no-progress`, `include-tag`) and emit the
   response sections: `acknowledgments` (with `ready`), optionally
   `shallow-info`, `wantref`, and `packfile`.
5. **Convergence.** Decide a cut point from the haves and say `ready` — the
   thing v0 single-ack cannot express, and the reason the object count drops.
   `commonHaves` (`internal/githttp/githttp.go:209`) already resolves which
   haves we hold; what is missing is choosing a base and signalling it.
6. **Sideband.** The v2 `packfile` section is delimited and progress rides
   band 2. gitweb advertises no `side-band` today and emits a bare pack
   (`ServeUploadPack`, `internal/githttp/githttp.go:149`), so this is new.
7. **Delimiters.** v2 uses `0001` (delim-pkt) and `0002` (response-end-pkt)
   alongside `0000`. The go-git pktline encoder in use writes flush-pkts; these
   need adding.

## Scope

**In scope:** `ls-refs` and `fetch` for upload-pack over smart HTTP, with v0
retained as a fallback.

**Out of scope:**

- **Push.** Protocol v2 defines no receive-pack command; push stays v0 even
  when the client negotiates v2 elsewhere. `DecodeReceivePack` is untouched.
- **Shallow/partial clone.** v2 makes them expressible; implementing them is
  separate work.
- **Protocol v1.** No value here — it is v0 with a version line.
- **The git:// and ssh:// transports.** gitweb serves HTTP only.

## Compatibility and rollout

Version selection is per request and driven by the client's header, so v0
clients are unaffected by construction. The risk is a v2 path that is subtly
wrong for some client, which argues for:

- keeping v0 as the default and treating v2 as opt-in until it has soaked;
- a kill switch (flag or env) to force v0 server-side, so a bad v2 path can be
  disabled without a rollback and redeploy;
- testing against the real `git` binary, not just unit tests. The existing
  end-to-end tests in `internal/server/fetch_test.go` are the pattern, and they
  should be parameterised over `-c protocol.version=0` and `=2`.

## Risks

- **All new protocol code, no library backing.** Everything in the list above
  is hand-written and easy to get subtly wrong; the `bad pack header` episode
  in `7f24700` is a reminder that protocol bugs surface as opaque client
  failures, not as clean errors.
- **Sideband is a new failure surface.** Framing the pack means a new way to
  desynchronise a client that currently receives a bare stream that works.
- **go-git may add v2 later**, which would make a hand-rolled implementation
  something to migrate off. Worth a look upstream before starting.
- **Interaction with the repo cache.** `ServeUploadPack` streams from a storer
  whose backing is kept alive by a deferred release
  (`internal/server/githttp.go`, `gitUploadPack` branch). A v2 response with
  multiple sections must keep the same lifetime discipline.

## Acceptance criteria

- A client with `protocol.version=2` negotiates v2 end to end: the
  advertisement is the v2 capability list, refs come from `ls-refs`, and the
  fetch uses `command=fetch`.
- An incremental fetch of a one-commit delta transfers **3 objects**, matching
  stock `git-http-backend` in the table above — verified end to end against the
  real git binary, not asserted from unit tests.
- `info/refs` for a many-ref repo returns the capability list only, not the ref
  list.
- A v0 client is byte-for-byte unaffected: existing `internal/githttp` and
  `internal/server` tests pass unchanged.
- Push is unchanged and still v0.
- `go test ./...` green, with end-to-end fetch tests run under both protocol
  versions.

## Impact

Medium. Nothing is broken — fetch works after `3bed207` and `7f24700` — but
every incremental fetch pays for it, and the cost scales with history depth
rather than with what changed. For the repos hosted today the absolute numbers
are small (175 objects, 51 KiB on `control.git`); the argument for acting is
that both costs grow with the repos and the ref counts, and that the transfer
half is fixable much more cheaply than the rest of v2.

## Rebuilding the measurements

The numbers above came from a local lab running stock `git-http-backend`, so
gitweb's behaviour is compared against a reference implementation rather than
against the spec. Nothing leaves the machine.

```bash
LAB=$(mktemp -d); mkdir -p "$LAB/cgi-bin" "$LAB/repos"
ln -s /usr/lib/git-core/git-http-backend "$LAB/cgi-bin/git-http-backend"

# 24-commit repo.
W=$LAB/work; git init -q -b main "$W"
for i in $(seq 0 23); do
  echo "v$i" > "$W/f.txt"; git -C "$W" add -A; git -C "$W" commit -qm "commit $i"
done
git clone -q --bare "$W" "$LAB/repos/deep.git"

# CGI server. CGIHTTPRequestHandler forwards a fixed set of headers, and
# git-http-backend selects the protocol version from GIT_PROTOCOL, so the
# Git-Protocol header has to be forwarded explicitly -- without this the
# server silently answers v2 clients in v0.
cat > "$LAB/serve.py" <<'PY'
import http.server, os
ROOT = os.path.dirname(os.path.abspath(__file__))
os.environ["GIT_PROJECT_ROOT"] = os.path.join(ROOT, "repos")
os.environ["GIT_HTTP_EXPORT_ALL"] = "1"

class H(http.server.CGIHTTPRequestHandler):
    cgi_directories = ["/cgi-bin"]
    def run_cgi(self):
        gp = self.headers.get("Git-Protocol")
        if gp:
            os.environ["GIT_PROTOCOL"] = gp
        else:
            os.environ.pop("GIT_PROTOCOL", None)
        return super().run_cgi()

os.chdir(ROOT)
http.server.HTTPServer(("127.0.0.1", 8799), H).serve_forever()
PY
(cd "$LAB" && setsid nohup python3 serve.py >server.log 2>&1 </dev/null &)

U=http://127.0.0.1:8799/cgi-bin/git-http-backend/deep.git

# Client one commit behind; count objects before and after the fetch.
git -c protocol.version=2 clone -q "$U" "$LAB/c"
git -C "$LAB/c" reset -q --hard HEAD~1
git -C "$LAB/c" update-ref -d refs/remotes/origin/main
git -C "$LAB/c" reflog expire --expire=now --all
git -C "$LAB/c" gc --prune=now -q
count() { git -C "$LAB/c" count-objects -v | awk '/^count:/{c=$2} /^in-pack:/{p=$2} END{print c+p}'; }
before=$(count)
GIT_TRACE_PACKET=1 git -C "$LAB/c" -c protocol.version=2 fetch origin 2>"$LAB/v2.trace" >/dev/null
echo "objects transferred: $(( $(count) - before ))"     # 3

# Swap -c protocol.version=0 for the v0 + multi_ack_detailed figure (also 3).
# Advertisement sizes:
for i in $(seq 1 200); do git -C "$LAB/repos/deep.git" branch "topic/$i" main; done
curl -s "$U/info/refs?service=git-upload-pack" | wc -c                               # 13458
curl -s -H "Git-Protocol: version=2" "$U/info/refs?service=git-upload-pack" | wc -c   # 131

pkill -f "$LAB/serve.py"; rm -rf "$LAB"
```

The gitweb figure (175) comes from the same shape against
`https://git.cosmocr.at/rpg137@gmail.com/projects/control.git`, which needs
credentials; any hosted repo with more than 16 commits reproduces it.