git@cosmocr.at/gitweb.git @ HEAD · summary refs log

HEAD / docs / FR-protocol-v2.md /

docs/FR-protocol-v2.md · 13.0 KiB · raw

FR: serve protocol v2 for upload-pack

Status: open — proposed, not started Filed: 2026-09-21 Affects: internal/githttp (upload-pack path), internal/server/githttp.go (routing) Builds on: 3bed207, 7f24700 — the v0 negotiation fixes, which made fetch work but left it transferring far more than the delta Measured against: stock git-http-backend (git 2.47.3) in a local lab, recipe at the end

Summary

gitweb speaks only protocol v0, and a restricted dialect of it: the advertisement offers no multi_ack, no multi_ack_detailed, no side-band. That combination makes efficient negotiation impossible over stateless HTTP, so an incremental fetch ships most of the history even when the client is one commit behind. It also means every connection re-advertises every ref.

A client that sends Git-Protocol: version=2 gets no v2 response today and falls back to v0 (internal/githttp/githttp.go:11); gitweb never reads the header at all. Since git 2.26 v2 is the client default, so this fallback is the normal path, not an edge case.

Symptom

Measured on git.cosmocr.at against rpg137@gmail.com/projects/control.git (57 commits, 304 objects), with a client one commit behind:

$ git fetch origin
Receiving objects: 100% (175/175), 51.44 KiB

175 objects to deliver a one-commit delta. The client already held every object but three. Before 3bed207 this fetch did not merely transfer too much, it failed outright with protocol error: bad line length character: PACK; the transfer volume is what is left after that fix.

Why v0 cannot be pushed further here

The two fixes so far took the v0 path as far as it goes. The remaining loss is structural, and the trace shows exactly where.

Over stateless HTTP git offers its haves in batches — the first holds 16 — and rebuilds its request buffer each round, sending only the haves it has not sent yet. This is directly observable: after a round in which gitweb replied ACK, the next request on the wire was

want b36f8ff… ofs-delta agent=git/2.47.3
0000
done

with no have lines at all.

That leaves a server advertising no multi_ack with two moves in a negotiation round, and neither works:

Reply to a `done`-less round	Result
`NAK`	Negotiation continues, but the client's next request carries only the haves it has not yet sent. The `done` round therefore presents an older slice of history, the pack is cut against that, and we get the 175 objects above.
`ACK <oid>`	Without `multi_ack` a plain ACK means negotiation over, pack follows. git acts on it at once, sends a bare `done` request with no haves, and reads that response as a raw packfile — the fetch dies with `protocol error: bad pack header`. Verified: this is what `7f24700` had to back out.

A real git server escapes because it can answer the first batch meaningfully: ACK <oid> ready. gitweb has no way to say "ready" in the dialect it advertises. Fixing that means adding a capability, and the two candidates are multi_ack_detailed (v0) or v2.

Measurements

Same 24-commit repo, same client, one commit behind, one commit to fetch. The lab server is stock git-http-backend; the recipe is at the end.

Server	Negotiation	Objects transferred
gitweb, v0 single-ack (today)	≥2 POSTs, no convergence signal	175 of 304
stock git, v0 + `multi_ack_detailed`	1 POST, `ACK … ready`	3
stock git, protocol v2	1 `command=fetch`, 16 haves, `ready`	3

Ref advertisement, same repo grown to 201 refs, bytes returned by info/refs on every connection:

	bytes
v0 (all refs, always)	13,458
v2 (`ls-refs` on demand)	131

The v2 advertisement carries no refs at all:

version 2
agent=git/2.47.3
ls-refs=unborn
fetch=shallow wait-for-done
server-option
object-format=sha1

The client then issues command=ls-refs with ref-prefix refs/heads/ and friends, so it asks for the refs it actually wants.

The honest comparison: v2 vs `multi_ack_detailed`

On transfer volume the two are equivalent — both measured 3 objects. If the only goal is to stop shipping the whole history, multi_ack_detailed on v0 is the smaller change and gets there. This FR should not pretend otherwise.

v2 earns its cost on the things multi_ack_detailed does not touch:

Ref advertisement. 13,458 → 131 bytes at 201 refs, on every connection. This is the win that grows with the repo rather than with the fetch.
It is what clients already ask for. v2 has been the client default since git 2.26. Every fetch today pays a fallback.
Room to grow. server-option, partial-clone filters, and wait-for-done are v2-only. Shallow clone — currently unsupported, see docs/ARCHITECTURE.md "Failure modes and limitations" — is negotiated inside the v2 fetch command rather than bolted onto the want list.
One command surface. v2 is a request/response command protocol (ls-refs, fetch) rather than a stateful conversation squeezed through stateless POSTs, which is a better fit for how this server is built.

Against that, v2 is substantially more code (see below), and every line of it is ours because go-git supplies none of it.

A defensible sequencing: land multi_ack_detailed first as a contained fix that captures the transfer win, then do v2 as a separate change for the advertisement and the feature surface. That keeps a large protocol rewrite off the critical path of a bug that is costing bytes today.

What implementing v2 requires

go-git v5.12.0 has no v2 support: plumbing/protocol/ contains only packp, and nothing in plumbing/ mentions command=ls-refs or version=2. Every piece below is new code in internal/githttp.

Version selection. Read the Git-Protocol request header and route on it. Nothing reads it today; it must reach both the info/refs GET (internal/server/githttp.go:34) and the git-upload-pack POST (internal/server/githttp.go:43).
Capability advertisement. Replace the ref list from AdvertisedReferencesContext (internal/githttp/githttp.go:65) with the v2 capability list when v2 is requested. v0 must keep working unchanged.
ls-refs. Parse ref-prefix, symrefs, peel, unborn; emit the matching refs. This replaces the always-everything advertisement.
fetch. Parse the command's arguments (want, have, done, thin-pack, ofs-delta, no-progress, include-tag) and emit the response sections: acknowledgments (with ready), optionally shallow-info, wantref, and packfile.
Convergence. Decide a cut point from the haves and say ready — the thing v0 single-ack cannot express, and the reason the object count drops. commonHaves (internal/githttp/githttp.go:209) already resolves which haves we hold; what is missing is choosing a base and signalling it.
Sideband. The v2 packfile section is delimited and progress rides band 2. gitweb advertises no side-band today and emits a bare pack (ServeUploadPack, internal/githttp/githttp.go:149), so this is new.
Delimiters. v2 uses 0001 (delim-pkt) and 0002 (response-end-pkt) alongside 0000. The go-git pktline encoder in use writes flush-pkts; these need adding.

Scope

In scope: ls-refs and fetch for upload-pack over smart HTTP, with v0 retained as a fallback.

Out of scope:

Push. Protocol v2 defines no receive-pack command; push stays v0 even when the client negotiates v2 elsewhere. DecodeReceivePack is untouched.
Shallow/partial clone. v2 makes them expressible; implementing them is separate work.
Protocol v1. No value here — it is v0 with a version line.
The git:// and ssh:// transports. gitweb serves HTTP only.

Compatibility and rollout

Version selection is per request and driven by the client's header, so v0 clients are unaffected by construction. The risk is a v2 path that is subtly wrong for some client, which argues for:

keeping v0 as the default and treating v2 as opt-in until it has soaked;
a kill switch (flag or env) to force v0 server-side, so a bad v2 path can be disabled without a rollback and redeploy;
testing against the real git binary, not just unit tests. The existing end-to-end tests in internal/server/fetch_test.go are the pattern, and they should be parameterised over -c protocol.version=0 and =2.

Risks

All new protocol code, no library backing. Everything in the list above is hand-written and easy to get subtly wrong; the bad pack header episode in 7f24700 is a reminder that protocol bugs surface as opaque client failures, not as clean errors.
Sideband is a new failure surface. Framing the pack means a new way to desynchronise a client that currently receives a bare stream that works.
go-git may add v2 later, which would make a hand-rolled implementation something to migrate off. Worth a look upstream before starting.
Interaction with the repo cache. ServeUploadPack streams from a storer whose backing is kept alive by a deferred release (internal/server/githttp.go, gitUploadPack branch). A v2 response with multiple sections must keep the same lifetime discipline.

Acceptance criteria

A client with protocol.version=2 negotiates v2 end to end: the advertisement is the v2 capability list, refs come from ls-refs, and the fetch uses command=fetch.
An incremental fetch of a one-commit delta transfers 3 objects, matching stock git-http-backend in the table above — verified end to end against the real git binary, not asserted from unit tests.
info/refs for a many-ref repo returns the capability list only, not the ref list.
A v0 client is byte-for-byte unaffected: existing internal/githttp and internal/server tests pass unchanged.
Push is unchanged and still v0.
go test ./... green, with end-to-end fetch tests run under both protocol versions.

Impact

Medium. Nothing is broken — fetch works after 3bed207 and 7f24700 — but every incremental fetch pays for it, and the cost scales with history depth rather than with what changed. For the repos hosted today the absolute numbers are small (175 objects, 51 KiB on control.git); the argument for acting is that both costs grow with the repos and the ref counts, and that the transfer half is fixable much more cheaply than the rest of v2.

Rebuilding the measurements

The numbers above came from a local lab running stock git-http-backend, so gitweb's behaviour is compared against a reference implementation rather than against the spec. Nothing leaves the machine.

LAB=$(mktemp -d); mkdir -p "$LAB/cgi-bin" "$LAB/repos"
ln -s /usr/lib/git-core/git-http-backend "$LAB/cgi-bin/git-http-backend"

# 24-commit repo.
W=$LAB/work; git init -q -b main "$W"
for i in $(seq 0 23); do
  echo "v$i" > "$W/f.txt"; git -C "$W" add -A; git -C "$W" commit -qm "commit $i"
done
git clone -q --bare "$W" "$LAB/repos/deep.git"

# CGI server. CGIHTTPRequestHandler forwards a fixed set of headers, and
# git-http-backend selects the protocol version from GIT_PROTOCOL, so the
# Git-Protocol header has to be forwarded explicitly -- without this the
# server silently answers v2 clients in v0.
cat > "$LAB/serve.py" <<'PY'
import http.server, os
ROOT = os.path.dirname(os.path.abspath(__file__))
os.environ["GIT_PROJECT_ROOT"] = os.path.join(ROOT, "repos")
os.environ["GIT_HTTP_EXPORT_ALL"] = "1"

class H(http.server.CGIHTTPRequestHandler):
    cgi_directories = ["/cgi-bin"]
    def run_cgi(self):
        gp = self.headers.get("Git-Protocol")
        if gp:
            os.environ["GIT_PROTOCOL"] = gp
        else:
            os.environ.pop("GIT_PROTOCOL", None)
        return super().run_cgi()

os.chdir(ROOT)
http.server.HTTPServer(("127.0.0.1", 8799), H).serve_forever()
PY
(cd "$LAB" && setsid nohup python3 serve.py >server.log 2>&1 </dev/null &)

U=http://127.0.0.1:8799/cgi-bin/git-http-backend/deep.git

# Client one commit behind; count objects before and after the fetch.
git -c protocol.version=2 clone -q "$U" "$LAB/c"
git -C "$LAB/c" reset -q --hard HEAD~1
git -C "$LAB/c" update-ref -d refs/remotes/origin/main
git -C "$LAB/c" reflog expire --expire=now --all
git -C "$LAB/c" gc --prune=now -q
count() { git -C "$LAB/c" count-objects -v | awk '/^count:/{c=$2} /^in-pack:/{p=$2} END{print c+p}'; }
before=$(count)
GIT_TRACE_PACKET=1 git -C "$LAB/c" -c protocol.version=2 fetch origin 2>"$LAB/v2.trace" >/dev/null
echo "objects transferred: $(( $(count) - before ))"     # 3

# Swap -c protocol.version=0 for the v0 + multi_ack_detailed figure (also 3).
# Advertisement sizes:
for i in $(seq 1 200); do git -C "$LAB/repos/deep.git" branch "topic/$i" main; done
curl -s "$U/info/refs?service=git-upload-pack" | wc -c                               # 13458
curl -s -H "Git-Protocol: version=2" "$U/info/refs?service=git-upload-pack" | wc -c   # 131

pkill -f "$LAB/serve.py"; rm -rf "$LAB"

The gitweb figure (175) comes from the same shape against https://git.cosmocr.at/rpg137@gmail.com/projects/control.git, which needs credentials; any hosted repo with more than 16 commits reproduces it.