ci: mirror release artifacts to Cloudflare R2 (#1114)

Mirrors every release artifact into Cloudflare R2 alongside the existing AWS S3
upload, so both buckets carry the same objects during a parallel period. S3 is
untouched and stays authoritative for now; once R2 is confirmed complete it can
be dropped by deleting the `blobs:` block and pointing this at R2 alone.

### Why `publishers:` and not a second `blobs:` entry

goreleaser's blob pipe authenticates from the global `AWS_*` environment and has
no per-entry credential fields, so two different credential sets (AWS S3 and
Cloudflare R2) cannot coexist in `blobs:`. Custom publishers do support
per-entry `env:`, which is the supported way to isolate the two.

### Why `curl --aws-sigv4` and not `aws`/`rclone`

The CI image `docker.gitea.com/runner-images:ubuntu-latest` ships none of `aws`,
`rclone`, `s3cmd` or `mc`, but does ship curl 8.5 with `--aws-sigv4`. This keeps
the change zero-install. Credentials are fed to curl through a config file on
stdin rather than argv, so they never appear in the process list.

### Behaviour

- Object layout is identical to S3 (`gitea-runner/<version>/<artifact>`), so
  migrating consumers later means changing only the host, not the path.
- Nightly gets R2 too, matching the existing nightly-to-S3 behaviour.
- Missing R2 configuration fails the build. Because custom publishers run as the
  very last step of the publish pipeline, a preflight `--check-config` step runs
  right after checkout so the failure happens before anything is built or
  published rather than after the release already exists.
- Verified against a local MinIO instance (site region `auto`, matching R2):
  successful upload byte-compared after download, plus the missing-variable,
  wrong-credentials, missing-file and bad-argument paths.

### Required repository secrets

These must be configured before the next release run, otherwise the new
preflight step will fail the workflow:

- `R2_ENDPOINT` (full base URL, e.g. `https://<account>.r2.cloudflarestorage.com`)
- `R2_BUCKET`
- `R2_ACCESS_KEY_ID`
- `R2_SECRET_ACCESS_KEY`

### Known, deliberate wart

The publisher runs 109 times for 73 distinct object keys: goreleaser's release
pipe already registers `release.extra_files` as `UploadableFile` artifacts, and
`internal/exec` appends the publisher's own `extra_files` on top with no
de-duplication. It cannot be globbed away because `gobwas/glob` has no
substring-exclusion matcher, so `./**.sha256` cannot be narrowed to exclude
`*.xz.sha256`. It is harmless since PUT is idempotent, and the redundant
`./**.xz` glob is kept on purpose so the publisher declares its own complete
file set instead of implicitly depending on the `release:` block's globs. This
is documented in a comment above the block.

Reviewed-on: https://gitea.com/gitea/runner/pulls/1114
Reviewed-by: Zettat123 <39446+zettat123@noreply.gitea.com>
This commit is contained in:
Lunny Xiao
2026-07-26 03:26:46 +00:00
parent b4a64b97dd
commit 94ab020204
4 changed files with 175 additions and 0 deletions

112
scripts/upload-r2.sh Executable file
View File

@@ -0,0 +1,112 @@
#!/bin/sh
# Copyright 2026 The Gitea Authors. All rights reserved.
# SPDX-License-Identifier: MIT
#
# upload-r2.sh uploads a single local file to a single object key in a
# Cloudflare R2 bucket, using curl's built-in AWS SigV4 signer (R2 is
# S3-API compatible).
#
# This is the R2 half of the release process's parallel S3+R2 upload
# period: goreleaser's `blobs:` pipe still uploads every release
# artifact to AWS S3, and this script is invoked once per artifact
# (via a goreleaser `publishers:` entry) to mirror the same artifact
# into R2. Once the migration away from S3 is complete, the `blobs:`
# block and the AWS_* secrets can be dropped without touching this
# script.
#
# Usage:
# upload-r2.sh <local-file> <remote-key>
# upload-r2.sh --check-config
#
# The second form only validates that the required environment
# variables below are set (it does not touch the network or the
# filesystem beyond that), and is meant to be run as an early
# preflight step in CI: goreleaser custom publishers run as the very
# last step of the publish pipeline, so without a preflight check a
# missing R2_* secret would only be discovered after the Gitea release
# has already been created and every artifact already uploaded to S3.
#
# Required environment variables:
# R2_ENDPOINT Base URL of the R2 endpoint, e.g.
# https://<account>.r2.cloudflarestorage.com
# R2_BUCKET Destination bucket name.
# R2_ACCESS_KEY_ID R2 access key id.
# R2_SECRET_ACCESS_KEY R2 secret access key.
set -eu
# check_env validates that all required R2_* environment variables are
# set and non-empty, printing a single "missing required environment
# variable(s): ..." message and exiting non-zero otherwise. Used by
# both the normal upload mode and --check-config, so the validation
# logic only exists in one place.
check_env() {
missing=""
if [ -z "${R2_ENDPOINT:-}" ]; then
missing="$missing R2_ENDPOINT"
fi
if [ -z "${R2_BUCKET:-}" ]; then
missing="$missing R2_BUCKET"
fi
if [ -z "${R2_ACCESS_KEY_ID:-}" ]; then
missing="$missing R2_ACCESS_KEY_ID"
fi
if [ -z "${R2_SECRET_ACCESS_KEY:-}" ]; then
missing="$missing R2_SECRET_ACCESS_KEY"
fi
if [ -n "$missing" ]; then
echo "upload-r2.sh: missing required environment variable(s):$missing" >&2
exit 1
fi
}
if [ "$#" -eq 1 ] && [ "$1" = "--check-config" ]; then
check_env
echo "upload-r2.sh: R2 configuration OK"
exit 0
fi
if [ "$#" -ne 2 ]; then
echo "usage: upload-r2.sh <local-file> <remote-key>" >&2
echo " upload-r2.sh --check-config" >&2
exit 1
fi
local_file="$1"
remote_key="$2"
if [ ! -f "$local_file" ]; then
echo "upload-r2.sh: local file not found: $local_file" >&2
exit 1
fi
check_env
# Strip a single trailing slash from the endpoint, if present, so that
# building the path-style URL below never produces a double slash.
endpoint="${R2_ENDPOINT%/}"
url="$endpoint/$R2_BUCKET/$remote_key"
# Credentials are passed to curl through a config file read from
# stdin rather than as a command-line argument, so they never show up
# in `ps` output.
#
# --fail-with-body (instead of plain --fail) still exits non-zero on
# HTTP errors, but also prints R2's XML error body, which is where the
# actual error code lives (SignatureDoesNotMatch, NoSuchBucket,
# AccessDenied, ...); with plain --fail that body is discarded and the
# failure is silent. --retry 3 (without --retry-all-errors) still
# retries the transient cases (5xx, 408, 429, connection failures);
# --retry-all-errors would additionally retry permanent 4xx responses
# three times with backoff, which only delays an inevitable failure.
printf 'user = "%s:%s"\n' "$R2_ACCESS_KEY_ID" "$R2_SECRET_ACCESS_KEY" | curl \
--config - \
--fail-with-body \
--silent \
--show-error \
--retry 3 \
--aws-sigv4 "aws:amz:auto:s3" \
--upload-file "$local_file" \
"$url"