Skip to content

fix(ci): run the S3 suites against RustFS #411

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
LKSNDRTMLKV merged 2 commits into from
Sep 24, 2026
Merged

fix(ci): run the S3 suites against RustFS #411

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
4b3de8f
fix(ci): run the S3 suites against RustFS
LKSNDRTMLKV Sep 24, 2026
491bfdb
fix(ci): generate the S3 test keys and gate on readiness
LKSNDRTMLKV Sep 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
58 changes: 36 additions & 22 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -204,43 +204,57 @@ jobs:
tool: nextest
- name: Create results dir
run: mkdir -p dpp-engine/results
# One MinIO for the whole job, rather than one per S3 test — the same
# One S3 server for the whole job, rather than one per S3 test — the same
# arrangement as the Postgres service above, for the same reason. The four
# `s3_archive` tests booted their own and spent about ten seconds each
# doing it, which put them on the wrong side of the slow-test budget
# whenever the runner was busy. `ODAL_TEST_S3_ENDPOINT` points them here;
# unset, they still start their own container, so a bare `cargo test`
# keeps working.
#
# A step rather than a `services:` entry because MinIO needs `server /data`
# as its command, and a service container can set `image`, `env`, `ports`,
# `volumes` and `options` but not a command. The alternative — an image
# that runs the server by default — would have CI and a local run testing
# two different MinIO builds, which is a worse trade than a longer step.
# The image and tag are the ones `s3_archive.rs` pins for its own container;
# the two must move together.
# A step rather than a `services:` entry so the readiness wait and the
# server's own log on failure sit in one place. The image and tag are the
# ones `s3_archive.rs` and `snapshot_static_tier.rs` pin for their own
# containers; the three must move together.
#
# `quay.io`, not Docker Hub: `docker.io/minio/minio` was removed. The Hub API
# answers 404 for the repository and every tag is denied, including this one,
# which this step pulled successfully until it vanished. quay.io is MinIO's own
# registry and carries this exact release — same publisher, same tag.
- name: Start MinIO
# RustFS, not MinIO: `docker.io/minio/minio` was removed, and on 2026-09-24
# `quay.io/minio/minio` stopped granting anonymous pulls — the registry's
# token for it carries no actions. RustFS takes the same shape (port 9000,
# an access key pair, `/data`), and enforces bucket policies, which the
# snapshot suite depends on. It fetches `version.rustfs.com` at every
# start and `RUSTFS_CHECK_UPDATES=false` does not stop that in 1.0.0, so
# the host resolves to loopback instead: a test double has no business
# reaching the network.
#
# The key pair is generated per run and handed to the tests through
# `ODAL_TEST_S3_ACCESS_KEY`/`_SECRET_KEY`, so no credential is written into
# the repository. The readiness probe (not liveness: the first thing every
# test does is create a bucket) carries its own `--max-time`, so a server
# that accepts and never answers cannot hold one probe past the deadline.
- name: Start S3 test server
run: |
docker run -d --name minio \
access_key=$(openssl rand -hex 16)
secret_key=$(openssl rand -hex 16)
echo "::add-mask::$access_key"
echo "::add-mask::$secret_key"
echo "ODAL_TEST_S3_ACCESS_KEY=$access_key" >> "$GITHUB_ENV"
echo "ODAL_TEST_S3_SECRET_KEY=$secret_key" >> "$GITHUB_ENV"
docker run -d --name s3 \
-p 9000:9000 \
-e MINIO_ROOT_USER=minioadmin \
-e MINIO_ROOT_PASSWORD=minioadmin \
quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z \
server /data --console-address :9001
--add-host version.rustfs.com:127.0.0.1 \
-e RUSTFS_ACCESS_KEY="$access_key" \
-e RUSTFS_SECRET_KEY="$secret_key" \
rustfs/rustfs:1.0.0
for _ in $(seq 1 30); do
if curl -fsS http://127.0.0.1:9000/minio/health/live >/dev/null 2>&1; then
echo "minio ready"
if curl -fsS --max-time 2 http://127.0.0.1:9000/health/ready >/dev/null 2>&1; then
echo "s3 server ready"
exit 0
fi
sleep 1
done
echo "minio did not become ready in 30s"
docker logs minio
echo "s3 server did not become ready in 30s"
docker logs s3
docker exec s3 sh -c 'tail -n 50 /logs/*' || true
exit 1
# Docker is pre-installed on ubuntu-latest; testcontainers uses it.
# The plugin-host suite includes the fuel-exhaustion sandbox test, which
Expand Down
29 changes: 29 additions & 0 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,35 @@ under the pre-1.0 conventions in [VERSIONING.md](docs/governance/VERSIONING.md):

## [Unreleased]

### Fixed

- **CI could not pull its S3 test server, so nothing could go green — again.**
`docker.io/minio/minio` was removed on 2026-09-13 and the pin moved to
`quay.io/minio/minio`; on 2026-09-24 that repository stopped granting
anonymous pulls. Its anonymous token carries an empty `actions` list, while a
control repository on the same registry grants `pull`, so this is the
repository closing rather than the registry failing. Every run failed at the
server's start, before a test ran.

The S3 suites now run against **RustFS 1.0.0** (`rustfs/rustfs`, Apache-2.0)
in CI and locally, pinned at the same three sites. Same shape — port 9000, an
access-key pair, `/data`. RustFS fetches `version.rustfs.com` at every start,
and `RUSTFS_CHECK_UPDATES=false` does not stop it in 1.0.0, so every container
resolves that host to loopback: the test double makes no outbound call. The
tests' own containers wait on `/health/ready` rather than a log line, because
RustFS logs to a file inside the container, and every probe carries its own
timeout. The `minioadmin` key pair the suites carried is gone: a container a
test starts gets a random pair, and the shared CI server's pair is generated
per run and passed as `ODAL_TEST_S3_ACCESS_KEY`/`_SECRET_KEY`.

**A new test pins what made the swap safe.**
`an_anonymous_read_is_refused_until_the_bucket_policy_allows_it` checks that an
unauthenticated read fails before the public-read policy is applied and
succeeds after. Every other snapshot test reads anonymously only after that
policy, so a server that let anyone read anything would have passed them all
while proving nothing about the policy the production bucket depends on — the
exact risk in replacing the server under them.

## [0.14.0] - 2026-09-24

### Breaking
Expand Down
148 changes: 102 additions & 46 deletions crates/dpp-node/tests/s3_archive.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,12 @@
//! Integration test: `S3ArchiveAdapter` against a real MinIO instance.
//! Integration test: `S3ArchiveAdapter` against a real S3-compatible server.
//!
//! Run: `cargo test -p dpp-node --features integration-tests`

#![cfg(feature = "integration-tests")]

use testcontainers::{
GenericImage, ImageExt,
core::{WaitFor, ports::ContainerPort},
core::{Host, ports::ContainerPort},
runners::AsyncRunner,
};

Expand All @@ -19,23 +19,24 @@ use dpp_domain::{
};
use dpp_node::infra::s3_archive::{S3ArchiveAdapter, S3ArchiveConfig};

/// The MinIO build every path here tests against.
/// The S3 server build every path here tests against.
///
/// One constant so the shared server CI starts and the container this file
/// starts locally cannot drift onto different releases. A suite that runs
/// against one MinIO in CI and a different one on a developer machine proves
/// less than it looks like it does.
/// `quay.io`, not Docker Hub.
/// against one server in CI and a different one on a developer machine proves
/// less than it looks like it does. The CI workflow and `snapshot_static_tier.rs`
/// pin the same pair; the three must move together.
///
/// `docker.io/minio/minio` was removed: the Hub API answers **404** for the
/// repository and a pull is denied for every tag, including this one, which CI
/// had been pulling successfully until it vanished. quay.io is MinIO's own
/// registry and carries this exact release, so the pin is otherwise unchanged —
/// same publisher, same tag. The CI workflow pins the same pair and the two must
/// move together.
const MINIO_IMAGE: (&str, &str) = ("quay.io/minio/minio", "RELEASE.2025-09-07T16-13-09Z");

/// Point this at a running MinIO and the suite uses it instead of starting a
/// **RustFS, because MinIO can no longer be pulled.** `docker.io/minio/minio` was
/// removed, and on 2026-09-24 `quay.io/minio/minio` stopped granting anonymous
/// pulls: the registry hands out a token whose `access` carries no actions,
/// while a control repository on the same registry grants `pull`. RustFS is an
/// Apache-2.0 S3 server configured the way MinIO was, and it enforces bucket
/// policies — which `snapshot_static_tier.rs` depends on, and a mock that
/// ignored them would pass while proving nothing.
const S3_IMAGE: (&str, &str) = ("rustfs/rustfs", "1.0.0");

/// Point this at a running S3 server and the suite uses it instead of starting a
/// container per test.
///
/// # Why this exists
Expand Down Expand Up @@ -66,17 +67,34 @@ const MINIO_IMAGE: (&str, &str) = ("quay.io/minio/minio", "RELEASE.2025-09-07T16
/// `cargo test` needs no orchestration.
const SHARED_ENDPOINT_ENV: &str = "ODAL_TEST_S3_ENDPOINT";

/// A MinIO to talk to, and the bucket this test owns on it.
struct Minio {
/// The shared server's key pair, read beside [`SHARED_ENDPOINT_ENV`]. Whoever
/// starts that server generates the pair; none is written into this file.
const SHARED_ACCESS_KEY_ENV: &str = "ODAL_TEST_S3_ACCESS_KEY";
const SHARED_SECRET_KEY_ENV: &str = "ODAL_TEST_S3_SECRET_KEY";

/// An S3 server to talk to, the bucket this test owns on it, and the key pair
/// that authenticates to it.
struct S3Server {
/// Held only on the container path — dropping it stops the container.
/// `None` when the endpoint came from the environment, where the server
/// outlives every test process and is not this test's to stop.
_container: Option<testcontainers::ContainerAsync<GenericImage>>,
endpoint: String,
bucket: String,
access_key: String,
secret_key: String,
}

/// A required variable of the shared-server arrangement, or a panic that says
/// which one is missing.
fn shared_key(name: &str) -> String {
std::env::var(name)
.ok()
.filter(|s| !s.is_empty())
.unwrap_or_else(|| panic!("{SHARED_ENDPOINT_ENV} is set, so {name} must be too"))
}

async fn start_minio() -> Minio {
async fn start_s3() -> S3Server {
// A bucket per test, so one shared server gives the same isolation a
// container per test gave. `ensure_bucket` creates it, buckets are cheap,
// and simple lowercase hex keeps the name inside S3's naming rules.
Expand All @@ -86,44 +104,82 @@ async fn start_minio() -> Minio {
.ok()
.filter(|s| !s.is_empty())
{
return Minio {
return S3Server {
_container: None,
endpoint,
bucket,
access_key: shared_key(SHARED_ACCESS_KEY_ENV),
secret_key: shared_key(SHARED_SECRET_KEY_ENV),
};
}

// `with_wait_for` is a `GenericImage` method; the `ImageExt` builders
// (`with_env_var`/`with_cmd`) convert to `ContainerRequest`, which has no
// `with_wait_for`. So set the wait condition before those calls.
// Pinned (not `latest`) for reproducibility — `latest` drifts its startup
// log (this release emits the `API:` banner on stderr).
let image = GenericImage::new(MINIO_IMAGE.0, MINIO_IMAGE.1)
// A fresh pair for a container that lives as long as this test, so no
// reusable credential is written down anywhere.
let access_key = uuid::Uuid::new_v4().simple().to_string();
let secret_key = uuid::Uuid::new_v4().simple().to_string();

// The image's default command runs the server on `/data`. RustFS fetches
// `version.rustfs.com` at every start, and `RUSTFS_CHECK_UPDATES=false` does
// not stop it in this release — measured, not assumed. Resolving the host to
// loopback does: a test double has no business reaching the network.
let image = GenericImage::new(S3_IMAGE.0, S3_IMAGE.1)
.with_exposed_port(ContainerPort::Tcp(9000))
.with_wait_for(WaitFor::message_on_stderr("API:"))
.with_env_var("MINIO_ROOT_USER", "minioadmin")
.with_env_var("MINIO_ROOT_PASSWORD", "minioadmin")
.with_cmd(vec!["server", "/data", "--console-address", ":9001"]);
.with_env_var("RUSTFS_ACCESS_KEY", access_key.clone())
.with_env_var("RUSTFS_SECRET_KEY", secret_key.clone())
.with_host(
"version.rustfs.com",
Host::Addr(std::net::Ipv4Addr::LOCALHOST.into()),
);

let container = image.start().await.expect("start minio container");
let container = image.start().await.expect("start the S3 server container");
let port = container
.get_host_port_ipv4(9000)
.await
.expect("minio mapped port");
.expect("S3 server mapped port");
let endpoint = format!("http://127.0.0.1:{port}");
wait_until_ready(&endpoint).await;

Minio {
S3Server {
_container: Some(container),
endpoint: format!("http://127.0.0.1:{port}"),
endpoint,
bucket,
access_key,
secret_key,
}
}

/// Poll the server's readiness route until it answers, for at most 30 seconds.
///
/// Readiness, not liveness: a server can be up before its storage is, and the
/// first thing every test does is create a bucket. Each probe carries its own
/// timeout, because a server that accepts a connection and never answers would
/// otherwise hold a single probe past any deadline. Not a log-line wait: RustFS
/// writes its log to a file inside the container, so nothing on stdout or
/// stderr marks the moment it starts serving.
async fn wait_until_ready(endpoint: &str) {
let url = format!("{endpoint}/health/ready");
let probe = reqwest::Client::builder()
.timeout(std::time::Duration::from_secs(2))
.build()
.expect("probe client");
let deadline = tokio::time::Instant::now() + std::time::Duration::from_secs(30);
while tokio::time::Instant::now() < deadline {
if let Ok(r) = probe.get(&url).send().await
&& r.status().is_success()
{
return;
}
tokio::time::sleep(std::time::Duration::from_millis(500)).await;
}
panic!("the S3 server at {endpoint} was not ready within 30s");
}

fn build_adapter(minio: &Minio) -> S3ArchiveAdapter {
fn build_adapter(s3: &S3Server) -> S3ArchiveAdapter {
S3ArchiveAdapter::new(S3ArchiveConfig {
endpoint: Some(minio.endpoint.clone()),
bucket: minio.bucket.clone(),
access_key_id: "minioadmin".into(),
secret_access_key: "minioadmin".into(),
endpoint: Some(s3.endpoint.clone()),
bucket: s3.bucket.clone(),
access_key_id: s3.access_key.clone(),
secret_access_key: s3.secret_key.clone(),
region: "us-east-1".into(),
})
}
Expand Down Expand Up @@ -179,8 +235,8 @@ fn make_passport() -> Passport {

#[tokio::test]
async fn archive_then_verify_integrity() {
let minio = start_minio().await;
let adapter = build_adapter(&minio);
let s3 = start_s3().await;
let adapter = build_adapter(&s3);
adapter.ensure_bucket().await.expect("create bucket");

let passport = make_passport();
Expand All @@ -200,8 +256,8 @@ async fn archive_then_verify_integrity() {

#[tokio::test]
async fn verify_wrong_hash_returns_not_ok() {
let minio = start_minio().await;
let adapter = build_adapter(&minio);
let s3 = start_s3().await;
let adapter = build_adapter(&s3);
adapter.ensure_bucket().await.expect("create bucket");

let passport = make_passport();
Expand All @@ -216,8 +272,8 @@ async fn verify_wrong_hash_returns_not_ok() {

#[tokio::test]
async fn retrieve_returns_original_passport() {
let minio = start_minio().await;
let adapter = build_adapter(&minio);
let s3 = start_s3().await;
let adapter = build_adapter(&s3);
adapter.ensure_bucket().await.expect("create bucket");

let passport = make_passport();
Expand All @@ -235,8 +291,8 @@ async fn retrieve_returns_original_passport() {

#[tokio::test]
async fn retrieve_unknown_passport_returns_none() {
let minio = start_minio().await;
let adapter = build_adapter(&minio);
let s3 = start_s3().await;
let adapter = build_adapter(&s3);
adapter.ensure_bucket().await.expect("create bucket");

let result = adapter.retrieve(PassportId::new()).await.expect("retrieve");
Expand Down
Loading
Loading