Home · ← SuperJ Manual EN|中文

S3-Compatible Multipart Upload

sj.aws and sj.aws.s3 implement AWS Signature Version 4 (SigV4) request signing and an S3 multipart-upload client, built directly on sj.net.NioAdapter's TLS transport (sj.http.HttpClient is plaintext-only). This tutorial covers credentials, wiring the TLS transport, the four multipart calls, the uploadFile convenience wrapper, and — first, because it changes how you'll actually use this — what it can and can't reach today.

Before you start: what this can actually talk to

connectTls does reach real AWS S3 (#4224). S3 negotiates the one suite SuperJ offers, TLS_AES_128_GCM_SHA256 + X25519 + ECDSA-P256, and its chain verifies against an ordinary CA bundle. Anchor trust in the OS bundle and name the host you expect:

byte[] pem = /* read /etc/ssl/cert.pem, or your distro's ca-certificates.crt */;
TlsConfig tls = new TlsConfig()
    .trustAnchors(Pem.certBundle(pem))
    .expectHost("s3.ap-northeast-1.amazonaws.com");

Use TlsConfig.pinnedPeerKey(...) instead for a fixed endpoint whose key you control (that is what the demo below does with its mock server), or trustAnchor(...) for a single CA.

Two things to know:

Temporary credentials work: AwsCredentials carries an optional session token and the signer covers it correctly (#4232). What is still on you is obtaining one — the IMDSv2 dance and expiry-driven refresh are not in the SDK. See Credentials.

The demo below stands a minimal mock S3 server up in-process and talks to it over a real loopback TLS connection, so it runs with no credentials and no network while every other detail — request building, SigV4 signing, response parsing — is exactly what a real S3-compatible endpoint would see. Run it yourself:

superj compile "$SJ_HOME/demo/sj/demo/S3Demo.sj" --sdk-path "$SJ_HOME/sdk" --link --output s3demo
./s3demo

--sdk-source "$SJ_HOME/sdk/sj" in place of --sdk-path builds the same demo whole-program; it works too, but spends ~100 s in clang (#5279).

Credentials

import sj.aws.AwsCredentials;

// Long-term keys.
AwsCredentials creds = new AwsCredentials(accessKeyId, secretAccessKey);

// Temporary keys — STS, or an IMDSv2 instance role.
AwsCredentials creds = new AwsCredentials(accessKeyId, secretAccessKey, sessionToken);

// Or from the environment: AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY, plus
// AWS_SESSION_TOKEN when it is set. Null if either key is unset.
AwsCredentials creds = AwsCredentials.fromEnvironment();

With a session token present, every request carries x-amz-security-token and includes it in the signature — SigV4 requires both, and doing one without the other fails as SignatureDoesNotMatch with nothing in the error to say which half was missing. You do not have to do anything to arrange that; S3Client and SigV4Signer agree on it between themselves.

What the SDK does not do is fetch or refresh those credentials. The IMDSv2 flow (PUT /latest/api/token, then GET /latest/meta-data/iam/security-credentials/<role>) and re-reading them before they expire are the application's job — build a fresh AwsCredentials and a fresh S3Client when they roll over.

Constructing the client

S3Client takes its NioAdapter by constructor injection rather than creating one internally — you drive the event loop, so anything else you're doing on that adapter (another connection, a server) keeps working normally while an S3Client call's own bounded poll() loop runs underneath it:

import sj.net.NioAdapter;
import sj.net.NioConfig;
import sj.net.TlsConfig;
import sj.aws.s3.S3Client;

NioAdapter adapter = new NioAdapter(new NioConfig(65536, 4096, 8, 65536, false, 16));
TlsConfig tls = new TlsConfig().pinnedPeerKey(pin);   // or .trustAnchor(caDer)

S3Client client = new S3Client(adapter, host, port, region, bucket, creds, tls);

Addressing is path-style (/<bucket>/<key>), not virtual-hosted — simpler, and works against any S3-compatible endpoint regardless of DNS setup.

The multipart calls

string uploadId = client.createMultipartUpload("my/object.bin");

string etag1 = client.uploadPart("my/object.bin", uploadId, 1, part1, 0, part1.length);
string etag2 = client.uploadPart("my/object.bin", uploadId, 2, part2, 0, part2.length);

PartETag[] parts = new PartETag[2];
parts[0] = new PartETag(1, etag1);
parts[1] = new PartETag(2, etag2);
client.completeMultipartUpload("my/object.bin", uploadId, parts);

completeMultipartUpload is not optional — it's the call that actually finishes the upload; parts sit unassembled (and billed) until it succeeds. If something goes wrong partway through, clean up with:

client.abortMultipartUpload("my/object.bin", uploadId);

Each part's SHA-256 payload hash is computed upfront (no chunked/streaming SigV4) — fine since the whole part has to be in memory to write it anyway. A non-2xx response from any call raises S3Exception, carrying the parsed <Error><Code>/<Message> when the response includes one.

uploadFile: splitting a local file for you

string uploadId = client.uploadFile("/path/to/local/file", "my/object.bin");

Splits the file into fixed-size parts (well above S3's 5 MiB minimum) via sj.io.MemoryMapped, uploads them sequentially with a bounded per-part retry, and calls completeMultipartUpload — or abortMultipartUpload if a part exhausts its retries or completion itself fails.

Uploading in a long-running process: the zero-allocation overload

SuperJ has no GC, so a string is retained for the life of the process. The overload above returns the upload ID as one, which costs about 48 bytes per call — permanently. That is nothing for a batch job and a slow leak for an archiver that runs for months. There is an overload for the second case:

ByteArray uploadId = new ByteArray(256);

while (moreFiles()) {
    uploadId.clear();
    int n = client.uploadFile(path, key, uploadId);   // retains NOTHING
    // the ID is uploadId.array()[0..n), or uploadId.flip().toString() if you
    // actually need it as text
}

Pass null for uploadIdOut if you do not want the ID at all. A buffer too small for it is reported by ByteArray's own bound, not silently truncated.

Everything else on the upload path already retains nothing — the request buffers, the TLS connection (reused across requests), SigV4 signing, the CompleteMultipartUpload XML, and the part ETags, which are kept as byte ranges rather than PartETag objects precisely so that no string is needed per part. examples/test_4590_zero_alloc_upload.sj asserts this as an equality — 200 uploads, 600 HTTP requests, zero bytes retained — and examples/test_4589_zero_alloc_request.sj does the same for a single request. Getting there took #4242, #4314, #4589 and #4590; the history is on #4636 if you want the accounting.

Reading objects

long size = client.headObject("my/object.bin");            // -1 if the server sent no Content-Length

byte[] buf = new byte[(int) size];
int n = client.getObject("my/object.bin", buf, 0, buf.length);

// Inclusive, as HTTP defines it — this reads 100 bytes.
int m = client.getObjectRange("my/object.bin", 0, 99, buf, 0, buf.length);

getObject throws rather than returning a short read if the object does not fit the destination — a truncated body reported as a complete one is how you end up writing a corrupt file and not finding out.

So for anything whose size you don't control, use downloadFile, the read-side mirror of uploadFile:

long written = client.downloadFile("my/object.bin", "/path/to/local/file");
long written = client.downloadFile("my/object.bin", "/path/to/local/file", 1 << 20);  // 1 MiB chunks

It HEADs the object, then walks it in ranged GETs straight to disk, so memory stays at roughly 2x the chunk size no matter how large the object is. That is what makes a multi-gigabyte cold-storage read possible here — there is no streaming response parser, so a whole-object getObject really does need the whole object in memory (twice, transiently).

What's deliberately not here

Each is a reasonable follow-up if you need it; none were needed for the use case this client was built for.

SuperJ — manual · generated from s3.md at pack time EN|中文