ByteBucket: S3-compatible object storage in one Go binary

ByteBucket is a self-hosted object store that speaks the Amazon S3 wire protocol. It ships as one Go binary with an embedded React admin UI, stores objects on an ordinary filesystem, and keeps users in an embedded BoltDB file. Existing S3 clients work against it unchanged: the AWS SDKs, the aws CLI, boto3, rclone, s3cmd and Terraform.

This post covers the parts of the implementation that carry most of the design: request authentication, the storage layout, durable writes, and the limits that keep resource use bounded.

Two ports, one set of handlers

ByteBucket listens on two ports.

• 9000 is the S3 surface. Requests are authenticated with AWS Signature Version 4, and bodies are XML in the shapes S3 uses, including the standard Error document.
• 9001 is the admin surface. It serves a JSON API under /api/*, the embedded dashboard at /, and Prometheus metrics at /metrics. Requests carry the admin access key and secret in two headers.

The storage handlers are written once and mounted on both routers. Port 9001 exposes every bucket and object operation under /api/s3/* with JSON instead of XML, which is what the dashboard uses. Because both surfaces share handlers, the test suite includes cross-surface parity tests that write through one surface and read through the other, and check that ETags and body bytes are identical.

The two ports have different trust levels. Port 9000 can face the internet. Port 9001 must not: the project's SECURITY.md says to bind it to localhost or a private network and put it behind a VPN, SSH tunnel or authenticated proxy if remote access is needed.

Verifying AWS Signature V4

Every S3 request on port 9000 either carries an Authorization: AWS4-HMAC-SHA256 ... header or is a presigned URL with the same fields in the query string. Header verification in internal/auth/auth.go works like this:

1. Parse Credential, SignedHeaders and Signature from the header. The credential must have five parts ending in aws4_request.
2. Require X-Amz-Date, and require that its date matches the date in the credential scope.
3. Require X-Amz-Content-Sha256. Unless the client declared UNSIGNED-PAYLOAD or a STREAMING-* mode, hash the body and compare.
4. Build the canonical request from the method, path, canonical query string and signed headers, and derive the string to sign.
5. Look up the user, decrypt their secret, derive the signing key, and compare signatures.
6. Reject timestamps more than 15 minutes from server time in either direction.
7. Check the user's access rules for the requested action and bucket.

Step 3 matters more than it looks. The signature covers the declared payload hash, not the payload itself. A server that skips the comparison accepts any body under a valid signature for a different one. ByteBucket reads the body, hashes it, and rejects a mismatch with XAmzContentSHA256Mismatch.

The signing key is the standard SigV4 derivation, an HMAC chain over the date, region, service and a fixed terminator:

func getSigningKey(secret, date, region, service string) []byte {
	kDate := hmacSHA256([]byte(date), []byte(secret))
	kRegion := hmacSHA256([]byte(region), kDate)
	kService := hmacSHA256([]byte(service), kRegion)
	return hmacSHA256([]byte("aws4_request"), kService)
}

The final comparison uses hmac.Equal on the decoded bytes, so its timing does not depend on how many leading bytes match. Presigned URLs follow the same path with values from the query string, plus an expiry check against X-Amz-Expires. They need no server-side state.

Why secrets are encrypted, not hashed

Passwords are normally stored as one-way hashes. SigV4 does not allow that: the server has to compute the same HMAC the client computed, which requires the plaintext secret. ByteBucket therefore encrypts each user's secret with AES-GCM under a 32-byte ENCRYPTION_KEY, with a random nonce stored in front of the ciphertext, and decrypts it per request.

The key is required on every boot. Losing it means losing every stored credential. Objects are unaffected, and the documented recovery is to delete users.db and start again with new credentials.

On first boot, and only then, ACCESS_KEY_ID and SECRET_ACCESS_KEY seed a super-user. After that, users are managed through the admin API, which generates each new secret on the server and returns it once. "Admin" is not a flag on the user. A user is an admin if their access rules contain an allow on all buckets and all actions; anything narrower is an S3-only user.

Objects on a plain filesystem

Everything lives under one data directory:

/data/
  users.db                  # BoltDB: users, access rules, encrypted secrets
  objects/
    <bucket>/
      <object>              # raw bytes
      <object>.meta         # JSON: ETag, checksums, user metadata
      <object>.tags.json    # JSON: tag set
      .acl.json             # bucket ACL
      .cors.json            # bucket CORS rules
  uploads/
    <bucket>/<uploadId>/    # in-flight multipart parts and manifest

Object keys map to file paths, so key validation is a security boundary. ValidateObjectKey uses an allowlist: it rejects empty, . and .. path segments, NUL bytes, and any segment that equals or ends with a reserved sidecar name. Without that last rule, a client could upload a key named .acl.json and overwrite the bucket's access policy. The repository includes a black-box harness in scripts/pentest that runs from a separate container and probes these cases, including URL-encoded and double-encoded traversal and attempts to overwrite sidecar files.

Durable writes

An upload is written to a temporary file in the destination directory. In one pass, the bytes go to the file, an MD5 hasher for the ETag, and a CRC32 hasher for the checksum. Then:

1. fsync the file.
2. Rename it over the destination. A crash never leaves a partial object.
3. fsync the parent directory, so the rename itself survives power loss.
4. Write the .meta sidecar.

Steps 1 and 3 are controlled by SYNC_WRITES, which defaults to on. An operator can turn them off to trade durability for throughput, from the environment or live from the admin settings page. If a crash lands between the rename and the sidecar write, the README documents that a missing .meta is recomputed on the next read.

Concurrent writes and deletes of the same key are serialized with a fixed array of 256 mutexes, selected by an FNV hash of the object's path. A per-key lock map would grow with the key space, which a hostile client controls. The fixed array keeps memory constant; the cost is that two unrelated keys occasionally share a stripe and wait for each other.

ETags

A single-part ETag is the hex MD5 of the object's bytes. A multipart ETag follows S3's composite format: the MD5 of the concatenated part MD5s, followed by a dash and the part count. Tags are stored in their own sidecar, so changing them never changes the ETag.

Keeping resource use bounded

Several limits exist to stop a client from turning a feature into a denial of service.

• Request size. Bodies are capped at 5 GiB on port 9000, the S3 single-PUT ceiling, and 100 MiB on port 9001. Headers are capped at 1 MiB.
• Timeouts. 10 seconds to read headers, 5 minutes for a read or write, 120 seconds idle.
• Rate limiting. Off by default. When enabled, it is a token bucket per client IP, shared across both ports so a client cannot double its allowance by splitting traffic. It runs before authentication, so a flood is rejected before any signature work or disk access. Over-limit requests get 503 SlowDown with Retry-After, which AWS SDKs treat as retryable.
• The limiter's own memory. The per-IP map is capped at 65,536 entries. When full, the least recently seen client is evicted, and a background sweep removes entries idle for more than 10 minutes. Without the cap, an attacker minting source addresses would grow the map until the process ran out of memory.
• Trusted proxies. RATE_LIMIT_TRUSTED_PROXIES sets how many X-Forwarded-For hops to trust, counted from the right. At zero, the header is ignored and the socket peer is the client.

On SIGTERM the server stops accepting connections and drains in-flight requests for up to 30 seconds. Every response carries an x-amz-request-id, and logs are one JSON line per request with the route template as the path, so object keys and signatures do not end up in log storage.

Limits

These come from the README and the code, and are worth knowing before choosing it.

• Single node. BoltDB is a single-writer embedded database. There is no clustering or replication.
• Not implemented: versioning, object lock, server-side encryption of object data, replication and lifecycle policies.
• Multipart is lenient. There is no minimum part size. S3 requires 5 MiB for every part except the last.
• Signed payloads are buffered. When a client signs the payload hash, the server reads the body into memory to verify it. Large files should use multipart upload, which most SDKs switch to automatically.
• Admin authentication is minimal. Credentials sit in the browser's localStorage and travel as headers. SECURITY.md lists server-side sessions, CSRF protection, a second factor and in-process TLS for the admin port as deferred work. That is the reason port 9001 stays private.

ByteBucket is licensed under the Server Side Public License.

Try it

Generate the credentials first, so you keep a copy of the encryption key. The server needs the same key on every start.

export ENCRYPTION_KEY="$(openssl rand -base64 32)"
export AK=admin
export SK="$(openssl rand -base64 32)"

docker run -d \
  --name bytebucket \
  -p 9000:9000 \
  -p 9001:9001 \
  -v bytebucket-data:/data \
  -e ENCRYPTION_KEY="$ENCRYPTION_KEY" \
  -e ACCESS_KEY_ID="$AK" \
  -e SECRET_ACCESS_KEY="$SK" \
  ghcr.io/byteink/bytebucket:latest

Then open http://localhost:9001 and log in with the same key pair, or use curl's built-in SigV4 signing:

curl -X PUT http://localhost:9000/my-bucket \
  --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AK:$SK"

curl -X PUT http://localhost:9000/my-bucket/hello.txt \
  --aws-sigv4 "aws:amz:us-east-1:s3" --user "$AK:$SK" \
  --data-binary 'hello'

Or point the AWS CLI at it, with AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY set to the same pair:

aws --endpoint-url http://localhost:9000 s3 cp ./big.bin s3://my-bucket/big.bin

SDK clients need path-style addressing (forcePathStyle: true in the JavaScript SDK, addressing_style: 'path' in boto3). The full API reference is in the repository. Our other public projects are on the open-source page, and the way we build and run products is described under product engineering.