Gabriel Tanksleygabescloud.com

← All work

Storage · Cross-cloud · Cost-aware design

Moving compliance log archives into S3 Glacier

Retained log archives had to move from a provider-hosted bucket into each organization’s own AWS account, in cold storage that could still answer an audit, without touching production. I owned the AWS side: architecture, build, execution, verification, and the runbook.

Role
AWS-side owner: design, build, execution, verification
Team
Three
Scope
Two organizations, one runbook
Status
Transfers designed and launched; deletion gated on verification

At a glance

How to read the tags
~20 TiB
Source-side baseline across two organizations
measured
~81%
Lower monthly storage cost than a hot-tier equivalent, at published per-GB rates, even after choosing the pricier archive tier
calculated
0
Production systems touched. The copy ran on a disposable host in the destination account, an architectural property
documented
Not measured
Final transfer wall clock. Monitoring moved shells before completion; the next run records actuals at the verification step
not measured

Storage and access design

  • S3 with Block Public Access on, server-side encryption with S3 Bucket Keys to cut per-object KMS requests at archive scale, a TLS-only bucket policy, a lifecycle rule that aborts incomplete multipart uploads, and a five-key cost-allocation and audit tag set.
  • On the source side, a single-purpose read-only identity per organization, bound to object-read on exactly one bucket, with a documented teardown step.
  • A credential relay that moves the source key through a managed secrets service onto the transfer host without the key material ever printing to a terminal.

Transfer host

The transfer host had AWS access through its instance profile only, with no AWS keys on disk. The one source-side read key arrived through the secrets relay. IMDSv2 was enforced, the root volume was encrypted, the security group had zero inbound rules, and access was through Session Manager instead of SSH.

Running the copy from a disposable instance in the destination account, instead of from production servers, was close to cost-neutral: source egress is billed the same either way, and ingress is free. It kept cross-cloud credentials off production hardware and removed the filesystem path-length limits that break long copy paths.

Verification gates

  • A pre-flight size and object-count baseline served as both the runtime estimate and the reconciliation target, and the transfer was gated on object count before any spend was committed.
  • A small canary per organization, with the storage class checked on real landed objects before the full copy. That check keeps a multi-terabyte write out of the wrong, far more expensive tier.
  • Verification reconciles object count and total bytes, then runs a one-way, size-only comparison. Size-only on purpose: reading content back means restoring every object first, which takes hours per batch and creates temporary copies.
  • Deleting source data requires all five: counts match, sizes match, zero differences reported, the verification log attached to the ticket, and written go-ahead from the lead or manager.

Key decision

Deep Archive, or Glacier Flexible Retrieval?

Deep Archive is materially cheaper per month and was the original plan. The first runbook draft was written against it.

  • S3 Glacier Deep Archive
    Lowest monthly cost. 180-day minimum storage duration, and retrieval that is slow and billed.
  • ChosenS3 Glacier Flexible Retrieval
    Higher monthly cost. 90-day minimum, and bulk retrieval at no charge.

Why

If an audit request or a litigation hold lands on this data, the retrieval path is what matters, not the monthly line item. The higher monthly cost bought half the minimum-commitment exposure and free bulk retrieval if that kind of request arrives.

Outcome

Still about 81% cheaper per month than hot-tier storage at the same volume.

What broke

  • Bucket name unavailable after deletion

    Recreating a recently deleted bucket failed outright. The global namespace holds names through a deletion window.

    FixDon’t plan on reusing a bucket name; switch to a new one immediately.

  • False zero-object reading after a session timeout

    Work spanned two cloud shells, a remote host over Session Manager, and a local terminal. A session timeout silently dropped exported variables and produced an alarming, empty monitoring result.

    FixIdentifiers are hardcoded in any command whose output drives a decision, and every command block is labeled with its execution context.

  • Propagation delay on new identity bindings

    Freshly created identity bindings return not-found for tens of seconds, which looks exactly like a bad command.

    FixRe-run once before debugging.

  • No progress output during the transfer

    Redirecting the log also silenced progress output during a multi-hour transfer.

    FixThe runbook documents the expected silence, so nobody kills a healthy job.

  • Idle transfer host cost

    A running transfer instance costs about $9 a day. Stopped, it costs under 30 cents.

    FixAn explicit stop step while waiting, with a note that the root volume keeps config and binaries across the stop.

Runbook reuse

After the first organization, the runbook was cut down to a proven-path version with a gotcha table built from errors that actually happened. The second organization ran as a clean single session on it.

Stack

  • S3 (Glacier Flexible Retrieval, Block Public Access, Bucket Keys, lifecycle)
  • EC2 (IMDSv2, instance profiles)
  • Systems Manager Session Manager
  • IAM
  • KMS
  • rclone
  • AWS CLI v2
  • bash

Organizations, people, dates, and identifying environment details are omitted or generalized. Each figure carries a tag for the kind of evidence behind it.