IAM · VPC endpoints · Production change
Least-privilege S3 access and a gateway endpoint in production
Log-shipping agents needed a write-only path into S3, a reporting server needed a credential-free read path, and S3 traffic needed to move off an inline inspection appliance and onto the AWS backbone. All three changes landed in a live, hardened production account. The ticket came to me as an escalation, and I was the sole engineer and the client-facing owner through handover.
At a glance
How to read the tagsWhat I built
- Write-only shipper identity. The agents run on managed appliances outside AWS and authenticate with an access key, so the job was to make that key as small as possible: a programmatic-only IAM user with
PutObjecton one prefix in one bucket. No list, no read, no delete, and a 90-day rotation scheduled at handover. - Credential-free read path. The reporting server got an EC2-trust role with
GetObjecton the log prefix andListBucketscoped by a prefix condition, attached through an instance profile swapped onto the running host. - Private egress. An S3 gateway endpoint on the one route table serving the reporting subnet moved S3 traffic onto the AWS backbone.
Endpoint policy scope
I left the endpoint policy at full access for this change. Scoping it to the one bucket would have broken the SSM agent’s own S3 calls, the CloudWatch agent, and the OS package mirrors, all of which read AWS-owned buckets. Tightening it correctly means a data-perimeter policy that allowlists those buckets, and that is a design with its own review, not a line added to a configuration ticket.
The write-only scope had a consequence of its own, and I wrote it into the handover before anyone hit it. Without list permission, aws s3 sync can’t work, because sync enumerates the remote side to compute a diff. The shipper has to do blind puts. Otherwise the restriction would have looked like a broken deliverable the first time someone tested it.
Key decision
Which path should S3 traffic take?
The subnet’s default route pointed at an inline inspection appliance. A gateway endpoint adds a more specific prefix-list route, which takes that route table’s S3 traffic off the inspection path.
Conditions attached
The blast radius had to be one route table, proven rather than assumed. Taking S3 off the inspection path went into the change record as an intended outcome, so a future security review finds it documented instead of discovering it. And sibling-AZ route tables were flagged as unchanged, because rebuilding one would silently revert that subnet to the appliance path.
Sequencing
The write credential shipped first, because it was the one deliverable that unblocked the team immediately.
What broke
Route-table query that hid the main table
I’d read subnet IDs out of route-table associations to prove only one subnet was affected. A main-table association carries a Main flag instead of a subnet ID, and the projection silently drops it, so a main table returns output identical to a non-main one. Had it been the main table, every unassociated subnet in the VPC, including future ones, would have inherited the route.
FixAdded an explicit main-table query before the change.
Security-group egress for S3
The first pre-flight pass didn’t check the host’s security-group egress. Gateway-endpoint traffic goes to public S3 addresses, so the egress rules had to allow the S3 prefix list.
FixPre-flight review covers every layer that can drop the traffic, security-group egress included.
Routing test run from the wrong layer
The plan checked the routing change from inside the guest OS, which can’t see VPC routing at all. It would have passed regardless of the outcome.
FixReplaced with checks that could actually fail.
Credential test exposed to inherited session variables
Inherited session variables would have let an SSO session’s token ride along, so the test could run as the admin identity and pass.
FixClear the session environment and assert the caller identity ARN before every credential test.
No CLI on the target host
The remote command exited 127. The host ran a minimal image with no AWS CLI or SDK, which also proved the remote execution channel worked.
FixVerification redesigned on the spot: instance metadata via curl to confirm the role, and an HTTPS reachability probe, where an unauthenticated 403 from a Block Public Access bucket is the passing result. The route itself was verified in the route table.
Negative tests printed as errors
Results that were denied exactly as designed printed as error lines.
FixNegative tests print a pass when the deny happens as designed, so nobody mistakes one for a failure and rolls back a working change.
Naming problems
An inline policy named after the role it lived on, and a shared endpoint named after the project that created it.
FixPolicies follow an
s3-<bucket>-<direction>-<scope>pattern, and shared infrastructure is named for what it is and tagged as shared.
Scope control
Access requests beyond the ticket got their own change records instead of being granted inside it. Out-of-scope issues on the host and account went to the owner in writing, neither quietly fixed nor ignored.
Stack
- IAM (users, roles, instance profiles, policy simulator)
- S3 (Block Public Access, prefix-scoped policies, lifecycle)
- VPC gateway endpoints
- Route tables and prefix lists
- Security groups
- Systems Manager
- CloudShell
- AWS CLI v2
- curl
- bash
Organizations, people, dates, and identifying environment details are omitted or generalized. Each figure carries a tag for the kind of evidence behind it.