Cross-cloud workload identity for Amazon Bedrock

Problem

The platform needed a second inference provider for capacity headroom, wired into the same chat front end. Two things made that harder than it sounds. The workload ran on Kubernetes in one cloud and had to authenticate to a managed AI service in another. And nobody wanted a static credential sitting in production to make that work.

What I built

• An IAM OIDC identity provider registered against the cluster’s issuer.
• An inference role whose trust policy requires the exact service-account subject and the STS audience. The common shortcut, a missing or wildcarded subject, lets any workload in the cluster assume the role.
• Permissions scoped to one model provider’s family, across the three regions its cross-region inference profiles route to.
• A rebuilt human identity layer: two purpose-separated managed policies and a permissions boundary, attached at group level, with self-service MFA enrollment, so the team could manage its own access within the boundary instead of filing a request for every change.
• The whole build and every verification ran from the CLI, so each step was reproducible and diffable.

Before anything went live, the IAM policy simulator had to agree with the permissions policy:

Capacity check

I pulled applied quotas from the Service Quotas API instead of quoting published defaults. They already met the team’s stated throughput target, with the top-tier model class several times over it. That answered the capacity question the same day, with no quota-increase requests.

Connectivity design

The usual starting point for cross-cloud connectivity is a private tunnel. I priced two private-network designs, including routing, DNS, and monthly run cost, and recommended in writing that federated identity over the public endpoint was the better fit. A tunnel would have been permanent infrastructure for the organization to run. The priced options stayed in the decision record, so the choice can be revisited with real numbers.

Key decision

What broke

• Alarm missing the model-ID dimension

  The service publishes invocation metrics only with a model-ID dimension. An alarm without that dimension matches no data, so it can never fire, and nothing about it looks broken at a glance. I caught it by confirming that real datapoints came back from the metrics API instead of trusting an alert-channel test.

  FixAlarms rebuilt to cover every model ID, each verified against returned datapoints before handover.

• Nonexistent IAM actions in a draft policy

  The conversational API’s calls aren’t IAM actions of their own; they authorize as the invoke actions. A policy that references a nonexistent action raises no error. It simply never matches.

  FixSimulate every policy before deploying it.

• Defects found in policy review

  Among them: privilege-management actions scoped to a wildcard resource, a trust-policy mutation on the allow side, and a permissions boundary wide enough that a repointed trust would have exposed object and secret reads.

  FixPre-deployment review of every draft policy. None of the four reached production.

• Self-service path blocked by a deny

  The self-service credential path failed because the policy denied identity creation by design, and the deny was broader than it needed to be.

  FixFixed the same day by narrowing the deny to block identity creation only when the permissions boundary isn’t attached.

• Issuer URL mismatch

  A trailing slash in the issuer URL broke the trust relationship, which requires an exact string match.

  FixFederation strings are copied from the source, never retyped.

• Model terms that conflicted with data retention

  After handover, one newer model’s terms conflicted with the data-retention requirements the platform was built on. Enabling it was one command, but it would have reversed a policy decision.

  FixEscalated to the executive sponsor and the identity lead instead of executing it.

Handover

The handover document separated what was built and verified from what was in use, and defined the check that proves the federated path is carrying traffic. It was written for the team that would run the platform next.

Stack

• IAM (OIDC provider, permissions boundaries, policy simulator)
• STS AssumeRoleWithWebIdentity
• Amazon Bedrock (cross-region inference profiles, data-retention controls)
• CloudTrail
• CloudWatch
• SNS
• AWS Budgets
• Service Quotas
• AKS workload identity
• Entra ID
• AWS CLI v2
• boto3