Three Secrets Domains on Purpose

A recurring question in security engineering review: “can’t we just have one place for secrets?”

No. In a large org there are three domains, and the separation is deliberate:

1. Developers — 1Password
2. CI — GitHub secrets
3. Workloads — AWS

They have different threat models, different blast radii, and different people holding the keys. Collapsing them into one vault means the laptop that gets stolen is also the thing that can deploy to production.

1. Developers: 1Password with op:// references

A developer needs credentials on their laptop. What they must not do is paste them into a .env file that lives on disk forever, gets cat’d into a Slack thread, and eventually lands in a git commit.

Use references, not values. With direnv and the 1Password CLI, .envrc holds a pointer:

# .envrc — safe to commit
export STRIPE_API_KEY="op://Engineering/stripe-test/credential"
export DATABASE_URL="op://Engineering/dev-db/connection-string"

Then wrap the command rather than exporting into your shell:

op run --env-file .envrc -- ./myapp

The secret exists only in the process environment, for the lifetime of the process. There is nothing on disk to leak, and .envrc is reviewable in a pull request because it contains no secret material — only the name of one.

Two things follow from this that people miss:

• op run masks secrets in stdout. It’s not perfect, but it stops the obvious echo $STRIPE_API_KEY accident from reaching a CI log.
• Access is revocable per-person. Someone leaves, you remove them from the Engineering vault, done. You do not rotate the secret. Compare that to a shared .env on twelve laptops, where the only correct response to a departure is rotating everything.

The 1Password vault is for human access to non-production systems. If a developer needs a production secret, that is a separate conversation with a separate approval, and usually the answer is that they need a break-glass role, not a secret.

2. CI: GitHub secrets, and preferably not secrets at all

GitHub Actions needs credentials to build, test and deploy. The instinct is to paste an AWS access key into repository secrets. Don’t.

Use OIDC. GitHub mints a short-lived JWT, AWS trusts it, and the job assumes a role. No long-lived key exists anywhere:

permissions:
  id-token: write
  contents: read

steps:
  - uses: aws-actions/configure-aws-credentials@v5
    with:
      role-to-assume: arn:aws:iam::111122223333:role/gha-deploy
      aws-region: eu-west-2

The trust policy is where the least-privilege work actually happens. This is the common mistake:

"StringLike": {
  "token.actions.githubusercontent.com:sub": "repo:myorg/*"
}

That lets any workflow in any repo in the org assume the deploy role, including a fork’s pull request build. Scope it to the repo and the ref:

"StringEquals": {
  "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
  "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:ref:refs/heads/main"
}

Better still, scope it to a GitHub environment (repo:myorg/myrepo:environment:production) so that deployment requires a required reviewer to approve before the token is ever minted.

What legitimately stays in GitHub secrets: third-party tokens with no OIDC story. Signing keys, npm publish tokens, a vendor API key. Keep the list short and audit it — every entry is a long-lived secret that a workflow change can exfiltrate.

3. Workloads: AWS, where the app never sees a credential

Running code should not read secrets from a file or an environment variable it inherited from a deploy pipeline. It should ask AWS at runtime, using an identity AWS already assigned it.

For AWS services — S3, DynamoDB, SQS — there is no secret at all. The task role or instance profile is the credential, rotated automatically by the metadata service:

import boto3
# No keys. The role is attached to the task.
s3 = boto3.client("s3")

For things AWS doesn’t own — a Stripe key, a database password — use Secrets Manager, and read it at startup:

import boto3, json

sm = boto3.client("secretsmanager")
secret = json.loads(
    sm.get_secret_value(SecretId="prod/payments/stripe")["SecretString"]
)

The IAM policy is the interesting part, and it should name the specific secret:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:eu-west-2:111122223333:secret:prod/payments/*"
}

"Resource": "*" on GetSecretValue means a compromised payments container can read the HR database password. This is the single most common least-privilege failure I see, and it is also the easiest to fix.

For RDS, skip the password entirely — use IAM database authentication and generate a token that expires in 15 minutes.

Worked example: the Stripe keys

Abstractions are easy to nod along to, so here is the whole thing for one real credential.

It isn’t one key

The test/live split is what makes “who stores it where” answerable — they are different secrets with different blast radii, so they get different homes. Conflating them is exactly how live keys end up in a shared vault.

Note rk_live_ rather than sk_live_. Stripe restricted keys are Stripe’s own least-privilege mechanism: scope one to charges:write and it cannot issue refunds. Do that first, because it is a tighter control than anything you write in IAM.

Test mode: the team stores it in 1Password

Anyone on the payments team can do this. It is a sandbox credential — treat it as a convenience, not a crown jewel.

op item create --category "API Credential" --vault Engineering \
  --title stripe-test \
  credential="sk_test_..." \
  'notesPlain=Test mode only. Live keys are in Secrets Manager, not here.'

That note matters more than it looks. It is what stops someone six months from now from “helpfully” filing the live key next to it.

Live mode: nobody stores it in 1Password

The service owner does this — the person accountable for payments, with Stripe live-mode admin. Not SecEng, not platform, not whoever is on call. Ideally two of them together, because the value is on screen exactly once.

Infrastructure defines the container, never the value:

resource "aws_secretsmanager_secret" "stripe" {
  name       = "prod/payments/stripe"
  kms_key_id = aws_kms_key.payments.arn
}

resource "aws_secretsmanager_secret_version" "stripe" {
  secret_id     = aws_secretsmanager_secret.stripe.id
  secret_string = "PLACEHOLDER"
  lifecycle { ignore_changes = [secret_string] }
}

ignore_changes is not a style preference. Terraform state stores values in plaintext, and state lives in a bucket with a far wider reader list than the secret does. Put the key in the .tf and it is now in state, in every plan output, and in your CI logs.

The human drops the value in separately:

aws secretsmanager put-secret-value \
  --secret-id prod/payments/stripe \
  --secret-string file:///dev/stdin

Paste, Ctrl-D. Via file:// rather than as an argument, so it never lands in shell history or in ps output on a shared box.

The important part: whoever writes it cannot read it

The role that operator assumes is write-only:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DropOnly",
      "Effect": "Allow",
      "Action": [
        "secretsmanager:PutSecretValue",
        "secretsmanager:UpdateSecretVersionStage"
      ],
      "Resource": "arn:aws:secretsmanager:eu-west-2:111122223333:secret:prod/payments/stripe-*"
    },
    {
      "Sid": "NeverRead",
      "Effect": "Deny",
      "Action": "secretsmanager:GetSecretValue",
      "Resource": "*"
    }
  ]
}

Two details people get wrong here. The trailing -*: Secrets Manager appends six random characters to the ARN, so an exact-match ARN silently matches nothing. And the explicit Deny, which beats any Allow the operator picks up from some other attached policy.

Then layer KMS on top. The customer-managed key’s policy grants kms:Decrypt only to the ECS task role, so reading the secret needs GetSecretValue and a KMS grant. An over-broad IAM policy on its own is no longer enough to leak it.

CI never touches it

This is the part that surprises people: GitHub secrets do not appear in this scenario at all.

Unit tests use stripe-mock. Integration tests that genuinely need test mode assume a role via OIDC and read the test key from the dev account. The value is never pasted into repository secrets, so a malicious workflow change cannot exfiltrate what the runner was never given.

Rotation, honestly

Stripe has no public API for minting API keys — you create them in the dashboard. A fully automatic Secrets Manager rotation Lambda is therefore not achievable here, and pretending otherwise gets you a rotation config that silently fails.

What works: Stripe lets you roll a key with a grace period, so the old one keeps working for up to 24 hours. Create the new key, put-secret-value, confirm the workload picked it up, revoke the old one. Semi-manual, a few times a year, every step auditable.

Who holds what

The test key deliberately exists in two places, and that is fine: two copies of a sandbox credential handed to two different identity domains. The live key exists in exactly one place, and no human is on its read path.

Short-lived credentials

Notice the pattern in all three domains. Nothing long-lived is issued to anything:

A leaked credential with a one-hour TTL is an incident. A leaked IAM access key with no expiry is a breach that lasts until someone notices — and the industry average for noticing is measured in months.

The practical test for any secrets design: if this value appeared in a public gist right now, how long would it be useful to an attacker? If the answer is “forever”, it is the wrong shape of credential and no amount of vaulting fixes that.

Corollary: stop rotating long-lived keys on a 90-day schedule and calling it security. Rotation is a mitigation for credentials that should not exist.

Principle of least privilege, in practice

Least privilege is easy to agree with and hard to implement, because the honest version requires you to know what your application actually does. Some things that make it tractable:

Start from deny. Write the policy with no permissions, deploy to staging, read the AccessDenied errors, add exactly those actions. Painful for an afternoon, correct forever.

Use IAM Access Analyzer. It generates a policy from CloudTrail history for a role. It is not a substitute for thinking, but it’s a much better starting point than PowerUserAccess.

Separate accounts, not just separate roles. An AWS account is the strongest isolation boundary AWS offers. Production and development in the same account means an IAM mistake is a production incident.

Permission boundaries stop a developer with iam:CreateRole from creating a role more privileged than their own — which is otherwise a trivial privilege escalation path and is why handing out IAMFullAccess is effectively handing out admin.

Make the audit real. Alert on GetSecretValue calls from principals that don’t normally make them. A policy nobody checks is a document, not a control.

Why three, not one

Three domains means three separate compromises are required to get from a phished developer to production data. One unified vault means one.

The friction is the feature. When someone asks why they can’t just put the production database password in the shared vault, that friction is the control working exactly as designed.