Service teams would get vended a set of accounts (e.g: dev, stg, prd) that contain it-security guardrails (e.g. predefined roles that can be assumed) or other resources that are commonly used within the organization. An example of this could be integration with CI/CD. Service teams would use the accounts (dev, stg, prd) to develop software, typically with a different IaC solution (e.g terraform or cdk). These tools offer better ergonomics for application development. - Olaf Conijn

Assuming three accounts {dev,stg,prd} accounts, how does a cross functional service team deploy to production in a rigorous manner?

1. Develop and commit to dev
2. Cut a dev commit to staging
3. Promote a tested staging commit to production
4. Rollback when needed

The simplest way to do it

Each {dev,stg,prd} account has an AWS Code{Commit,Build,Pipeline} that triggers a build/deployment pipeline once code has been pushed to it.

The workflow is simple:

1. git commit and git push # dev is default
2. git push stg # cut a dev commit to staging account
3. git push prd # promote the tested stg commit to production

Need to rollback? With git push $env $prev_commit, you effectively practice gitops since deployments map to your git code state.

Pros:

• Simple
• Code is the source of truth
• Dependencies tracked with a lock file or “vendored”
• Full git history with integrity
• Isolated environments
• Easy to add Approval stage for PO sign-off in the Production account
• Changes happen via Git API
• No cross role setup required, state can be managed in the account
• Minimises code duplication (no difference between environments given a commit)
• Empowers service team
• CI/CD is managed by AWS, no third party set up or trust required

Cons:

• In each Environment a build is required and that might be slow or non-reproducible in some languages
• Without some checks in place (check stg has commit) or approval stages, a git push prd could happen
• Each stack (e.g. Infra, App) typically is a separate Code{Commit,Build,Pipeline} and inevitably some manual co-ordination is needed. I.e. Infra pushed before App
• Assumes trunk based development - complex branching strategies might make the flow … complex

Using Github

Workflows are great, but to effectively deploy to your {dev,stg,prd} environments, Github needs to be externally trusted.

Pros:

• Workflows have far more mindshare than AWS’s basic buildspec.yml
• Easy to add a workflow, to enforce some organisational policy
• Managed!
• Github comes with lot of great developer experiences and tooling

Cons:

• Could dissuade teams from using trunk based development
• Third party Github effectively has a back door to your service which is a security risk

Github’s access can be limited to say the Container Registry, i.e. builds instead of deployment, nonetheless there is a supply chain issue.

Using some CI/CD tool in a dedicated account in your AWS Organisation

Also described as a One stack definition managed with a pipeline, however the problem is that “the pipeline” is often centralised unlike “The simplest way to do it” above, where the pipeline is minimal, native to each AWS account and driven exclusively via git push.

Pros:

• Focuses on build artefacts which might be easier to work with than code
• Should speed up the pipelines, since we are working with artifacts
• Should allow for almost any complex workflow / pipeline with (non-code) dependencies et al
• More guard rails, e.g. ensuring that every change has been applied to each environment (i.e. less mistakes)
• Often comes with a nice dashboard and visualisation map
• It can be simplified, for example when a shared AWS managed Docker Container Registry is used in that “dedicated account”, and each container image is marked with a commit hash

Cons:

• The fact that it can accommodate more complexity, means that the CI/CD server will be far more complex
• The CI/CD tool becomes over arching and bloated, responsible for co-ordinating the whole path with roles to deploy in their respective environment
• Some team probably has to support it
• The IAM roles and networking will be non-trivial!
• CI/CD and artefact store is typically centralised which can become a bottle neck or security target
• Source of truth might be less obvious in complex, multi-step pipelines

Summary

The top heavy, high maintenance, complex, centralised CI/CD pipelines in organisations is in stark contrast to the proposed AWS managed CI/CD aka AWS Code* in each {dev,stg,prd} account.

Other managed CI/CD management options like Github are considered, but that requires strong external trust and integration.

AWS supplies a decentralised, fault tolerant, auditable path to production coordinated by git pushes with Code*.

This path stresses reproducible builds and uses a git repository’s commit hash as the source of truth. Furthermore it empowers the service team since they can take responsibility for their Continuous Integration build rules and deployment. That said, production deployment approval and further guard rails can be introduced with a self service pattern.