Episode 5: Trunk based infrastructure with feature flags

Well, good morning, Vincent.
>> Good morning.
>> It's been uh couple of weeks, hasn't it?
Or [snorts] something.
>> Yeah.
>> Yeah. Could be at least 10 days, I
think.
>> Yeah. I wanted to start by asking you a
couple of questions. Maybe
an info um let I mean, we're both
infrastructure engineers, so I have an
infrastructure question for you.
>> Okay. So this is a wonderful GitHub
account of this person here.
>> I was just trying to
>> so imagine there's like an AWS landscape
like this where where where the fruits
here are basically different teams,
right?
>> Yeah. So the so some teams have a dev
account, some some have a some have the
whole you know the whole shebang.
Um and actually this is not even
accurate like sometimes they can have
deployments in different regions right
could be U1
>> I should have sorry this is not the best
example but you know I was I had to be
quick about it. [snorts] Um so so each
team like potentially has
has a a number of deployments so we say
right I think that's the the correct
terminology because they could have a
deployment in USC East1 Singapore
same account
different environments and each
environment is mapped to an account.
Yeah. So the my question is is that like
I have a landscape like this in a in a
big company to manage and often enough
I'm like you know does team Mango have
stuff in this region and like I don't
know. Um but it's even worse. I mean if
I had this uh up to date Jason JSON I
probably could easily quickly answer
yeah okay they should have stuff in new
east one
>> but the trouble is that sometimes it's
it's up to the different teams to deploy
their own infrastructure so it's like
then again I don't even know if they've
actually really done deployments you
know what I mean it's like so as a
platform engineer I have to like sort of
keep an eye on what people are doing but
it's very difficult for me to keep it so
I thought to myself this example is one
I struggle with on a daily basis. Does
Tacos or your tool or I guess I mean
Terraform Cloud will probably be the
greatest tool for this but like how do I
track?
>> Yeah. So with with the grid what I I do
and I think other tools uh like Terra
Team or Digger or Spaceelift they also
have the concept of labels. So teams can
attach labels to states and you can
track if um if they are having states
for resources inside a certain region.
Now the problem could be if you are
doing multi-reional
terraform right if you could if you have
one state that creates um for example
VPC peering between two regions and then
creates the end points on one side and
then does the
>> that's a very good example that that
that's a kind of like special case I was
going to talk about next
>> but at the moment we've gone um yeah the
state file is separated per region yeah
>> which makes sense and another thing
before you start with
uh in my previous organizations we also
had the first question we ask do we
organize by team then by by by stage. So
for example in this case you have team
Apple has stages dev pro and u or do we
go by stage and then by team because um
do you first have like dev pro stage and
then within dev you have team A, team B,
team C and then within stage you have
team A. We had we had a I mean the
previous incarnation of of this cloud
deployment on the on this particular
client they had they had it whereby
there was like one dev account and all
the teams were on it
>> but we the people were stepping on each
other's toes all the time.
>> It's not necessarily like with well the
problem we were facing is from a dev sec
ops perspective the rules that apply to
non-production uh stages are different
from production. It's easier if you're
in AWS or manager to basically define
your um what's it called like the units
or the organizational units or use on
the stages level so that we could apply
security policies on the stages level.
Oh, you mean like stages as in these
environments?
>> Oh, use. Yeah, we have nested to use in
in AWS or right. So, we decided that it
would make more sense because we are
attaching our organization.
>> I never thought about that. I never
thought about that. That's a that's a
bloody good idea actually. What? I've
never done that before now. So, yeah.
So, you have a tree.
Uh how do I share that with you? where
we had a discussion between the platform
team lead and the dev sec ops team and
then how do we set up our OUS and how do
we manage our security policies in our
AWS and we had an existing setup where
we had like team A, team B, team C and
then we had dev staging and prod under
the teams and we switched it over to the
other way because we wanted to manage
our policies that way.
>> Yeah. Yeah. Yeah, that makes sense. Um
did so did AWS support you? No, not
really.
>> With AWS OUS, uh they can be like
nested, right? And you can attach
policies to the OU. So you can attach
the non-prod policies to the nonpro OUS
and then all of the teams operate within
those and then when you go to prod,
there's more guard rails and and
security.
>> So yeah, I see what you're saying. So
the policies
apply on top level top level I'm not
sure that's the right term stage.
>> Yeah.
>> And then and then that trickles down.
Yeah, that makes a lot of sense. Why
haven't I done that? I feel dumb for not
having even done that.
>> I mean because you came from an
environment where they were sharing an
account and it was causing issues. But
AWS is actually at some point I read
this book from uh like microservices
from Sam Newman I think and he was like
saying in the book how AWS even goes so
far as to give every environment an
account. So that you know like team A
prod has one AWS account and team B prod
has a separate AWS account. Uh actually
even worse like team A has two services
FU and bar and then team A FU stage goes
into one account and team uh A bar stage
goes into another account which is like
so you need to do all of the peering and
like you know you can only connect uh or
communicate across account boundaries
which is crazy from like from a small
startup's perspective to to be to have
to have all of this AWS account
management overhead right [snorts] um
Another thing is what we wanted was the
ability to
um
[clears throat] vent an AWS account for
a team for a limited amount of time
>> and then once they were done with it
like return it to a pool. So we wanted
on demand account creation.
>> You return it to a pool then you just
destroy it.
>> Well that's the thing. Originally AWS
didn't have a lot of AWS uh a lot of API
for account management. So destroying an
account was very difficult.
>> Yeah. Yeah. I see what you're saying.
>> So, so the idea is to run
>> I made a video about that.
>> Yeah. So, the idea is is to run like
nukes and and and basically have a pool
of accounts and and cycle through them.
Uh which is a lot easier and there's a
lot of tools to do that.
>> Do you so you've used AWS Nuke? I think
it changed.
>> Uh there's two of them. I think I use
the other like there's one from Terra
Grant which is very popular or um not
Terra Grant but the Grunt Works nuke uh
that has been forked. I don't I don't
think I liked
>> you.
>> Yeah, I think the fork was better and
there's the grunt work one wasn't good.
There's a cl there's a cloud nuke uh
that supports multiploud. I I remember
going through all of them and only one
of them was really wellmaintained and
and well structured.
>> This is the one that is used in my
employer if I remember correctly.
I have one in my repo as well like it's
set up in lift and shift because I have
tags which is like uh nuke don't destroy
you can put a tag so it's 100% not going
to destroy some resources so I have
>> ah I I have cloud nuke and I have AWS
nuke which one am I using
>> initial p was done with cloud nuke which
was a best practice from terror test for
cleaning up but it was really bad and so
then I used reby AWS nuke which worked a
lot
Um, and then Ericson AWS Nuke. Yes. So
Christristen AWS Nuke is a fork from the
original Dby.
>> Yeah.
>> Yeah. I'm I'm a big fan of uh nuking uh
accounts or thermal environments. I made
a video about that again.
>> Big fan.
>> Yeah. So Okay. So what I'm Okay.
Okay. You you told me about this uh
stage level. Um
>> Yeah. U organization. That's cool. I
didn't uh think about that for some
reason or wasn't exposed to that. But
but still the question remains how how
do you how do I mean if I mean this is
going to be duplicated on each stage. Um
and then and then of course I mean let's
just be a little bit explicit then then
then again this is going to be you know
AP southeast one or something crazy
u east one they're going to have
different resources in different regions
US uh EU west two or something and then
and then just to make things even extra
spicy I'm not too sure if it's worth is
it worth putting it here yeah it is
because things are split by region. Like
for example, they have a they use a
database vendor. I'm not going to tell
the name of the vendor, but the vendor
itself has has this notion of uh
different sites. So you can have like a
London site
and uh I don't know an island site
and things on their AP southeast.
>> Yeah. So So um okay, let's just put over
there. So, so
yeah, it doesn't make sense there. It
should be over there. I can't get over
there very easy. Well, let me just US
one or something. So what I'm trying to
say here is that like I I think
hopefully you see the pain is like at
the moment like do we separate the state
by you know this uh path
or do we clump the state together even
though the database is actually designed
>> uh potentially to be separate actually
>> because it's like a backup
>> and like I'm just like my head is
turning like just tracking this is
making my head spin. Never mind uh you
know uh
the other aspects of operating this plat
this platform.
Yeah. I mean I think the easy way
forward always and this is why I think
infrastructure as code and really being
able to to treat infrastructure as code
um it involves things like refactoring,
right? So I think we should just go
forward with merged like dry
>> yagny you ain't going to need it type of
approach merging everything.
>> Yeah.
>> And then and then at the point that that
it's like oh actually we're facing
issues because of contention on the
locks state locks or whatever or because
of um the terapform plan takes too long
or whatever at that point there then
refactoring needs to be uh done. So of
course we need to have like a road map
and a target And it's such a pain. It's
such a pain to refactor cuz we we have
like kind of like teams bothering us all
the time saying no you can't change this
or like you know this is going to be
disruptive.
>> Uh I mean uh I mean it's not that
disruptive but like just
>> getting uh you know some of these
account uh some of these state files are
so like high security that it's like a
pain just to move from one place to
another.
>> Okay, I can understand that. But one
thing right from experience running with
Terraform a lot is that you must be able
to rename things. So that's why I want
to decouple my logical identities from
their actual physical identities. But
then at the same time uh my original
plan with with like the library that I
wrote is that you don't get to determine
your identities right if you use Google
cloud or other providers usually you
create a resource you don't determine
the ID behind the scene it's determined
for you. So if you change a name or a
description, it's not only recreated.
Um, and that's good for a lot of
scenarios, but it doesn't work like for
S3 bucket or an RDS maybe, but even then
for an RDS, it shouldn't you shouldn't
force the identity on it because then
when you need to move things, then it
becomes disruptive. But ideally moving a
resource between two different states
where it's being managed shouldn't be
disruptive. It should just be like I'm
moving. It's an interesting little thing
that you said there because like we have
a com we have a config YAML which we
essentially give to each team because
each team essentially manages I think
what we call a data product
um they could have multiple data
products but let's just say
it's one to one even though it's not and
in that config.yaml YAML
we there's a whole bunch of
configuration as as we all know and like
for example
uh feel free to disagree with this on
but like at at this point in time we
actually allow them to set what the ids
are and these ids are you know cascade
and in in all sorts of weird and
wonderful ways
>> to name different resources and you know
like you know how like AWS likes dashes
and then
>> another thing likes underscores and
doesn't allow you to use those
characters.
>> Other things can't be longer than a 20
characters or something crazy like that
or whatever.
>> But like at the at this point in time,
we actually allow team Mango like I mean
you know if Mango wanted to be a pair
the next day we allow that but it causes
absolutely mayhem honestly.
>> Absolutely. Yeah. Uh so this is what I
mean I come from an like accounting
background right and accounting entries
when you want to slice and dice and
analyze your data you use dimensions. So
you attach in in AWS parlins this is uh
tags right labels and and and all of
this is mutable. So you can create
reports that aggregate by certain keys
and then um that's why a tagging policy
is super important and and that's why
also the same with the like the um
the example here of like how do you
organize your states? Do you organize
them by that and then by by team or vice
versa? Those should all be labeling. Uh
>> yeah, we we do tag we do tag quite
aggressively. But like but but does
tagging really even help here? Like if
if if some if team mango decided to
become team pair or something,
>> it shouldn't it it shouldn't be any
problem then, right? If if the team team
tag is is really a tag and not tied into
any resource identity, changing the
identity, sorry, changing the tag
doesn't affect the identity. So
everything that depends on it is still
pointing the original identity.
>> Oh, I think I see what you're saying. So
instead of instead of like focusing on
changing I don't know the C resource or
the stack name you change the the tag
name.
>> Yes. And then you can like always
redefine or because I worked at everyone
worked at organizations and business
development teams or organ or the a like
the the sorry the the C level decides to
reorganize and and and rename the teams
merge the teams and it causes a mayhem
on the infrastructure level because of
um logical
>> not good advice. physical identity,
sorry, physical identity should never be
um attached to u should never be
dictated by by the the organization or
the the product name. Um and I think
some clouds don't allow you that, right?
The problem is a ter provider and AWS
APIs allows you to do that and they even
have like this pre uh
>> this prefix like ID prefix thing. That's
all.
>> You're so right, Vincent. Jesus, I wish
I was working with you. I'm learning a
lot just just to get chatting with you
every hour.
>> The problem that you're going to get
weeks
>> I tried to introduce this at a company
and I got a massive um push back from
engineers because they are used to use
to do click ops in the AWS console and
they're like I can't find my resources.
It was cloud for stacks and the way the
cloud form like um calculates the
physical ID name was very confusing
because of the nested stack way of doing
it that causes its own set of problems
because then you lose resources just
because you rename a stack now the
resources logical identity no longer map
and it just loses track of it. So they
they are adding more features to like
manage refactoring of resources which
Terraform already does for you. Uh but
then the push back was from the
engineers that are used to go into the
AWS console and I can't find my specific
radius instance or my specific load
balancer because the name is so and I'm
like you the problem is the dashboard if
you don't have a proper dashboard that
shows you the tags and shows you to like
slice and dice your resources by the
tags
>> to data dog's credit data dog actually
does a good job of servicing all the
tags
>> and that's what it should be right
>> console doesn't like I swear to god I
was dealing I can't remember what it was
but I was dealing with something the
other day in in AWS and it didn't have
bloody tags or it didn't show the tags.
>> Yeah, tags are not everywhere in AWS. Um
and and also not everywhere cuz there's
a convention a lot of resources if you
put a tag with the key name it shows up
in the UI as the actual resource name uh
nicely like for example on EC2 instance
you cannot control the instance ID but
you put on a tag with a name and then
the dash the the the dash
>> the web stores it nicely. Yeah,
>> but some of them don't like the RDS
because the RDS and the end point URL
for the RDS as well.
>> These are teams because every team works
independently, right?
>> Yeah. There should be like, you know, a
bezos whatever mandate, make sure that
everything's bloody tag or tagable.
>> Yeah. So I think they're doing a lot of
work there with like the resource
manager uh that the t you have now the
ability to like review all of your
resources across your across your AWS
orc uh using the resource manager and
then using tags. So there are separate
services focused just on that on like
managing resources and finding your
resources.
>> Cool. Cool. Okay. So
>> I really like your tips here, but but
still the question remains given a
platform like this with lots of uh teams
and things like this. I is is the is the
grid the right tool to make to to keep
an eye on all all this stuff or what
would you suggest?
Because I need to know what's going on
here sometimes. Otherwise,
>> when people ask me, "Has Gene Mango got
the thing in London?" I'll be like, "Um,
>> I think you should research to system."
I can't speak. I think you should re
reach out to system initiative because
he's often posting about like um they
have a programmable graph that [snorts]
allows people to manage their
>> system initiative. I
>> so Adam and uh his colleagues uh demoed
it to to to us
>> works.
>> Yeah. And uh to be honest, I I was a
complete [ __ ] but you know, you got
to give me credit because I
>> as usual. [laughter]
>> So, so he Yeah. So, system initiative,
they demoed this like, you know, like
this big square. I don't know why that's
like that.
>> And then you have your your VPCs and
things like this. Like, first off, I was
like, ah man,
>> I hate it when you you start
infrastructure from a networking
perspective, but I get it. But I I hate
it like
like you know like if you want Yeah. VPC
this VPC that
because I just I just hate it. I hate
like a a focus on you know network based
infrastructure. I prefer to think of
things as I don't know like this level
and then and then it goes on like
whatever to um eventually you know EC2
and things like this. Um but then the
thing that really got me was like okay
so you you're creating resources la
>> and then I was like how do you like
maintain the state and all that stuff?
He says oh this is just this is what it
is right now.
>> It was like a onetoone mapping and then
I go like oh so how do you do that? How
do you like connect to the terraform
provider? He says oh we just run AWS CLI
CLI commands under the
>> Yeah. Yeah. And I was like
>> this is crazy. No, no, this cannot work.
>> I I initially wanted to contribution
of this year at the beginning of the
year was to like find an open source
project and contribute to it. And I was
like, okay, SI is it. I'm going to look
into it. And the moment they showed me
like linked me to the TypeScript under
the hood, which is literally just
calling the AWS CLI, I was like, there's
so much autogenerated APIs like the
telephone provider autogenerated. Um why
would you like go down to just scripting
around the CLI like literally TypeScript
TypeScript OSX um the thing is that's I
think is an implementation detail
because the idea of every
um component there being uh it's a a
programmable graph right the point is
that there are simple um imperative type
of scripts I think it works better for
people that are used to simple that are
like like to write uh a script in YAML.
Um basically every arrow, every
component has like um their own create,
read, update hooks and those invoke the
scripts and they build a whole bunch of
components with all of their CRUD hooks.
For example, a VPC that has the create
hook, the read hook, the update hook
that directly maps to the AWS CLI
commands. And that's basically SI built
the graph with the hooks that are
invocable like functions. And then they
built the whole bunch of like components
that are using that graph. So then when
they create the VPC, it goes and runs
that AWS CLI command, that AWS CLI
command when you want to update it. And
I mean I I think it comes back to the
the question like why doesn't Terraform
do that also, right? Because Terapform
is just talking to the AWS API. It could
just go and find all of the resources
that supposedly exists based on tags
because that's what um Hashimoto
originally did for Terraform. uh they
were just creating resources and using
tags on the resources to find them.
>> It's funny you mentioned it. I was
randomly watching a video and he talks
about Terraform as a graph API and like
well okay but where's your
representation of the graph?
So, so there's this interesting
Terraform issue where uh they ask
Hashimoto like why didn't you just um
you know attack like why do you need
state like get rid of the state file and
he highlighted all of the issues that
you get if you don't have a state file
if you don't um I think one of them was
speed um there was others because the
physical identity changes if the tag
changes um so they tried it and they
said and it didn't really work so um
basically you're saying that Hashimoto
did try something what system initiative
did but it was just not going to work
>> a long time ago right and I think things
changed a lot so I don't know um I think
there's some interesting ideas uh but I
had the same reaction when I looked at
the system initiative when I really
lifted up the the whole
>> feel for dismissing the whole idea
because I just thought like it wasn't
going to work just instinctively but but
now yeah now I mean I started this
conversation saying that I need I need
to see a visualization and then we just
go back to system initiative.
[snorts] Uh yeah, I mean I guess they
they could convince us in the end. I
think this is one of the main what I
learned from building my library is that
the framework in itself maybe is
approaching the problem from a certain
way but the actual utility comes from
the library of components that comes
with it because like you can interact
with the cloud however you want like you
want to use a state file or you want to
use a kubernetes operator or you want to
build pulley um you know typescript to
every possible language sure go ahead do
all of that the problem is if you cannot
give me a nice library of components. If
you like when I looked at crossplane,
there was no proper um you know
available publicly available set of
crossplane um resources that I could use
with AWS. I would need to build
everything out of the box. Uh and the
only framework that really has this is
AWS CDK, right? Because it comes out of
the box not just with the low-level AWS
API interface but also with the
integration patterns of how you use
these services together. And then the
other framework that does that is SSI
which is uh sorry SST not SSI what the
hell uh SST which is basically using
Palumi those basic components and then
building highle system integration
patterns on top of it
>> just to service integration
>> just to repeat what my understanding
what you just said you just like you got
to use the tool that has the best
coverage of the of the resources that
you're trying to use.
>> Yeah. And the services uh the
integrations have um integration
patterns between services because if you
use AWS you must know every single
service integrated with IM kind of
practical look at it.
>> Yeah and and and this is what what
system initiative I think spent 5 years
doing they started they had to spend
after they have the basic framework they
have to spend all this time to building
all of the components for AWS writing
every single AWS CLI script for every
single hook that a component um needs to
you know invoke whenever couldn't have
gone down that path of uh what did you
show me that was autogenerated? Oh my I
forgot the name of it.
>> There there is AWSC uh like cloud
control provider that's autogenerated
from the AWS API but again on its own
that doesn't really help because
creating a single API resource does not
give you the functionality that you need
out of the AWS API because the AWS API
effectively is a combination of many
resources working together in a certain
configured way. But the but system
initiative since they since they
rely on AWS CLI the AWS CLI surely is
their interface and that's there.
>> Sure. But the AWS CL CLI is also just
autogenerated wrapper around most of the
AWS APIs, right? It doesn't give you
like the idea of AWS CDK. The example
that I like to give is I have an EC2
instance and I want it to give it
permission to an IM uh to an S3 bucket.
I want I have an EC2 instance and I want
to give it access to an S3 bucket.
Right? So now look at how many resources
you need to create. Right?
>> Maybe you say, "Oh, but that's a very
common pattern in my organization. So I
create a terapform module that combines
the two and then later down the line you
have a situation where the S3 bucket
actually needs to trigger a lambda
whenever the the S3 instance does
something or whatever or needs to be
hooked up to an SNS and you keep
extending that original very simple
basic pattern until it becomes a massive
like module with you know 40 different
parameters to configure whatever use
cases you now have after two years of of
using AWS and evolving your product. Um
I think the right way is what what they
are doing with AWS CDK is the actual
service teams and the community building
proper um patterns around these things
such as you know with abstractions such
as I have an I principle I have an uh
resource and I can grant a principle
access to a resource and if the I
principle is an AWS instance and it
needs to implement this this this
interface that means it must have an IM
role that I can refer to. So who can
build up under the
>> hood idea of encapsulated u I guess my
favorite one in AWS is the the
serverless SAM stuff.
>> Yeah.
>> Here's an example that got my head
spinning the other day. Um
so a colleague was telling me that when
you use ECS it creates like a bucket
behind the scenes. And the trouble is in
our in in the organization that I'm what
I'm working with, they're so strict
about all their their bucket policies
and things like that. Like for example,
I think you have to have a bucket policy
with some deny on it or something like
that. Like just off the bat when you
create a but the trouble is that ECS
creates a bucket and it doesn't it
relies on that bucket not having the
policy for it to work. So, so basically
this colleague of mine was telling me
that they were using ECS and it wasn't
working and they couldn't figure out
what the hell the problem [laughter] was
because like if you look around ECS
documentation, you won't you won't see
that you know ECS requires this bucket
to work.
>> It's kind of hidden away.
>> Yeah. And then so so basically that's to
show I mean that's that's me trying to
give come up with a dumb counter example
to these abstractions because then you
just don't know what the hell's going on
sometimes.
>> Um especially if they use
>> so here the extraction is the user
interface that AWS creates and when you
create the button under the hood it
calls many different APIs to create the
bucket and whatever. And when you then
need to capture that into infrastructure
as code now suddenly you have to go and
figure out bucket policy get cascades
down into that bucket you're screwed.
You know what I mean?
>> Okay. Yeah. What you're saying is
>> a stupid example. It shouldn't really
happen, but it it happened.
>> No, but like these things happen all the
time. Like I am I'm very annoyed when
people click ops things into a
non-production envir uh sorry into a
production environment because there's a
time limit. There's a time constraint
and they didn't have time to do to do
the ISC. they clicked it in on in dev
then they clicked it in in staging and
now there's a deadline and they need to
do it um because the IC doesn't work for
example
>> when you when you click something in
>> it will maybe enable a service uh uh
linked role into the AWS account
automatically so like you you start
using an AWS service if you use user
>> interface it does a lot of things and
you don't even know what it is it's so
hard to capture
>> and that's what I like about AWS CDK
because it actually capt captures all of
those actions. It's like a user
interface but a programmatic one. So
it's built to cap like it's almost the
same. I mean honestly when you use AWS
CDK and you have a bucket and you have a
lambda and you need to connect it to
under the hood it's going to make sure
that there's a resource policy that
there's an IM lambda permission whatever
if the bucket needs
>> are you confident that like the the the
AWS console and the CDK are are
synologist or onetoone because I'm I'm
always a bit doubtful that they are
>> very doubtful
>> I am very confident that
>> 90% of the time they are because I
believe AWS uses CDK internally as well
to build their products. I believe that
the teams that build the API interfaces
of those services actually contribute to
AWS CDK to capture these things.
>> And
>> so did you say 80% because like yeah
that's a lot better than say Terraform
where I'm probably like 10% you know.
>> Oh absolutely. Terraform is a nightmare.
Um and and and having an autogenerated
Terraform provider is not going to solve
the problem again because the the
low-level API resources are not the
solution. It's a combination of them.
And and then the comment I get a lot on
Reddit is like, "Yeah, but if you're
using a programming language to do
infrastructure, you're doing it wrong."
And the reality is again
the the the cloud is not as basic as it
used to be. It's not just about creating
network subnets and and EC2 instances
and allocating network interfaces. uh
and that used to be probably quite
simple but now you have like IBV6 then
you have like your flow logs then you
have uh a flow log bucket and then you
have all kinds of uh integration
patterns across different services and I
mean case in point
>> so sorry can you just rephrase that like
who said like if you if you're using
infrastructure you're doing it wrong
what
>> no if you're using programming languages
like full
>> oh right
>> like full uh what's it called to
incomplete programming languages to to
write infrastructure as code you're
doing it That's a very common um
argument against using general purpose
programming language. The the most
common argument is um we just need a
config file. We just need to do a
declarative and we need to keep it
simple and it will be fine.
>> Y yeah
to be honest I'm usually in that camp
but uh but it bites all the time.
>> Well you you can be in that camp until
you do something actual real work and
then you realize that that that's a very
naive approach. And then every time I I
I I come with that argument, people say,
"Oh, I worked at like hundred different
country large organization and we have
no problem." I don't believe it. I don't
believe you have a problem or I don't
believe you work at such an
organization. If you think that it's so
simple to just use terapform
configuration.
>> Yeah, I'm just trying to capture what
we're saying in a nice graphic
in a nice slide. Uh
>> so there's this these efforts from the
community to use basic terapform modules
to capture all of these usage patterns
and very quickly those modules become
just I mean what I was going to say
>> one
>> yeah what what I was going to say is
case in point um the argument that the
terraform sorry the argument that the
cloud is simple and we just need to use
basic declarative configuration just go
look at the of the VPC module just go
look at the VPC module look at how many
um variables there are input variables
because you can do um IMAP uh sorry
what's it called when you have IP
manager um IP man no the the basically
managed IP EPOM yeah IP uh yeah that
sounds right so you can do you can set
up a cider to control the you know cider
subet allocations
um you can do your flow locks you can so
the amount of of of things that you can
do is so big that if you look at that
module
>> uh alone you will notice that there's
like you know I don't know so many
different Torah form configuration files
each one has a locals block that has
multiple indirection layers on top of
the uh putting Julio variable that needs
to be set to true before the whole
module can be used u but then you have
so much in there that
you know you can't tell me that you just
need a very basic and then and then the
um the response is usually oh no problem
I just take that beautiful module that
has all of these things because people
contribute epam I need IPv6 support. I
need net instance support because I
don't want to pay through my nose for
net instances. Uh sorry, I don't want to
pay for net gateways. I just want to
spin up an instance and and save 20% um
sorry and save 80% of the cost of of
net. Um so I need all of these features.
So I just take that module, I remove
what I don't want and I keep it simple.
But then you have to maintain your own
version of that module. And as things
are being introduced, you need to like
merge upstream down into your stuff. And
basically everyone is just forking um
shared resources and then keeping their
own private copy of it and then how many
people actually contribute back to it.
Yeah,
>> I think that's that's just a
>> in my current client we have so many
different constructs and uh
>> they're using AWS CDK, right? and they
are trying
>> well we're mix now we're using a mix
between CDK and ter but historically
they were using CDK
>> yeah it works but yeah it's but but we
have that problem as you describe so
there's nothing that to help me
visualize this Vincent [ __ ]
>> well does
>> you have access to the grid
>> um but
uh
>> so what the grid does is right now I
mean the grid But can the grid can can
the grid I remember the grid taking in
terraform state but can it can can it
also take in CDK state because this is a
mix between CDK and Terraform right now.
This is this is another thing that makes
my life extra exciting.
Uh um right now the grid
>> new requirements new requirements.
>> So the grid is you can look at the grid
as like two two way you have one is the
control plane and two is a data plane.
The control plane is the is the way of
creating states and metadata around the
states such as the labels such as the
team or the the stage or the
>> why do you why do you say labels is just
labels and
>> tax is a very AWS specific terminology
>> okay carry on
>> labels is closer to K cubernetes way of
doing things like in in Kubernetes you
have labels and annotations I just um
have the key value pair
>> what's annotation again then
>> annotations you can apply to resources
then I think there's two different um
programmatic parts. One of them is meant
to be like the label queries are meant
to be much faster to query across the
state and the annotation is less
queryable. I think that's why they
separate the two.
>> Wow. An annotation is um
>> some information that you can like the
annotations you were using for like on
an ingress manifest. You may have an
annotation that the controller is
enginex ingress for example and another
ingress manifest might be controlled by
another um um ingress controller. So
that goes into annotations but the
labels is like um you want to do label
queries against the resources to find
them. I know that when it when they were
asked about like in Kubernetes why do
you separate labels and annotations and
the response was one is much more f like
performs much better to query.
>> Yeah because you know everything in
Kubernetes is uses label queries to to
select resources. Right.
>> Right.
>> So you don't you don't have Okay. So
there's no CDK. Okay. There's there's
nothing that that will help.
>> No. Right. Now the the grid is purely
for Terraform state. The problem with
CDK is it's cloud form, right? Cloud
form manages the stacks.
>> Couldn't you add a source for cloud for
>> I think I think if you look at spacelift
or um mass driver, they may they support
both. I think the way that you describe
your um your your your your employer's
um what do you call it infrastructure is
that you you had you had CDK and then
you and then and then you oh and then
you get it and then you and then you
converted to to Terraform. Oh [ __ ] You
don't use you don't use ter you don't
use CDK at all do you? [ __ ] I now
remember your your setup.
>> You translate everything.
>> I worked at different companies.
>> Okay. Okay. So everything's terraform
basically.
>> I'll tell you that my my issue was um
before we have cloud formation on one
team and terraform on another team and
as a platform team for example we like
to create our landing zone using
terraform. So we have our networks or
clusters that maybe we have a shared
Kubernetes cluster that both teams need
to land into. That's your landing zone.
So what we need to be able to do is
expose those integration points, right?
So team B needs to be able to read
where's my cluster ID and how do I or
how do I hook up my product repository
so that the Argo CD deployments going
into the right cluster.
>> So you do that so sorry if I I didn't
quite catch the beginning. So you do
that in terraform the the landing zone.
So what I'm I I'm just giving you a
context of like different teams use
using different technologies and the
problem I had was I tried to create an
integration registry and we talked about
this because he says at um at your
organization they use a massive JSON
file that keeps track of the integration
points between different um components.
So everything registers just writes
information out to a JSON file. And I
did similar things like I had AWS CDK
write to SSM parameter store and I had
Terafform read from SSM parameter store
and I had terapform right to SSM
parameter store and AWS CDK reading from
parameter store.
>> Yeah. I prefer the central as I mean
this this in reality has lots of other
fields. Yeah.
>> Yeah. So so the problem with with using
a JSON file you're restricted to a um
usually a a git repository, right?
either you go to a massive monor repo or
uh you do a subm module across all of
the repository so that they have that
JSON file or I don't know how you're
going to get the file in there and
version control it
>> I I kind of re I shouldn't say this on
online but I kind of rewrote something
that we used at work I I mean I didn't
uh steal the code I absolutely wrote it
from scratch using
uh using um
I guess a bit of claude magic So, so
essentially what happens is that we have
an act a reusable action which which
just feeds in the account group or the
the team name
>> and the environment and then out pops
uh
uh things like oh sorry this is me using
the action um
sorry the other repo actually had the
implementation
there's there's an action that basically
just out uh looks up the style the stuff
in the actions JSON.
It's it's nice and fast. It doesn't do
like a full checkout or something weird.
And uh and then just sets the extra
information that that that the um that
the Terraform or whatever your
infrastructure needs like the account,
the region, a key for this, a key for
that.
>> Yeah. The problem there is like you're
you're using a registry and then you
don't know like how stale is that value
and who is consuming the value. That was
my problem. Um [snorts]
>> well I mean if it's centralized the
source of truth is pretty sweet. I mean
>> sure but if it changes how do you
trickle down that change or identify who
needs to be updated? Yeah, this is this
is why I'm going back to this all the
time because I need to know
>> what [snorts] how fresh they are here.
>> Yeah, never mind what's where. I need to
know what version they're running.
>> Um
>> or what version they have viewed or
consumed.
>> Yeah. Like we we in our repos we tag the
this for example and then we kind of
allow them to
tag their deployments or what do you
call it you know use a branch or
whatever to to pin sorry pin was the
word I was looking for but then yeah
they've pinned you know version one here
and like now we're on version five like
how do
>> exactly so so this is also something I
learned about um this version pinning
scenario is that if you start to deal
with these type of deeply nested
structures where you have 40 different
states.
>> Just hold that thought. I got to reply
to my wife.
So the kids need need to go to school. I
think my wife will take them. So yeah.
Yeah. Hold the thought resume.
>> Yeah. So what we were saying is that
version pinning is important,
>> right? and and when you start I started
my career around DevOps with Kubernetes.
So I learned very early that immutable
infrastructure your docker image needs
to be hard you know hard pinned so that
you can do rolling updates because
Kubernetes doesn't allow you to use a
mutable um label. So version pinning is
very important. You want to pin your
docker image. Kubernetes doesn't allow
like uh at latest or whatever the docker
>> condition or officially the way that
Kubernetes is designed is to use
immutable tags and then the way that it
detects that that um a deployment needs
to be rolled is when you change
something right you change the tag and
so it spins up a separate replica
controller starts
>> I didn't realize it enforced it yeah I
know best practice is to use a is to use
>> yeah and there's a lot of workarounds
like keep or ke sorry to basically um
track um you know a docker hub image
repository and identify if a new um
metadata if a new checksum has been
pushed for a certain latest tag and then
automatically trigger a rolling update
but then you have to work around it by
by changing the labels and then it will
start rolling out pods with new labels
those are all workarounds the original
design of Kubernetes is very simple your
image tag is what determines the version
that runs so you cannot use a mutable
tag. You have to specify which version
you want to run and if you want to
update it, you change Kubernetes
manifests. What what helps you update
them? Sorry. Like how do you what's the
workflow for people like do they get
like a dependabot like oh you need to to
bump.
>> Yeah. So we're we're going back to the
point that the design when you start
with Kubernetes and Docker and immutable
infrastructure is that version pinning
is very important, right? And and as you
just highlighted also when you get a lot
of different states and a new version
has been published but um you have all
of these um you know very niche um
little states that haven't been touched
for a long time and they're still on
version one right and now you and then
for example you have a big um
organizationwide initiative that you
need to upgrade the provider and then
you find out that some really deep uh
nested areas are still using a very old
version and now they suddenly have to go
all the way to production and they're
like
the answer. I [laughter] need to know
the answer. How you keep
>> So, so funny funny enough. Funny enough,
uh, one of my previous colleagues, he
just got rid of all of the versioning.
He trunk based just
>> Well, this is this is what I like to do
in a in a sense. Just tell everyone you
got to use latest.
>> See you.
>> Yeah. Latest or at stable or something
or there's probably some other type,
>> right? And but if you're in a monor repo
and and you're you're running off git,
you just point to head, right? So so
when you run your IC for example and it
checks out the repository and it it gets
head and so it gets all the latest
version of your modules together with
head of your um because like imagine
that you have a infrastructure mono and
you have a GitHub action or Atlantis
running to to do your IC terapform plan
apply. So the first thing it does on
your branch or of master it will check
out the repo and it will take head right
and then instead of version pinning
every single module which is the
recommended right way to do instead of
version pinning any every module you
point everything relatively within the
repo to your current like head.
>> Got it.
>> Right. Which is trunk based development.
Right.
>> Right. And at that point that means
every single change you make to a module
automatically gets procreated across
your whole infrastructure.
>> Yeah, that means but that means you have
to deploy in a in a with some rigor some
schedule, right?
>> No, you you have to use feature flags
and you have to release features like
for example if you're going to so this
is what we do at my organization right
now. We we have gotten rid of all the
modules.
So trunk based trunkbased infrastructure
with feature facts.
>> Absolutely.
I did a a presentation about that other
conference.
>> This is something that Thoughtworks
preaches for software development, but
for some reason I never thought about
this in the context of infrastructure.
>> Yeah. So it's it's it's the same thing.
The first thing you have to do is you
point all of you get rid of all the
module versions. you point to the
relative path or you make sure that
you're always checking out the latest
version of the modules, right? And then
when you run your Terraform plan, of
course, you have to control how those
features get rolled out, right? So the
best way to control how features get
rolled out across your infrastructure is
by putting in feature flags, right?
>> So we're adding a new, for example,
we're adding a new we're doing a
database migration. I'm changing my
module to to provision the new database
as well as keep the old database
instance running and it's behind a
feature flag. So when this gets applied
into dev, it's going to spin up a new
database. It's still going to have the
old database there and we're going to
start pointing our customers um to the
new one, our client to the new one. If
we're doing this with a cut over, uh we
obviously need
>> How do you do this with Terraform? But
carry on.
[clears throat]
>> It's very simple on your module. You
just put a variable that that um sets
migration enabled true and in production
is disabled and then you have another
variable that says uh delete the old
database and then you you you toggle the
the flags as you are getting confident
you have the same code running in depth
you have tested it then you're going
into staging you've tested the cut over
everything then you ready go to prod
you're doing the first flag on the new
database gets created you re reconfigure
your clients you turn on the second flag
the old database gets deleted your cut
over is done and then you go back to
your module, you delete folds variables
and across your real estate everything
is replanned with the latest version of
that module and it works because you've
tested it everywhere, right? Your your
module is immediately planned everywhere
and it shouldn't show you any diff when
you remove them. So, it's feature flag
development, you know, with u
>> I've never heard of this in all honesty.
I thought I've never seen
>> I do that with Atlantis only. Um um at
>> does Atlantis have some some sort of
support for feature flags.
>> No, but I basically just do an enable
auto plan and I automatically I have
Atlantis project detection integrated
within my repo bootstrapper. So when I
set up my repository, it automatically
generates the configuration for every
single directory for Atlantis to figure
out whenever this module changes, you
need to replan all of the states that
depend on that module. And so whenever
anyone makes a change to a module, it's
immediately tested everywhere. Like if
it's going to have an impact on pro,
that PR is immediately blocked because
you haven't put in a feature flag on the
variable. So you need to set up a
variable to make sure that it's by
default acting like originally and then
you go and update the call sites where
you want this feature to be deployed and
then you set now enable it here and then
you test it in dev. It's okay. Then we
can go um and and and push this module.
We can get the PR merged because then we
do multiple PRs to roll it across our
environments and that way we don't have
like some really old state using a
really old module. We always have
everything on head.
>> That's the goal.
>> Yeah.
>> Yeah. Yeah, I like that a lot. Wow.
>> Wow.
>> And so
>> the requirement is to have the
capability to define what is the
dependency or which modules a state
depends on. Is it if you don't do that
right, you're back to your original
situation where everything is version
pint and then now you could discover
that one particular team is still using
version 1.0 and if you're doing sambar,
right, you're on version 2.0 means that
you have a breaking change, right? So
they have to
don't I mean or
>> or uses it as an excuse to break an API
which is
>> yeah but I'm I'm so I'm doing that
turnbased development on my IC with
feature flex but I am still every single
time doing sar on my modules because I'm
snapshotting the module version in case
anything happens I can always pin it so
this is something that happens sometimes
is this module this change is
significant and it's we can't just put
it behind a feature flag because that's
the the the the flip side of feature
flags, right? You cannot control the
interaction of the flags if you have so
many flags on them. So, at some point um
the risk is that maybe one team is
adding a variable for their use case and
other teams another variable and how do
those interact with each other? You
don't know how they interact with each
other. So, it's nice to say like, oh, we
know all of the possible configurations
of the feature flags together. We know
that it's going to work. we don't if we
don't clean up feature frags and if we
if we are not careful we we have other
complexities to deal with. Um
>> yeah so it's not it's not there's no
silver bullet of course I think I know
what you mean there like [laughter]
clean yeah like then you have you know
new problem uh
>> so so every single module change comes
with a server like change log and an
actual snapshot of that module pushed
into an S3 bucket. So if we need for
whatever reason we can pin a module but
it's highly discouraged if you do like
>> if you do go ahead and pin it then I set
up um a scheduled action that that
detects any states that have a pinned
module and then it goes and uh tries to
unpin it and creates a PR to unpin it
and then the idea is that we need to get
those PRs merged as soon as possible to
basically unpin the modules otherwise
these particular states will start to
back behind and become more of a
headache and lead you down yak shaving
paths down the line. Right. [snorts] Um,
>> cool, man. Thanks. Thanks for all that
wisdom, man.
>> But that all works if you're in a monor
repo, right? Um
>> and and and and then the other
alternative actually to just this week
config hub it was announced like last
week but this week they they did an IT
next article where they go into details
about the problem of YAML manifest and
Argo CD. They're basically highlighting
how a lot of people are abusing Argo CCD
uh with Helm to you know in memory on
the Kubernetes cluster um expand all of
the YAML manifests and then fire it off
at the Kubernetes cluster and then they
have no idea what's happening. Um so
they are they're introducing the concept
of um infrastructure as data. No
configuration as data. C A yeah it's CA
A and
I really still don't know exactly how
their platform but their platform.
Yes. Uh oh there's another one Napio but
there's one from
today. It's definitely it's Brian Grant
startup. There's a one that was posted
like like maybe 12 hours ago from config
hub where they go into the problems and
the solution of configuration as data.
Um and in in the problems they are
highlighting like the ren the renovate
bot approach. So if you version pin your
modules or version pin your helm charts
then you set up something like renovate
to automatically create pull requests to
update your things.
>> Yeah. I mean actually at my workplace
they've written a beautiful like 100
line uh Python script and what it does
it's it's called distribute py and you
you can have a config to say like you
want these paths pushed to these repos
or even I think even it supports like
hierarchy. So, so basically like if if
they want to make sure that every team
in dev has this new config set, they
push it out or a new file in their in
their repos, they just push it out
>> so that everyone has a new workflow that
they they they can use or the or their
workflow in their in their repos.
Actually, this is what this thing is
missing repositories.
>> Yeah. All of that mapped across repos.
But like uh I guess the repro would sit
here but like yeah it it pushes it out
and they do this and they said like I
said like wow I've never seen this
before. So you didn't want to get the
pendot like a like a pullbase flow going
like he says no pullbase will not work
in our organization. We need a push we
need to be fast and it needs to be
updated like immediately
>> and I was like wow I've never seen that
before.
>> When they say push do they do do they go
and create pull requests in the other on
the other ones? Direct commit.
>> Direct commit. Yeah, they have like a
bot and it just like just overwrites
whatever they have.
>> So, this is my biggest issue with all of
these like cookie cutter and
bootstrappers. Most of them they do a
onetime bootstrap and they never
consider the the day to
I created an issue about this in GitHub.
Um, let me see if I can
>> Oh, forget about GitHub template repos.
I hate them. Every time I see somebody
use them, I I just immediately want to
shut that down. It's like don't don't
>> um yeah like here's my question here. Hi
there. How do you should a repository be
generated from a template keep up to
date with an evolving template?
>> Yeah,
>> it says GitHub templates are designed
for onetime scaffolding.
Yeah. So
>> I created a worm in my previous
organization.
>> So what do you mean to update stuff?
>> Yeah. So whenever I created a PR on my
on my template repo, it would
automatically create commits in my other
repositories to pull it in to the to
pull in the diff from that PR. Uh except
that it would copy to itself and then
trigger itself. And I had to like
explicitly allow uh GitHub action
workflows to trigger themsel. And then I
I I created like 70 plus emails per
minute across the whole email system.
And then my C after like half a day my
my CTO sent me a message like what's
happening? [laughter]
>> What what do you mean you're sending
emails to different people?
>> Yeah.
>> No, because GitHub action run started
like failing and sending emails like the
the run failed and so I was literally
spend three hours killing killing action
runs.
>> Uh [laughter]
it was a lot of fun. So I I had this
idea as well like originally with with
template repos maybe like five years ago
to to automate this push down to all of
the repos that all.
>> So I I I mean in all in all honesty at
this particular client I think the push
mechanism works quite well because you
know the relationship they have with the
teams like
>> you know like these are data teams they
don't know what they're doing. They need
this thing up to date now.
>> And then the other thing is the moment I
started using progen projen.io IO which
is popular within the AWS CDK ecosystem.
Um, Proen.io
uh is kind of like the Pandaot. When you
bootstrap a repository with with this
tool, it sets up an update uh scheduled
update uh system. So you can define like
once a week, daily, and you can also set
up automerge and all that. and and and
so what it does is it it runs the it it
downloads the latest version of the
template and it runs it and it basically
mod like selfmutates and and keeps the
repo up to date and then automerges. So
it's pool based but it happens
automatically
>> but like but
I I'm not too sure how we got here. We
were talking about uh configuration
>> unpinning and pushing.
>> I'm actually not sure dependabot is a
very scalable solution. Um well it
>> uh so one thing I don't like about
dependabot is that it creates so many
PRs and and that's why I usually
disabled it but you can actually set up
grouping.
>> Yeah I know there was one repository
where I think I set up auto like
>> auto actually how did I do that again?
Uh oh,
>> you have to create a a GitHub token for
an automerge uh bot to to
>> I'm not too sure how I can find my own
automerge. What What should I search
for? H PR
>> I use dependabots with um release please
and automerge on my Golang CLI for
Terraform. It's a public repo. You want
to see it?
>> Um okay. If you Well, I mean I've done
it before. I just can't remember where I
done it.
>> Yeah.
>> Okay. The problem um with dependabot is
originally my biggest issue was all of
the different PRs but you can set up
grouping rules. You can say like for
this buckets manager you group all the
PRs um instead of creating like five PRs
you create one for go dependencies for
example and and then you set up the
>> the question remains like you can't
expect all these different teams to
basically
>> merge the PRs and that's where things
really That's why I don't think that
what you mentioned earlier about the
push flow is automatically committing to
master. I think what it's doing is it's
creating a branch. It's creating a PR.
It's it ts it with a label and then it
said it's pushing it straight to main.
They usually updating workflows. They
only usually do it for updating
workflows.
>> I think it's either it's pull or push.
It doesn't really matter much. Um, but I
have the same flow which it just creates
u a PR that the PR is labeled as
automerge and then you have an automerge
bot that will auto approve actually
sorry the PR is labeled as auto approve
and then that bot will auto approve
those PRs and then it will enable the
automerge as soon as the checks pass. So
I will never merge or commit straight to
master.
>> That's quite a complex setup but I guess
it's
>> I mean let me share my screen and I'll
show you the repository. It is a public
report.
Uh
oh, not what I
>> is requesting you to allow multiple
presenters to share. Okay, whatever.
>> Okay, I'm sharing.
Which one am I sharing? This one. I
guess
>> I think we got to wind up in a couple of
minutes just because I've got I got so
much work today. Oh my god.
>> And we never talked about we never
talked about AI. Oh god.
>> [snorts]
>> So here I have this Golang binary. It's
a fork from
>> Are you sharing? I can't see it.
>> Oh, sorry. You can see it now.
>> Yeah, I can see it now.
>> Maybe let me drag it out of all these
tabs so it's a bit less confusing. So um
I forked this from the Chan Zuckerberg
Foundation, the wife of um some Facebook
dude. I don't know.
>> Yeah, I don't know.
>> And they create they have a um
>> Oh, do do quick sign up. Do you know um
Swix? Uh what's his real name? He's
Singaporean guy.
>> No, I mean you met him.
>> Um I forgot his real name. I just call
him by his uh his handle. He did an
interview with Mark Zuckerberg like last
week.
>> Oh, nice.
>> Um so you know, one one connection away.
Okay. Yeah. Carry on.
So, so this is a nonprofit that uses
Terraform to bootstrap some of the
services that they use and they have a
very uh they have a lot of tooling there
uh some of Golang like tooling and this
one is a very opinionated Terraform
bootstrapper. Um, the reason I like it
is because after using Terra Grant,
having all of your different Terraform
state configurations spread across the
repo, it was very hard to keep track of
like exactly how things were organized.
And so this uses a central manifest, a
central YAML file that basically defines
the layout of the repository. Um the
idea is that this gets used across
product repos so that every every team
is basically bootstrapping terraform the
same way and then you can update the
templates and then every team updates
their fog version and then it updates
the repository across right. Um I'm
using it in a monor repo way which is
not really the intended design. So I
added features to like distribute the
config file. So I ended up with kind of
like an in between. Um but yeah, I I
basically forked it and I've been adding
a lot of features. But what we're
talking about is how do the releases
happen here, right? So in this case, I
also have this uh dependabot
configuration to automatically update my
go dependencies and it doesn't directly
merge I mean it doesn't commit to
master, right? It creates a different PR
and it does a terapform group as you
see, right? It groups all of the
dependencies that are related to
Terraform uh modules and groups them all
into one place. So it picks up like um
we're getting the latest go-getter uh
and if there's another module that is
related to the terapform group, it will
be part of this PR and then what happens
is it automatically lets adds the bot
merge label and that gets auto approved
but it's using my pat so it's showing me
as the approver so it's using my GitHub
token to approve this and as soon as all
of the the the CI/CD passes on the pull
request only then does it merge right
it's it enables automerge and when
everything like see it enables automerge
And when all is good and all the tests
pass, then it gets merged. So you get a
very controlled way of of deploying.
>> Your profile looks like the opposite of
my profile in GitHub. That's weird. Like
I'm looking one direction, you're
looking the other direction.
>> This is Scarface, right?
>> Oh, that's not you.
>> Not me. [laughter]
>> Yeah. Um, so so that I like this. I like
this a lot. And what happens is all of
these u dependency updates get merged
onto master and then Google release
please keeps track of them and basically
tells me the next release that's going
to happen is going to include all of
these bumps.
>> But have have you used this in anger? I
mean have you used this with several
teams? I'm just wondering if this is
going to work.
>> So we use this CLI across our Terraform
repos but they're large. So we have like
two or three of them and basically fog
is dictating the the the layout of the
terraform repo and integrating with
Atlantis and also now integrating with
Terra constructs. So it's bootstrapping
TypeScript projects, it's setting up the
TS config, it's setting up ES linting
rules, everything. So so all of those
templates are inside fog and I I still I
don't recommend it because uh we're
using it and it's working for us but
some people might not really like some
of the things it does, right? Um but
yeah it does the full repo remote and it
controls some part of what the grid will
do. It controls how these components
link to each other. So it determines
what is the remote state for that
component. So if you have a landing zone
component and you have a cluster
component and the cluster needs to to
understand where is the network created
by the landing zone then it will read
those outputs from each other. So that's
through the remote state right so so
it's a very opinionated way again
originally from from Chan Zuckerberg but
I added a lot of features on top of
that.
>> That's interesting. That's really
interesting. Nonetheless, I think I
think in in my current gig, the
the different teams are not really
owning the infrastructure. So, I don't
need them to do a lot of complexity. I
just need them to do the data type
stuff. And I think I can control that
better not using this. But I don't know.
This is really interesting. But, okay, I
got to wind up here because I got to I
got to go go straight into into work
mode.
>> Okay. Great talking with you. Um,
unfortunately we didn't talk anything
about AI.
I don't know, maybe maybe another
morning we can I can do the same thing.
Just let me know.
>> [ __ ] Actually, I I I found out remember
yesterday we talked and I said that
there's um I was doing like a quick
dashboard because I'm using beats to
track issues for the LLM.
>> Yeah, you showed me. That was great.
Yeah.
>> But there are actually like six or seven
people doing the same thing.
>> That's the beauty of AI, isn't it? It's
so beautiful.
>> Yeah. So, so I found some u kind of
working dashboard and it's pretty neat.
It gives you like a full overview of
your large features and your
>> epics. Let's pick up on that next. Don't
no more talking about infrastructure
full AI next talk. Okay.
>> And uh I'm happy to wake up in the
morning. Just let me know what day suits
you.
>> Yeah, sounds good.
>> Could be could even be tomorrow. I don't
I don't mind. See you.
>> Sounds good. Yep. See you. Bye.