Episode 1: Terralith: the Future of Terraform Automation and Collaboration? (Re-Upload)

The following is a conversation with
Vincent Smith on the latest developments
around Terraform.
Sorry, I I couldn't resist the Lex
Friedman inspired introduction.
Well, I know Vincent from uh Singapore.
He's he's out there somewhere in Asia.
He's a he's a thorough veteran of all
things infrastructure, as am I. I'm
based in the UK
and an infrastructure engineer is my day
job. What is an infrastructure engineer?
I help run those software services that
you use. I sort of help uh provision
those cloud and manage and govern those
cloud services usually AWS.
And uh what one way to provision those
resources is with a popular uh project
called Terraform which fairly recently
uh got bought by IBM and relicicensed uh
under like a commercial license. So the
thing that everyone uses the terraform
is now uh in a sort of a interesting
place and the conversation um that
follows is Vince and I just chatting
about it and hopefully you enjoy it.
Please comment below and um I will uh
try my best to answer all your
questions. Thank you.
>> So what's the tea?
>> What's the tea? So you had some you had
some juicy infrastructure gossip.
>> Okay. So maybe some background like
there's quite a few companies built
around Terraform.
Some of them being Space Lift, M0, you
know, you know the whole deal with the
the licensing change around Terraform
and um a lot of companies forking the
last uh Mozilla public licensed version
of Terraform and then maintaining it as
a fork under open tofu. Right.
>> Right. And um aside from those
companies, there's also Digger. I'm not
sure how much Digger.dev is involved in
this open TOFU effort. I'm not sure
actually who are the founding companies
behind OpenTofu. Um the the the term
Taco Terapform automation TA
collaboration CO and software. So Tacos,
right? T A C O S Terafform automation
collaboration software.
>> Okay. Okay.
>> Okay. So a lot of this Terapform
automation and collaboration software
examples are Teraphform Cloud where you
push code into Git and then it triggers
a a run of Terapform plan and apply. Um
and and so Atlantis is a famous uh open
source tacos.
>> Yeah, I I just have Digger open right
now and it looks like it just reminds me
of Atlantis.
>> Yeah, so Digger is an alternative to
Atlantis with the premise being why do I
need a separate EC2 instance when I hub
a whole bunch of CI/CD runners. So what
they do is they give you CI/CD workflows
for um GitLab and GitHub actions and
then they simply have an orchestration
API that those runner workflows can you
know be orchestrated with. So whenever
um a an event comes into the GitHub, it
maybe triggers through the orchestration
layer and the orchestration layer would
then dispatch uh dispatch uh GitHub
workflow. So it actually uses your
GitHub runners that you may have
self-hosted or that you are having part
of your GitHub hosting to to run the
terraform jobs, right? You don't have a
separate terapform instance. So
>> so their view on how is is the minimal
like impact like the easiest fit into
existing uh company um workflows. So
they build digger in that uh frame and
then they mentioned like you know what's
really missing and a big problem with
Atlantis as well is the ability to just
manage the orchestration. So Hashi Hashi
conference happened last week where
Hashikov is going to make a lot of
announcements. So there was actually
quite a lot of um things or news
happening around Hashik um from other
companies trying to you know use the
event to also
>> Wow, that must be such a cool event.
Atlanta. Okay, cool.
>> Okay.
>> Yeah.
>> Yep.
>> So So last year on Hashicon they
announced
uh stack beta. I think the two years ago
they announced uh terraform stacks
private beta and then public beta and
now it's finally ga general available.
So what is terraform stacks? Steraphhone
stacks is basically the the ability to
define multiple terraform states in one
stack. So that they for example a common
scenario is you have a network and then
you have a Kubernetes cluster and then
you have some Argo CD deployment that
needs to depend on that Kubernetes
cluster right so you can
those are all all individual components
right the network has to exist and
create all the subnets private and
public subnets for the application nodes
to be provisioned in. So the Kubernetes
cluster can only be created after the
network has been created. So Terapform
STS will help you to orchestrate across
multiple terapform states.
>> So this is something that is cloud only,
right? They will this is not part of the
Terapform CLI until the most recent
release where you can actually use the
Terraform CLI to query the cloud stacks.
But cloud stacks is cloud only except
for some CLI to get some information
about your stacks. But but wait a
minute, Terraform must have had this
feature before to depend on another
stack for another I mean you could you
can fish out values from another
>> Yes, you can fish out values and I
believe they just use workspaces and
outputs between workspaces but they
didn't have a crossworkspace
orchestration uh um feature as far as
I'm aware.
>> What what does orchestration mean like
like a like a like a
>> this coordination
>> like depends on on things like this?
>> Yes. like I want to provision a new
environment and that environment
consists of a network, a cluster and a
githops controller and the uh network
needs to be created first then the
cluster then the controller. Okay.
>> So yeah
>> so when you say orchestration I'm just
like mapping it into GitHub workflows.
Yeah. But okay carry on. when yeah when
I say orchestration I mean like
coordination between different um you
know activities that need to be ran and
and you know orchestrated you know the
events between them that trigger the
activities or actions to be
>> executable something
maybe more like a make file in my you
know Dumbo brain you know
>> a make file gives you a couple of that's
more like a task runner right a make
file gives you a couple of targets to
execute and then you you are the
orchestrator usually that says first I
need to run this make file has like some
minimum orchestration
>> well it has like files so you you you
say like you know I need this file food
to be built before this file bar so so
it it will run the target for fu but to
be honest make files are never quite
used that way but uh anyway we're
getting distracted here so orchestration
I'm I'm I'm all aboard on GA
>> all right so there's been GA now great
um however
A lot of these Terraform automation and
collaborations software or tacos have
switched to a resource under management
pricing mechanism where you're being
charged for every resource that um they
manage for you and so if you have a lot
of Terraform resources then you are
getting a very big bill over time right
>> yeah yeah I mean that's that's really
evident in my current client with data
dog I mean when when you look at each
resource cost. It's cheap,
>> but when you have a zillion resources
and the sprawling cloud of which you
inevitably do, it's really expensive,
>> right? So, one of the arguments
of um you know the advantage of of using
a a collaboration software like you can
use Atlantis, you can run it selfhosted
um but it doesn't really do cross tech
orchestration all this tech feature
doesn't exist. Um, so I've also
personally been thinking like how do I
add on orchestration on top of Atlantis
for example because Atlantis whenever
you open a pull request it will um
identify any GitHub um it will use git
to identify any changes like based on
the pull request compare compared to the
base. It will identify which terraform
projects may have changed. It can auto
detect that or you can be very explicit
on how it needs to track which projects
need to be run and where those projects
live. Um and then Terraform will
basically try to run all of those
projects. If you want, you can auto
plan. So every time a pull request is
opened, it will auto plan any projects
it believes that may have changed and it
runs them all at the same time. So
obviously if you make a pull request
that creates three different components
and they depend on each other, then one
of them will plan successfully, the rest
of them will fail because the other
components don't exist, right? So you
have the ability to do some type of
orchestration if you manually um define
or uh execution groups. So Atlantis has
the ability to say execution group one
and then execution group two then
execution group three. A little bit like
Argo CD uh deployment waves. Uh it's the
same same.
>> Okay. Yeah.
>> Uh it's execution order group.
>> Okay. Wow.
>> Yeah. So it's for ordering. And then uh
about two years ago another PR came in
which was the ability to define
dependency. So you uh which basically
you can use order group to determine
when to run the which projects only
after a certain other projects have been
run. And then the dependency was that uh
Atlantis would fail or bail if uh one of
the projects dependencies hasn't been
applied yet. So it has a very basic
tracking of of of a project. It says
like okay this one is planned this one
is uh
>> applied
>> um and and if it tries to if you try to
run a plan and it and the dependencies
have not been resolved it will fail
because it has these dependencies of
other projects that must be um you know
resolved first. Yeah. But in in reality
those two concepts don't really map
directly to like the ability to
orchestrate like to say uh run this and
then run that right. Um because that
should be sufficient to just say
dependency. You don't need execution
group and dependency. treat it
separately. One is to make sure the
dependency has been resolved. One is to
make sure that one
>> I mean ordering is like very basic uh
you know you're you're on a maturity
journey that that's when I see people
doing that that's like very basic. It's
like kind of manual but when you have a
a proper tool like any like make or
depend or or an orchestrator as you say
then it becomes
>> spacelift or m0 they all have the
ability to do that even teragrant like
even teragrant within the repository you
can define dependencies between your
terafr grunt um you know states and then
you uh teragrant can do a run all uh to
orchestrate it so basically run
everything in in in order so it
>> it basically builds a graph
uh across your different Terraform
states. So, so within a diff within a
Terraform state, Terraform builds a
graph between the resources, right? this
resource need to be created and then it
that resource depends on it and then and
it does its own parallelization and like
execution and and sometimes you need to
be you need to terraform you need to
tell terraform hey actually you know
there's like for example an API gateway
when you create the API gateway and all
of its integrations in AWS um if you
then try to trigger an API uh gateway
deployment if that if that uh actual
integration hasn't been resolved or
doesn't or the lambda hasn't been
attached and you trigger the deployment
it will fail AWS API I will say got it
>> this gateway doesn't have a an
integration so it will fail and and in
that case there's no direct dependency
because the deployment has a dependency
implicit dependency on the gateway
because you use the gateway ID uh the
API gateway identity so when you you use
that link terapform knows I need to
create a gateway then I need to create
the deployment but it does not have an
explicit dependency between the
deployment and for example an
integration on the gateway so when when
you don't do that then um you get a race
condition. So it may manifest um
sometimes it may not manifest right u
but but the right way is to really say
like okay this deployment also depends
on these other resources you know and
you have to explicitly say it depends on
this gateway integration with this
lambda and so on right
>> yeah got it
>> so that that becomes very um that works
really well with terraform within a
terraform state
>> so terapform is amazing um in handling
these resource dependencies sometimes
you have to give it a little bit of a
hint uh now from a higher level outside
of the state. Um, creating dependencies
between states adds a lot of like
additional orchestration overhead that
makes it difficult sometimes. Like one
of the most common problem is I'm going
to change my network for example and I'm
going to change my output and now my
subnets are no longer a map of key value
pairs. Maybe they become an ordered list
and then maybe I have another component
which reads a key and then gets an error
because there's no longer a map there.
it's
right I understand the complexity
>> that happens um so why do people
separate everything right why why do
they create one state for the network
one state for the cluster and that's
often because of like life cycle um or
ownership
>> right
>> yeah so a network usually has a much
longer life cycle than the clusters
within or even the deployments on on
clusters
>> I mean I usually say blast radius to to
to clients like that's why we have so
many goddamn stacks.
>> Yes. Uh blast radius and um with that is
also for example if you use and again
this comes back to like a lot of
discussions within the terraform space
which is like do I always uh only apply
after I merge. So um you know only apply
off the master branch or do I like do I
allow people to run apply from a from a
branch and basically the branch is
modifying the states which is shared
between the master branch and then if
somebody runs an apply then they may
like inadvertently undo my changes
because my merged yet right
>> I see that all the time in my current
client we have a channel called
Terraform stack wars or something like
that people basically announce that I'm
going to make a change to the monitoring
stack or the networking stack.
and then they create a branch and imply
it in dev and uh of course people need
to somehow lock it so that they don't
stamp on each other
>> right so Atlantis has a very good
feature there and a lot of organizations
follow Atlantis way of doing things
which is as soon as you open a pull
request and Atlantis runs a plan it
basically keeps a lock on that
particular state
>> so if another pull request goes against
that particular state like Atlantis
detects a project and and tries to plan
it, it will get an error saying, "Oh,
this pull request number 34 owned by
Vincent owns the lock on this particular
project.
The lock is quite needs to be quite long
lived because like
>> as long as the PR holds the lock so it's
in a database like like an object
database keeping track of the lock and
um and so basically until somebody goes
in and clears the lock from that PR that
PR holds the lock.
>> Nice.
>> Right. And that's because Atlantis tries
to be deterministic. So when you when it
auto plans it will write a plan on disk
and when somebody approves it he will
approve exactly that plan that run right
and if something changes um for whatever
reason if you if you try to trigger an
apply and you know Terraform tries to
and this is a terapform feature right
Atlantis passes the previously generated
and approved plan from disk into the
Terraform apply and then the Terapform
will immediately reject it if it
discovers that something changed and the
plan is no longer valid. So the plan has
become stale. So then you need to like
rerun plan to get an approval. It can be
>> depending depending you can be flex. You
can say as soon as it you know the PR
has received one one approval it's okay.
You can use the GitHub approval
requirements um you know whenever a
commit is pushed maybe it invalidates
the PR. So you use the you know your uh
version control um gates on your pull
request.
>> Yeah.
It sounds like Atlantis has been
evolving over the years that I I mean I
haven't used it. Uh so that sounds good.
I mean does it have does Atlantis for
example have a feature where if you
don't destroy anything like you know if
then just approves
>> that like some sort of policy is
>> auto approval.
>> Um
that's not actually no not that I'm
aware of. It's probably possible but it
would I mean technically everything is
possible right but it would require a
little bit of additional I don't have
any um such setup currently
>> but this is definitely like another
requirement that um people ask it's like
whenever like I want an auto approve or
an auto deploy flow so I want Atlantis
approval to automatically happen and
>> yeah I mean I mean and there's probably
there's probably AI in the mix here you
know like
>> you don't know I would avoid AI in those
situations to be honest.
>> Well, I've I've been using Copilot in
GitHub Copilot um as a reviewer in in
PRs in in the team and I'm actually
quite surprised how good it is in some
ways. It's it's better than any like I
mean is it better it complements any
like linting tool and things like this,
doesn't it?
>> We can go deeper on like all of the
things that we can do with context
because this really relates to the tea
that I want to talk about.
>> Okay. Okay.
>> Okay.
>> Oh my god. 20 minutes in. Yeah. So, so
you you'll have to edit this heavily,
but we've basically explained some of
the, you know, options because what you
just mentioned is we have some manual
coordination to lock PRs. Atlantis does
that for you. Um, but this means that
Atlantis locks the PR and then it has a
apply before merge philosophy and a lot
of organizations are like this is
unacceptable. uh you know trunk is the
source of truth and um we cannot have
anything applied until it landed on
trunk that's where it should be
>> merge uh okay
>> yeah because Atlantis keeps the lock on
the PR as soon as everything is applied
within the PR then that needs to be
merged as soon as possible needs to go
into trunk right because now the state
file reflects whatever is on that branch
yeah and if that is not merged and
somebody you know runs Terafform from
trunk from their laptop without you know
having this at
>> I Atlant's lock is purely on the
Atlantic side. It doesn't
>> contentious but it doesn't it doesn't
seem like such a big deal to me really
because you're working.
>> Yeah. So some organizations are very
strict and so some uh some workflows
they they say apply before merge is
incorrect. It should be merge and then
apply. So my main I am from the Atlantis
like I grew up with Atlantis. that glad
this solved a lot of my problems and and
for me I have experienced many times
that the plan looks perfect but when you
try to apply it fails right for whatever
reason because you know a terapform plan
doesn't really validate like
>> yeah yeah yeah
>> I guess I guess terraform crud reads
only the attributes and then does this
internal validation of like this needs
to change and then um the actual ter AWS
provider may get an invalid response I
mean to be honest I I prefer a workflow
where we don't even plan because because
>> plan doesn't catch probably I'm making a
crazy figure about AI it plan doesn't
catch like 80% of the the issues that
you might get when you when you're
deploying infrastructure. I mean it it
it doesn't really help you move fast. Um
>> okay right so so so there's just I
wanted to highlight that Atlantis is one
solution. It lacks a lot of the
orchestrations, gives you a lot of other
stuff. Um,
>> and it's open source and it's it's it's
easy. You don't have to worry about
>> can run it on $15 EC2 instance. You can
go to the digger docs where they go into
like pretty good arguments against it
saying like, well, you know, it's never
just one Atlantis instance. They have
like a a doc where they compare the
problems with Atlantis. I believe they
say uh Atlantis is never Yeah. compared
to Atlantis. I like their arguments.
It's like it's never just one instance.
Where is
I can't see it now. But there was or
maybe I I remember it from another um
tacos. Uh
>> but but since but since Atlas is just
doesn't is is not like cloud pricing. I
mean that's a huge win in my book. They
don't even mention the price. I I talk
to to friends of mine that are running,
you know, 10 thousands of Terraform
resources of, you know, very big scale
and when they look at any RUM quotes,
it's just like blows up their cloud bill
because it's it goes from I'm I'm paying
$20 a month from a, you know, a decent
EC2 instance running Atlantis or
running, you know, on a Kubernetes
container. So, it's even less um, you
know, CPU memory usage and I can, you
know, so I'm not paying anything right
now. I'm paying like 20 bucks max and
I'm going to be facing $10,000 a month
if I switch to like the cloud just
because I want that like additional
policy as code or I want additional
orchestration features or arbback like
that's not in Atlantis right you don't
have proper arback you can do like some
code owner setup with your GitHub PRs to
you know only validate that if that
person approves then it can be applied
we do some conf test in my organization
where we we do invalidate like or we
block a PR if there's any changes to an
IM policy so that the security has to
come in and review these changes to IM
policies for example. So we have some
conf test like that
>> some some routings going on.
>> Yes and conf test is also um like
hardcoded into Atlantis right you can
Atlantis has their own workflows and the
default workflows includes confest so it
runs an OPA um policy against your um
Terapform plan but if you don't want to
use confest you can use other OPA or um
other scanners. I I don't think I've
even used comp like like why is it
claiming to be
>> secure by design and
>> uh maybe because when you run Atlantis
you're running an EC2 instance that may
that has a lot of permissions like an IM
policy that has administrative like uh
permissions and maybe that's like you
you are responsible for securing your
Atlantis because your Atlantis in the
end is the one that takes or assumes the
profile to make all the cloud changes
and then that instance It's quite normal
though. That's pretty normal.
>> Yeah, it's
>> and so I mean it's
>> because you're hosting an EC2 instance,
you need to like you can't make it
publicly accessible, but it needs to
receive GitHub web hook. So it must have
some ingress path for
>> conversely, I'm I'm assuming that if
you're using digger, you have to hand
over the crown jewels to digger just
like CICD to to the ter.
>> No, they don't. So, so they are purely
like the orchestration layer that just
dispatches the the workers and your
workers are your CI/CD system. So, you
can you can have those hooked up to your
GitHub like you have runners for for for
stuff anyway, right? So, so basically
they hand off like
>> I don't Okay. Okay. Let's let's get to
the the gossip then.
>> Okay. So, so aside from Atlantis,
there's Digger, there's Spaceelift,
there's M0, there's Terot Team, there's
quite a few. And so Digger has been
posting these um thought experiments on
Reddit where they talk about the states
manager. What if a terraform
orchestration like Tacos wasn't a
runner? Like most of these like
Spacelift, M0, they're all providing
runners um basically triggered by web
hooks. But what if you decouple the
runner infrastructure and just focus on
the state management part? So the
biggest problem that we have because we
are splitting states. Um the biggest
problem is how do we like manage the
dependencies between states and and
identify you know if if one state
changes what else is impacted because
for example I make a change to my
network now I need to invalidate
everything that depends on my network so
it may need
>> so so we we're go we're going back to
Terraform stacks aren't we here if
you're talking about states
>> it's kind of related to the stacks
stacks capture the dependencies between
states but the argument from digger on
Reddit like the thought experiments were
about like let's imagine an API that you
only use it to define dependencies
between your states and it gives you
endpoints to you know identify any stale
states. So a state that needs to be
replanned because something it depends
on has changed. So basically track the
edges, color the edges when outputs
change and then replan or like give an
basically give back a list of jobs to
run because there's these states need to
run, right?
>> Color color the edges. That's an
interesting expression. Yeah.
>> Color the Yeah. Um it literally what it
does it's a a state multigraph because
it keeps track of multiple edges between
states for each
>> out. Where are these discussions
happening on terraform or something? on
the terraform subreddit which was like I
think posted around maybe June.
>> Okay.
>> So state management calendar edges I
think I'm I'm with you.
>> Right. So so I was very interested and I
talked with um with with the the team
because I'm building something that
really needs this state orchestration
and not only the state orchestration but
I need to hook into it because I
generate terapform. So I want to be able
to read the outputs and I want to be
able to see what's the shape of an
output. So I want something that can
give me um or can help me track
dependencies or outputs and then
identify what is the source of that
output and things like that. So I talked
to them like hey um I want to know what
you guys like I'm interested to help
build with this. So I had a call with
them and they presented to me uh
something that was they were building in
stealth at the time and they were just
wanted to get some feedback and it was
called open taco. Okay interesting. So
that was about like two weeks ago. Then
not much after Terra team the other
tacos provider announced uh an
interesting project called state
graph.dev state graph.dev.
Yes. So terraform without the state file
bottleneck. Like they their argument is
basically the the reason we are
splitting states is because we don't
have proper tools to manage our
terapform state because the terraform
tool is only able to manage a single
state. The real problem is if we were to
create like a massive teralith basically
one big state where all of your
environments are and then we improve the
actual tooling around this single
massive state and we're able to like
terraform apply uh slices of it and
we're able to you know control arbback
of individual resources like we're able
to track ownership and and approvals
against the individual resources. we're
able to solve a lot of the problems why
we are having this blast radius and and
these arguments.
>> Yeah.
>> I've been argument is
>> yeah that I've been thinking this we
need like a monolithic
kind of state
that problem.
>> Yeah.
>> Yeah. So they announced like let's get
rid of this state you know multi-state
problem. Um and at the same time you
have uh Tara Grant or Grunt Works
promoting these uh kill the terit
advertisement or workshops. It's very
funny. My LinkedIn feed is like on one
side Grunt Works you know advertising of
kill the monolith uh kill the ter
>> break
>> breaking the ter
workshops and they're like um you know
terits are killing you. You must have
responded like
>> a monolithic terapform state.
>> Well,
no one ever does that but uh I don't see
the problem there.
companies like Grunt works they're like
consultancy and they're going into a lot
of places where people you know like in
in-house develop their own terapform
setup and then they face all kind of
issues and basically teragrant is quite
extreme because when you use teragrant
you pretty much get one state per module
so you build one module and then you
invoke that module and that's one state
you cannot like have multiple modules
this is a bit crazy right I mean if you
imagine how small modules sometimes are
I can tell you in a previous place I
worked at we like three years of terror
grants and we had like 300 or 400
terraform states. It was insane.
>> So, so, so I don't understand. So, if
they do it in such a fragmented way,
then how can they claim that that the
monolithic terraform state is a problem?
>> No, they're coming into organizations
and they're saying, uh, we will help you
break down your terror because all of
your problems are because of the terith.
>> That's crazy to me, but okay.
>> Right. And they have they also launched
a cloud product called Terra Terracale
or Teragrant scale. So they they
launched a product uh basically to
migrate you into and then orchestrate.
>> Oh okay. So it's like a importing blah
blah blah
>> right. Okay. So so that was just an
anecdote because I'm seeing on one site
um Terra team constantly posting about
don't break down your ter is not your
problem. The problem is the tooling. And
on the other side I have like grunt work
sponsor saying we have workshops to help
you break down the term.
Yeah, I'm definitely in camp um state
graph. Definitely.
>> Right. Yeah. So, so they hit hacker news
uh front page the moment they posted
this and um they Yeah.
>> Nine point. Oh, yeah.
>> That one. Yeah. Like 13 days ago.
>> That's quite respectable,
>> right? And then they also got like the
mailing like they they said they have
hundreds of organizations interested.
They're looking for design partners to
to help build out what people want from
a state graph and what it looks like.
They're even hiring Okamel developers,
but they won't train you. I try I asked
they said no, we don't have time to
train you. Um, so you must know Okamel
and then um they're building this their
launch date, their road map is for
quarter 16.
>> Atlantis is written in what? Python, I
guess, or something.
>> Golang, a massive Golang source code.
Oh, nice.
>> Quite complicated. I don't know about
Golen for monolithic like
>> it gets really complicated.
>> It's not that comp. I love go.
>> The problem with with Atlantis is that
it's so ingrained into the pull request
flow that everything is like based off
the idea that it's coming from a pull
request event which is horrible. Like
for example, they try to add an API
endpoint to run Terraform plans and it's
really painful because it like oh you
have a pull request, right? Where else
are we going to run a plan on? Right.
>> Yeah.
>> So it's so
>> okay. So but anyway, Atlantis is not
um sorry my my head space is in this
whole monolithic versus
>> Atlantis doesn't really dictate how you
organize your state. Yeah, you know has
this auto discovery to to find your hash
your terapform configuration files. You
even have like integration with terror
so that you can automatically like you
can let terror actually run the
terraform across your like repository
and then you can Atlantis can hand off
some of that. The most important part
about Atlantis is basically locking pull
requests posting plan into the
>> requ. So um yeah so sorry I I'm just
trying to track in my brain. We talked
about Atlantis and Digger and now we're
talking about Sterraph and Terror Grant.
Well, is there some relationship?
>> Yes, the relationship is that Digger was
posting all these thoughts experiment on
on on Reddit. Uh suddenly there's this
announcement of state graph. I found it
quite funny because you know the terror
team is actually quite active on Reddit
and I see them commenting on these posts
and then I see them announcing state
graph and I thought like like it's quite
similar. Uh so so I talked to them and I
was like that's interesting like I need
something that I can query. So either
it's it's the thing that digger is
building or it's the thing that um the
terror terror team is building whichever
right I need to query the state I want
to know I want to retrieve information
>> so yeah it's about it's between digger
and state graph for you okay
>> right right so hashik happened uh the
week last week hashik um is going to be
by the end of the week at the start of
the week they announces we have some big
announcement this week we have a really
big announcement this week
>> infograph
>> infograph is announced Hashikov which is
very similar and again to the point of
AI why do all of these people suddenly
or all these organizations are suddenly
so interested in building all of these
massive state graphs and dependencies
between states
>> because AI can easily gro it properly.
>> Exactly. Because if you want to build an
NLM agent, every single organization
right now is positioning themselves like
we have the context. We've got all of
the information about your
infrastructure to for you to query.
>> Context is king.
>> Exactly. So, so on one side, Terra team
announces state graph, perfect for
building a context. Digger is like we're
building the state manager because you
need to be able, you know, we we need to
be able to query, you know, keep track
of dependencies, right? Yeah. We all
want to keep track of dependencies, use
our product, right? And and then the big
announcement on Wednesday, I think a day
before Hashikov, Digger opens uh this
very aggressive video. If you go to
digger.dev/opentaco, dev/open taco.
That's basically where the project was
an open taco.
So, a couple of things I remember from
that video.
Um, basically they start with like this
kind of like apocalyptic error messages
like people are being held hostages,
resource under management pricing is
criminal. Um, you know, this is not an
open standard. Um, and we have the
solution. We are building the future of
uh infrastructure as code. the open
standard. You see the first line there
introducing project open taco the open
standard for terraform open automation
right and then scroll down there's like
this road map the pillars I think the
pillars are are amazing the state
manager is the first thing that we just
talked about right then they said well
you need a private registry for modules
you need rolebased access control you
need to define your dependencies between
your stacks and you need to have uh
insights right reality map to code and
all that
>> cool I I really like um sites that like
advertise their principles or pillars or
whatever.
>> It's I think it's a it's a really nice
idea. Um
>> but the the response
>> w
even the negative responses can be
capitalized on. Right.
>> Okay. Well, okay. Let's look at what
happened next. Right.
>> What happened next?
Heat. Heat.
[Music]
Heat. Hey, Heat.
Heat. Heat.
[Music]
What happened next is they posted this
on Reddit and
and they posted on on on Hacker News. I
think on Hackin News they had three
uploads. I keep track of all the links
and on Reddit there was an immediate uh
you know very strong backlash from the
community. Um there's a few Reddit
accounts that were you know linked to
organizations I me mentioned before
>> and they were very critical. They were
like why do you talk about an open
standard like doesn't the standard
involve communication and like
contribution and a community like where
is the discussion? where is the you know
the RF the request for uh comments the
RFC where is it you know the
specification design um and why does it
is it like declared as the open standard
but is does it look like a product like
there's a product road map even like how
is this an open standard how can you
declare to be the standard when you
haven't like you know there's no
community behind it right
>> and I personally didn't really see that
but like apparently that that resonated
with a lot of people because that was
pretty much like there weren't that many
comments maybe 10 and like four out of
them or maybe 50 50% I mean were like
this is a product stop you know making
it look like it's an open standard uh
it's a product blah blah blah right so
they open source the source code uh and
one of the main things that's
interesting is they completely integrate
with the Terraform cloud uh back end so
if you're familiar with Terraform as a
back end you can use S3 you can use um
Postgress you can use an HTTP remote
backend or You can use Terapform cloud.
So the Teraphform cloud back end is a
very strict protocol that uses like
authentication. So you use Terapform
login when you run Terapform plan. You
can even have it remote.
>> So it uses the same back end.
Interesting.
>> And they implemented the Terapform cloud
back end because it's like you know
Terapform code is open source right
source available. So people replicated
all the API endpoints that Terapform
uses and they implemented the back end
of their own version. And that was like
whoa cool. They use the whole Terraform
back end. They have like SSO
integration. quite small. So they
>> and then yeah
>> the creator of F or open terraform like
cloud back end comes in and says
>> okay guys I expect in this day and age
you you know people copy code but you
guys do it without like any attribution
and you can't even be bothered to to
change the comments like you literally
copy the file from my project
>> with the original comments right in it.
>> What is the project name? F
>> yeah.
>> Ouch. Ouch. Indeed.
Yeah. Leg zero uh leg 100f comes into
the Reddit comment and says, "I'm the
creator of F. You guys lifted my files.
>> I didn't even know about I didn't even
know about this project. This is
interesting." Ah,
>> I bring I bring receipts. I have links
to all of these Reddit comments. I can
share them.
>> So, there was definitely some hot
controversy.
>> H Yeah. The funny thing is that most of
the like harsh backlash
um I can always link back to the same
accounts.
It's always like two accounts that are
always going like you guys say you're no
standard but this is a product code. You
guys are breaking the trust of open
source code.
>> You hinted about it earlier but like on
Reddit um I mean I have one account on
Reddit and I stick to that one account
but I know loads of people that have at
least two or three accounts on Reddit.
two people going at it from four
accounts
>> and the and the moderator because what
happened to the post the big
announcement of open taco it got deleted
got removed I'm very curious now like
why unless like I'm very oblivious
>> unless I'm really oblivious and and I
really don't see that much as a really
bad like situation I look at this from
like docker point of view right docker
came in they built contain Linux
containers they built a whole tool
framework around it and then OCI was
created open container their initiative
they created the standard. So for me
like if you look at open taco it's a
product and you can like create a
standard out of a product like docker
did right and being so aggressive and
saying you but okay I understand that
there is first of the message like the
YouTube was very aggressive then second
the um you know declaring they are the
open standard that clearly like upset a
lot of people right then obviously the
massive failure of having this um you
know code that happened So what was the
response of um of of Digger to this uh
backlash and the post being removed?
>> A apology, right? So they did an apology
post
>> where
>> um
>> Oh, let me
>> Oh,
>> postmorton open taco using code from
without attribution. This is the chain
of events.
>> This is how it happened. We we we did a
stub five Y's
um and they post it. So I I want to hear
your opinion about what you feel about
this this this um
>> so why did the the code was moved from
as is from an internal
okay that doesn't really excuse anything
why
>> and they open source the internal to
full to show the full commit history of
how that was built when it was built who
was doing the PRs and all that
>> it still doesn't explain yeah
because I'm curious how you would feel
about this because the the response was
also Interesting.
Well, I mean,
I'm Is it Is it called Akams Akum's
razor? You know, don't attribute malice
to what could be excused as, you know,
incompetence.
Is that is that the right phrase? um
razor is is more about like if if
something if there's a simpler
explanation then that's the right
explanation but don't attribute malice
to what could be attributed to like
stupidity or or incompetence that's uh
that's some I don't know exactly what
>> okay something else but um I I would
probably go on the more positive side of
things because because uh you know like
when you're in a hurry and they're
ripping code from there I mean it could
This is even more the case with AI
nowadays. Like it wouldn't be surprised
me if you're writing some code with AI
and it just steals some code from some
code base on GitHub and then and then
you're taking flak because like uh you
have a comment from another GitHub repo
and I'll be like, "Oh shit." It sounds
like they've fixed this issue, I'm
guessing. So
I I mean it's it's a mistake. It's a
balls up, but I wouldn't uh I would move
on from it. That's my initial feeling.
>> Right. The response was this is just
another product marketing, not really an
apology. Half of your apology talks
about the product or something like
that. Like I'm trying to find the
thread. Yes. And you can guess who was
putting those comments. The same people.
>> So this is where we are. I mean, how
long ago was this? 26th of September.
>> That was Saturday. Yeah. Okay. That's
This is hot hot hot
steaming gossip
tea. I've been like I've been having a
ball this weekend, you know? I was
literally eating popcorn like what's
going to happen next?
Whoa, this happened. Like um I honestly
like my personal opinion.
I mean, I don't really think it's fair
the the way that it's being um responded
to. Maybe I'm biased because I really
want something like this to exist. And
um yeah, I know of course
the the main like post the main the main
complaint of of the people complaining
was like, you know, trust in open source
has been eroded constantly. Projects are
relicensing. Um you know people are very
reluctant to to adopt a project now and
we need to build trust and trust is
earned by doing things right and
stealing code and you know relicensing
it because is a modilla public license
and digger and therefore the open taco
code as well is actually Apache 2
license so you can't rely it there's no
attribution so there's like so many nos
in this in this action like this
completely destroys trust this is
unacceptable So, so wait a minute.
>> Uh, Vincent, their repo was open source
too, right?
>> No. So, they Well, yes, they they moved
the they moved the open taco into their
main repository. Um, which is a massive
Golang monor repo. So, if you go to
digger. So, the actual
>> where is the
>> this is the open taco.
>> So, this is where they stole the co the
code, right? that they
>> this is the this is the internal one
that did they did some quick PC on the
internal code and then they copied all
of that into their main digger
repository. So this is the open taco
that was private
>> and then there's if uh there was it
wasn't into their internal if you go
back to their profile and you go to
their main monor repo the main one
there. So there you have the license
zone I guess
>> which is Apache 2.
>> Oh it's MIT.
>> Oh it's better even.
Um,
okay. Well, thanks for the gossip. I
think I got a a taste of uh the what do
you call it? The digestion system of
Terraform or something. You know, what's
going around and what's going to come
out. And
>> I I I like some of the like first off, I
think the name is the name is very funny
because you know, you have open tofu,
you have the concept of tacos and then
open taco kind of plays on that open to.
>> So So who coined tacos then? What's the
>> tacos is a known way of describing
Terraform automation and collaboration
software? Tacos has been around for like
two.
>> So like if I search for like tacos
site hashp.com
do they
>> sure
don't think they call it tacos but space
lift zero and so on. If you search for
like terome automation, then you're
going to find a lot of like websites
that refer to that as tacos,
>> right? Taco provider like Spaceelift.
Yeah. So, they do have it on but it's on
the discussion. It's on their forum.
It's not
>> interesting. Interesting.
>> Yeah.
>> Well, anyway, that that was a really
good conversation. I think I I need to
um transition into work. Uh, I got stuff
to
to uh cards cards to move into different
lanes and stuff like that.
>> So, anyway, if anyone is watching this
to this far, I congratulate you. Okay.
All the best, Vincent. Let's catch up.
>> Some links if you want to add it in some
links.
>> Yeah, context is king. Context is king.
>> Yeah, but it's more to discuss. We
definitely have to do another session.
>> Yeah, we definitely have to do another
session. Okay, great. See you, man.
>> Okay. All right.