Episode 26: If it Hurts, do it More Often (with AI)

Hey, hey, how are you, man? Good. Well,
I heard I heard you say this is going to
be a short call. Yeah, actually I just
realized like I'm I just jumped off one
call, but then I see that there was
another one back to back, but like I'm
going to be I don't know. There's just
too many calls, you know. Well, you're
busy busy guy. Well, I'm good for you
good for you. I guess you're spinning a
few plates, right? With Specular and
I didn't want to do a Specular thing.
Yeah, and like I think it would be cool.
Uh so for example, I did one 2-hour
session with a friend who
came up with a cool idea and then he
built a little mock-up in V0 with static
data and I asked him like is there
anything you want to build and then we
do a 2-hour session and if if all goes
well by the end of it I'll you'll have a
extra little piece of code on top of it
that works. Uh basically and and you
understand a little bit more the Spec
Driven Development approach.
>> Yeah.
I'm keen to do that. Like the thing that
bothers me, I've got real furor around
that whole like you hear developers who
say that like yeah, I've got some agents
going overnight and you know, while I go
for to the gym, that the the agents are
running. But
when I'm using an agent, it's just like,
you know, I ask it a couple of things
and then it stops. Have you managed to
like run the Spec Ledger overnight and
you come up with and and then in the
morning there's a
there's a there's a fresh new
implementation. Months ago they had this
like gas town spice must flow this Dune
references with like the ideas of
keeping agents in a loop if they stop
until the task is complete. Even lately
auto research is going berserk
everywhere about people like you give it
a
some constraints some experiments to run
and it iterates against those goals and
overnight you get like I don't know 45
50 experiments. I'm just making up
numbers here.
>> What's what is auto research? Sorry, I
didn't get that. Uh auto research is
from that ex-OpenAI guy and that the guy
who coined the term fine-coding. I am
afraid to mispronounce his name,
Karpathy. Andrej Karpathy. Yeah. So, he
he he released auto research and he said
like he does a lot of LLM fine-tuning
manually and finds optimizations based
on how he understands the algorithms.
And he basically formalized the process
and now he has LLM agents executing
this. And he says he already had a
highly manually optimized
stack and the AI found a lot of
improvements overnight. They talk about
numbers. I don't remember them like 30%
improvement.
>> This is the work where I feel like I'm
missing it. I mean, do do you do that
sort of stuff at Spec Ledger? Get the
agent working all the time or
I mean, if we continue a little bit
about auto research, it's very
interesting because people are taking
this from fine-tuning models to actually
building out features as well. And so,
you see a lot of posts on Reddit and
LinkedIn and everywhere from people
saying I took auto research and I made
it work on this like product feature.
Like
I think there was one famous company, I
don't remember the name, but they had
manually fine-tuned templates for their
template engine. I think it was Shopify.
Might have been Ruby, I'm not sure. And
they they had auto research
improve the performance of those
templates. So, it basically can work on
on all kinds of like it's it's it's a
framework. It's a bit like Ralph Ralph
Wiggum. It's kind of a loop to to get
that going.
>> Okay, I feel like I need to try this. I
mean, you mentioned
>> terms of your question that you asked
three times already, which is have you
set up agents to run overnight with Spec
Ledger? And I feel like I'm not like
Oh, man. Don't let me down.
>> I just told you this before that even if
I let if I write the specs and somebody
else builds it, it's not up to my
expectation. Even last night, I did
something really fun. There's like I use
a certain sauce and they have a CLI but
it doesn't expose some of the UI
features. You have to go to the web
interface to click on this then go there
and it's very annoying because I have to
click a whole bunch of buttons to just
see the error message. Plus it's then
beautified but it doesn't actually give
me raw data and ultimately I need that
data back into the shelf to give it back
to my agent. So to go and click buttons
and then to copy out data put it in the
shelf it's annoying. Yeah. So um
the agent or Claude Code really cleverly
found a whole bunch of curl endpoints. I
I think maybe from public API specs and
it does a lot of like it fetches my
credentials from the credential file. It
puts it you know does a curl with
authorization like bearer token and then
it finds the details but there are so
many curls and then sometimes makes
mistakes that I just decided hey I want
to capture all those curls and put them
into a CLI like a go lang CLI. And just
to come back to the point right? I
started with first documenting like some
markdown so that I could point agent to
it here's how we did this particular
step. These are the curls you do this
particular step. These are the curls you
do. And so I took that doc and I gave it
to Spec Ledger say write the user
stories right as an as an engineer or an
AI agent I need the command line
interface to you know trigger this
particular action and to go fetch the
logs and to get the the error messages
and things like that. So it's it's not
like GitHub actions something different
because GitHub actions CLI does this
beautifully right? So um so anyway I got
Spec Ledger to write the user stories.
Then I went one step further because I
had like like two weeks almost of me
doing this. So I went and I told it now
you have the user stories you have this
initial thing. Now go through all of my
Claude Code sessions look for this
particular API endpoint and find any
usage scenario where this was where this
was being used and what was the usage
what was the use case how was it was it
being used why was it being used?
>> That's pretty much what you said.
>> Look through your cloud sessions but
carry on. And yeah I mean like I like I
said I I am not using the automation and
often I have way too many tokens right
now. So, I'm like, just go ahead and use
as many tokens as you want because I'm
running out like it's the end of the
hitting the end of the week and I still
have like used only 40% of my tokens.
So, just like over 2 hours just go and
explore. And it found like three or more
usage scenarios where the AI agent was
like, yeah, your document talks about
these things, but there's a couple of
cases where the agent was trying to do
this and and it didn't work well with
the curls. So, it added those in. And
then ultimately I had a beautiful spec,
right? I had a wonderful spec. This is
the CLI. I have my whole design of how I
want it. I wrote the plan and there was
just like 40 tasks which like in the
past I would hit 80 tasks, right? And
after 80 tasks I would say 80 tasks is
too much. It will definitely go RE, it
will not go well. And I included E2E
like it uses Go lang so it uses VCR
cassettes to like replay HTTP because it
I'm I'm I'm working against an API
server, right? So, I'm running
end-to-end with
>> VCR cassettes I've never heard of that.
I've heard of HTTP test.
Yeah, I I think it's using like a local
HTTP test server, but it's replaying
responses from like cassettes from like
on on on disk snapshots of what the API
returns from what it observed in the
past. So, wonderful, right? 40 tasks,
VCR cassettes, cassettes, end-to-end
testing, CLI fully defined. This is
perfect, you should go to sleep, right?
And you should just let it let it run.
But, you know, 40 cloud takes maybe 30
minutes to implement this stuff, right?
Very fast. And then it finished the
first session. I think I did cut off
after the the spec, but I did keep the
same session because I'm using 1 million
tokens and I was only at like 15% so
that's like 150,000. And if I hit 20%
that's 200,000 tokens, then I'm already
at like
50 over 50% of my 400k even though I'm
on 1 million tokens, when I hit 20% I
usually clear it because even if you
have 1 million, if you go over 20% it
starts to degrade. Are you using Sonnet
or uh Opus. I'm on Max, right? So, I'm
on like on Opus 4.6 with 1 million token
context window.
Yeah, I got I got promo because like in
Copilot like Opus says 3x, so I don't
click Opus and hence I'm not getting the
good stuff.
Yeah, so so what happened then is it
completed the task very quickly,
implemented the CLI. I asked it to use
the CLI and it did great. Like I gave it
the original curl scenarios and it says
run these scenarios with the CLI and it
did everything and it worked
wonderfully. And then I asked any
divergence from plan. Oh, yeah. The plan
said I should put everything under
package golang directory, but I put it
all under under internal, which means
it's an internal module you can't use it
from other golang library as a library,
right? I said, "Okay, not a big deal,
but why did you diverge from plan,
right?" It says, "Oh, somewhere it said
something like it didn't put it in where
I expected it." I guess the agent
decided this was an internal utility and
therefore it shouldn't be an external
contract, therefore it needs to go in
internal. This is a diversion from plan.
It pisses me off, right? Cuz I give you
a very clear plan and for some reason
you do it differently. Then the second
thing is, "All right, now run the end to
ends.
Run the end to end, right? With a
clearly told you to build end to end."
Oh, yeah, "No, no, end to end were
marked as out of scope." What are you
talking about? It wasn't out of scope.
>> So, this seems to be a theme. We've
talked about this before where the AI
doesn't quite do what you want it to do.
So, are you getting Do you think you're
getting any
any better at controlling the AI from
going off piste?
>> Builds. I haven't built the the Gerkin
integration yet, right? What we talked
about BDD. Do you think Gerkin will pull
it straight? Hopefully, because it
should write them first, right? It
should write all the BDDs first and then
it should not stop until the BDDs pass.
>> sense. Like you you have your spec and
then you have have your your test BDD.
And then that should be the guardrails
going forward. Yeah, so that's the thing
that pisses me off and that makes me
reluctant to let an agent's team run off
like loose.
>> I see. I see where your your point is
that you don't want to make it run
overnight because you think it's just
going to go wrong anyway. Yeah.
Um,
but to come back to like Claude has made
a lot of releases that make it much
easier to leave your machine and then,
you know, check in and then kick it back
off.
Yeah.
I haven't really activated them yet, but
like you can /remote and then it
will it makes your your laptop
accessible through the app. So, you can
be on your phone and then you can check
if the what's the status Yeah, those
those features sound great. I mean,
unfortunately, my employer gives me
Anthropic via like an API key. So, all
those like quality of life features like
voice, mobile, all those features are
actually not enabled in my Claude code.
Yeah. Do you do you use voice yourself?
Yeah, um, I did it on the demo, right?
With my friend and I like it, but I
rarely use it because
to me, while I type, I formulate my
thoughts and when I speak, I will
misspeak and then I have to go back and
delete part of the Oh, that's not clever
cuz like I think some of them are kind
of clever enough to like
I think it's a very fine line.
>> respond or something. It's a fine line
between being like clever and and like
shortcutting and clearly seeing where
the user what said something and they
meant something cuz sometime it
misunderstands me because of my
beautiful accent and then it comes up
with this weird word and then it tries
to make sense of it and then it would
delete half my sentence, not Claude
code, but I've seen that happen in some
other tools and I was like, okay, Yeah,
the dictation is always one of those
things that never quite works very well.
I mean, like I was just thinking
yesterday like my as try as try as I may
to like optimize my my workflow. Like
when I when I write my to-do list I'm on
a piece of paper in the morning that
that just seems to work the best for me.
Like I want to dictate some ideas, but
and then they they sort of get lost in
voice recorder or they never
they never comes back out. So does that
Did I answer your question about
>> Yeah, I guess you did, but like
blame
I need to figure this out myself,
obviously. Can you hit your table? Hit
your table.
I I
Wow.
I got it on like a stand now. So I
listen to the feedback. Thank you,
whoever gave me feedback.
Well, it bothered you and honestly, I
also noticed your camera shaking last
time. I
>> Yeah, so I fixed
And
are you going to fix your your headset
or you going to get a road headset or
something? Like a proper What do you
mean? Like a proper microphone? Is Yeah,
like the road stuff is is pretty good
from Australia.
Australia?
>> No.
Oh, the weather is so bad here. How's
the weather in Vietnam? Just tell me
it's
>> You're the third person who asked me
this.
Tell me it's beautiful.
It's beautiful. Um oh yeah, thanks for
sharing those Reddit links about the
trivia thing. I always thought to myself
that
that one day someone's going to do a
sneaky attack with the
get orphan commits because I even I've
noticed that myself sometimes that when
I'm messing around the branches and I
delete a branch, if you know the commit
hash, you can you can pull stuff out
from
from the nether. Yeah, it's crazy,
right? It's like even even very early on
in my career when I pushed my
credentials to the to get up like
everyone does.
Not me. I'm not a fool. Oh, yeah. Well,
not yet then. So um
So then you learn very early on that to
remove credentials from a get commit is
not sufficient to to just like go and
rewrite rebase
>> Actually, I've never done it. So what do
you What do you have to do? Just delete
the repo and start again or what did you
do? You have to rotate your credentials.
They are gone.
Even if you go and rebase interactively
and rewrite history and push that,
GitHub doesn't garbage collect those
commits. So there are bots that are
watching for for events and they will
find those commits and then they will
delete the data. GitHub does check for
credential stuff nowadays, I do believe.
Yeah, and a lot of yeah, even definitely
if you push a personal access token,
right? It will immediately go send you a
notification like we revoked your PAT.
>> Yeah.
Cuz you pushed it. God damn it. That's
what I I wanted you to share the PAT.
Well, I'm trying to actually work it
Have you ever worked with GitHub apps?
That's the thing I'm supposed to do at
work, but it's quite confusing and I'm
just wondering
>> Yeah, GitHub apps are my favorite
>> Something weird about it. way to
interact with the GitHub because so
there's yeah, I mean GitHub apps are the
ideal way to manage automation because
you can create them inside your
organization as an app, keep them
internal, and then you can install them
into repositories, and there's no single
like if you if you say you use the old
way, which is create a machine user,
which create an account into your org.
>> Well, basically a PAT. What do you mean
Yeah, you could give your personal PAT,
but usually you don't do that, right?
So, you create a dedicated
>> people create automation accounts for
it, but yeah, that's
>> but then you have to buy a license seat
for it. So, you have to pay for that as
a license seat.
>> Yeah. And then you have to like find a
way to share access to that account. And
you have to log in to access the PAT and
control like if you even use this create
what's it called granular Yeah, and make
a long you have to make a long-lived
PAT. It's very very yeah, annoying. And
then if you use GitHub apps, you
basically can dedicate app
administrators within the organization,
and they can then modify and request
permissions, and then they can be
reviewed and applied, which is way more
better than a PAT. It is better, but
like for some reason I think maybe it's
just there's some disconnect between the
powers that be who administer this
GitHub Enterprise that I'm
it there's some weird permission problem
and I can't roll the specific Yeah, I
wouldn't want to be like the first time
I worked with GitHub
PAT like sorry, not personal access
tokens, but actual proper apps. I had
full admin, so I could see everything.
And I wouldn't want or wish upon anyone
to try and roll out GitHub apps without
having visibility in like what that
Yeah, yeah. I I hate that sort of like
dark pattern when you log in and you
think you you can see everything.
>> tell any anyone to say go and do this, I
need this permission if you can't see
what is exactly missing.
>> Exactly. And really this is this is the
thing I'm hitting my head on when I
Yeah, so I would recommend create your
own GitHub app and your own account and
play around with that to understand how
it works. I think the most important
thing to understand is that a GitHub app
has an identity and it needs to use that
identity against an installation to get
a token for then doing activity within a
repository. So the app identity itself
does not give you any permission. It's
the identity plus the installation
identity, so the app identity plus the
installation identity together that you
need to use to get access to a
repository. That's the only way that you
need to get um a short-lived token
from your installation. And and that's
basically the hardest thing to
understand because you got a app ID, you
got a secret key, then you have an
installation ID, and then you need to
get a a short-lived token
to to be able to do things. And it's
like That makes sense. Spea- Speaking of
this sort of authentication, so another
another thing I'm I'm working on is like
we we have an estate with like 50-plus
accounts. And uh previously I was being
quite happy with AWS SSO, but I think
there's been some advancements. I I
don't know. Have you ever had the I
don't know what the new AWS switching
user experience is like. I mean
do you ever use multiple sessions and
switch accounts?
>> I'm I've I've found something that works
really well for me and I have not been
following what are the new ways to do
things. I used AWS Vault, which is from
9 Designs, but they no longer
maintaining it. And it was moved over to
another maintainer, which probably is
scary about the last
>> I mean you assume or something like
that? Which one? Assume? No. I've heard
about other There was one famous one
that even the AWS people were using and
it was something It was the idea of like
you you basically have your credential
management as a platform and then you
just allocate it to users and then the
users could just very quickly switch
roles. It was a service, but then they
became full paid. It was used to be
free. It's something like Jump or
something. Yeah, yeah, that sounds
familiar. That's
>> But I never used that. I used AWS Vault.
I've been so happy with it because it
can even run as a demon. Like if you
want to do a Terraform apply, it can
refresh tokens in the background. It can
um
it can do a lot of things.
>> I I guess when I I got I have a bit of
PTSD when it comes to Vault. You know
It's not HashiCorp Vault.
>> It's not HashiCorp Vault.
No, no, AWS Vault is a Go binary that
uses your keychain on my quest Darwin it
uses your keychain. But if you're on
Ubuntu, you can you have to have a a
secure backend and the most common or
default one on Ubuntu is pass. Uh sorry,
pass, p a s s. So that takes a bit of
time to set up. So I just have onboarded
someone a while ago and he and he was on
Windows and I told him set up WSL and
then set up Ubuntu VM and then use pass.
And there's actually a couple of steps
to like set up GPG and all that to to
encrypt your your secrets backend. If
you're on macOS, it's all keychain and
no problem. You can you know finger
fingerprint authenticated. It's pretty
cool.
>> Yeah, I
I've I've used
like that before. I was just thinking
more AWS native solutions. And then the
other Speaking of AWS native solutions,
I've always found like when you have a
bazillion AWS accounts,
like grabbing CloudWatch logs from from
all those accounts and putting them in a
one central place. That's usually There
is some sort of like AWS pattern for it,
but it's just a nightmare if I remember
correctly. Like you have to set up
Kinesis. I can't remember what you have
to do exactly. And then most people just
end up setting up DataDog or or and then
paying through the nose for DataDog. I
was just curious if you have a go-to way
of collecting logs from multiple
accounts. Oh, huh. You know what happens
when you set up control tower on AWS
organization, it sets up this really
advanced um log like audit account and
it sets up forward like rules
automatically like like the moment you
activate control tower, creates like you
have your maintenance account, it
creates your audit account, it creates
your security hub account
>> so it like sets up like a
well-architected Yeah, it immediately
like uh deploys bunch of cloud formation
stacks all over the place.
>> I don't think I have access to that in
my organization, so that's not cool, is
it?
>> Yeah. Um
and then it creates a centralized log
account and it and it sets up the all
the forwarding for it as well. I
remember that it I had to like figure
out how that worked to kind of disable
things because it was costing money and
like I I don't know there were some
things, but like I was impressed by the
amount of configuration it did to
basically fetch all of the logs and
across all of the organization accounts.
It's not trivial when you do it
manually, I must say. I've I've done it
in the past and it was pain. So when I
when I onboard like when I have
Terraform onto an org, I usually have my
Datadog integration module, which I it's
a wrapper around the cloud formation
stacks that Datadog maintains because
they don't maintain Terraform modules,
they give you a cloud formation stack.
>> Oh yeah. So what I did is I parsed out
the stack the stack variables into
Terraform variables and I can invoke the
module and control all of the services
that are enabled on the accounts like
disable everything by default explicit
opt-in because if you have a Datadog
enabled on your AWS account
>> to pull the metrics and it starts to
scrape everything and then you get a
huge AWS bill because Datadog constantly
keeps talking to those endpoints.
>> One thing that we ran into work is that
the the SLA or whatever for for the
metrics is actually really slow. I think
they like it's about 15 minutes by
default or something. So like if you
have it in production That's CloudWatch.
That's like the default AWS um uh you
have you can you have to go like you
have to go and say I want more
aggressive, but then you have to
acknowledge that it will cost money.
>> Yeah. So the 15 minute one is default.
It's not a DataDog problem to be clear,
but it's but it ultimately when you set
up DataDog on on AWS you see this
problem often like why why didn't we see
this problem earlier? Like well, this is
the default I'm afraid the 15 minute
lag. Yeah. Yeah.
Yes. Okay, so so basically what just to
summarize what I set up a multi-account
set up for AWS you probably you want to
set up control tower to bootstrap it and
then I mean if you're happy to live with
that control tower structure. It
depends. So I mean it will it will set
up your security hub account. It will
set up like Yeah, all the defaults.
So that everything gets aggregated like
your your your findings from all the
accounts get aggregated in like a
centralized dashboard. That's really
nice. I mean that's I haven't done that
for the last three or four years. Yeah,
I'm done with it so I don't know what
where they are at now. What they are
doing now. Like like many things
in the cloud is that cloud sprawl and
then every every organization I work on
it like it gets a bit crafty after a
while and I just I just wish we could
have
a firm all accounts and or firm all
organizations where we just
move everything to new org but like I
guess well, also at the same time I've
worked in a number of enterprises that
everyone is absolutely allergic when you
say something like migration. Everyone
goes no, no, we're not migrating
anything. This is a funny thing when I
had discussions about platform with some
people and they were like okay, so next
month we'll finally have everything
nicely like migrated, right? And we'll
have everything in order. And I'm like
most likely not cuz cuz there's like or
there's this discussion about like we
want to go towards this, but we're in
the middle of a migration. So if you're
going to do this it's going to like
cause a migration on top of a migration.
And there's this tendency to be like
don't do that. We going to wait. And I'm
like, it's always like that. Like,
you're never done. There's always
something.
>> a thing that I wonder if there's like a
culture in an organization which says
like that's insane enough to go like,
"Hey, we're moving from AWS to Azure.
Like, and we're doing it over the
weekend. And all our data is uh 2
terabytes of whatever our data is going
to be moved." And then And then every,
you know, week or month they can almost
do that again and again and again
because ultimately people get calcified
in whatever hosting platform they're on.
And they never can move.
>> DevOps mantra? Like, if something hurts,
do it more.
I've never seen it myself. I've never
seen it myself. But like, if you don't
like, just the practice of restoring a
backup and testing a backup. Like, how
many people restore a backup? It's so
rare in reality, in my opinion, um that
people have the rigor to just Just
shouldn't say that out
I know. Hopefully no one at work is
watching. Hopefully no one at work is
watching. No, like I was I mean, because
the cloud gives you so much sense of
security, cuz we're not doing the
on-prem data backup tapes that we have
to like constantly restore. We kind of
trust AWS backup to work. But they have
very clear like well-architecture check
marks. It's like, how many times are you
restoring your backups? How many times
are you restoring a backup in a
different region? How many times are you
That's all part of the checklist if you
want to do the well-architected. They're
very very very long checklists. And I'm
sorry to think like a cool startup up
with idea would be like validating that
restore process, you know, like There's
quite a few.
>> Like, when my my friend was going
towards becoming like an AWS partner
with with an organization, you have to
pass those all those checklists. And
there was a company he found that that
did everything. So, you just hook it up
to your account. It goes and find any
vi- violation, and then deploys a cloud
formation stack to like rectify it. And
then he said the moment you stop paying
or deactivate the account connection,
all of the cloud it rolls back all of
the cloud formation stacks. Like you
can't use the the platform to like
become the compliant and then disconnect
it because it kind of like rolls
everything back and you lose all of the
things. So he was like, "God damn it,
like I spent so much time." Because he
was trying to reverse engineer the
stacks and snapshot them so that he
could redeploy them and things that I
don't know.
Just to do it quick, but like there's
quite a few companies that That's kind
of a dark pattern. I mean
Maybe with AI today would be so fast to
do this kind of thing.
>> exactly. Like you could give a role to
this restore okay company called restore
test and maybe a subset of the
architecture and data somehow. Just It's
very very difficult with AI now. Because
when when you look at disaster recovery,
there's so many different strategies
that you can use that you can do
depending on your RTO and your other
like requirements. Every service has
different ones. Like do you want
point-in-time recovery? Do you want
constant live streaming? Do you want
complete failover? Do you want pilot
light? Do you want to pay how much you
want to pay?
Yeah, so so AWS has these white papers
because I had to do disaster recovery
setup and they have a very detailed
breakdown of like four strategies. Pilot
light, live like Yeah, yeah, yeah.
active passive type of setups. And
there's so many scenarios they meant to
protect with and the white paper is
basically don't choose any strategy
unless you have well defined your RTO,
your RPO, whatever. And also what are
your attack scenarios? You can't build a
disaster recovery plan if you don't
define what are the things that you're
protecting against. Is it a hack? Is it
an intrusion? Is Yeah, you need to do
some threat modeling.
I love I love this thing that I just
talked about.
>> That's the white paper. Yeah. But like
what I'm trying to say here is like
restoring stuff
test a lot of that is quite tedious,
right? If if AI can do
can run through this exercise without
people manually cuz like no one wants to
do a restore because it's boring as hell
and tedious.
But it was actually a very fun exercise
when we did it. It was really fun. Well,
you you enjoyed but like that's probably
just a one-off thing. It's like you
know, I'm doing this for the first time.
Yeah, but it's very important because
when we did it we're like we had a plan
and then when when it happened we we
simulated it in in non-prod but then we
actually did it in life as well.
Um
it was like
Uh yeah, it's very clear what my plan is
but then when you're actually doing it
there's this all additional stuff that
were like, yeah, but am I absolutely
sure? Oh, wait a minute. I need to
double-check that. Oh, no, this process
is actually very cumbersome. Like it was
designed like this and I have to do this
thing but it wasn't clear to me how to
validate it. So doing it live, even if
you have completely automated backup
restore procedures,
um the actually validating the the plan
and the buttons you click is so
valuable. Yeah. Like if if we could just
if in light of this AI error that we're
living in right now, if you could just
list all the tedious things that we've
that we couldn't really automate before
like backup restore testing. If we could
just list that and now and basically
prioritize them in a sense cuz now now
these things are possible without dying
of boredom to basically get AI to
validate your backup or to bootstrap
your your AWS account. I mean, that
let's be honest like bootstrapping an
AWS account or organization with control
with with the with control tower, did
you say? I forgot the name of the
product already.
>> tower from AWS.
>> I mean, can it even be driven by
automation actually? These are the
things that
>> So this is where I'm not up to date
because there's many ways to bootstrap
your AWS account. I I mean, if you put
in place or your organization, if you
put in place control tower, you already
have like a first step but then you can
also hook it up with way more things
like control tower has landing zone and
originally when I tried it out 5 years
ago it was horrible. It was using
because the account provisioning was so
it took so long so much time and you
couldn't even specify what's the VPC ID.
Like imagine that you have this
requirement that every VPC ID uh cider,
sorry not ID but cider, is separate
because you need a transit gateway to
connect everything or interconnect. You
couldn't do that with control tower
landing zone. You you had to define the
cider lanes of the VPC that was going to
be provisioned in advance in the
template. You couldn't say like when the
account gets created go here and
allocate a cider range and then create
the VPC with that cider range. I can see
there's a lot of some automation gaps.
Like one thing that's always frustrated
That was 5 years ago. I'm pretty sure
they already fixed some of that stuff. I
made a video about this. Like just
closing an AWS account.
They also fixed that a lot.
Before it was not possible there was no
API but they added APIs and everything.
>> Yeah, now they have APIs but still
there's like a very like it's weird.
Like the the account is like Yeah, the
recommended practice But it's still
there for like at least a month or
something.
Yeah, the recommended practice for years
has been you keep a pool of accounts and
you you allocate the account and then
you release the account once you're done
with whatever project that needed the
account. There's even a whole
release the account open source. Not
deleting it. No, no. So you keep a pool
and you have a
basically you allocate the account to a
project, provision it and then when it's
done you basically new call of it and
you release it back to the pool. So you
never destroy it. The account never gets
created or That's been There's an
There's an open source framework that
does that for you. It's been around for
like I don't know 8 10 years and it's
still one of the most popular ways. Like
I talked to someone he says like oh I've
been working on like account pool
mechanisms like just a month ago and I
said oh I remember there was this
framework he said yeah that's probably
the one I'm using and it's still the
same one. But what's Why not just nuke
it and create a new account in the org?
I don't understand. You just mentioned
the account stays around so long. I
mean, it's always been problematic to
close the account. And Okay, so this is
kind of like what I'm working on.
>> Yeah. Yeah, okay. Okay. By the way,
yeah, thanks for sharing that thing
about State Graph. It does look I'm
really I'm really impressed by State
Graph's
landing page. It's so clear what what
it's doing and how to use it and what
the what you need to do to start
you know, how advertisers value and
things like that. Like This is really
good. Yeah, they they've been doing a
lot of demos
talking about
what they're trying to achieve. I was
surprised that because I thought the
whole idea of State Graph was was to go
towards a Terralit so that you can
resolve all of your dependencies
within a single state, but they added
multi-state support. And I was like,
"Oh, it seems you changed your mind."
They said and they said, "No, no, no,
that was always the plan. The ability to
have multiple state within the
individual state and and be able to plan
everything without mocks because if you
do that in Terragrunt, you're going to
have to specify output mocks
and things like that.
Have you given it well? I mean, you
mentioned it cuz I mean, I'm guessing
that they want it released, right? It's
There's It's been
It's been released.
Like last week. Okay. The blog doesn't
seem to reflect that, but whatever.
>> because there was a release. I think
they released it just before QCon, but
like I don't know if they made it open
source, right? They they said that they
they couldn't figure out the licensing.
And this is by the way some very
interesting article I saw today this
morning about today there's this
website. It's a joke, I guess, and I'm
not sure it's for fun, but what they
what it does is like
don't like the Pesky license? Just let
AI do enough change so that you you make
it look like it's a different project,
but it's and you can relicense it under
your own license that you want. So, they
said this is like in the enterprise
today, you don't want to get affected by
AGPL a fair
uh GPL that basically is very
and GPL in in itself like whatever it
touches you're supposed to then open
source and so on. So people enterprise
are very much against it. So any project
that's GPL or AGPL usually is a no in
enterprises. So you don't like the
license? Great. Here's a workflow. Just
fork it, make it your own license and
just run it how you want.
Well, I just get AI to re-implement
everything I suppose. Is that what
you're saying, right?
>> Yeah, I I can share the the link. It's a
very funny one because I was having this
discussion
um with someone. Basically, honestly,
there was this project and I can read
the source code because it's MPL Mozilla
public license, but I can ask an AI to
rewrite it in a different language. And
where do I where where do I need to keep
the license? How do I need to keep the
attribution if I'm basically rewriting
it in a different language like I guess
Yeah, I I've seen that.
>> AI is reading all the algorithms. I
mean, that that's assuming big things,
right? Because I mean, we've been
chatting about it. I mean, this is what
your spec ledger project is about in a
way. It's like I don't think people have
figured out how to ship with AI, really
have they? So like You think that the
ideal pointing an AI agent to an
existing project and asking it to
rewrite is not I mean, it's possible.
It's it's it's bold talk. I mean, it but
it's it's talk. You want me to share the
link?
>> I don't think anyone's really done that
yet. I mean, it's just hypothetical at
this point, right? At the risk of
causing a
shitstorm, you just showed state graph,
right? So let me just share my screen.
And that's not even open source. Maybe
it's Hold on, screen. Entire screen.
It's going to It's going to You can see
it, right? You can see state graph,
right? Yeah, yeah. I was showing that
earlier.
>> So you see this the the landing page.
Yo, yo, yo, yo. With like sequential
allocations versus multi-state and uh
see the difference. It's all SQL.
>> I just love that landing It's so good.
Now look at this. Well, he changed it.
Maybe somebody complained. Well, first
what I like what I think is funny, I
didn't even realize you need to do
squares with the little dots inside.
Well, there's dot pattern.
I thought this was very funny. Oh, this
one is very similar.
>> cop- copying a He changed it a little
bit. Before it was way more
>> page is one thing. Copying
an actual
uh Oh, no, this thing works.
This thing works. It works?
>> Yeah, yeah. I actually integrated like
the CDKTF and you see stack ledgers in
there as well. So, I actually did the
CDKTF test generator and then I run all
of it all of these um sample So, what's
your relationship with this project? You
just know the guy or something?
>> I because Okay, so this this is
basically doing the same thing as State
Graph, but it's in Rust and it's open
source. And it's just like he just
>> What? The idea is very interesting
because he just re-implements the config
parser reading the original Terraform
configuration and then generates the um
and then invokes the Terraform providers
directly. So, he implements the the RPC
protocol that Terraform talks with the
plugins, right? So, you have the Go lang
Terraform talking to the protocols and
um it's
it just invokes the Sorry, the Go lang
execution core of Terraform.
Instantiates the providers and
communicates to them with the protocol.
So, this thing does the same thing. It
It instantiates the protocol Sorry, the
providers and then it communicates to
them through gRPC. So, every CRUD action
that you need to do for every resource
configuration just initializes Yeah. It
just calls out straight to the provider.
>> Okay, but there's also like a database
aspect to it. Yeah, so this thing is
like State Graph. It basically uses
Postgres as the uh SQLite as a as a
state storage. So, it doesn't support
like It doesn't do the remote backend
setup that you can do with S3 um or or
other of those
>> Is this
plagiarism or I'm not too sure how to
Yeah, this goes back to the to the
original question. Like this is not even
looking at the source code. This is just
looking at the at the landing page and
duplicating the capabilities. And when I
talk with him he says the main reason is
because they didn't have a plan to open
source it. And I really like the idea.
And I I didn't think it was that hard to
build. And if you if you can tell it's
all AI. And you said like all that's all
like bold bold talk about AI rebuilding
products. It's a fact, okay?
You can rebuild a complete product. Have
you Have you Have you tried this in
anger? Is this going to work?
>> I did I did work it. There was one thing
that I discovered and I'm not sure if
they fixed it.
>> Have you actually compared it with State
Graph? Oh no. I I Okay. I mean, honestly
State Graph's been worked on in private
in in in um
how you call it stealth mode since
September. Okay, so a team's been
working on that for almost half a year
or 8 months. I cannot tell that this
project which is maybe 1 month old or
like a month and a half old is anything
comparable in terms of like stability
and features. But the fact is that it
works. I was able to clone it. I was
able to use it with Terraform JSON. And
I was able to Terraform sorry, Oxid
apply my Terraform configs. And it was
creating against the Terraform provider.
It was doing all I guess I guess this is
the difference between
a commercial product and something that
you can just clone out and give a whirl.
Yes. So so so you you are right in a way
that you that like that is a pattern
that people often say. I think there was
a post about it recently, which is
people think that they can just rip it
and then move away from the
well-established and tested library. And
they replace it in 30 minutes. And then
they spend months fixing bugs and and
basically supporting it. Yeah, so So
this is the thing I wanted to because
you you diverted me. You said it's it's
it's all it's a bold claim, but it it
holds no reality. But this is the the
the website which is I think a bit of a
a joke.
Um and honestly I think they use the ant
track thing.
>> News. I think I read it.
>> Yeah, it's funny how they like put a
[ __ ] smear on it.
>> it is slightly I I mean
It's hyperbole.
>> I actually want State Graph to succeed,
but like when I look at the pricing,
when I think about the enterprises that
I work with, it's really hard for them
to adopt um
a proprietary product. And uh what How
do you pronounce this other project?
Oxid or something?
>> Yeah, Oxid. I should put the State Graph
on on here, like I think. Like Oxid is
actually even though it came later and
re-implemented State Graph or something
like that,
um
it actually stands a a lot better chance
of being adopted because
because people the you know, the
engineers on the ground can just clone
it out and try it out. There's like
So the person that built Oxid, um he
told me that he has a couple of like
fintech clients that are looking to roll
this out across their organization
because they have such a big Terraform
state and Terraform is extremely slow.
And they are seeing like a 10x or you
know, 20x improvement Yeah, I mean I've
had this problem so many times and this
this The trouble is is that none of my
my projects in GitHub
um I don't think will give this a proper
whirl a whirl, like you know, like no
one has no one has a private project
that that replicates
um you know, an enterprise.
You got rid of the ant track thing?
>> up, do they? Unless I have my home lab
my home lab somehow expand dramatically.
So one of the things that was missing
was this the the context exposed to the
the parser, the language parser was
missing these internal mod like um
path uh modules for this type of stuff.
So, I identified there were several like
in in the context it's missing the path
module, the path root, the path uh
current working directory, and the
workspace name. So, those were a couple
of things.
>> hate those those sort of like And this
also was not implemented, but again, I
opened this issue February 18th. So,
maybe they've already implemented
because I don't think anyone can use
this if you don't have these functions.
Right? You need these functions like
file, template file, dirname. Like these
were all missing on the initial version.
So, I'm not sure if he actually fixed
that. I haven't really looked at the
latest commits. Yeah. And Taco's guru,
you're going to have you said you
mentioned you're going to put
you're going to put State Graph there.
You're going to you're going to put
Oxide there, too? Yeah, oh, another
thing, he has a platform called Ops Zero
AI, which is like an agent um
driven um like for Terraform. So,
uh he said that he already has Oxide as
as a back end, like an execution engine
integrated there. So, this is new
website again. My gosh.
>> So, he when I told I showed him Taco's,
he said, "Why don't you put Ops Zero on
there?" Because Ops Zero is also
providing that. So,
so I think both Ops Zero and Terra team
Sorry, uh State Graph should go on there
uh and show like the cost. Cuz like here
it says 10,000 resources at 12K.
Um no idea about like what's the
Is there any because the licensing is
usually on either the team size, right?
If you reduce it, you can see Open Taco
is by user. So, if you have a large
team, Open Taco quickly becomes way
expensive uh most expensive if you go
over 50 users. And then you have the
then you have the like number of
resources. So, in my organization we're
around 25K.
If we if we look at State Graph pricing,
they they say 10K. So, I'm already at
like 12, which is what $1,000 a year?
$1,000 a month is not that bad. I'm just
looking at your your tacos guru repo
here. Like you must
I mean you said it was AI generated. I
mean what you have or you have the
evaluation criteria. Yeah, it was fully
vibe coded. Zero spec ledger even. But
can I take over the presentation here?
Yeah, yeah. So I just I think I'm a
little bit confused here.
So you basically rate the fact that it
has no Kubernetes requirement as higher.
I thought you
>> You know what? Yeah, right? No, why? Why
would you need a Kubernetes cluster to
run your tacos? Yeah, yeah, yeah, but
it's it's confusing in the sense that
you might you might use your terra you
might use Terraform to bootstrap
Kubernetes. I know what you
>> Yeah. So you got like but like someone
might look at this and say I
>> Yeah, yeah, yeah, no, no, the way that
it's formulated it's a bit weird. Um
but the idea is no k eight means
basically disqualifies uh I think cube
terra terra cube, which is a which is an
open source that you can run on but you
need Kubernetes, right? You can't you
can't run it without Kubernetes.
>> this is this is quite opinionated. Like
actually on the topic of Why would you
need
Yeah, I I I get you. I get you. But it's
No, that's why the weights are are open,
right? No, no, no, but look look if you
don't agree with this weight, you just
reduce the weight. Yeah, okay, okay.
That's why the weights are all
configurable. Really? That's clever. You
go to tacos guru. If you don't agree
with the weight of this particular item,
it's opening you're only saying it
Oh, so it's telling you in the window.
Yeah. That's really clever. And it's
it's affects the the order, right? If
you say like for example The order
doesn't seem to change here in the top
right.
>> tofu. Change open tofu weight to zero.
Let's say that you don't care about it
being Terraform or OpenTofu. Immediately
see ACP Terraform go up. Then change CDK
TF to zero. Again, CD ACP Terraform goes
up. It went up three three spots.
Terraform went down one.
Yeah.
>> You need to put this in the You need to
put this as a arguments though on the on
the URL.
>> if you go on mobile, it tells you a
little bit more.
And even also on the I really like that.
Did you get this Did you come up with
this idea yourself or you you copied
another site? I think I need to credit
all my ideas to my principal architect
in in in uh
No, like basically what happened is I
needed to evaluate platforms to migrate
to and I had to establish criteria that
mattered to me. So, based on my story, I
told Opus like, "Okay, this is how we've
started. This is how we evolved. This is
what we have established. This is what
matters to us. CDK TF we use heavily, so
it's important." And it came up with a
bunch of criteria for me, right? And
then it came up with the with the
formula to to calculate the score of
each platform using weights. I think I
might have suggested to use weights,
right? And then I I I presented a
markdown document with with the weight
calculate like the weights to each
criteria and somebody said it would be
nice I I said to them, "Do you agree
with these weights?" And they said, "It
would be nice if we could change the
weights and see how it impacts the
score."
>> Nice. Nice. And so I went back to Opus.
I said like, "Hey, we want to make this,
you know, an Excel spreadsheet is going
to suck. So, I'm going to like you might
as well build a website and then and
then came up with all of this."
>> Well, I love what you've done here. It's
really good. Uh
the I we we I remember the like
we play these agile games back in the
day when you were in the office, you
know, play points. Uh where you would uh
poker sort of have sliders in a in a
sense. You'd ask your client, "How
important is this to you? How important
is that to you?" You should see what
what Opus did with um ThoughtWorks
um Tech Radar, by the way. I I gave it
the original Tech Radar open source repo
and I told it I have analyzed my
internally technology, right? I I I gave
it access to look at the GitHub. I
haven't posted it uh but I kept it
internally. So, and then I said, "Yeah,
but this is very nice to see the Tech
radar and why the dot is there and we
can click on it and we can see a little
bit of a discussion why this particular
technology is here because we haven't
implemented across most of our
repositories." Cuz you're on a VPN. I
don't know. Okay, that's a problem.
You're you're planning your your holiday
to to Spain. It's definitely an
Okay, that's a bug.
That's interesting. So, so basically I
gave Opus the GitHub repository and then
I was like making it to give a
breakdown. So, whatever tech radar item,
there's several criteria for this item
to be there like in that in that in that
band. And I wanted to show that
technology being used in these projects
and these repositories. So, it created
another page. When you click on any dot,
it will then show like the bat like the
number of reports this this technology
is active in.
Cool. Cool. Cool. Cool. Opus again just
expanded on this with the ability to
give like a slice and dice kind of
experience of
Yeah, I like what you've done there. I
should probably use AI for more
visualizations like that. And it's hard
to do it like you can try to create a
slides or you can try to create an Excel
spreadsheet or you can try to ask it
with the color X color draw to to draw
some diagrams, but the real power, even
the one that you shared earlier when you
look at like
and we never we only did the top row of
like common agent like bad practices,
when you hover every single one of them
gets like a little interactivity. Like
this type of stuff is so much easier to
build with um
with with AI. Yeah. Yeah, I need I need
to build something like this.
Uh
for I mean this I feel like
when I interact with my open claw, half
the time I'm just I'm just asking like,
"Should I buy, you know, this product or
should or is this product better?" And
half the time when I talk with AI, I'm
trying to evaluate some options. So, it
would be great
to
have AI just flesh it out a bit. I
didn't This was also not my original
idea because the
um
Brickman, Evgeny Brickman from Grant
Works, he posted a blog on uh on
LinkedIn saying that he was evaluating,
I don't know what it was, uh different
backup uh storage arrays for his home um
system. And he was waiting them against
pricing and features and whatever. Uh I
don't I'm just making up what exactly it
was, but it was something like that. It
was something like backup solutions. And
he had said, "I I used AI to create a
little like a comparison chart and the
ability to modify like my criteria." And
And before I started on tacos.guru, I
did go back and search for his blog
post, uh which was on his personal blog,
and then look again exactly what it was.
But um
I didn't remember it exactly right
because it was a little bit more limited
than what I wanted. So, I did then just
take the idea and and and build uh
tacos.guru. So, so yeah. Yeah. Uh quick
side note,
if you were to if you had the
requirement from a client to to build
out Kubernetes, would you use Terraform
or to Yeah, I think so.
>> bootstrap it? Yeah, yeah, yeah, yeah.
You wouldn't You wouldn't go CDK? I
mean, CDK is better fit for AWS
constructs, right? AWS
uh
services.
>> I have to say
>> There's the EKS blueprints repository,
which uses AWS CDK,
but it was built before the Kubernetes
CDK. So, there's a CDK for Kubernetes,
and it automatically creates like the L1
bindings and then creates L2s, and it's
really nice.
Um what that means is that somebody in
the EKS blueprint repository kind of
wrote some TypeScript interface to
represent a Kubernetes manifest, like
what are the mandatory fields that you
need to do. Um which is very rudimentary
and that's how they they bootstrap EKS
clusters with like very rudimentary um,
representation of what Kubernetes
objects are like compared to CD gates
which automatically
builds uh, a type script representation
of every open API open API schema in
Kubernetes. Okay, so this is this is
fully EKS but but you would still use
Terraform, right? Or Yeah, so I I I
really like CD gate for Kubernetes to
manage my Kubernetes deployments but
like not many unless you're in the CD
gate ecosystem it's probably not worth
it, right?
Oh, okay. So you use this to manage your
deployments but not but not the
>> Not the actual bootstrap. So what I did
3 years ago was I implemented
um, clusters
using some of the EKS blueprint patterns
which is like they have like builder
pattern and other patterns to to to
build with AWS CD gate. Um, but I didn't
feel it was such a great experience. It
became it was not that good. Um,
Terraform modules are very well
established like the Terraform module
for EKS is is well established. Um, I do
not like it. It's way complicated. Um,
but it's way more widespread. I mean
>> Okay, so so if you're deploying
Kubernetes on
on AWS you would use this Terraform
thing. I mean I I wouldn't, right? I
mean I would I would land in a team that
does it and I would tell them I think
I'll I think I used this in the past and
I thought it was
really heavyweight but It is very
heavyweight and and it's impossible to
support every because you know, EKS team
keeps adding features like now they have
completely managed add-ons.
Yeah, I'm surprised you
you you didn't gravitate to using CD
gate for this. I mean it's just I guess
it's just an experience. My goal with
Sarah constructs was to port EKS
blueprints for AWS CD gate to Terraform
CD gate to get rid of the stupid module.
So, my goal was to build and I have
already got EC2 instance profile.
Um I've got load balancers, everything.
The only thing I don't have I have auto
scaling group also, but I don't have
um
No, I think I don't have auto scaling
group. And I need auto scaling group to
do um self-hosted um configurations.
The thing is the AWS CDK L2s for EKS
depend on every single EC2 um every
single load balance like it depends on
so many other modules that to get to
that point, I have to port everything
else.
Um but I told you that I like
>> with the CDK, surely. If all these Yeah,
I think so. I I think so because
honestly they it's way more intuitive
and it's where the the EKS product team
is probably contributing. Like they
they're adding the features into
CloudFormation and then they're adding
the feature into AWS CDK. And then
somebody from the community needs to add
it into Terraform AWS EKS, which becomes
more and more bloated uh and is an
absolute nightmare, but for some reason
still has 5K stars. Yeah. Yeah, I mean
he's This guy's done a few things. I'm
trying to remember all the things he's
done. Um he wrote um the three command
one which which is which is a which is a
great example of why you should use CDK.
Exactly. And the thing is when when he
was doing YouTube streams on on working
on this Lambda module, I showed him CDK
for Terraform. I showed him Terra
Constructs. I showed him complete
bundling solution with complete IAM
permissions preset and he said he really
liked it, but he wouldn't spend any time
on it.
Well, I think he's made a name for
himself for Terraform. Yeah, he also
shared he also shared Terra Constructs.
>> Serverless TF as a newsletter, yeah.
Yeah. No, this was Yeah, he was building
Serverless TF and I was like, "Why would
you do that?" And then he he built
Compliance TF, which I think makes
sense. He He's He's built it for a year.
He wants to like 12 or I don't know how
many conference talks. It's to you you
need you can select what sort of
compliance you want and then it's going
to bootstrap the modules uh his
Terraform modules and then it's going to
apply the compliance rules and it's
going to make make sure that they are
compliant to your compliance
requirements. This actually sounds
useful for at least one thing that I
used to work on. Yeah, so for me
personally
I feel this is a waste of time and you
should you should use aspects and you
should use CDK aspects. Okay, oh yeah.
Oh actually that's one thing I need to
work on today. I need to I need to prove
that that some level two level three
constructs were deployed in certain
stacks. Oh yeah, you asked me last time.
Yeah, I think you we talked about it but
uh so if I can if I can produce a bill
of materials to say that this construct
is here, that's what I need to do. I
mean compliance half the time it is
about
giving a nice list of a bill a bill of
materials as I like to call it.
I mean maybe I'm but I am
I am uh what do you call it Trivy
fuzzing everything here but yeah, a bill
of materials I feel is is the critical
concept. Okay, cool man. We didn't talk
about a Trivy hack. We did. I mean You
mentioned about the fork. No, we you
just quickly short mentioned that you
could access fork. I mean that's how
that's how uh Unleash itself. I mean Oh
man, it's insane to know. Sorry, I
I can't really share
WhatsApp, can I? Do you do you have a
link to that? Yeah, maybe we can at
least put a link to it. Yeah, so the
most interesting
um
I I I should put the link there but the
most interesting breakdown I mean first
of all was three three
events, right? The first one at the
beginning of March where the
claw bot like the hacker bot, the bot
that was like a claw bot that was
configured to find vulnerabilities
basically
um proved a vulner like a problem with
the way that the Trivy GitHub workflows
were set up. I mean it it got some
compromise into the Trivy set up, right?
That was the first event.
Um
They had to like erase
maybe the cloud bot hitting five or six
different targets was to to hide the
actual Trivy
um foothold because I think at that
point they got a foothold, they got some
credentials, and they got um access to
Trivy, right? And um
Trivy had Trivy had nuked the repository
and had to re- re- re-enable it. Like,
Trivy was gone. The GitHub action was
gone for for a few days. And or and
people were like, "Oh, my Trivy things
are failing." All of it's because the
repo was nuked and things like that,
right? Oh, my god. Yeah, and then the
second thing that happened is people
started to realize the binary had
compromised, and it started stealing
secrets. So, anyone who had run the
Trivy action had exfiltrated their
secrets. Like, it had several paths to
identify Yeah, but the people don't
understand the impact because the Trivy
binary and the GitHub Trivy GitHub
action and that scanner live in the
Datadog agent, live in the Go Teleport
privileged access management repository.
It is so widespread across the community
that all of these repositories should
step forward and highlight how they were
affected by the hack, if at all. Yeah, I
mean, I don't want to mention names, but
I know some big enterprises use Trivy,
and I just don't know I don't want to
even think about what the hell's
happened there.
Yeah, so it's Luckily, I did I did I did
search around GitHub to see if anyone in
my org was using it, and no one was
using it. But, we don't use containers
very much, thank god. It's Trivy doesn't
scan only containers. It was a
replacement to Terraform scanner
uh after Terraform scanner was acquired
by I don't know, was it Snyk? Well, I
just did a I I did a GitHub search for
Trivy, and we don't use it, so. Okay.
So, so it's used for IAC scanning, it's
used for virtual machine scanning, like
AMIs. You can run on your Linux Linux
machine for file system it's downloaded.
So,
>> I have always wanted to deploy it.
Uh but I've used it.
>> Everyone's been quiet now. Like, no we
don't use Trivy. No, I I I wanted to
deploy it and it's I have it on some
Jira tickets where like we should use
Trivy for this cuz we pay for snake and
crowd strike and all this stuff and then
and then I'm like, why, you know, it we
can get Trivy. Trivy is good.
But like, okay, let's let's run through
this a little bit. So, okay, say there's
a GitHub action and it steals
um secrets inside the GitHub action flow
where the Trivy scan is happening. I
mean, usually A, they're short-lived and
B, like what would you even do with
these secrets? Cuz a lot of the secrets
like what can you think of a well,
something that it's looking for like an
Anthropic API key or
>> mentioned that you still use PATs. Yeah,
but aren't PATs Oh, yeah, I on a GitHub
in GitHub PATs are pretty long-lived
usually, aren't they?
>> Yeah.
And then the second thing that what I
see that that actually connects very
well what we were saying earlier with
GitHub because
>> PATs. PATs is like a hole in the system,
isn't it?
>> Yeah, but also I don't think
using GitHub apps
like the way that I see GitHub apps
being used mostly
is you use the app ID and its secret key
to get an installation short-lived
token. That means
but that's probably in a separate job,
you actually do run
a short step where you get the
Oh, where
>> Sorry, I think I need to Sorry, my kid
is call- calling me. I have to take you
to school. I think mommy says you have
to take you to school. Deflection. I
think my wife told me Deflection. Okay,
I think let's let's end it there. Sorry,
I got to take my kids to school. Just
finish your point and then and then I'll
close the call. I wonder how effective
Oh, I think I know what you're saying.
The PAT becomes a GitHub token. Is the
PAT still exposed? I think you're you're
to say. No, the GitHub app the GitHub
app
requires a long-lived secret, also. So,
I don't know how you like you need a a
the GitHub app key to get a token. So,
you need to have a key exposed somewhere
into a workflow that gets a temporary
token, and then you can use that token
to do things. So, you need to really
segregate those two, right?
Anyway.
So, so it's still dangerous. It's still
like still easy to shoot yourself in the
foot. If in the same runner, you are
getting the key, getting a token, then
if that the key is still in that same
runner, something could accidentally
trade it. You should really have like a
separate job, a completely different
runner, to get a temporary token for for
the GitHub app, and then feed that into
your job that needs to do the activity.
I think the I mean the risk the risk is
here is that that someone can can get
into your GitHub account and go crazy, I
suppose, given a and can can use the
GitHub app
permissions to do whatever the GitHub
app
>> most people most big organizations have
approvals and things like that.
>> GitHub apps usually yeah, they're very
they they are very fine-grained tokens,
and every single permission that the
GitHub app has is way way visible. Like
it's not like a user that creates a I
assume you mean GitHub apps, which is
really hard to set up for some reason.
You should be using GitHub apps. Yeah,
it's it's definitely a lot more secure.
>> We all should be using GitHub apps. And
apparently you shouldn't be pinning your
commits to to your GitHub actions, cuz
that was a long long time established
pattern, and that's how they got in as
well. People are now like, "Yeah, you
can't use a sha because the sha can be
pulled from a fork, and then you can
compromise the repo." That's the second
problem. Anyway.
Yeah, okay. Well, anyway, great speaking
with you, Vincent. Yep.
We'll catch up another time. See you.
>> Yeah, yeah, yeah. See you. Bye.