WEBVTT

00:00:02.440 --> 00:00:05.120
Well, this is episode 27 of the AI

00:00:05.120 --> 00:00:07.120
infrastructure podcast.

00:00:07.120 --> 00:00:09.400
This is an intro coming from Cornwall, a

00:00:09.400 --> 00:00:11.080
windy one.

00:00:11.080 --> 00:00:13.760
And today I'm driving to London, to the

00:00:13.760 --> 00:00:15.240
big city

00:00:15.240 --> 00:00:18.840
for the AI engineer conference. So, if

00:00:18.840 --> 00:00:21.040
you're there Wednesday, Thursday, Friday

00:00:21.040 --> 00:00:22.600
in London,

00:00:22.600 --> 00:00:24.560
do say hello.

00:00:24.560 --> 00:00:27.160
Otherwise, I will be

00:00:27.160 --> 00:00:29.200
uh reporting back from that event. I'm

00:00:29.200 --> 00:00:31.320
sure it will be very interesting to meet

00:00:31.320 --> 00:00:32.360
lots of

00:00:32.360 --> 00:00:34.560
AI enthusiasts.

00:00:34.560 --> 00:00:36.120
And um

00:00:36.120 --> 00:00:38.520
enjoy the pod. I think

00:00:38.520 --> 00:00:40.240
it was a bit of a

00:00:40.240 --> 00:00:43.120
weird one where

00:00:43.120 --> 00:00:44.560
I was talking about the documentation

00:00:44.560 --> 00:00:45.960
problem again

00:00:45.960 --> 00:00:47.440
that AI

00:00:47.440 --> 00:00:49.880
presents.

00:00:49.880 --> 00:00:51.840
And uh we were just rambling about other

00:00:51.840 --> 00:00:54.760
things. So, hopefully enjoy it.

00:00:54.760 --> 00:00:56.200
Um

00:00:56.200 --> 00:00:58.680
if you didn't, comment below. If you

00:00:58.680 --> 00:01:00.320
did,

00:01:00.320 --> 00:01:02.400
like it and comment below.

00:01:02.400 --> 00:01:04.400
Otherwise, please enjoy the pod.

00:01:04.400 --> 00:01:07.280
Good morning, Vincent. Good morning.

00:01:07.280 --> 00:01:10.040
And it's Good Friday. That's nice.

00:01:10.040 --> 00:01:10.360
I didn't

00:01:10.360 --> 00:01:12.280
>> You're on holi- You're on holiday? No.

00:01:12.280 --> 00:01:14.920
No, I have to do some work. But it's

00:01:14.920 --> 00:01:16.840
okay, I have to catch up on some work.

00:01:16.840 --> 00:01:18.320
It's pretty interesting work, actually.

00:01:18.320 --> 00:01:22.440
Okay. So, I I'm just editing the uh doc

00:01:22.440 --> 00:01:25.800
that we share. I was adding some uh

00:01:25.800 --> 00:01:27.800
things to talk about. Oh, let me just I

00:01:27.800 --> 00:01:30.280
guess this will be edited out.

00:01:30.280 --> 00:01:32.040
Uh so, perhaps I I

00:01:32.040 --> 00:01:34.960
could we could start with a quick fire

00:01:34.960 --> 00:01:36.880
questions for you.

00:01:36.880 --> 00:01:38.240
I'm going to probe the depths of your

00:01:38.240 --> 00:01:41.200
mind. Yeah. Um okay, the the

00:01:41.200 --> 00:01:42.600
the first couple of bits are related to

00:01:42.600 --> 00:01:44.400
work and which I might have uh delete,

00:01:44.400 --> 00:01:47.160
but uh whatever. Uh sorry, um

00:01:47.160 --> 00:01:49.400
Claude Code is blocked. I need to be

00:01:49.400 --> 00:01:52.480
unblocked as soon as possible.

00:01:52.480 --> 00:01:55.200
Sorry. Cloud com- Claude Code comes

00:01:55.200 --> 00:01:55.800
first.

00:01:55.800 --> 00:01:57.440
>> Just it asked me something. For local

00:01:57.440 --> 00:01:59.400
compose test, so we should keep the test

00:01:59.400 --> 00:02:01.360
for a secret command

00:02:01.360 --> 00:02:04.600
being available, I believe.

00:02:04.600 --> 00:02:07.360
Okay. Um yeah, sorry. Go ahead. We

00:02:07.360 --> 00:02:09.399
talked about docs uh

00:02:09.399 --> 00:02:12.680
last pod and you recommended keeping

00:02:12.680 --> 00:02:15.280
documentation in the source code.

00:02:15.280 --> 00:02:17.840
And you showcased some pretty cool

00:02:17.840 --> 00:02:19.960
abilities. I just wanted to say that

00:02:19.960 --> 00:02:22.480
like at work, we did manage to convert

00:02:22.480 --> 00:02:24.480
all our Confluence to markdown, which

00:02:24.480 --> 00:02:27.080
was great. But then, once it was in

00:02:27.080 --> 00:02:29.640
markdown,

00:02:29.640 --> 00:02:34.000
uh it was too tempting to to to to use

00:02:34.000 --> 00:02:36.720
AI to basically improve the docs. And

00:02:36.720 --> 00:02:38.960
then, of course,

00:02:38.960 --> 00:02:40.320
Uh pardon?

00:02:40.320 --> 00:02:41.880
And last time you mentioned that Yeah,

00:02:41.880 --> 00:02:43.160
and then people created different copies

00:02:43.160 --> 00:02:44.880
with different type of angles, right?

00:02:44.880 --> 00:02:47.440
And Yeah, I wouldn't say co- We didn't

00:02:47.440 --> 00:02:50.840
go down the route of copies, but we got

00:02:50.840 --> 00:02:52.760
we got into almost

00:02:52.760 --> 00:02:55.520
a worse situation in a way. Perhaps it

00:02:55.520 --> 00:02:57.640
is the same as copies, but but since it

00:02:57.640 --> 00:02:59.800
was rewritten, we couldn't decide what

00:02:59.800 --> 00:03:01.680
the source of truth was because people

00:03:01.680 --> 00:03:03.240
were were saying like, "Well, no one's

00:03:03.240 --> 00:03:06.280
actually gone through all these 500 uh

00:03:06.280 --> 00:03:09.160
files of rewritten documentation. So, we

00:03:09.160 --> 00:03:11.680
can't like sign it off cuz it's just too

00:03:11.680 --> 00:03:14.800
big of of a rewrite. Essentially, we

00:03:14.800 --> 00:03:16.840
couldn't decide we couldn't like we

00:03:16.840 --> 00:03:19.480
couldn't call it a a success because

00:03:19.480 --> 00:03:28.040
because

00:03:28.040 --> 00:03:31.480
so, I guess it maybe is a is a story to

00:03:31.480 --> 00:03:32.360
to

00:03:32.360 --> 00:03:35.120
to say around AI where maybe we should

00:03:35.120 --> 00:03:37.000
just go slower or something like that

00:03:37.000 --> 00:03:39.440
just to get the buy-in. But at the same

00:03:39.440 --> 00:03:43.640
time, it's just so easy to to tell AI

00:03:43.640 --> 00:03:46.200
uh rewrite all the docs in different

00:03:46.200 --> 00:03:49.200
ways. And it works for one person, but

00:03:49.200 --> 00:03:50.640
it doesn't work for a team is what I was

00:03:50.640 --> 00:03:52.320
trying to say. Well, okay. Maybe you

00:03:52.320 --> 00:03:53.760
don't have anything to say to that, but

00:03:53.760 --> 00:03:54.960
I just wanted to

00:03:54.960 --> 00:03:57.680
Why didn't it work for for for a team?

00:03:57.680 --> 00:03:59.920
Because the team

00:03:59.920 --> 00:04:02.640
the the Confluence or the documentation

00:04:02.640 --> 00:04:04.240
that we have was was pretty extensive.

00:04:04.240 --> 00:04:07.480
There's like 500 pages. Yeah. And no one

00:04:07.480 --> 00:04:08.400
was going to

00:04:08.400 --> 00:04:11.040
Most of the team were were not prepared

00:04:11.040 --> 00:04:14.080
to like go through the documentation

00:04:14.080 --> 00:04:15.160
and

00:04:15.160 --> 00:04:17.600
and basically, you know, approve it.

00:04:17.600 --> 00:04:19.640
They just did they felt uneasy. Approve

00:04:19.640 --> 00:04:21.560
it because the changes were too too much

00:04:21.560 --> 00:04:22.920
like it needed additional review and

00:04:22.920 --> 00:04:24.360
approval of like the changes that were

00:04:24.360 --> 00:04:26.000
introduced. Yeah, but

00:04:26.000 --> 00:04:27.920
when you're in a big team like

00:04:27.920 --> 00:04:30.000
effectively, the whole team had to get

00:04:30.000 --> 00:04:31.160
behind the documentation cuz the

00:04:31.160 --> 00:04:33.400
documentation sort of describes how to

00:04:33.400 --> 00:04:35.360
use the platform, how to do this and

00:04:35.360 --> 00:04:37.400
that and the other across all different

00:04:37.400 --> 00:04:39.960
sort of aspects across a very large

00:04:39.960 --> 00:04:42.800
landscape. So,

00:04:42.800 --> 00:04:45.080
so basically, it wasn't because one

00:04:45.080 --> 00:04:47.000
person needed to make a decision. It it

00:04:47.000 --> 00:04:49.280
was like the whole team really needed to

00:04:49.280 --> 00:04:51.560
get behind the rewrite and it was just

00:04:51.560 --> 00:04:52.680
impossible.

00:04:52.680 --> 00:04:54.640
>> Okay. No, because like there's this

00:04:54.640 --> 00:04:55.840
couple of

00:04:55.840 --> 00:04:58.200
interesting sharings that I read just

00:04:58.200 --> 00:05:00.080
today one and then there's another one

00:05:00.080 --> 00:05:02.520
that um I don't forgot forget the

00:05:02.520 --> 00:05:04.760
context. Um but you mentioned something

00:05:04.760 --> 00:05:08.560
about if you create like docs

00:05:08.560 --> 00:05:10.560
all over your repo in markdown, they

00:05:10.560 --> 00:05:13.160
become maybe harder to discover than

00:05:13.160 --> 00:05:14.440
Confluence

00:05:14.440 --> 00:05:16.440
if you just use Git as a as a as a

00:05:16.440 --> 00:05:18.560
storage mechanism, which is something I

00:05:18.560 --> 00:05:19.600
always say, right? You should have your

00:05:19.600 --> 00:05:22.200
docs in Git. But also, people are saying

00:05:22.200 --> 00:05:23.720
you need to have your docs available to

00:05:23.720 --> 00:05:26.040
the agent because ultimately, the agents

00:05:26.040 --> 00:05:27.920
are going through these faster than

00:05:27.920 --> 00:05:29.640
humans.

00:05:29.640 --> 00:05:30.560
>> Yeah,

00:05:30.560 --> 00:05:32.840
I mean, step one was that we we did

00:05:32.840 --> 00:05:35.640
manage to get our docs into markdown

00:05:35.640 --> 00:05:36.800
um

00:05:36.800 --> 00:05:40.840
available to the agent. But then,

00:05:40.840 --> 00:05:42.760
that that was that was all done, you

00:05:42.760 --> 00:05:44.640
know, that was done. But the second step

00:05:44.640 --> 00:05:46.760
is like the

00:05:46.760 --> 00:05:51.280
um the the generated docs,

00:05:51.280 --> 00:05:53.920
we we just couldn't agree on it cuz

00:05:53.920 --> 00:05:55.520
cuz

00:05:55.520 --> 00:05:57.440
like what

00:05:57.440 --> 00:05:58.800
um

00:05:58.800 --> 00:06:00.040
Okay.

00:06:00.040 --> 00:06:01.200
Let let me just flesh this out. You

00:06:01.200 --> 00:06:03.440
know, like one way is that you could you

00:06:03.440 --> 00:06:09.320
could like if you want uh to find info,

00:06:09.320 --> 00:06:13.000
you could you could always ask uh the

00:06:13.000 --> 00:06:16.120
the AI agent. Right. So, the core source

00:06:16.120 --> 00:06:17.560
of truth should always remain, right?

00:06:17.560 --> 00:06:19.200
There should not be like derivative

00:06:19.200 --> 00:06:19.400
docs.

00:06:19.400 --> 00:06:20.720
>> Yeah, but the trouble the trouble is the

00:06:20.720 --> 00:06:23.040
docs they are quite expen- they're quite

00:06:23.040 --> 00:06:24.840
extensive. There's 500 pages and there

00:06:24.840 --> 00:06:27.560
are actually like mistakes in them. But

00:06:27.560 --> 00:06:30.040
okay, that aside, the trouble with that

00:06:30.040 --> 00:06:31.960
approach is that like um

00:06:31.960 --> 00:06:34.840
not everyone is prepared

00:06:34.840 --> 00:06:36.200
uh

00:06:36.200 --> 00:06:38.520
is is is prepared to to read docs that

00:06:38.520 --> 00:06:40.440
way. Okay, so what you're saying is you

00:06:40.440 --> 00:06:42.160
moved them out of Confluence into

00:06:42.160 --> 00:06:43.880
markdown, so they became available to

00:06:43.880 --> 00:06:46.480
agents. You then had one approach of

00:06:46.480 --> 00:06:48.840
asking agents to generate different

00:06:48.840 --> 00:06:50.720
derivative docs, but that's not good. We

00:06:50.720 --> 00:06:52.600
discussed it and you said we should keep

00:06:52.600 --> 00:06:55.000
a single source of core docs. And then

00:06:55.000 --> 00:06:57.640
you discover docs by asking agent. You

00:06:57.640 --> 00:06:59.360
find the information by asking agent,

00:06:59.360 --> 00:07:01.880
but not everyone agreed to that. Yeah.

00:07:01.880 --> 00:07:04.320
And then with with the core docs, the

00:07:04.320 --> 00:07:08.560
you know, huge temptation to let AI, you

00:07:08.560 --> 00:07:11.320
know, fix things. And then um which you

00:07:11.320 --> 00:07:13.640
did actually in and I I think it did a

00:07:13.640 --> 00:07:15.280
sterling job. I couldn't actually find

00:07:15.280 --> 00:07:17.960
any major issues. But then then the

00:07:17.960 --> 00:07:20.200
issue is that not not everyone in the

00:07:20.200 --> 00:07:22.120
team wanted to approve it. wanted to

00:07:22.120 --> 00:07:24.720
approve it because it basically

00:07:24.720 --> 00:07:26.760
uh

00:07:26.760 --> 00:07:28.360
since

00:07:28.360 --> 00:07:32.240
it touches like 500 files.

00:07:32.240 --> 00:07:32.960
Yeah, but

00:07:32.960 --> 00:07:35.320
>> That that sounds like Okay, because when

00:07:35.320 --> 00:07:37.400
you said that earlier, it felt to me if

00:07:37.400 --> 00:07:39.240
you're moving out of Confluence and

00:07:39.240 --> 00:07:40.960
you're moving it under version control,

00:07:40.960 --> 00:07:43.040
the migration should be a one-to-one.

00:07:43.040 --> 00:07:44.840
There can be no argument about this is

00:07:44.840 --> 00:07:47.480
not the same. It's exactly the same. And

00:07:47.480 --> 00:07:49.200
then you need to decouple. And if you

00:07:49.200 --> 00:07:51.080
say we're going to let AI fix things,

00:07:51.080 --> 00:07:52.920
and if if there is this hard requirement

00:07:52.920 --> 00:07:55.000
within the organization that any change

00:07:55.000 --> 00:07:57.680
to these docs now needs to be approved,

00:07:57.680 --> 00:08:00.240
um you have to be um and reviewed, you

00:08:00.240 --> 00:08:02.440
have to break it down in in in

00:08:02.440 --> 00:08:05.240
reviewable chunks, right? So, if you let

00:08:05.240 --> 00:08:08.280
the agent makes the fixes, then you have

00:08:08.280 --> 00:08:11.720
to ensure that the fixes remain um you

00:08:11.720 --> 00:08:13.600
know, small PRs that are reviewable so

00:08:13.600 --> 00:08:15.200
that you can get buy-in and maybe

00:08:15.200 --> 00:08:16.560
distribute the review work across

00:08:16.560 --> 00:08:18.120
different people

00:08:18.120 --> 00:08:19.920
uh in a way. I I I'm just thinking. I

00:08:19.920 --> 00:08:21.440
mean, I constantly have this problem

00:08:21.440 --> 00:08:23.800
with AI that I keep blowing up the scope

00:08:23.800 --> 00:08:25.760
and pulling in more changes. And it's

00:08:25.760 --> 00:08:28.520
very hard to just say no. I mean, if

00:08:28.520 --> 00:08:30.480
you're talking about code, we've come to

00:08:30.480 --> 00:08:31.880
the conclusion that a lot of people say

00:08:31.880 --> 00:08:33.840
we don't look at the code, right? Even

00:08:33.840 --> 00:08:37.680
um Adam Jacob's talk at the config man

00:08:37.680 --> 00:08:40.000
configuration camp Yeah. Yeah, I can't

00:08:40.000 --> 00:08:41.560
believe you just watched that one. I I'm

00:08:41.560 --> 00:08:43.560
sure I shared it with you 3 months ago.

00:08:43.560 --> 00:08:45.640
>> went an hour When I watched it, it was

00:08:45.640 --> 00:08:48.040
an hour on YouTube. So, Yeah, I I

00:08:48.040 --> 00:08:49.880
included a timestamp.

00:08:49.880 --> 00:08:52.160
No, it was an hour ago posted on YouTube

00:08:52.160 --> 00:08:54.280
when I watched it. Was it on YouTube?

00:08:54.280 --> 00:08:55.960
Yeah, it was. I don't know. I didn't

00:08:55.960 --> 00:08:57.800
because when I watched it yesterday, I

00:08:57.800 --> 00:09:00.400
can I should go check. I thought it was

00:09:00.400 --> 00:09:02.040
not on YouTube before. Anyway, it

00:09:02.040 --> 00:09:03.600
doesn't matter when or what I watched

00:09:03.600 --> 00:09:06.240
it. Yeah, I Adam Adam's I love his

00:09:06.240 --> 00:09:08.960
talks. Well, it's a very good opening

00:09:08.960 --> 00:09:11.640
talk uh to to ex-

00:09:11.640 --> 00:09:13.480
to um

00:09:13.480 --> 00:09:15.120
convince people, to give people a

00:09:15.120 --> 00:09:16.560
summary of where the world's been

00:09:16.560 --> 00:09:18.040
headed. And it sounds like he's

00:09:18.040 --> 00:09:20.560
basically spending 35 minutes convincing

00:09:20.560 --> 00:09:23.000
the room that yes, AI is coming. Yes,

00:09:23.000 --> 00:09:24.720
it's bigger than you think. And yes,

00:09:24.720 --> 00:09:26.320
it's changing everything and the way you

00:09:26.320 --> 00:09:28.080
think. But here are the things that I

00:09:28.080 --> 00:09:31.000
learned the last 3 months using it. And

00:09:31.000 --> 00:09:32.640
and it's great for that. But even though

00:09:32.640 --> 00:09:34.960
it was recorded in January, so when you

00:09:34.960 --> 00:09:37.520
watch it today, it's already outdated.

00:09:37.520 --> 00:09:38.960
Um he draws

00:09:38.960 --> 00:09:40.040
>> How?

00:09:40.040 --> 00:09:42.640
he draws the boxes of what has

00:09:42.640 --> 00:09:45.560
completely been dominated by AI agent.

00:09:45.560 --> 00:09:47.960
And he shows like the human is in in

00:09:47.960 --> 00:09:50.080
response for design. The human is

00:09:50.080 --> 00:09:52.400
responsible for plan. And then the

00:09:52.400 --> 00:09:54.200
agents are is responsible for

00:09:54.200 --> 00:09:56.920
implementation, validate, testing, and

00:09:56.920 --> 00:09:58.200
deployment, or whatever. I don't

00:09:58.200 --> 00:10:00.520
remember the exact boxes, right? And I

00:10:00.520 --> 00:10:02.600
just thought when I saw those boxes that

00:10:02.600 --> 00:10:04.920
I haven't really me personally reviewed

00:10:04.920 --> 00:10:07.600
a plan generated by an agent. I just ask

00:10:07.600 --> 00:10:10.600
another agent like adversarial review

00:10:10.600 --> 00:10:12.480
agent. So, you just give another agent

00:10:12.480 --> 00:10:15.120
complete blank context. That that do the

00:10:15.120 --> 00:10:17.880
cross reference.

00:10:17.880 --> 00:10:20.320
And I posted it as a comment and he

00:10:20.320 --> 00:10:22.080
agreed with me. He said like, "Yeah,

00:10:22.080 --> 00:10:24.680
that's already a case for for for his

00:10:24.680 --> 00:10:26.240
experience as well. They are not

00:10:26.240 --> 00:10:27.480
reviewing the plans anymore. They're

00:10:27.480 --> 00:10:29.840
doing adversarial agent reviews." And so

00:10:29.840 --> 00:10:31.560
now we are just part of the design.

00:10:31.560 --> 00:10:34.880
Yeah, adver- sorry. Adversarial, sorry.

00:10:34.880 --> 00:10:36.360
Adversarial, I keep saying it wrong.

00:10:36.360 --> 00:10:38.400
>> Adversarial What's the the whole phrase,

00:10:38.400 --> 00:10:41.760
sorry? Adversarial agents or agent

00:10:41.760 --> 00:10:44.040
reviews.

00:10:44.040 --> 00:10:46.800
Here they talk about networks, but it's

00:10:46.800 --> 00:10:48.560
it's definitely a

00:10:48.560 --> 00:10:50.920
a hot keyword right now in in the AI

00:10:50.920 --> 00:10:53.560
bubble. Yeah, I'm not I'm not 100% sure

00:10:53.560 --> 00:10:57.280
I can get agree and get behind this yet

00:10:57.280 --> 00:11:01.000
because I haven't managed to

00:11:01.000 --> 00:11:02.920
uh

00:11:02.920 --> 00:11:04.720
Like for example, I don't know if you

00:11:04.720 --> 00:11:07.200
shared it with me um or I shared it with

00:11:07.200 --> 00:11:11.160
you or I just found it by myself.

00:11:11.160 --> 00:11:12.280
um

00:11:12.280 --> 00:11:15.080
Oh, there's this thing called cook

00:11:15.080 --> 00:11:18.440
and it it essentially makes it easy to

00:11:18.440 --> 00:11:19.320
um

00:11:19.320 --> 00:11:22.400
I can't find it now. It makes it easy to

00:11:22.400 --> 00:11:23.880
to do this adversarial thing in the

00:11:23.880 --> 00:11:26.400
sense that like spawn an agent that

00:11:26.400 --> 00:11:29.000
reviews the other agent. And I used it

00:11:29.000 --> 00:11:31.520
and I thought to myself that I wasn't

00:11:31.520 --> 00:11:32.960
convinced because it it made the

00:11:32.960 --> 00:11:35.080
solution more complicated if anything.

00:11:35.080 --> 00:11:37.440
It It just It just

00:11:37.440 --> 00:11:39.560
It didn't get to the

00:11:39.560 --> 00:11:41.920
to the

00:11:41.920 --> 00:11:44.000
to the solution that I wanted, but that

00:11:44.000 --> 00:11:45.960
was just one anecdotal data point,

00:11:45.960 --> 00:11:48.440
right? So, anyway, we were talking about

00:11:48.440 --> 00:11:50.800
adversarial agents, why? Because things

00:11:50.800 --> 00:11:52.440
are outdated, but okay, you're talking

00:11:52.440 --> 00:11:54.520
about Adam Jackson's talk. Yeah, I was

00:11:54.520 --> 00:11:56.000
also just wondering a minute ago how did

00:11:56.000 --> 00:11:57.240
we get there, but because you were

00:11:57.240 --> 00:12:00.360
talking about docs and breaking down

00:12:00.360 --> 00:12:02.920
chunks into smaller reviews and I was

00:12:02.920 --> 00:12:04.200
saying that that's one of the biggest

00:12:04.200 --> 00:12:07.000
issues I have with scoping down the

00:12:07.000 --> 00:12:08.600
amount of work that the agent submits

00:12:08.600 --> 00:12:10.800
because because I then trailed off in

00:12:10.800 --> 00:12:12.640
the thought that we all agree that we're

00:12:12.640 --> 00:12:15.160
no longer reviewing the the implemented

00:12:15.160 --> 00:12:17.839
like the code, but we we now supposedly

00:12:17.839 --> 00:12:19.640
only review the plans generated by the

00:12:19.640 --> 00:12:21.720
agent and I'm just saying we don't even

00:12:21.720 --> 00:12:23.680
review that anymore.

00:12:23.680 --> 00:12:26.040
Um so so basically, I think we got there

00:12:26.040 --> 00:12:28.160
because we were talking about what needs

00:12:28.160 --> 00:12:31.080
to be reviewed by humans, right? So, so

00:12:31.080 --> 00:12:33.000
if you need to produce something with AI

00:12:33.000 --> 00:12:34.680
that needs to be reviewed by humans, you

00:12:34.680 --> 00:12:37.080
have to resist the temptation to let it

00:12:37.080 --> 00:12:38.560
become a massive amount so that they

00:12:38.560 --> 00:12:40.800
will have a you know, rejection because

00:12:40.800 --> 00:12:42.280
it's too much. They have no time to

00:12:42.280 --> 00:12:43.880
review it. Basically, that's the point I

00:12:43.880 --> 00:12:44.440
was making.

00:12:44.440 --> 00:12:46.120
>> Yeah, and

00:12:46.120 --> 00:12:49.080
and to me that's like a trap. And to be

00:12:49.080 --> 00:12:50.720
honest, I think we even went down that

00:12:50.720 --> 00:12:53.560
way at work for this documentation. And

00:12:53.560 --> 00:12:54.760
we were just saying like, "Hey, we got

00:12:54.760 --> 00:12:56.200
this view. We got this view. We got this

00:12:56.200 --> 00:12:57.960
view." You don't need to It's It's too

00:12:57.960 --> 00:12:58.520
much to

00:12:58.520 --> 00:13:00.280
>> Why are you talking about this view and

00:13:00.280 --> 00:13:01.720
this view? Are you creating from the

00:13:01.720 --> 00:13:03.440
same core docs different views? Because

00:13:03.440 --> 00:13:05.240
we already agreed don't do that. Okay,

00:13:05.240 --> 00:13:08.440
well, I'm just I'm using it as a as a as

00:13:08.440 --> 00:13:11.320
a terrible example of like like I got to

00:13:11.320 --> 00:13:13.760
a stage where you where where where we

00:13:13.760 --> 00:13:16.640
wanted to say don't review and then and

00:13:16.640 --> 00:13:18.240
then then we have a right mess,

00:13:18.240 --> 00:13:20.400
basically. Okay, so it sounds like

00:13:20.400 --> 00:13:21.839
because this is discussion we had last

00:13:21.839 --> 00:13:23.040
time and I'm not sure if we still want

00:13:23.040 --> 00:13:25.000
to go deep into this topic right now,

00:13:25.000 --> 00:13:28.400
but we agreed that at the time that

00:13:28.400 --> 00:13:30.520
there are a certain source truth core

00:13:30.520 --> 00:13:32.360
documents that are the single source of

00:13:32.360 --> 00:13:35.400
truth and then you said we we said we

00:13:35.400 --> 00:13:37.600
should use agents to query the docs and

00:13:37.600 --> 00:13:39.560
find and just basically create like

00:13:39.560 --> 00:13:41.960
ephemeral views

00:13:41.960 --> 00:13:43.760
that that gets caught thrown away and

00:13:43.760 --> 00:13:44.800
then you said some people are not

00:13:44.800 --> 00:13:47.240
confident because I can I can I can see

00:13:47.240 --> 00:13:49.000
that, you know, why would you not assume

00:13:49.000 --> 00:13:50.720
the agent is hallucinating in one of

00:13:50.720 --> 00:13:53.000
those ephemeral views, right? Unless you

00:13:53.000 --> 00:13:55.520
generate a more like

00:13:55.520 --> 00:13:57.520
persistent view that is actually

00:13:57.520 --> 00:14:00.000
reviewed by a human to confirm that it

00:14:00.000 --> 00:14:02.520
is not any hallucination in there. And

00:14:02.520 --> 00:14:04.320
then you would start creating

00:14:04.320 --> 00:14:07.640
um other views that are persistent, but

00:14:07.640 --> 00:14:10.839
it's not good. So, it's really um

00:14:10.839 --> 00:14:13.200
Okay, make conundrum. Yeah, it is quite

00:14:13.200 --> 00:14:15.959
a conundrum. uh And yes, the talk was

00:14:15.959 --> 00:14:17.720
posted on YouTube a month ago. I just

00:14:17.720 --> 00:14:20.000
checked. So, my mistake, not an hour.

00:14:20.000 --> 00:14:21.079
And

00:14:21.079 --> 00:14:23.040
here's the what I was trying to talk

00:14:23.040 --> 00:14:25.160
about earlier. Here's this CLI that I

00:14:25.160 --> 00:14:27.240
was using.

00:14:27.240 --> 00:14:28.360
So,

00:14:28.360 --> 00:14:29.600
when when you first set it up, you

00:14:29.600 --> 00:14:31.880
basically say which which

00:14:31.880 --> 00:14:33.640
which models you want to use like Claude

00:14:33.640 --> 00:14:36.240
and Codex. I mean, all you know, OpenAI

00:14:36.240 --> 00:14:38.760
and Claude. And then and then you ask it

00:14:38.760 --> 00:14:40.720
something and then it has these little

00:14:40.720 --> 00:14:41.800
like

00:14:41.800 --> 00:14:43.640
uh tags or I don't know what you call

00:14:43.640 --> 00:14:45.280
these little arguments where you can

00:14:45.280 --> 00:14:47.640
basically instruct the three parts other

00:14:47.640 --> 00:14:51.000
agent how to review the code or

00:14:51.000 --> 00:14:53.839
or you know, go three times

00:14:53.839 --> 00:14:55.520
or how to

00:14:55.520 --> 00:14:57.480
what what the evaluator function is or

00:14:57.480 --> 00:14:58.920
something.

00:14:58.920 --> 00:15:00.360
I and I used it and it

00:15:00.360 --> 00:15:02.360
works beautifully. But my friend's

00:15:02.360 --> 00:15:04.640
observation was a lot of all of a lot of

00:15:04.640 --> 00:15:06.880
these frameworks and he he started using

00:15:06.880 --> 00:15:09.000
get done, GSD.

00:15:09.000 --> 00:15:10.959
And he says it's very funny because it's

00:15:10.959 --> 00:15:13.680
literally just a whole bunch of markdown

00:15:13.680 --> 00:15:15.680
prompting and

00:15:15.680 --> 00:15:18.240
thin little layer of JavaScript. Yeah,

00:15:18.240 --> 00:15:19.959
and and a little loop you know, like

00:15:19.959 --> 00:15:21.400
>> And a lot of the these things are like

00:15:21.400 --> 00:15:23.520
that. Yeah, exactly. Exactly. I mean,

00:15:23.520 --> 00:15:24.720
And he says like, "What are we doing?

00:15:24.720 --> 00:15:27.400
Like what's the value add here?"

00:15:27.400 --> 00:15:29.440
>> yeah, and mine

00:15:29.440 --> 00:15:31.560
I mean, I love it in the sense that like

00:15:31.560 --> 00:15:33.640
it's beautiful to see these agents

00:15:33.640 --> 00:15:35.360
working in in concert.

00:15:35.360 --> 00:15:38.400
>> Mhm. But the results

00:15:38.400 --> 00:15:40.520
were not good, but maybe I need to give

00:15:40.520 --> 00:15:43.320
it a a better try. And another

00:15:43.320 --> 00:15:45.839
you know, feedback I I got because um my

00:15:45.839 --> 00:15:47.000
friend

00:15:47.000 --> 00:15:49.200
has been mostly using cursor or a little

00:15:49.200 --> 00:15:51.200
bit of agentic in cursor and he's been

00:15:51.200 --> 00:15:53.760
happy with it. And I've been asking him

00:15:53.760 --> 00:15:55.400
like, "Hey, you know,

00:15:55.400 --> 00:15:57.400
I'll use spec ledger with you and then

00:15:57.400 --> 00:15:59.360
you'll get to experience what is spec

00:15:59.360 --> 00:16:01.560
driven development and I will deliver

00:16:01.560 --> 00:16:03.640
some feature that whatever you want gets

00:16:03.640 --> 00:16:05.640
implemented." I told him. And he he

00:16:05.640 --> 00:16:07.440
wanted to drive, so I said, "If you want

00:16:07.440 --> 00:16:09.079
to drive, then you should have." And he

00:16:09.079 --> 00:16:11.600
says, "But I want to use chat GPT

00:16:11.600 --> 00:16:12.959
models. I'm not going to pay for

00:16:12.959 --> 00:16:14.760
Anthropic. I have already chat GPT." And

00:16:14.760 --> 00:16:16.040
I said, "Okay, then you should use open

00:16:16.040 --> 00:16:17.760
code." "Why not Codex?" He says, "I am

00:16:17.760 --> 00:16:19.200
not I don't support

00:16:19.200 --> 00:16:21.360
Codex CLI. I only support open code

00:16:21.360 --> 00:16:23.320
because it's more similar to Anthropic

00:16:23.320 --> 00:16:25.000
Claude Code in terms of they are trying

00:16:25.000 --> 00:16:27.240
to match the tool usage and stuff in the

00:16:27.240 --> 00:16:27.880
agent realm.

00:16:27.880 --> 00:16:30.200
>> Okay, okay. So so he then decided,

00:16:30.200 --> 00:16:31.800
"Okay, forget it. I'll just use cloud

00:16:31.800 --> 00:16:33.600
code then." And he's been using cloud

00:16:33.600 --> 00:16:35.959
code for a week now with spec driven

00:16:35.959 --> 00:16:38.120
development and he I think maybe he

00:16:38.120 --> 00:16:40.200
might have used another one, but he was

00:16:40.200 --> 00:16:41.839
frustrated, so he started using get

00:16:41.839 --> 00:16:44.600
done. And he said like within 1 week 2

00:16:44.600 --> 00:16:47.160
weeks, he was like, "This is amazing.

00:16:47.160 --> 00:16:48.800
Like why didn't I do that before?" And

00:16:48.800 --> 00:16:51.160
then he's he's already like running

00:16:51.160 --> 00:16:54.120
agent he's using get work trees and sub

00:16:54.120 --> 00:16:56.280
agents. And he also subscribed to the

00:16:56.280 --> 00:16:59.079
Claude Code next plan and um So, he's

00:16:59.079 --> 00:17:01.440
been red pilled.

00:17:01.440 --> 00:17:03.959
Like and he's asking me now how do you

00:17:03.959 --> 00:17:06.320
use like sub agents and work trees and

00:17:06.320 --> 00:17:08.439
agent teams and review agents and I'm

00:17:08.439 --> 00:17:10.326
like, "I'm not there."

00:17:10.326 --> 00:17:10.679
>> [laughter]

00:17:10.679 --> 00:17:12.280
>> I'm like heads deep in just the spec

00:17:12.280 --> 00:17:13.640
driven development side of things. I

00:17:13.640 --> 00:17:16.439
haven't really like I I have on purpose

00:17:16.439 --> 00:17:18.439
avoided going into this whole agent team

00:17:18.439 --> 00:17:20.120
orchestration because I believe that

00:17:20.120 --> 00:17:21.600
there's so many people doing it that

00:17:21.600 --> 00:17:23.240
soon enough it will be like, you know,

00:17:23.240 --> 00:17:24.520
perfect and I just need to wait a

00:17:24.520 --> 00:17:25.760
little.

00:17:25.760 --> 00:17:28.240
Yeah, so get so get done is is like

00:17:28.240 --> 00:17:30.800
that cook I was just sharing, right? Is

00:17:30.800 --> 00:17:32.760
it like that? I mean, He says it's a

00:17:32.760 --> 00:17:35.800
spec driven development toolkit that

00:17:35.800 --> 00:17:37.880
that is more like proactively keeps the

00:17:37.880 --> 00:17:39.320
agent running until the things are

00:17:39.320 --> 00:17:42.480
completed. Cuz a lot of the like spec

00:17:42.480 --> 00:17:44.720
driven development frameworks, they

00:17:44.720 --> 00:17:46.880
aren't really like keeping the agents in

00:17:46.880 --> 00:17:48.480
loop until it's done.

00:17:48.480 --> 00:17:50.360
Uh and apparently get get done does

00:17:50.360 --> 00:17:52.800
that more. Like you see, I've done spec

00:17:52.800 --> 00:17:55.040
kit open spec, but get done is

00:17:55.040 --> 00:17:57.000
actually delivering the results. I find

00:17:57.000 --> 00:17:58.800
that a little bit difficult. I'm

00:17:58.800 --> 00:18:00.880
definitely a a Peter Steinberger kind of

00:18:00.880 --> 00:18:03.920
guy. I just chat with the with the AI

00:18:03.920 --> 00:18:06.720
until it's done. This one seems to I

00:18:06.720 --> 00:18:08.360
wouldn't refer to myself as Peter

00:18:08.360 --> 00:18:10.920
Steinberger if I don't run if I don't

00:18:10.920 --> 00:18:12.320
have five

00:18:12.320 --> 00:18:14.080
you know, open code accounts and I'm

00:18:14.080 --> 00:18:16.040
running you know, 20 terminals at the

00:18:16.040 --> 00:18:18.480
same time. I wouldn't do you know,

00:18:18.480 --> 00:18:20.200
because this guy is on another level.

00:18:20.200 --> 00:18:22.320
Yeah, he he's obviously very good at

00:18:22.320 --> 00:18:24.040
context switching. I mean, I I like to

00:18:24.040 --> 00:18:25.280
think I'm pretty good at context

00:18:25.280 --> 00:18:26.360
switching.

00:18:26.360 --> 00:18:28.720
Like I can I can you know, spin three

00:18:28.720 --> 00:18:32.480
plates, maybe not five or 10. Mhm. Um

00:18:32.480 --> 00:18:33.919
well, I don't know what to say to this,

00:18:33.919 --> 00:18:35.560
but

00:18:35.560 --> 00:18:37.440
I guess I just need to give it a try.

00:18:37.440 --> 00:18:39.800
But it doesn't seem quite the same as as

00:18:39.800 --> 00:18:42.000
though that cook thing, but maybe maybe

00:18:42.000 --> 00:18:44.040
it's worth looking. I mean, there's

00:18:44.040 --> 00:18:45.640
I mean, this is the crazy thing why we

00:18:45.640 --> 00:18:47.440
we need to talk every week because

00:18:47.440 --> 00:18:48.200
there's a

00:18:48.200 --> 00:18:50.040
bazillion projects, aren't there? Yeah,

00:18:50.040 --> 00:18:52.600
it's moving really fast and and also

00:18:52.600 --> 00:18:54.440
what I took away from my my friend's

00:18:54.440 --> 00:18:55.919
experience is that even if you haven't

00:18:55.919 --> 00:18:57.840
looked at this, it takes maybe a week or

00:18:57.840 --> 00:18:59.919
two and you can be at the front, you

00:18:59.919 --> 00:19:02.360
know, trying to use the latest of the

00:19:02.360 --> 00:19:03.000
latest.

00:19:03.000 --> 00:19:05.000
>> Yeah. And maybe it's a good thing

00:19:05.000 --> 00:19:06.440
because if you've been using it for 8

00:19:06.440 --> 00:19:07.760
months, you're like stuck in your old

00:19:07.760 --> 00:19:09.440
behaviors and the models have moved on.

00:19:09.440 --> 00:19:11.120
And if you come in a fresh, you're like

00:19:11.120 --> 00:19:12.520
no idea where to start, but this is

00:19:12.520 --> 00:19:13.880
looks like what what people are doing,

00:19:13.880 --> 00:19:15.000
so I'm going to start doing this.

00:19:15.000 --> 00:19:17.000
element of beginner's luck. It's not

00:19:17.000 --> 00:19:18.960
like beginner Yeah, well, it's it's just

00:19:18.960 --> 00:19:21.560
being able to shed your your legacy or

00:19:21.560 --> 00:19:24.280
tribal knowledge and being able to cuz

00:19:24.280 --> 00:19:26.720
even Boris Chef Ni, he says that he's

00:19:26.720 --> 00:19:28.600
stuck in a lot of old ways when he's

00:19:28.600 --> 00:19:30.000
doing troubleshooting. He always gives

00:19:30.000 --> 00:19:31.600
this example where he says there was

00:19:31.600 --> 00:19:33.960
this memory leak and and he he has to

00:19:33.960 --> 00:19:36.560
avoid going into his habit of going

00:19:36.560 --> 00:19:39.080
going into an heat map to identify the

00:19:39.080 --> 00:19:41.320
memory leak and then somebody younger

00:19:41.320 --> 00:19:42.679
who joined the team and has really

00:19:42.679 --> 00:19:44.720
adopted Claude Code, you know, just went,

00:19:44.720 --> 00:19:46.320
"Hey, you know, I just asked Claude and

00:19:46.320 --> 00:19:47.880
within 5 minutes it found the memory

00:19:47.880 --> 00:19:49.760
leak. Whereas you started going into

00:19:49.760 --> 00:19:51.919
your heat map and you know, 20 minutes

00:19:51.919 --> 00:19:54.000
later you still nowhere and Cloud

00:19:54.000 --> 00:19:55.960
already fixed it. So it's it's really

00:19:55.960 --> 00:19:57.560
part of and then Boris says it, it's

00:19:57.560 --> 00:19:59.640
part of the models are so good. I

00:19:59.640 --> 00:20:01.960
constantly have to remind myself to like

00:20:01.960 --> 00:20:03.840
cut those old habits and just ask the

00:20:03.840 --> 00:20:05.000
model.

00:20:05.000 --> 00:20:07.240
Yeah. Speaking of Claude Code, I I think

00:20:07.240 --> 00:20:08.920
I don't know if this is real but I did

00:20:08.920 --> 00:20:11.240
see some comment that someone analyzed

00:20:11.240 --> 00:20:13.440
the the source code drop and there was

00:20:13.440 --> 00:20:15.360
some talk that there's a way more

00:20:15.360 --> 00:20:18.280
powerful model in the in the works but I

00:20:18.280 --> 00:20:19.520
guess that's always going to be true

00:20:19.520 --> 00:20:21.400
really. But like I don't have my faults

00:20:21.400 --> 00:20:23.040
or Metas.

00:20:23.040 --> 00:20:25.120
Yeah, when the new model drops how is it

00:20:25.120 --> 00:20:27.120
going to change things? I saw a

00:20:27.120 --> 00:20:28.600
screenshot on Reddit which could be

00:20:28.600 --> 00:20:30.520
completely fake but like the

00:20:30.520 --> 00:20:33.040
improvements compared to Opus were

00:20:33.040 --> 00:20:34.800
significant. I mean the way it was

00:20:34.800 --> 00:20:37.160
discovered when I I heard about it

00:20:37.160 --> 00:20:38.920
before the drop was because Anthropic

00:20:38.920 --> 00:20:40.880
apparently left information in a public

00:20:40.880 --> 00:20:43.560
database. It wasn't like a meta dot dot

00:20:43.560 --> 00:20:45.720
dot database. It left some information

00:20:45.720 --> 00:20:48.600
about this Metos project or Capybara or

00:20:48.600 --> 00:20:50.360
something like that.

00:20:50.360 --> 00:20:53.080
And and that's how I heard it was uh

00:20:53.080 --> 00:20:54.640
But it says quite a lot of like first of

00:20:54.640 --> 00:20:56.480
Anthropic like up time being such a

00:20:56.480 --> 00:20:57.640
horrible

00:20:57.640 --> 00:20:59.080
I heard a lot of people moved away from

00:20:59.080 --> 00:21:00.560
Anthropic because they are not able to

00:21:00.560 --> 00:21:04.800
deliver their reliability. Yeah. Yeah.

00:21:04.800 --> 00:21:07.840
They had a huge

00:21:07.840 --> 00:21:09.400
They had this huge

00:21:09.400 --> 00:21:10.760
issue with

00:21:10.760 --> 00:21:13.840
token caching so your usage would

00:21:13.840 --> 00:21:15.640
drop significantly like people would

00:21:15.640 --> 00:21:17.160
send one or two messages and they would

00:21:17.160 --> 00:21:20.560
hit their 5-hour budget because um

00:21:20.560 --> 00:21:22.680
there was a bug in the way that the two

00:21:22.680 --> 00:21:25.120
the cache was working and it was also

00:21:25.120 --> 00:21:27.120
happening when you restarted Cloud. So

00:21:27.120 --> 00:21:28.840
people were like finding the bugs and

00:21:28.840 --> 00:21:30.840
even when the short source code leaked

00:21:30.840 --> 00:21:32.680
people started to debug those bugs.

00:21:32.680 --> 00:21:34.560
There's a memory leak I'm going So not a

00:21:34.560 --> 00:21:36.640
memory leak. It was two significant bugs

00:21:36.640 --> 00:21:38.520
that the community highlighted and and

00:21:38.520 --> 00:21:39.880
that they had to fix. So they have been

00:21:39.880 --> 00:21:42.280
having a very hard time. Then all of

00:21:42.280 --> 00:21:44.400
this like internal information leaking

00:21:44.400 --> 00:21:47.280
about models being underway. So people

00:21:47.280 --> 00:21:49.360
are like right now on Reddit it's very

00:21:49.360 --> 00:21:51.560
annoying because every day there's three

00:21:51.560 --> 00:21:54.160
or four posts of people saying you know

00:21:54.160 --> 00:21:55.840
First of all weeks they complain about

00:21:55.840 --> 00:21:58.320
the the token usage budgets depleting

00:21:58.320 --> 00:22:01.760
too fast and now Anthropic has released

00:22:01.760 --> 00:22:05.520
.90 and .91 fixing some of these issues

00:22:05.520 --> 00:22:07.920
and now people are posting yeah that's

00:22:07.920 --> 00:22:10.000
not fixed until you refund you need to

00:22:10.000 --> 00:22:12.360
refund because we couldn't use it.

00:22:12.360 --> 00:22:15.000
You're all in on Anthropic right still

00:22:15.000 --> 00:22:17.080
with your max plan? I don't want to

00:22:17.080 --> 00:22:20.080
support Open AI. I like to think I'm

00:22:20.080 --> 00:22:21.760
being a bit balanced like I've been

00:22:21.760 --> 00:22:25.080
moving between Open AI and Anthropic. I

00:22:25.080 --> 00:22:27.040
still have ChatGPT. I canceled it they

00:22:27.040 --> 00:22:28.480
give me one month free but I haven't

00:22:28.480 --> 00:22:30.280
recanceled it. So you need to cancel it

00:22:30.280 --> 00:22:32.320
then they give you one month free then

00:22:32.320 --> 00:22:34.080
you if you Yeah, yeah. You need to

00:22:34.080 --> 00:22:35.880
cancel it for real after one month and I

00:22:35.880 --> 00:22:37.720
haven't canceled it because my wife

00:22:37.720 --> 00:22:41.160
really likes using ChatGPT. Mhm. The I

00:22:41.160 --> 00:22:43.120
think the voice mode is also Much better

00:22:43.120 --> 00:22:44.000
than

00:22:44.000 --> 00:22:47.160
Yeah. Um phone app. Okay, let's change

00:22:47.160 --> 00:22:50.480
topic ever so slightly. I wanted to

00:22:50.480 --> 00:22:52.880
ask you quickly if you've ever worked on

00:22:52.880 --> 00:22:54.920
a data platform. I worked for some

00:22:54.920 --> 00:22:56.760
engineering

00:22:56.760 --> 00:22:57.720
I mean I

00:22:57.720 --> 00:22:59.040
I think it's

00:22:59.040 --> 00:23:00.640
>> Enough time has passed and I can talk

00:23:00.640 --> 00:23:04.360
about my Gojek experience. Mhm.

00:23:04.360 --> 00:23:08.440
I I worked briefly at Gojek and uh

00:23:08.440 --> 00:23:10.120
it didn't work out to be honest at the

00:23:10.120 --> 00:23:12.440
end of the day. Not going to lie. But

00:23:12.440 --> 00:23:13.760
the

00:23:13.760 --> 00:23:15.880
but when I when I worked there

00:23:15.880 --> 00:23:16.720
uh

00:23:16.720 --> 00:23:18.440
as a data plat I was a data platform

00:23:18.440 --> 00:23:21.160
engineer. It was interesting um in the

00:23:21.160 --> 00:23:22.880
sense I I don't know if you've ever seen

00:23:22.880 --> 00:23:24.679
this sort of model like this. We we had

00:23:24.679 --> 00:23:28.320
data engineers and about 5 or 6 7 years

00:23:28.320 --> 00:23:30.920
ago data engineers was like like for

00:23:30.920 --> 00:23:32.880
some reason like really popular. I'm in

00:23:32.880 --> 00:23:34.800
fact I think you you still see it today

00:23:34.800 --> 00:23:37.080
like you do these like boot camps and

00:23:37.080 --> 00:23:38.760
then people learn how to become a data

00:23:38.760 --> 00:23:40.000
engineer.

00:23:40.000 --> 00:23:41.720
You know what I mean? Like it seems to

00:23:41.720 --> 00:23:44.679
be a role unto itself.

00:23:44.679 --> 00:23:46.360
Oh oh I can see you're distracted by the

00:23:46.360 --> 00:23:47.520
Claude Code.

00:23:47.520 --> 00:23:49.320
Now what what what would happen is that

00:23:49.320 --> 00:23:51.560
the the data engineers would get some

00:23:51.560 --> 00:23:54.160
data or get some task to to wield some

00:23:54.160 --> 00:23:56.080
data. They would jump in the then

00:23:56.080 --> 00:23:58.560
Jupiter notebooks and then they would

00:23:58.560 --> 00:24:02.120
come up with a POC about how the data is

00:24:02.120 --> 00:24:03.880
going to look or how it's going to be

00:24:03.880 --> 00:24:05.760
end up or something like this.

00:24:05.760 --> 00:24:07.520
And then and then that becomes a data

00:24:07.520 --> 00:24:09.960
product. But the trouble is is that they

00:24:09.960 --> 00:24:12.240
since the the these data engineers were

00:24:12.240 --> 00:24:15.400
kind of like fresh grads mostly or they

00:24:15.400 --> 00:24:17.920
were just not really uh I don't know

00:24:17.920 --> 00:24:20.720
what the reason was exactly. Well they

00:24:20.720 --> 00:24:22.600
they lack some skills to productionize

00:24:22.600 --> 00:24:24.480
it. So so they they would hand over

00:24:24.480 --> 00:24:26.880
their work essentially to data platform

00:24:26.880 --> 00:24:28.320
engineers and then we would

00:24:28.320 --> 00:24:30.080
productionize it. Have you ever seen

00:24:30.080 --> 00:24:31.440
something like this before Vincent? I

00:24:31.440 --> 00:24:33.800
mean I've never really worked inside of

00:24:33.800 --> 00:24:36.240
data team. I have supported a data team

00:24:36.240 --> 00:24:38.880
as a platform team um

00:24:38.880 --> 00:24:41.360
for one of the data teams I worked with

00:24:41.360 --> 00:24:44.280
I was managing their Airflow deployments

00:24:44.280 --> 00:24:46.040
and also helping

00:24:46.040 --> 00:24:47.960
them define how they would get secrets

00:24:47.960 --> 00:24:50.480
to the jobs in Airflow and also That's

00:24:50.480 --> 00:24:53.400
typical. It has secrets to I mean since

00:24:53.400 --> 00:24:56.760
since Airflow is like a shared thing

00:24:56.760 --> 00:24:58.960
I the Cloud Composer I think is is

00:24:58.960 --> 00:25:00.760
basically Airflow. All right. Yeah,

00:25:00.760 --> 00:25:02.760
essentially Airflow is quite an

00:25:02.760 --> 00:25:05.960
infrastructure challenge because it's I

00:25:05.960 --> 00:25:09.000
think it's got some horrible assumptions

00:25:09.000 --> 00:25:10.679
that everything is run on a shared

00:25:10.679 --> 00:25:12.240
instance and then of course when you

00:25:12.240 --> 00:25:14.240
have lots of data products. Uh I was

00:25:14.240 --> 00:25:15.320
running

00:25:15.320 --> 00:25:17.840
So at Honestbee back in 2016 they were

00:25:17.840 --> 00:25:20.640
using Airflow and the

00:25:20.640 --> 00:25:22.360
or they adopted Airflow at least around

00:25:22.360 --> 00:25:24.000
2017

00:25:24.000 --> 00:25:25.120
and

00:25:25.120 --> 00:25:27.679
they hired someone an ex-DevOps engineer

00:25:27.679 --> 00:25:30.080
who wanted to go into data and he joined

00:25:30.080 --> 00:25:32.040
the data team and they only give him one

00:25:32.040 --> 00:25:34.200
job which was to deploy Airflow. So he

00:25:34.200 --> 00:25:35.800
was pretty pissed.

00:25:35.800 --> 00:25:38.560
But but he was used to run Mesosphere.

00:25:38.560 --> 00:25:40.520
So he was running they I was running

00:25:40.520 --> 00:25:43.280
Kubernetes for the the the product

00:25:43.280 --> 00:25:44.800
which were like several big Ruby on

00:25:44.800 --> 00:25:47.760
Rails applications and they approached

00:25:47.760 --> 00:25:50.760
the the platform team for uh you some

00:25:50.760 --> 00:25:53.160
type of job running framework and at the

00:25:53.160 --> 00:25:56.640
time 2017 the job API like the resources

00:25:56.640 --> 00:25:58.160
of for running jobs in Kubernetes were

00:25:58.160 --> 00:26:00.080
like alpha or beta they were very

00:26:00.080 --> 00:26:02.720
unstable. So I said yeah we can do jobs

00:26:02.720 --> 00:26:03.520
but

00:26:03.520 --> 00:26:05.080
and so they decided okay I'm just going

00:26:05.080 --> 00:26:07.120
to go with what I know and rather than

00:26:07.120 --> 00:26:09.280
to talk to us they just deployed

00:26:09.280 --> 00:26:11.520
Mesosphere. I was like what you know why

00:26:11.520 --> 00:26:14.400
you deploying a competing like another

00:26:14.400 --> 00:26:16.560
container orchestration system. Anyway

00:26:16.560 --> 00:26:18.120
they ended up with having this huge

00:26:18.120 --> 00:26:20.040
Mesosphere cluster

00:26:20.040 --> 00:26:22.640
that was mounting the file system.

00:26:22.640 --> 00:26:25.640
>> to Airflow right to jobs? Yeah because

00:26:25.640 --> 00:26:27.320
of Airflow. They was just because they

00:26:27.320 --> 00:26:28.640
didn't run anything because they were

00:26:28.640 --> 00:26:30.080
using That's that's a typical problem

00:26:30.080 --> 00:26:32.600
like everyone is on the shared Airflow

00:26:32.600 --> 00:26:34.840
instance and then they go like oh we

00:26:34.840 --> 00:26:37.880
have secrets we have things data we need

00:26:37.880 --> 00:26:39.600
to containerize it and then they come up

00:26:39.600 --> 00:26:41.800
with some hodgepodge containerization

00:26:41.800 --> 00:26:42.679
solution.

00:26:42.679 --> 00:26:44.520
>> because I managed Airflow at another

00:26:44.520 --> 00:26:47.440
company like four years later and it was

00:26:47.440 --> 00:26:49.320
completely different. None of that what

00:26:49.320 --> 00:26:51.160
we just mentioned. In Honestbee we had

00:26:51.160 --> 00:26:52.080
what the problem that you just

00:26:52.080 --> 00:26:54.040
mentioned. They had they basically

00:26:54.040 --> 00:26:55.960
containerized Airflow by creating a

00:26:55.960 --> 00:26:58.360
massive image and then mounting a file

00:26:58.360 --> 00:27:01.280
system into it with all of the DAGs. So

00:27:01.280 --> 00:27:04.040
the whole all of the DAGs was just one

00:27:04.040 --> 00:27:06.280
massive volume. So the image was just

00:27:06.280 --> 00:27:08.040
something like a shell Yeah. I don't

00:27:08.040 --> 00:27:09.679
think that's that's a that's a good

00:27:09.679 --> 00:27:11.040
idea. That's like containerizing

00:27:11.040 --> 00:27:13.200
Kubernetes. And the only reason I know

00:27:13.200 --> 00:27:15.640
is because and everything. The Yeah yeah

00:27:15.640 --> 00:27:18.040
they they basically run Airflow which I

00:27:18.040 --> 00:27:19.520
also thought like why do you need to run

00:27:19.520 --> 00:27:21.640
a job runner on top of a job runner?

00:27:21.640 --> 00:27:23.440
Like why do you need Mesosphere to run

00:27:23.440 --> 00:27:25.920
Airflow? Okay, okay. Okay. So when I

00:27:25.920 --> 00:27:28.120
inherited it but then I can tell you one

00:27:28.120 --> 00:27:30.040
thing okay doesn't have to be that way

00:27:30.040 --> 00:27:32.800
with Airflow because I deployed Airflow

00:27:32.800 --> 00:27:35.760
in 2020 which was like

00:27:35.760 --> 00:27:38.120
four years later and and at that time

00:27:38.120 --> 00:27:40.360
Airflow had a native Kubernetes job

00:27:40.360 --> 00:27:42.400
runner and it has a native Kubernetes

00:27:42.400 --> 00:27:44.760
orchestrator. Cuz originally Airflow was

00:27:44.760 --> 00:27:46.760
like using Redis to distribute jobs to

00:27:46.760 --> 00:27:49.040
runners but with with

00:27:49.040 --> 00:27:51.000
with the Kubernetes integration it was

00:27:51.000 --> 00:27:53.000
using Kubernetes as a job runner

00:27:53.000 --> 00:27:54.400
interface and it would talk to the

00:27:54.400 --> 00:27:56.800
Kubernetes API to spin up a pod which

00:27:56.800 --> 00:27:59.800
was like an Airflow worker node and then

00:27:59.800 --> 00:28:02.200
you it would get the the DAGs

00:28:02.200 --> 00:28:04.720
from I think we put them on S3. Yeah I

00:28:04.720 --> 00:28:05.960
don't remember how we delivered the

00:28:05.960 --> 00:28:07.080
DAGs.

00:28:07.080 --> 00:28:08.720
But it was definitely not like a huge

00:28:08.720 --> 00:28:11.320
massive like file system. So then

00:28:11.320 --> 00:28:13.560
actually it was really manageable. It

00:28:13.560 --> 00:28:15.760
was really very smooth and the secrets

00:28:15.760 --> 00:28:18.600
was easy because we were using like um

00:28:18.600 --> 00:28:20.080
secrets directly for jobs and they were

00:28:20.080 --> 00:28:21.480
just getting injected when they needed

00:28:21.480 --> 00:28:23.320
it. Not shared nothing like that. And

00:28:23.320 --> 00:28:25.880
that was in 2020 so Okay. So

00:28:25.880 --> 00:28:27.560
so sorry I I I was a little bit

00:28:27.560 --> 00:28:28.960
distracted. So how did what was the

00:28:28.960 --> 00:28:31.159
solution ended ended up being again?

00:28:31.159 --> 00:28:33.280
Airflow supports Kubernetes as a as a

00:28:33.280 --> 00:28:35.560
job Really? All right okay that's

00:28:35.560 --> 00:28:36.800
interesting.

00:28:36.800 --> 00:28:39.159
It has for a long time.

00:28:39.159 --> 00:28:41.200
I actually need to investigate that. We

00:28:41.200 --> 00:28:42.640
were just

00:28:42.640 --> 00:28:44.440
like back then we're using the the

00:28:44.440 --> 00:28:47.040
managed GCP version and then recently

00:28:47.040 --> 00:28:48.080
I've been

00:28:48.080 --> 00:28:50.520
running the AWS managed version called

00:28:50.520 --> 00:28:53.880
managed workflow I don't know MWAA. But

00:28:53.880 --> 00:28:56.159
it it's

00:28:56.159 --> 00:28:58.240
Okay, that's something to look up. I

00:28:58.240 --> 00:29:01.360
mean what what I wanted to sort of

00:29:01.360 --> 00:29:04.760
perhaps tell you was perhaps

00:29:04.760 --> 00:29:07.240
um not nothing AI related it's more to

00:29:07.240 --> 00:29:09.520
do with organization. I mean as much as

00:29:09.520 --> 00:29:11.159
I didn't I actually didn't really like

00:29:11.159 --> 00:29:12.960
this model at all because the data in

00:29:12.960 --> 00:29:14.480
there was so there was a lot of data

00:29:14.480 --> 00:29:16.320
engineers and and the data platform

00:29:16.320 --> 00:29:18.480
engineers which I was a part of we were

00:29:18.480 --> 00:29:20.880
really like overloaded with work because

00:29:20.880 --> 00:29:23.200
unfortunately we needed to know what

00:29:23.200 --> 00:29:25.280
they were doing because sometimes they

00:29:25.280 --> 00:29:27.720
would just throw over the wall

00:29:27.720 --> 00:29:30.200
um some some code that just basically

00:29:30.200 --> 00:29:32.240
didn't even work and we would have to

00:29:32.240 --> 00:29:35.720
like fix it and own it from end to end.

00:29:35.720 --> 00:29:38.480
So we were like basically working on new

00:29:38.480 --> 00:29:40.520
products all the time and as well as

00:29:40.520 --> 00:29:42.320
supporting old products. It was quite

00:29:42.320 --> 00:29:43.960
stressful.

00:29:43.960 --> 00:29:46.280
Yeah.

00:29:46.280 --> 00:29:48.280
But nowadays I think I think back to

00:29:48.280 --> 00:29:49.560
those five years ago and I think to

00:29:49.560 --> 00:29:51.200
myself like a lot of the work that we

00:29:51.200 --> 00:29:53.640
were doing wasn't so

00:29:53.640 --> 00:29:55.920
like inventive. You know what I mean?

00:29:55.920 --> 00:29:58.520
Like we we didn't come up with the the

00:29:58.520 --> 00:30:00.240
products themselves. They That was kind

00:30:00.240 --> 00:30:01.600
of part of the day data engineers

00:30:01.600 --> 00:30:04.000
things. But we were part of like

00:30:04.000 --> 00:30:06.640
productionizing it. And now with AI,

00:30:06.640 --> 00:30:09.640
productionizing probably becomes

00:30:09.640 --> 00:30:10.880
easier?

00:30:10.880 --> 00:30:13.440
Debatable. I mean, I don't think AI's

00:30:13.440 --> 00:30:15.840
super proven to ship, that isn't it? And

00:30:15.840 --> 00:30:18.040
productionizing is about shipping. But I

00:30:18.040 --> 00:30:20.120
still I still maintain that this world

00:30:20.120 --> 00:30:21.440
is probably going to look pretty

00:30:21.440 --> 00:30:25.160
different in 2026. That's for sure.

00:30:25.160 --> 00:30:27.400
Um

00:30:27.400 --> 00:30:29.280
And then And then also I just wanted to

00:30:29.280 --> 00:30:32.360
contrast that with what I saw what I saw

00:30:32.360 --> 00:30:34.440
kind of recently on a on a gig where

00:30:34.440 --> 00:30:37.480
basically um

00:30:37.480 --> 00:30:39.920
Yeah, basically people are expected to

00:30:39.920 --> 00:30:43.120
to know a lot more to do the job.

00:30:43.120 --> 00:30:46.000
Like it we created, you know, a proper

00:30:46.000 --> 00:30:48.400
platform where they where they had you

00:30:48.400 --> 00:30:51.000
know, had to create a Git repo and all

00:30:51.000 --> 00:30:52.960
that sort of stuff. And to be honest, I

00:30:52.960 --> 00:30:54.640
I think it was just too difficult for

00:30:54.640 --> 00:30:57.320
for for people to do it that way. I

00:30:57.320 --> 00:30:58.960
don't know if you've ever created a a

00:30:58.960 --> 00:31:00.880
platform where

00:31:00.880 --> 00:31:04.080
you know, there's a config YAML and

00:31:04.080 --> 00:31:06.400
you know, product teams are are expected

00:31:06.400 --> 00:31:09.200
to to deploy their own app.

00:31:09.200 --> 00:31:11.680
I I think to I think to myself that

00:31:11.680 --> 00:31:14.160
that's just too hard. It's too hard. You

00:31:14.160 --> 00:31:16.240
They've got to like It's got to be a lot

00:31:16.240 --> 00:31:17.880
easier for different in a big

00:31:17.880 --> 00:31:20.200
enterprise. It's got to become really

00:31:20.200 --> 00:31:25.000
really like AI easy for people to create

00:31:25.000 --> 00:31:27.800
software or a solution for for their

00:31:27.800 --> 00:31:29.840
particular problem. Can't expect people

00:31:29.840 --> 00:31:34.040
to like read docs, set up GitHub repos,

00:31:34.040 --> 00:31:37.000
run workflows. It's just too hard.

00:31:37.000 --> 00:31:39.040
That's That's all I wanted to share. I

00:31:39.040 --> 00:31:40.880
definitely have problems where the

00:31:40.880 --> 00:31:44.040
platforms I create around Terraform

00:31:44.040 --> 00:31:46.400
are for some people

00:31:46.400 --> 00:31:47.800
too

00:31:47.800 --> 00:31:51.240
um let's say niche. Yeah, yeah. That's a

00:31:51.240 --> 00:31:52.880
typical one cuz like like we find

00:31:52.880 --> 00:31:55.160
Terraform easy.

00:31:55.160 --> 00:31:57.600
But I know I know like as soon as you

00:31:57.600 --> 00:31:59.000
you tell

00:31:59.000 --> 00:32:01.200
a data um a product team like, "Oh, you

00:32:01.200 --> 00:32:02.880
have to deploy Terraform." Then mind

00:32:02.880 --> 00:32:04.640
just explodes. Yeah, but the problem

00:32:04.640 --> 00:32:06.920
with Terraform is that it's not a very

00:32:06.920 --> 00:32:08.600
good product.

00:32:08.600 --> 00:32:12.000
Like it has the Go lang mentality of

00:32:12.000 --> 00:32:13.880
it's better to rewrite something or

00:32:13.880 --> 00:32:16.160
duplicate it than to create coupling,

00:32:16.160 --> 00:32:18.840
right? Um what we are doing often with

00:32:18.840 --> 00:32:20.240
Terraform is we are creating

00:32:20.240 --> 00:32:22.640
abstractions that then create coupling

00:32:22.640 --> 00:32:24.640
between different parts because we're

00:32:24.640 --> 00:32:27.240
trying to reduce the the boilerplate and

00:32:27.240 --> 00:32:29.560
the repetitive repetitive

00:32:29.560 --> 00:32:31.360
repetition. So this is kind of like

00:32:31.360 --> 00:32:33.320
where it starts to then become

00:32:33.320 --> 00:32:35.080
complicated for people because we create

00:32:35.080 --> 00:32:37.800
these meta layers around Terraform. Mhm.

00:32:37.800 --> 00:32:41.000
So Terraform in itself can be bad.

00:32:41.000 --> 00:32:43.400
And modules Modules in Terraform also

00:32:43.400 --> 00:32:45.760
just flummox people then they like don't

00:32:45.760 --> 00:32:48.480
know what's going on inside them.

00:32:48.480 --> 00:32:51.200
Terragrunt reached V1, by the way. What?

00:32:51.200 --> 00:32:52.880
Thinking about layers that complicate

00:32:52.880 --> 00:32:55.440
Terraform, Terragrunt was released

00:32:55.440 --> 00:32:58.120
with an official V1 like stable thing.

00:32:58.120 --> 00:32:59.720
>> But Terragrunt has been around for a

00:32:59.720 --> 00:33:01.320
long time. Yeah, but they never did a

00:33:01.320 --> 00:33:03.440
stable release. So they finally released

00:33:03.440 --> 00:33:06.800
Terragrunt V1 and they included stacks,

00:33:06.800 --> 00:33:09.080
um which But they've been working on

00:33:09.080 --> 00:33:11.560
stacks. Well, I used But this isn't exa-

00:33:11.560 --> 00:33:14.000
exactly an example of where Terraform is

00:33:14.000 --> 00:33:16.440
meant to be simple and then it becomes

00:33:16.440 --> 00:33:18.560
too unmanageable because of the amount

00:33:18.560 --> 00:33:20.320
of config files and repetition that you

00:33:20.320 --> 00:33:22.440
get. And then people build layers around

00:33:22.440 --> 00:33:23.920
it like Terragrunt and it's a prime

00:33:23.920 --> 00:33:28.040
example of creating complete like super

00:33:28.040 --> 00:33:29.880
hard things to understand because of the

00:33:29.880 --> 00:33:33.360
complexity it it allows you to do.

00:33:33.360 --> 00:33:35.280
I don't know, they probably improved a

00:33:35.280 --> 00:33:36.560
lot of it,

00:33:36.560 --> 00:33:39.600
um but I used Terragrunt in in one

00:33:39.600 --> 00:33:42.560
organization which was rather large. And

00:33:42.560 --> 00:33:44.880
I feel it was very successful in the way

00:33:44.880 --> 00:33:46.320
that it managed our infrastructure as

00:33:46.320 --> 00:33:48.679
code, but I also feel it was a massive

00:33:48.679 --> 00:33:51.040
blocker for anyone that wasn't platform

00:33:51.040 --> 00:33:52.120
to then contribute to that

00:33:52.120 --> 00:33:53.880
infrastructure as code. Yeah, I guess

00:33:53.880 --> 00:33:55.720
that's a good example. And the same with

00:33:55.720 --> 00:33:57.840
Terra mate and all the others. Anything

00:33:57.840 --> 00:34:00.160
that's a Go lang CLI around Terraform is

00:34:00.160 --> 00:34:02.200
like that. Yeah. And do you think I

00:34:02.200 --> 00:34:05.960
mean, compared to CDK? Uh good.

00:34:05.960 --> 00:34:09.200
Um because CDK gets rid of the Terraform

00:34:09.200 --> 00:34:10.960
modules, which to me I think is the

00:34:10.960 --> 00:34:13.280
biggest problem. Most of these

00:34:13.280 --> 00:34:15.800
orchestration system try to work with

00:34:15.800 --> 00:34:18.800
Terraform modules. But Terraform CDK or

00:34:18.800 --> 00:34:22.840
AWS CDK completely gets rid of it by

00:34:22.840 --> 00:34:24.600
creating a higher level abstraction

00:34:24.600 --> 00:34:27.000
because it has it is not just like

00:34:27.000 --> 00:34:29.359
a scripting library, something that can

00:34:29.359 --> 00:34:31.040
do simple loops or things like that.

00:34:31.040 --> 00:34:33.480
It's an actual complete programming

00:34:33.480 --> 00:34:35.120
language language and it allows you to

00:34:35.120 --> 00:34:38.120
create object-oriented things such as

00:34:38.120 --> 00:34:40.679
extending an a base class, implementing

00:34:40.679 --> 00:34:43.280
interfaces, it's allowing you to do

00:34:43.280 --> 00:34:46.040
dependency injection by just making sure

00:34:46.040 --> 00:34:48.440
that the constructor takes in the

00:34:48.440 --> 00:34:50.600
something that implements the interface.

00:34:50.600 --> 00:34:52.760
So you get a lot more capabilities than

00:34:52.760 --> 00:34:54.520
what you can do with Terraform modules.

00:34:54.520 --> 00:34:57.000
So we're there CDK is not stuck in this

00:34:57.000 --> 00:34:58.560
like I just have to script around the

00:34:58.560 --> 00:35:00.240
whole bunch of stupid Terraform

00:35:00.240 --> 00:35:02.000
>> converted. I'm converted.

00:35:02.000 --> 00:35:04.160
We talked about this. We talked

00:35:04.160 --> 00:35:05.960
I think one

00:35:05.960 --> 00:35:07.920
Terragrunt was comes in view because of

00:35:07.920 --> 00:35:09.720
TerraTest. Do you have you ever used

00:35:09.720 --> 00:35:11.480
TerraTest? I am a heavy user of

00:35:11.480 --> 00:35:14.280
TerraTest. Really? Yeah. I just today

00:35:14.280 --> 00:35:15.840
was walking through

00:35:15.840 --> 00:35:18.320
We're having AMIs built. So I used

00:35:18.320 --> 00:35:21.200
TerraTest mainly for I think TerraTest

00:35:21.200 --> 00:35:26.040
is amazing. If you have home-cooked AMIs

00:35:26.040 --> 00:35:28.480
that come with a

00:35:28.480 --> 00:35:30.960
pairing paired module. Like for example,

00:35:30.960 --> 00:35:32.760
you know that there's this thing called

00:35:32.760 --> 00:35:36.080
FCK net, which is like a feasible cost

00:35:36.080 --> 00:35:37.600
um

00:35:37.600 --> 00:35:40.320
net gate like net instance. So instead

00:35:40.320 --> 00:35:43.120
of running Let me Google this. FCK what?

00:35:43.120 --> 00:35:44.520
>> net NAT,

00:35:44.520 --> 00:35:46.920
network address translation. It's like a

00:35:46.920 --> 00:35:50.640
10% cost of an AWS NAT gateways.

00:35:50.640 --> 00:35:53.040
>> Oh, so it's just like a a NAT gateway

00:35:53.040 --> 00:35:53.920
replacement.

00:35:53.920 --> 00:35:57.000
>> AWS VPCs support NAT instances. So you

00:35:57.000 --> 00:35:58.600
can set up your routing tables to route

00:35:58.600 --> 00:35:59.880
all the traffic through one EC2

00:35:59.880 --> 00:36:02.160
instance. You don't need a NAT gateway.

00:36:02.160 --> 00:36:06.080
Mhm. So basically I I forked this AMI

00:36:06.080 --> 00:36:08.240
because it's literally like a

00:36:08.240 --> 00:36:11.040
50-line bash script delivered through a

00:36:11.040 --> 00:36:14.160
systemd one-shot to set up your IP

00:36:14.160 --> 00:36:17.400
routing tables and um I'm sure people

00:36:17.400 --> 00:36:19.400
are not doing that. It's just IP tables

00:36:19.400 --> 00:36:22.480
and it's just registering your ethernet

00:36:22.480 --> 00:36:25.600
interface to act as a egress interface,

00:36:25.600 --> 00:36:26.040
right?

00:36:26.040 --> 00:36:27.800
>> Okay. So it's very very basic small

00:36:27.800 --> 00:36:29.600
little thing and it's amazing.

00:36:29.600 --> 00:36:31.880
Um and I have a it together with a

00:36:31.880 --> 00:36:34.120
module. I have a Terraform module that

00:36:34.120 --> 00:36:35.400
basically

00:36:35.400 --> 00:36:37.760
deploys this in several configurations.

00:36:37.760 --> 00:36:41.320
It can be a single NAT instance in one

00:36:41.320 --> 00:36:43.480
subnet and then all the other subnets

00:36:43.480 --> 00:36:45.760
routing tables are routed towards that

00:36:45.760 --> 00:36:47.920
NAT instance and then you would do cross

00:36:47.920 --> 00:36:50.120
AZ that way. Or it can be one NAT

00:36:50.120 --> 00:36:52.840
instance per subnet. So the module has

00:36:52.840 --> 00:36:54.840
many configurations and they need to be

00:36:54.840 --> 00:36:56.800
tested because it's a contract that you

00:36:56.800 --> 00:37:00.400
support. TerraTest is amazing. Yeah.

00:37:00.400 --> 00:37:01.680
Um so so what

00:37:01.680 --> 00:37:03.359
>> I I I was just trying to remember. So

00:37:03.359 --> 00:37:05.680
TerraTest is basically te-

00:37:05.680 --> 00:37:07.760
is testing your

00:37:07.760 --> 00:37:08.960
your

00:37:08.960 --> 00:37:10.280
what do you call it? Your deployed

00:37:10.280 --> 00:37:12.200
infrastructure, right? So what TerraTest

00:37:12.200 --> 00:37:14.400
is is nothing but a bunch of Go lang

00:37:14.400 --> 00:37:15.960
modules.

00:37:15.960 --> 00:37:18.840
And what it shines through is the Go

00:37:18.840 --> 00:37:20.640
lang testing

00:37:20.640 --> 00:37:23.840
capabilities. Go test? Yes, Go test

00:37:23.840 --> 00:37:25.520
because Go test can run test in

00:37:25.520 --> 00:37:28.120
parallel, it can, you know, do all kinds

00:37:28.120 --> 00:37:30.040
of like subnets tests, table-driven

00:37:30.040 --> 00:37:31.600
tests,

00:37:31.600 --> 00:37:33.520
and you can defer so clean up functions

00:37:33.520 --> 00:37:36.640
can be deferred. So Terra- TerraTest is

00:37:36.640 --> 00:37:39.720
just leveraging Go lang test framework

00:37:39.720 --> 00:37:42.480
and it's just a a whole bunch of like

00:37:42.480 --> 00:37:45.880
AWS SDK API implementations so that you

00:37:45.880 --> 00:37:46.880
can

00:37:46.880 --> 00:37:49.040
you know, um

00:37:49.040 --> 00:37:51.200
on one side it it does OS exec

00:37:51.200 --> 00:37:53.840
Terraform, OS exec Packer to test your

00:37:53.840 --> 00:37:55.480
Terraform module and your Packer build

00:37:55.480 --> 00:37:57.200
script. And on the other side it has a

00:37:57.200 --> 00:37:58.800
whole bunch of assertions that you can

00:37:58.800 --> 00:38:01.640
do. Is the instance registered with SSM

00:38:01.640 --> 00:38:03.240
>> in the past, but like I mean, the

00:38:03.240 --> 00:38:06.320
landscape I'm using it. So since

00:38:06.320 --> 00:38:08.840
Terraform has added Actually, I have a

00:38:08.840 --> 00:38:10.400
presentation about this that I delivered

00:38:10.400 --> 00:38:12.640
in a conference in in Singapore. I think

00:38:12.640 --> 00:38:14.080
there's different layers of testing that

00:38:14.080 --> 00:38:16.160
you can do with Terraform. And the first

00:38:16.160 --> 00:38:17.840
one you should use is the built-in test

00:38:17.840 --> 00:38:19.680
command, right? Mhm. Because the

00:38:19.680 --> 00:38:22.240
built-in test command is pure in memory.

00:38:22.240 --> 00:38:24.560
So compared to TerraTest, TerraTest is

00:38:24.560 --> 00:38:26.600
going to provision everything inside a

00:38:26.600 --> 00:38:28.480
file system and it's then going to run

00:38:28.480 --> 00:38:30.359
Terraform in it and it's going to take a

00:38:30.359 --> 00:38:32.240
while. Whereas if you use the Terraform

00:38:32.240 --> 00:38:33.720
test command, it's going to create

00:38:33.720 --> 00:38:34.800
everything in time. It's you used the

00:38:34.800 --> 00:38:36.560
Terraform test command?

00:38:36.560 --> 00:38:38.200
It spins resources up.

00:38:38.200 --> 00:38:40.000
>> Yeah. So I use both. So I I used the

00:38:40.000 --> 00:38:41.920
Terraform tests

00:38:41.920 --> 00:38:44.920
files in my module for some basic cross

00:38:44.920 --> 00:38:47.359
variable validation like does a variable

00:38:47.359 --> 00:38:48.720
condition

00:38:48.720 --> 00:38:50.359
is it triggered as expected? For

00:38:50.359 --> 00:38:51.920
example, I have a module and you should

00:38:51.920 --> 00:38:53.760
set one variable and not the other. And

00:38:53.760 --> 00:38:55.520
if you set both, you should get an

00:38:55.520 --> 00:38:59.000
error. So I have a test around that.

00:38:59.000 --> 00:39:00.600
So do you

00:39:00.600 --> 00:39:04.040
So you use TerraTest with your CDK

00:39:04.040 --> 00:39:07.240
your Terraform CDK stuff? With my my

00:39:07.240 --> 00:39:09.600
library that I built, basically porting

00:39:09.600 --> 00:39:12.359
the AWS CDK L2s, I used TerraTest to

00:39:12.359 --> 00:39:14.400
validate that the L2s work actually

00:39:14.400 --> 00:39:16.840
deploying. Yeah. Really cool. You should

00:39:16.840 --> 00:39:19.440
mention that on your CDK trends.

00:39:19.440 --> 00:39:20.960
It is.

00:39:20.960 --> 00:39:22.280
Uh not on CDK terrain, it's on

00:39:22.280 --> 00:39:24.560
TerraConstructs. Okay, okay. Which is

00:39:24.560 --> 00:39:27.240
the L2 porting on top of CDK terrain.

00:39:27.240 --> 00:39:28.359
But

00:39:28.359 --> 00:39:30.680
there's some really exciting like things

00:39:30.680 --> 00:39:32.359
in the work around that. So soon

00:39:32.359 --> 00:39:34.040
TerraConstructs will will be no longer

00:39:34.040 --> 00:39:36.080
needed, hopefully, and we can have

00:39:36.080 --> 00:39:38.280
something directly that is going to have

00:39:38.280 --> 00:39:40.400
all of the coverage of AWS CDK. And

00:39:40.400 --> 00:39:41.920
that's going to be amazing cuz then you

00:39:41.920 --> 00:39:43.760
can use it with AWS CDK with OpenTofu

00:39:43.760 --> 00:39:47.160
directly. No more CloudFormation. Okay.

00:39:47.160 --> 00:39:48.720
Can you imagine that? That you can use

00:39:48.720 --> 00:39:51.600
all of those AWS CDK L2s directly with

00:39:51.600 --> 00:39:54.120
OpenTofu. That's it's going to be

00:39:54.120 --> 00:39:56.920
amazing. Okay. Let's That sounds

00:39:56.920 --> 00:39:59.040
interesting, but that's that's upcoming,

00:39:59.040 --> 00:40:00.960
right? That's upcoming.

00:40:00.960 --> 00:40:02.840
Well, I think it's it's it's actually

00:40:02.840 --> 00:40:04.960
more realistic now to talk about it

00:40:04.960 --> 00:40:07.280
because if you if you're aware, there's

00:40:07.280 --> 00:40:10.160
this one guy Kanto. I think he's a Japan

00:40:10.160 --> 00:40:12.480
based. He's a heavy contributor to AWS

00:40:12.480 --> 00:40:14.640
CDK and he like writes a lot of blog

00:40:14.640 --> 00:40:17.320
articles about how AWS CDK works and

00:40:17.320 --> 00:40:19.000
some of the features within and he's

00:40:19.000 --> 00:40:20.640
contributing a lot of really really big

00:40:20.640 --> 00:40:22.560
features. And he just announced

00:40:22.560 --> 00:40:25.120
something called AWS CDK direct. And

00:40:25.120 --> 00:40:29.400
what he does is he lets you run AWS CDK

00:40:29.400 --> 00:40:31.880
without CloudFormation. So it reads the

00:40:31.880 --> 00:40:34.600
CloudFormation YAML file and then issues

00:40:34.600 --> 00:40:38.240
AWS CLI commands. Like it just calls

00:40:38.240 --> 00:40:40.760
Oh, I see. I see. Something like that.

00:40:40.760 --> 00:40:42.480
Yeah, and he uses a state file as well.

00:40:42.480 --> 00:40:44.800
I might be a CloudFormation

00:40:44.800 --> 00:40:46.640
apologist here, but like CloudFormation

00:40:46.640 --> 00:40:48.480
kind of works. I mean, do you really

00:40:48.480 --> 00:40:50.480
want to replace it? CloudFormation? I

00:40:50.480 --> 00:40:51.920
mean, they made a lot of improvements

00:40:51.920 --> 00:40:53.400
allowing you to refactor and I think

00:40:53.400 --> 00:40:55.640
we're in a broken record, but it's it's

00:40:55.640 --> 00:40:57.400
pretty painful when you have to deal

00:40:57.400 --> 00:40:59.400
with logical ideas. Well, maybe that's

00:40:59.400 --> 00:41:01.200
no longer a problem because now you can

00:41:01.200 --> 00:41:03.240
tell CloudFormation that some resource

00:41:03.240 --> 00:41:05.240
has changed its identity, so it's not

00:41:05.240 --> 00:41:06.840
really need to be deleted. But one of

00:41:06.840 --> 00:41:09.080
the biggest issues is a crash loopback.

00:41:09.080 --> 00:41:11.720
Not Oh, yeah. The update crash fallback.

00:41:11.720 --> 00:41:14.120
Rollback. Yeah, that's been broken. But

00:41:14.120 --> 00:41:15.760
how does how does Terraform Sorry, how

00:41:15.760 --> 00:41:17.520
does Terraform doesn't have that. Just

00:41:17.520 --> 00:41:19.280
because of the planning stage will catch

00:41:19.280 --> 00:41:20.520
that or something. No, just because

00:41:20.520 --> 00:41:22.320
Terraform cannot rollback. You cannot

00:41:22.320 --> 00:41:24.040
rollback Terraform. Well, you you can

00:41:24.040 --> 00:41:26.600
disable the rollback uh

00:41:26.600 --> 00:41:27.960
you know, you can pass an argument to

00:41:27.960 --> 00:41:29.920
CloudFormation so it doesn't rollback.

00:41:29.920 --> 00:41:32.320
>> Maybe that's new. Not last time I used

00:41:32.320 --> 00:41:34.160
it, which was like 2 years ago, I think.

00:41:34.160 --> 00:41:36.440
Okay. Well, if if there is something

00:41:36.440 --> 00:41:38.120
coming out, I guess it's worth

00:41:38.120 --> 00:41:40.600
evaluating. Like I think someone asked

00:41:40.600 --> 00:41:43.120
me the other day like so with CDK you

00:41:43.120 --> 00:41:45.360
get more linting, you you get more shift

00:41:45.360 --> 00:41:47.320
left, right? I mean, we can we can

00:41:47.320 --> 00:41:49.040
because well, I guess you have the best

00:41:49.040 --> 00:41:50.680
you have the best of both worlds, don't

00:41:50.680 --> 00:41:51.960
you? I mean,

00:41:51.960 --> 00:41:54.040
when I talk with Adam Jacobs and I told

00:41:54.040 --> 00:41:56.040
him I don't know. I told him about

00:41:56.040 --> 00:41:58.040
because he's he's basically saying we

00:41:58.040 --> 00:42:00.120
don't have to live with legacy of

00:42:00.120 --> 00:42:02.080
Terraform providers. We can just rewrite

00:42:02.080 --> 00:42:03.760
the Terraform providers on demand with

00:42:03.760 --> 00:42:05.920
AI whenever we need it with whatever we

00:42:05.920 --> 00:42:06.320
need it.

00:42:06.320 --> 00:42:08.000
>> As someone as someone who's developed a

00:42:08.000 --> 00:42:10.400
Terraform provider, it's not easy. No,

00:42:10.400 --> 00:42:12.320
but and I told him like why would you

00:42:12.320 --> 00:42:13.560
need to do that? Like you're throwing

00:42:13.560 --> 00:42:15.640
away the baby with the bathwater. And he

00:42:15.640 --> 00:42:17.120
says, "No, the model is completely

00:42:17.120 --> 00:42:18.920
wrong. We need to get rid of it." So I

00:42:18.920 --> 00:42:20.880
was like, "Okay." I just said I don't

00:42:20.880 --> 00:42:23.720
understand that reply. So maybe you

00:42:23.720 --> 00:42:27.160
I'm not going to go further there.

00:42:27.160 --> 00:42:28.720
Okay.

00:42:28.720 --> 00:42:31.280
Um hold on. I want you to get on the

00:42:31.280 --> 00:42:32.720
topic whereby

00:42:32.720 --> 00:42:35.360
Sorry. What ecosystem gives you the max

00:42:35.360 --> 00:42:38.240
sort of shift left ability? Like one

00:42:38.240 --> 00:42:39.920
thing I

00:42:39.920 --> 00:42:42.080
noticed Let me just share this. How do I

00:42:42.080 --> 00:42:44.480
share my Like there's been a few local

00:42:44.480 --> 00:42:46.480
stacks sort of replacement

00:42:46.480 --> 00:42:48.400
>> Today there was like this mini stack.

00:42:48.400 --> 00:42:51.000
Yeah, there's there's been a few.

00:42:51.000 --> 00:42:52.800
I just noticed them popping up here,

00:42:52.800 --> 00:42:54.160
there, and everywhere. Yeah, since

00:42:54.160 --> 00:42:56.000
LocalStack is like changing their

00:42:56.000 --> 00:42:56.800
pricing model.

00:42:56.800 --> 00:42:59.000
>> called Flocky. Oh, yeah. I've seen that

00:42:59.000 --> 00:43:02.280
one. Um and then

00:43:02.280 --> 00:43:04.720
I noticed this this AWS developer I

00:43:04.720 --> 00:43:07.040
follow and I noticed I think randomly on

00:43:07.040 --> 00:43:09.000
GitHub. Sometimes GitHub can just show

00:43:09.000 --> 00:43:11.280
you what people are working on.

00:43:11.280 --> 00:43:12.680
And

00:43:12.680 --> 00:43:13.800
um

00:43:13.800 --> 00:43:15.600
hold on. What is my username? And I

00:43:15.600 --> 00:43:17.120
noticed he was working on this thing

00:43:17.120 --> 00:43:18.840
called

00:43:18.840 --> 00:43:22.160
um yeah, this Eman Fatih.

00:43:22.160 --> 00:43:25.520
Um he he This I mean, this this comes in

00:43:25.520 --> 00:43:27.480
my opinion, since he works at AWS, this

00:43:27.480 --> 00:43:28.720
comes out of AWS.

00:43:28.720 --> 00:43:30.800
>> No. No. No. No.

00:43:30.800 --> 00:43:31.920
You cannot.

00:43:31.920 --> 00:43:33.640
I can I mean, can you imagine working at

00:43:33.640 --> 00:43:35.000
a big old company then everything you

00:43:35.000 --> 00:43:36.640
does gets associated with it? That would

00:43:36.640 --> 00:43:38.920
be so annoying.

00:43:38.920 --> 00:43:39.720
I would hate that.

00:43:39.720 --> 00:43:41.760
>> Obviously he vibed it. I mean, you can

00:43:41.760 --> 00:43:42.680
tell.

00:43:42.680 --> 00:43:44.200
I was about to say and I like that

00:43:44.200 --> 00:43:45.520
there's Gherkin in it. That's super

00:43:45.520 --> 00:43:46.760
interesting. I want to have a look at

00:43:46.760 --> 00:43:48.280
that. Yeah, so

00:43:48.280 --> 00:43:49.800
um

00:43:49.800 --> 00:43:51.440
Okay, let me get my thoughts together.

00:43:51.440 --> 00:43:52.400
So

00:43:52.400 --> 00:43:54.800
you you use you use CDK, use Terraform.

00:43:54.800 --> 00:43:56.920
Okay, what is the best ecosystem that

00:43:56.920 --> 00:43:59.680
gives you the fastest iterations and

00:43:59.680 --> 00:44:02.560
before you even deploy on the cloud,

00:44:02.560 --> 00:44:05.359
that gives you the max amount of, you

00:44:05.359 --> 00:44:07.160
know, verification that what you built

00:44:07.160 --> 00:44:11.680
works. So to me it's it's something like

00:44:11.680 --> 00:44:13.720
using uh I guess in the past, since I

00:44:13.720 --> 00:44:15.480
don't have that much CDK experience

00:44:15.480 --> 00:44:17.359
except my last gig, I would use

00:44:17.359 --> 00:44:19.720
Terraform and I've maybe used uh

00:44:19.720 --> 00:44:22.120
LocalStack and and then I would just,

00:44:22.120 --> 00:44:24.040
you know, iterate from there and then I

00:44:24.040 --> 00:44:26.080
would deploy.

00:44:26.080 --> 00:44:27.720
That would probably be the fastest shift

00:44:27.720 --> 00:44:30.320
left way. But have you come across any

00:44:30.320 --> 00:44:32.359
faster ways of doing it? And I I don't

00:44:32.359 --> 00:44:33.480
know exactly what happened to

00:44:33.480 --> 00:44:35.400
LocalStack. Did they just go really

00:44:35.400 --> 00:44:37.960
pricey or something? No, they changed

00:44:37.960 --> 00:44:39.960
the license for

00:44:39.960 --> 00:44:40.840
um

00:44:40.840 --> 00:44:42.560
Before you could download the Docker

00:44:42.560 --> 00:44:44.720
images of the community edition and you

00:44:44.720 --> 00:44:46.760
could just run them, but now they

00:44:46.760 --> 00:44:48.680
removed those. They deprecated those

00:44:48.680 --> 00:44:50.720
Docker images and they only support the

00:44:50.720 --> 00:44:52.920
enterprise Docker images and you have to

00:44:52.920 --> 00:44:54.960
give it a token. So they are tracking

00:44:54.960 --> 00:44:56.840
your usage. And another thing that they

00:44:56.840 --> 00:44:58.840
did is if it detects that it's running

00:44:58.840 --> 00:45:01.359
inside GitHub Actions or in CI, it just

00:45:01.359 --> 00:45:03.560
shuts down. So you can use it locally,

00:45:03.560 --> 00:45:05.359
but you can't use it in CI at all. So

00:45:05.359 --> 00:45:07.440
that's where I think people got upset.

00:45:07.440 --> 00:45:09.480
One, you can't use it anymore without

00:45:09.480 --> 00:45:11.280
creating an account and giving it a

00:45:11.280 --> 00:45:13.920
token so that they can track you. Two,

00:45:13.920 --> 00:45:15.920
you can't run it at all in CI anymore.

00:45:15.920 --> 00:45:18.920
Okay, that's a good summary. Yeah.

00:45:18.920 --> 00:45:20.720
Um

00:45:20.720 --> 00:45:22.040
that sucks.

00:45:22.040 --> 00:45:23.960
So what in your opinion is the fastest

00:45:23.960 --> 00:45:26.960
shift left way of of doing it? I mean,

00:45:26.960 --> 00:45:28.680
I've always been intrigued by LocalStack

00:45:28.680 --> 00:45:30.160
and I actually like to use it for like

00:45:30.160 --> 00:45:31.720
take-home assignments because it doesn't

00:45:31.720 --> 00:45:34.080
require a candidate to have any account

00:45:34.080 --> 00:45:37.080
and I give them like a very basic AWS

00:45:37.080 --> 00:45:39.920
infra that allows them to um you know,

00:45:39.920 --> 00:45:43.160
do everything within the challenge that

00:45:43.160 --> 00:45:44.520
um

00:45:44.520 --> 00:45:46.760
that doesn't require that works without

00:45:46.760 --> 00:45:48.480
a LocalStack license or anything like

00:45:48.480 --> 00:45:51.040
that, right? But I have never used it

00:45:51.040 --> 00:45:53.560
for anything like

00:45:53.560 --> 00:45:54.720
um

00:45:54.720 --> 00:45:56.600
you know, [snorts] validating a large

00:45:56.600 --> 00:45:58.359
serverless setup. I think LocalStack

00:45:58.359 --> 00:46:00.240
only makes sense if you are really

00:46:00.240 --> 00:46:01.840
heavily depending on like serverless

00:46:01.840 --> 00:46:04.240
services, right? That you cannot Like

00:46:04.240 --> 00:46:06.280
otherwise you would just spin up like a

00:46:06.280 --> 00:46:08.840
compose environment or you would spin up

00:46:08.840 --> 00:46:09.640
a

00:46:09.640 --> 00:46:10.240
you know, the services

00:46:10.240 --> 00:46:12.280
>> Well, it's also good for all the like

00:46:12.280 --> 00:46:15.920
AWS native, you know, Kinesis and So I I

00:46:15.920 --> 00:46:18.000
have I built like this the Terraform

00:46:18.000 --> 00:46:21.280
provider that uses SQS and I wanted to

00:46:21.280 --> 00:46:23.760
validate it really fast and you can you

00:46:23.760 --> 00:46:25.800
can actually have like complete

00:46:25.800 --> 00:46:28.960
emulators of of SQS. I'm not sure about

00:46:28.960 --> 00:46:31.040
Kinesis, but I'm I would be surprised if

00:46:31.040 --> 00:46:32.840
you don't have that. And the funny thing

00:46:32.840 --> 00:46:35.040
is those are like dedicated projects,

00:46:35.040 --> 00:46:37.320
which I think are better than going with

00:46:37.320 --> 00:46:39.120
like a massive library that tries to

00:46:39.120 --> 00:46:40.800
track everything. Yeah. Yeah. Yeah. I

00:46:40.800 --> 00:46:42.800
know what you mean. Like there's a few

00:46:42.800 --> 00:46:46.359
like DynamoDB type Yeah, and and and the

00:46:46.359 --> 00:46:50.680
the one that I was using for SQS was um

00:46:50.680 --> 00:46:54.240
Was it ask Yeah, it was SQS. Was the was

00:46:54.240 --> 00:46:57.000
written in I think Elixir or or um

00:46:57.000 --> 00:46:59.359
something like Java. It actually Yeah, I

00:46:59.359 --> 00:47:01.040
think it was using the JVM under the

00:47:01.040 --> 00:47:03.400
hood. Uh is it Scala? It was something

00:47:03.400 --> 00:47:05.000
crazy. Like I would never use it. I was

00:47:05.000 --> 00:47:06.160
like, "Ah, I can't use that

00:47:06.160 --> 00:47:07.440
because I don't I don't

00:47:07.440 --> 00:47:07.640
understand."

00:47:07.640 --> 00:47:09.320
>> self-contained as well. Yeah, it was

00:47:09.320 --> 00:47:11.040
absolutely like yeah, it didn't didn't

00:47:11.040 --> 00:47:12.160
matter what it was written in.

00:47:12.160 --> 00:47:15.000
>> of most of AWS is probably some insane

00:47:15.000 --> 00:47:17.120
Java. Yeah, but I mean, this was a this

00:47:17.120 --> 00:47:19.640
was like a a public thing that just um

00:47:19.640 --> 00:47:21.440
exposes the same API endpoints. And I

00:47:21.440 --> 00:47:22.920
had good success with it. I was very

00:47:22.920 --> 00:47:25.280
happy. Like at least I was able to test

00:47:25.280 --> 00:47:26.600
this one little thing that needed that

00:47:26.600 --> 00:47:28.359
one little service. So I could run it

00:47:28.359 --> 00:47:30.440
locally. Um

00:47:30.440 --> 00:47:31.840
but that's why I never use something

00:47:31.840 --> 00:47:34.320
like LocalStack uh except for like

00:47:34.320 --> 00:47:35.680
candidate take-home assignments or

00:47:35.680 --> 00:47:37.680
something like that. Mhm.

00:47:37.680 --> 00:47:39.040
So it sounds like you haven't really

00:47:39.040 --> 00:47:42.359
explored this shift shift shift left

00:47:42.359 --> 00:47:44.680
north stuff that hard. I mean, to me

00:47:44.680 --> 00:47:46.600
this sounds like the biggest problem to

00:47:46.600 --> 00:47:48.600
solve, really.

00:47:48.600 --> 00:47:50.480
Especially now with AI because you want

00:47:50.480 --> 00:47:52.800
AI agents to be in a little Why don't

00:47:52.800 --> 00:47:54.480
you just give them an AWS account? Cuz

00:47:54.480 --> 00:47:56.640
I'm not insane. No, I mean, why don't

00:47:56.640 --> 00:47:58.640
you just give a developer an AWS account

00:47:58.640 --> 00:48:00.400
so and a sandbox so that he can, you

00:48:00.400 --> 00:48:02.359
know, test it against the real AWS API

00:48:02.359 --> 00:48:04.359
services? Why would you try to emulate

00:48:04.359 --> 00:48:05.760
it? Like I don't get that.

00:48:05.760 --> 00:48:08.520
>> Because of speed. Speed and

00:48:08.520 --> 00:48:09.680
safety and

00:48:09.680 --> 00:48:11.760
>> you mock your tests, you get mock

00:48:11.760 --> 00:48:14.280
confidence. Yeah, I I

00:48:14.280 --> 00:48:16.720
I still maintain that this needs to be a

00:48:16.720 --> 00:48:19.040
bit explored a bit more further. I'll

00:48:19.040 --> 00:48:21.160
I'm going to bolden it. Oh my gosh.

00:48:21.160 --> 00:48:23.080
Okay, last topic.

00:48:23.080 --> 00:48:24.320
Cuz I need to run out and buy some

00:48:24.320 --> 00:48:26.240
bread. Orchestration

00:48:26.240 --> 00:48:29.000
uh can I assume that you that you and

00:48:29.000 --> 00:48:30.920
like most people nowadays just use

00:48:30.920 --> 00:48:32.400
GitHub workflows when you want to

00:48:32.400 --> 00:48:34.600
coordinate something or do you use

00:48:34.600 --> 00:48:37.520
something fancier to coordinate? I'm

00:48:37.520 --> 00:48:39.359
and you know, Is this the last topic? I

00:48:39.359 --> 00:48:41.333
don't know. I want to change the topic.

00:48:41.333 --> 00:48:42.000
>> [laughter]

00:48:42.000 --> 00:48:43.680
>> Can I change the topic?

00:48:43.680 --> 00:48:45.120
Sure. So you know, because I think

00:48:45.120 --> 00:48:46.320
there's more interesting with the trivy

00:48:46.320 --> 00:48:48.760
hacks and and basically the axios hack

00:48:48.760 --> 00:48:50.520
that happened and axios being on the

00:48:50.520 --> 00:48:54.359
registry with a infostealer for 2 3

00:48:54.359 --> 00:48:56.000
hours before it was detected and

00:48:56.000 --> 00:48:58.240
removed. So I think right now and and

00:48:58.240 --> 00:48:59.520
actually it's funny because I just got a

00:48:59.520 --> 00:49:02.920
DM from someone that we both are trying

00:49:02.920 --> 00:49:05.200
to build something and I said I'm sorry,

00:49:05.200 --> 00:49:06.960
but I've been busy. I I've not been

00:49:06.960 --> 00:49:09.160
working on this because of other stuff,

00:49:09.160 --> 00:49:10.680
but if you want to work me to focus on

00:49:10.680 --> 00:49:12.320
this, I can like pick it back up. And he

00:49:12.320 --> 00:49:13.800
replied,

00:49:13.800 --> 00:49:15.800
"I've been really busy with these supply

00:49:15.800 --> 00:49:17.800
chain attacks." And I think it's it's

00:49:17.800 --> 00:49:19.840
actually a very interesting topic

00:49:19.840 --> 00:49:21.680
because everyone needs to change

00:49:21.680 --> 00:49:24.680
completely the way that we handle

00:49:24.680 --> 00:49:26.760
CVE and patch and

00:49:26.760 --> 00:49:28.600
you know, releases. So maybe I can give

00:49:28.600 --> 00:49:30.320
some background, right?

00:49:30.320 --> 00:49:33.080
The the established practice today to

00:49:33.080 --> 00:49:34.280
handle

00:49:34.280 --> 00:49:38.080
CVEs in the past was basically to adopt

00:49:38.080 --> 00:49:40.200
those fixes as soon as possible. So, if

00:49:40.200 --> 00:49:42.400
you use semantic versioning, you have

00:49:42.400 --> 00:49:43.760
three numbers. The first one is the

00:49:43.760 --> 00:49:44.680
major

00:49:44.680 --> 00:49:46.280
release, the second one is the minor

00:49:46.280 --> 00:49:48.120
release number, and the last one is the

00:49:48.120 --> 00:49:49.720
patch release number.

00:49:49.720 --> 00:49:49.960
Mhm.

00:49:49.960 --> 00:49:52.000
>> And so, usually if there's a CVE, which

00:49:52.000 --> 00:49:54.760
you're not introducing any new um

00:49:54.760 --> 00:49:57.440
features or you're doing a bug fix, you

00:49:57.440 --> 00:49:59.640
you bump that patch version number, the

00:49:59.640 --> 00:50:01.200
last number, right? If you're

00:50:01.200 --> 00:50:02.960
introducing new features, you do the

00:50:02.960 --> 00:50:04.760
minor, and if it's a massive breaking

00:50:04.760 --> 00:50:07.040
change, you do the the major uh release

00:50:07.040 --> 00:50:08.840
bump. So,

00:50:08.840 --> 00:50:10.800
the way that you use semantic versioning

00:50:10.800 --> 00:50:13.320
as a consumer of dependencies is by

00:50:13.320 --> 00:50:16.240
setting up some type of rules around or

00:50:16.240 --> 00:50:18.120
constraints around the packages that you

00:50:18.120 --> 00:50:20.640
accept, right? So, you can set up your

00:50:20.640 --> 00:50:22.520
your project, and this is something that

00:50:22.520 --> 00:50:24.280
people learned quickly when

00:50:24.280 --> 00:50:26.400
um they didn't put any constraints on

00:50:26.400 --> 00:50:28.480
the dependencies, and they just adopted

00:50:28.480 --> 00:50:30.960
minor and major um versions, and then

00:50:30.960 --> 00:50:33.200
they got hacked or they got uh impacted

00:50:33.200 --> 00:50:35.080
by those. So, people very quickly

00:50:35.080 --> 00:50:37.040
learned to put a constraint around

00:50:37.040 --> 00:50:39.800
dependency that you only accept uh patch

00:50:39.800 --> 00:50:42.440
releases or maybe sometimes minor

00:50:42.440 --> 00:50:44.640
releases, but usually just patch because

00:50:44.640 --> 00:50:46.240
minor release Well, not minor release

00:50:46.240 --> 00:50:47.840
shouldn't break you, right? Yeah, yeah,

00:50:47.840 --> 00:50:49.680
yeah, yeah, yeah. So, so the best

00:50:49.680 --> 00:50:51.200
practice right now that has been

00:50:51.200 --> 00:50:53.080
established long by Dependabot,

00:50:53.080 --> 00:50:55.520
Renovate, and others is to basically

00:50:55.520 --> 00:50:57.480
constantly check if any of your

00:50:57.480 --> 00:50:59.720
dependencies has a patch release. And

00:50:59.720 --> 00:51:01.640
then that usually indicates, because

00:51:01.640 --> 00:51:03.200
there's a CVE, you should upgrade as

00:51:03.200 --> 00:51:05.680
soon as possible. And so, Dependabot

00:51:05.680 --> 00:51:07.920
detects it, creates a pull request, and

00:51:07.920 --> 00:51:09.200
if you're really fancy, and I think a

00:51:09.200 --> 00:51:10.480
lot of people are, definitely if you

00:51:10.480 --> 00:51:12.280
manage many projects or open source

00:51:12.280 --> 00:51:15.680
projects, they set up an auto merge if

00:51:15.680 --> 00:51:17.920
all of the CICD pass, it immediately

00:51:17.920 --> 00:51:20.200
merges, right? So, that means that as

00:51:20.200 --> 00:51:21.760
soon as a patch release is published

00:51:21.760 --> 00:51:24.160
>> not quite fancy, but yeah, I mean No,

00:51:24.160 --> 00:51:26.480
all of the CDK, TN, all of the CDK,

00:51:26.480 --> 00:51:28.800
Terrain is auto bumping like that.

00:51:28.800 --> 00:51:30.720
>> I When I say fancy, I mean that's

00:51:30.720 --> 00:51:32.640
actually quite mature, but yeah, carry

00:51:32.640 --> 00:51:34.920
on. Yeah. Cool. So, you

00:51:34.920 --> 00:51:37.680
is the established best practice until

00:51:37.680 --> 00:51:40.520
today. Yeah. Because with the recent

00:51:40.520 --> 00:51:43.160
supply chain attacks, and a lot of those

00:51:43.160 --> 00:51:44.840
basically what happened is some projects

00:51:44.840 --> 00:51:47.040
get compromised, and the attackers are

00:51:47.040 --> 00:51:50.080
smart. They just push a patch release,

00:51:50.080 --> 00:51:51.680
right? So, everyone that's running

00:51:51.680 --> 00:51:54.120
Dependabot updates and pulls in this

00:51:54.120 --> 00:51:57.120
this package from the registry. So, this

00:51:57.120 --> 00:51:59.000
this best best practice works against

00:51:59.000 --> 00:52:00.760
people. And in the age of AI, there's

00:52:00.760 --> 00:52:02.520
been a lot more supply chain attacks, a

00:52:02.520 --> 00:52:04.720
lot more info stealers, a lot more like

00:52:04.720 --> 00:52:06.240
>> so there must be something to be said

00:52:06.240 --> 00:52:08.320
for people who don't update, right? I'm

00:52:08.320 --> 00:52:09.680
just looking at my notes.

00:52:09.680 --> 00:52:10.840
>> because if you don't update, you're

00:52:10.840 --> 00:52:12.960
still exposed to CVEs, which are also

00:52:12.960 --> 00:52:14.720
being detected first faster thanks to

00:52:14.720 --> 00:52:16.840
AI, because AI is analyzing codebases

00:52:16.840 --> 00:52:18.840
and finding CVEs faster. So, there are

00:52:18.840 --> 00:52:20.560
more patch releases. Yeah, it's between

00:52:20.560 --> 00:52:22.240
a rock and a hard place, isn't it? So,

00:52:22.240 --> 00:52:24.240
what you need to So, and so, I was

00:52:24.240 --> 00:52:26.120
asking around, how do I protect my

00:52:26.120 --> 00:52:27.960
project? Because we just forked, I don't

00:52:27.960 --> 00:52:29.440
want to get like a supply chain attack,

00:52:29.440 --> 00:52:30.960
and then all of the trust in the project

00:52:30.960 --> 00:52:32.520
dies. How do I make sure that this

00:52:32.520 --> 00:52:34.360
project is not on the newspaper? Maybe

00:52:34.360 --> 00:52:36.680
good and any news is better than than no

00:52:36.680 --> 00:52:38.680
news at all, right? I could just let it

00:52:38.680 --> 00:52:40.200
get hacked and then say, "Ah, people

00:52:40.200 --> 00:52:42.280
like Finally know there's a fork."

00:52:42.280 --> 00:52:44.440
No such thing as a bad publicity. Yeah,

00:52:44.440 --> 00:52:46.000
so anyway, I I don't intend to have it

00:52:46.000 --> 00:52:48.920
in the publicity in that way. So, I was

00:52:48.920 --> 00:52:51.520
asking around, and I really like how

00:52:51.520 --> 00:52:53.040
we're trying to solve this problem. It's

00:52:53.040 --> 00:52:55.200
an interesting thing to discuss as well.

00:52:55.200 --> 00:52:57.160
Cuz basically, a lot of these package

00:52:57.160 --> 00:52:59.280
managers, aside from you obviously

00:52:59.280 --> 00:53:02.120
fixing your long-lived credentials so

00:53:02.120 --> 00:53:03.680
that you they don't get stolen and using

00:53:03.680 --> 00:53:05.480
OIDC publishers so you get a short-lived

00:53:05.480 --> 00:53:07.360
credential to publish so that even if an

00:53:07.360 --> 00:53:08.680
info stealer got that key, it's already

00:53:08.680 --> 00:53:10.320
expired by the time it tries to use it.

00:53:10.320 --> 00:53:12.200
Yeah. Uh aside from first of trying to

00:53:12.200 --> 00:53:13.680
make sure that you don't get compromised

00:53:13.680 --> 00:53:15.760
that way, switch everything to OIDC for

00:53:15.760 --> 00:53:17.600
publishing. The other way is so that you

00:53:17.600 --> 00:53:19.160
don't get compromised as a consumer of

00:53:19.160 --> 00:53:21.280
these dependencies is to make sure that

00:53:21.280 --> 00:53:23.840
you use the feature that they have

00:53:23.840 --> 00:53:26.400
added, which is one, you can demand that

00:53:26.400 --> 00:53:27.040
the

00:53:27.040 --> 00:53:30.280
dependency must be older than X days or

00:53:30.280 --> 00:53:33.360
seconds. So, you can say to Dependabot

00:53:33.360 --> 00:53:36.040
or to the package manager, do not accept

00:53:36.040 --> 00:53:37.960
any package that hasn't been on the

00:53:37.960 --> 00:53:41.680
registry more than less than 3 days.

00:53:41.680 --> 00:53:44.480
Yeah, that sounds like a a very good

00:53:44.480 --> 00:53:46.120
approach, right? Because now if the

00:53:46.120 --> 00:53:48.720
package gets compromised, then hopefully

00:53:48.720 --> 00:53:51.160
it get discovered, and by the time you

00:53:51.160 --> 00:53:53.680
run, either it's been deleted or, you

00:53:53.680 --> 00:53:55.480
know, you shouldn't have it. Yeah, I

00:53:55.480 --> 00:53:57.720
guess that's like the Debian stable

00:53:57.720 --> 00:53:59.440
model, like it has to go through an

00:53:59.440 --> 00:54:01.920
unstable phase before it drops down into

00:54:01.920 --> 00:54:03.840
stability.

00:54:03.840 --> 00:54:05.240
And then the second thing is because I

00:54:05.240 --> 00:54:06.480
don't think that's enough, right? Maybe

00:54:06.480 --> 00:54:08.000
it was detected, but it wasn't taken

00:54:08.000 --> 00:54:09.160
down because of I don't know what

00:54:09.160 --> 00:54:11.280
reason, but usually NPM, G, actually the

00:54:11.280 --> 00:54:12.640
package registry managers will

00:54:12.640 --> 00:54:14.680
immediately take it down. Mhm.

00:54:14.680 --> 00:54:16.560
The second thing is you need to gate

00:54:16.560 --> 00:54:19.840
your CI, because your CI is not your

00:54:19.840 --> 00:54:22.280
unit test passing is not sufficient

00:54:22.280 --> 00:54:24.400
anymore for an auto merge. You have to

00:54:24.400 --> 00:54:27.200
run a security scan in your CI, right?

00:54:27.200 --> 00:54:28.160
And I was asking

00:54:28.160 --> 00:54:29.520
>> do that. A lot of people do that, but

00:54:29.520 --> 00:54:30.040
like

00:54:30.040 --> 00:54:31.560
>> Yeah, yeah. You were just saying it was

00:54:31.560 --> 00:54:33.080
fancy.

00:54:33.080 --> 00:54:34.400
Well, a a lot of people A lot of people

00:54:34.400 --> 00:54:36.960
do, but like I I I mean, I'm so worn

00:54:36.960 --> 00:54:38.840
down by Well, I'm not going to I'm going

00:54:38.840 --> 00:54:41.560
to say the name, Snyk. Like so many

00:54:41.560 --> 00:54:45.040
There's so many like noisy Snyk alerts

00:54:45.040 --> 00:54:46.320
Well, actually, because they have an

00:54:46.320 --> 00:54:48.920
open source supports option, I was

00:54:48.920 --> 00:54:50.520
thinking about, but I don't want to

00:54:50.520 --> 00:54:52.280
because we use it at work and I hate it.

00:54:52.280 --> 00:54:53.640
I don't even want to use it for my own

00:54:53.640 --> 00:54:54.840
source.

00:54:54.840 --> 00:54:56.000
>> What what what What are you talking

00:54:56.000 --> 00:54:57.560
about? Something What you just

00:54:57.560 --> 00:54:58.080
mentioned.

00:54:58.080 --> 00:55:00.120
>> Oh, okay, yeah. I don't I don't I don't

00:55:00.120 --> 00:55:02.280
mean to cast shade, but like it's

00:55:02.280 --> 00:55:03.800
probably like a user error, like we

00:55:03.800 --> 00:55:05.600
didn't configure it correctly or someone

00:55:05.600 --> 00:55:07.040
didn't configure it correctly. Some

00:55:07.040 --> 00:55:08.120
organ-

00:55:08.120 --> 00:55:10.640
inevitably org administrator applies it

00:55:10.640 --> 00:55:12.920
to every repo. So, every time you look

00:55:12.920 --> 00:55:15.520
at a repo, there's a bazillion security

00:55:15.520 --> 00:55:17.720
errors. You can hardly

00:55:17.720 --> 00:55:20.200
work through the mess. Okay. So, then

00:55:20.200 --> 00:55:22.680
then what I asked around and people told

00:55:22.680 --> 00:55:25.600
me is that actually NPM audit, in this

00:55:25.600 --> 00:55:26.960
case because we're doing the TypeScript

00:55:26.960 --> 00:55:29.960
projects, right? NPM audit is pretty

00:55:29.960 --> 00:55:31.840
good. So, Yeah, it is It is a great

00:55:31.840 --> 00:55:34.000
tool. So, when I talked about this auto

00:55:34.000 --> 00:55:36.160
update system that automatically merges

00:55:36.160 --> 00:55:38.280
the PR if everything's good, actually

00:55:38.280 --> 00:55:40.200
that's all built into Projen. Like you

00:55:40.200 --> 00:55:41.760
just have to enable a flag. You just

00:55:41.760 --> 00:55:43.960
say, "Yes, I want weekly updates. Yes, I

00:55:43.960 --> 00:55:45.960
want you to to make sure

00:55:45.960 --> 00:55:49.160
>> Okay. Yeah, and and Projen even switched

00:55:49.160 --> 00:55:51.640
from Yarn Classic because apparently

00:55:51.640 --> 00:55:53.200
Yarn Classic is not maintained and

00:55:53.200 --> 00:55:54.560
doesn't have any of those features. So,

00:55:54.560 --> 00:55:57.040
if you're on Yarn Classic, get off it,

00:55:57.040 --> 00:55:59.359
right? So, I'm moving everything from

00:55:59.359 --> 00:56:02.080
this inherited project from from CDK,

00:56:02.080 --> 00:56:04.040
TF, it's all using Yarn Classic. I'm

00:56:04.040 --> 00:56:07.280
migrating everything over to PNPM. Uh

00:56:07.280 --> 00:56:09.160
Projen migrated from Yarn Classic to

00:56:09.160 --> 00:56:10.800
NPM. They said it's good enough, it has

00:56:10.800 --> 00:56:12.520
everything we need, and it's less of a

00:56:12.520 --> 00:56:14.600
another hurdle for people to learn if

00:56:14.600 --> 00:56:16.560
they don't not familiar with it with

00:56:16.560 --> 00:56:19.560
with Yarn, Berry, or PNPM. But I'm using

00:56:19.560 --> 00:56:21.760
PNPM, I really like it. And Projen has

00:56:21.760 --> 00:56:24.320
support for PNPM dev dependency flag.

00:56:24.320 --> 00:56:26.040
So, you can just say like it must be

00:56:26.040 --> 00:56:28.960
older than 3 days, and I want you to run

00:56:28.960 --> 00:56:31.840
an NPM audit in the CI of that that job

00:56:31.840 --> 00:56:33.680
before it auto merges. It sounds like

00:56:33.680 --> 00:56:36.760
NPM in a way or no that that that

00:56:36.760 --> 00:56:38.960
ecosystem has the best tooling, because

00:56:38.960 --> 00:56:42.120
the the tooling in Python, I would say

00:56:42.120 --> 00:56:44.359
there isn't like an NPM audit. And

00:56:44.359 --> 00:56:46.160
that's what I was saying. Just that What

00:56:46.160 --> 00:56:47.680
do you want those guys to do for

00:56:47.680 --> 00:56:47.880
security?

00:56:47.880 --> 00:56:50.440
>> Yeah, that goes back to me bemoaning the

00:56:50.440 --> 00:56:53.240
data platform stuff. I mean, UV UV

00:56:53.240 --> 00:56:55.120
Astral would probably have those things,

00:56:55.120 --> 00:56:55.600
right?

00:56:55.600 --> 00:56:56.960
>> No, it doesn't really. Like the whole

00:56:56.960 --> 00:56:59.120
Astral update story is still a mess.

00:56:59.120 --> 00:57:01.920
Okay. That's not good. That's not good.

00:57:01.920 --> 00:57:03.400
So, yeah, I I think it was really cool

00:57:03.400 --> 00:57:07.560
because it's like um a really like

00:57:07.560 --> 00:57:09.880
really hot and and really interesting

00:57:09.880 --> 00:57:11.560
really. Uh because now you get like to

00:57:11.560 --> 00:57:12.920
play balance. But then, of course, the

00:57:12.920 --> 00:57:15.200
comment the first idea that most people

00:57:15.200 --> 00:57:16.680
have is like, "But if everything

00:57:16.680 --> 00:57:18.560
everyone is waiting to adopt a package,

00:57:18.560 --> 00:57:20.440
how are we even going to know if it's

00:57:20.440 --> 00:57:22.160
compromised?" Yeah, yeah, yeah. Cuz now

00:57:22.160 --> 00:57:23.600
everyone's going to wait 3 days before

00:57:23.600 --> 00:57:24.800
they they download it, and then they're

00:57:24.800 --> 00:57:25.520
only going to find

00:57:25.520 --> 00:57:28.200
>> imagine that that the NPM What whatever

00:57:28.200 --> 00:57:31.280
happens in NPM is going to be

00:57:31.280 --> 00:57:32.560
the the

00:57:32.560 --> 00:57:34.280
What is the the expression? It's like

00:57:34.280 --> 00:57:36.400
necessity is the mother of invention.

00:57:36.400 --> 00:57:38.120
The stuff that happens in NPM is on the

00:57:38.120 --> 00:57:39.720
forefront, right? And then it will

00:57:39.720 --> 00:57:42.080
trickle down to other ecosystems or the

00:57:42.080 --> 00:57:44.600
supply chain mitigations. So, so I have

00:57:44.600 --> 00:57:46.200
a lot of work to do in that aspect,

00:57:46.200 --> 00:57:48.040
actually. Like I still have to switch I

00:57:48.040 --> 00:57:50.000
have already enabled OIDC publishing on

00:57:50.000 --> 00:57:52.000
all the on all the registries, but I

00:57:52.000 --> 00:57:53.920
haven't switched over the actual

00:57:53.920 --> 00:57:55.560
publishers, the GitHub runners, to use

00:57:55.560 --> 00:57:58.359
OIDC. And Okay, you're like an NPM

00:57:58.359 --> 00:58:00.640
publisher deluxe, then I suppose. No,

00:58:00.640 --> 00:58:05.120
I'm like a NPM, NuGet, Maven, and PyPI

00:58:05.120 --> 00:58:07.480
publisher all across the board now.

00:58:07.480 --> 00:58:09.880
Yeah, that's That sounds kind of risky,

00:58:09.880 --> 00:58:12.040
actually. Yeah. I think I think I've got

00:58:12.040 --> 00:58:12.800
like

00:58:12.800 --> 00:58:15.120
I've got a couple of PyPI packages, and

00:58:15.120 --> 00:58:17.080
I've got a couple of NPM packages. I

00:58:17.080 --> 00:58:19.200
haven't uploaded to them for a while,

00:58:19.200 --> 00:58:21.000
but but yeah, I think they're both just

00:58:21.000 --> 00:58:24.000
passwords that I've never updated.

00:58:24.000 --> 00:58:26.160
Yeah. But then to to address the concern

00:58:26.160 --> 00:58:27.840
of people saying like, "But now

00:58:27.840 --> 00:58:29.320
everyone's waiting, so how are we going

00:58:29.320 --> 00:58:31.600
to know it's compromised?" But I think

00:58:31.600 --> 00:58:33.840
that that security companies have a real

00:58:33.840 --> 00:58:35.359
incentive, because every time And now

00:58:35.359 --> 00:58:36.880
they're like all over Reddit, like Step

00:58:36.880 --> 00:58:39.160
Security, but there's a couple of them,

00:58:39.160 --> 00:58:40.920
Roll Security. There's like a lot of

00:58:40.920 --> 00:58:42.520
security companies, and every time this

00:58:42.520 --> 00:58:44.720
happens, they're on Reddit posting,

00:58:44.720 --> 00:58:46.440
"There was this massive supply chain

00:58:46.440 --> 00:58:49.760
attack." They have an incentive to find

00:58:49.760 --> 00:58:52.359
to scan the packages published and find

00:58:52.359 --> 00:58:55.480
any info stealer, any type of

00:58:55.480 --> 00:58:58.080
you know, in injected whatever they they

00:58:58.080 --> 00:59:00.080
usually identify things like the package

00:59:00.080 --> 00:59:03.240
is unusually larger than than before.

00:59:03.240 --> 00:59:05.640
Yeah. The package has some base64

00:59:05.640 --> 00:59:07.560
encoded like strings in there that are

00:59:07.560 --> 00:59:09.800
weird. So, they do this this uh

00:59:09.800 --> 00:59:12.359
heuristical scans to very quickly detect

00:59:12.359 --> 00:59:14.160
them. So, I don't think we need to worry

00:59:14.160 --> 00:59:16.680
about if everyone waits 3 days, it's not

00:59:16.680 --> 00:59:19.440
going to make detection slower. And then

00:59:19.440 --> 00:59:20.880
somebody said gave me an example, "What

00:59:20.880 --> 00:59:23.720
about exit utils?" I think exit util is

00:59:23.720 --> 00:59:25.600
is like a completely different scenario,

00:59:25.600 --> 00:59:29.000
right? What was the example?

00:59:29.000 --> 00:59:32.280
X XZ utils, which was a

00:59:32.280 --> 00:59:35.000
Yeah, yeah, yeah. I mean that that got

00:59:35.000 --> 00:59:37.240
that took months before it was detected

00:59:37.240 --> 00:59:38.600
and only because some Microsoft

00:59:38.600 --> 00:59:40.840
researchers saw his CPU spikes

00:59:40.840 --> 00:59:42.320
>> maybe it's worth

00:59:42.320 --> 00:59:43.960
maybe it's worth plotting this in a in

00:59:43.960 --> 00:59:45.960
Excalidraw. Let me just

00:59:45.960 --> 00:59:47.600
public searchable cuz I do you think

00:59:47.600 --> 00:59:49.760
it's worth capturing? Which one? Like Z

00:59:49.760 --> 00:59:52.480
utils or what we just discussed? What we

00:59:52.480 --> 00:59:55.480
just discussed like um I'm just sharing

00:59:55.480 --> 00:59:57.240
my screen. And now we I just want to

00:59:57.240 --> 00:59:58.320
capture it.

00:59:58.320 --> 01:00:00.080
I just wanted to capture some

01:00:00.080 --> 01:00:01.760
roughness. So I thought it was really

01:00:01.760 --> 01:00:04.120
cool, interesting. Aside from obviously

01:00:04.120 --> 01:00:07.200
the Claude Code source code map leak,

01:00:07.200 --> 01:00:10.360
which has been talked about for death.

01:00:10.360 --> 01:00:12.360
Um

01:00:12.360 --> 01:00:14.120
I think the the scenario the whole

01:00:14.120 --> 01:00:16.560
situation around Trevi being a complete

01:00:16.560 --> 01:00:19.080
Is it it's Aqua Security that's that was

01:00:19.080 --> 01:00:20.960
completely compromised, no? Okay, so

01:00:20.960 --> 01:00:23.440
doing forgot how to use this thing.

01:00:23.440 --> 01:00:25.040
So the

01:00:25.040 --> 01:00:26.960
Uh we started by saying that like

01:00:26.960 --> 01:00:32.320
developers with uh you know, a password

01:00:32.320 --> 01:00:35.760
can can get compromised. Since they

01:00:35.760 --> 01:00:39.200
aren't using OIDC. So that that could

01:00:39.200 --> 01:00:41.720
happen. So that's that's one angle of

01:00:41.720 --> 01:00:43.320
of attack. I'm just I'm just wanted

01:00:43.320 --> 01:00:45.160
since they aren't using short-lived

01:00:45.160 --> 01:00:47.000
credentials. Yeah, yeah. It's not

01:00:47.000 --> 01:00:50.160
necessarily just OIDC, right? Oh my god.

01:00:50.160 --> 01:00:52.200
Aqua Security posted an update yesterday

01:00:52.200 --> 01:00:54.960
as well on on Wednesday, 2 days ago. The

01:00:54.960 --> 01:00:57.880
other angle is that people are not using

01:00:57.880 --> 01:00:59.760
uh security scanners. Yeah, but if you

01:00:59.760 --> 01:01:01.960
use Aqua Security scanner

01:01:01.960 --> 01:01:03.960
you got compromised.

01:01:03.960 --> 01:01:05.440
If you were using Trevi, you got

01:01:05.440 --> 01:01:07.080
compromised.

01:01:07.080 --> 01:01:09.400
Trevi is the exception.

01:01:09.400 --> 01:01:11.280
I I just wanted to plot like the the

01:01:11.280 --> 01:01:14.040
typical ways that supply chain I mean,

01:01:14.040 --> 01:01:15.360
there's probably a better list somewhere

01:01:15.360 --> 01:01:16.640
already.

01:01:16.640 --> 01:01:18.160
Um at least the ones that we talked

01:01:18.160 --> 01:01:20.720
about. Um and then there was a third

01:01:20.720 --> 01:01:21.840
one. What

01:01:21.840 --> 01:01:24.400
Sorry, I I lost I was too busy typing.

01:01:24.400 --> 01:01:26.840
What's What is the third one? Well,

01:01:26.840 --> 01:01:28.360
well, I guess what what you just said

01:01:28.360 --> 01:01:31.320
like the NPM will will will take that

01:01:31.320 --> 01:01:34.040
will will will catch it.

01:01:34.040 --> 01:01:35.280
Uh

01:01:35.280 --> 01:01:40.120
given you know, X Y X days of cool down

01:01:40.120 --> 01:01:42.080
or something like that. You know, like

01:01:42.080 --> 01:01:44.000
like the like the XC or something.

01:01:44.000 --> 01:01:47.040
>> It's something called um

01:01:47.040 --> 01:01:49.480
age. The package The age of the package

01:01:49.480 --> 01:01:51.760
has to be above. Let me find I did a

01:01:51.760 --> 01:01:54.960
pull request on the CDK terrain repo um

01:01:54.960 --> 01:01:57.200
provider project's old one. All these

01:01:57.200 --> 01:02:00.120
security companies have Has Has anyone

01:02:00.120 --> 01:02:02.160
Have these security companies actually

01:02:02.160 --> 01:02:04.600
said that we have stopped attacks on

01:02:04.600 --> 01:02:07.240
NPM? I want I I mean, does NPM use a

01:02:07.240 --> 01:02:08.800
security scanner? Yeah, I don't think

01:02:08.800 --> 01:02:10.440
they do because I think

01:02:10.440 --> 01:02:12.760
when when Microsoft owns NPM

01:02:12.760 --> 01:02:15.960
Microsoft owns NPM. Oh, really? Yeah.

01:02:15.960 --> 01:02:16.520
Let me check.

01:02:16.520 --> 01:02:18.640
>> even know that. NPMGS I mean, like the

01:02:18.640 --> 01:02:21.000
official registry owned by GitHub. So

01:02:21.000 --> 01:02:22.640
yes, Microsoft.

01:02:22.640 --> 01:02:23.760
You understand what I'm saying? Like

01:02:23.760 --> 01:02:24.880
when I

01:02:24.880 --> 01:02:26.960
maybe I have a bad memory, but when I

01:02:26.960 --> 01:02:28.960
published NPM, it's like pretty much

01:02:28.960 --> 01:02:31.120
instant. So that means there's no

01:02:31.120 --> 01:02:33.320
there's no gate, right? It It They They

01:02:33.320 --> 01:02:35.160
just publish it immediately. I think so,

01:02:35.160 --> 01:02:37.920
yeah. So it makes makes me think is this

01:02:37.920 --> 01:02:39.880
step two people are not using security

01:02:39.880 --> 01:02:41.080
scanners?

01:02:41.080 --> 01:02:42.440
Include does

01:02:42.440 --> 01:02:44.640
does NPM do this? Something to look

01:02:44.640 --> 01:02:46.280
into.

01:02:46.280 --> 01:02:47.640
Anyway, it's a it is an interesting

01:02:47.640 --> 01:02:50.800
space. It's a space So if you use PNPM,

01:02:50.800 --> 01:02:54.400
the flag is config.minimum release age

01:02:54.400 --> 01:02:56.320
and you have to provide the number in

01:02:56.320 --> 01:02:59.000
minutes. So I've just set it up as 4

01:02:59.000 --> 01:03:01.120
days.

01:03:01.120 --> 01:03:03.400
Which is 5,760 minutes.

01:03:03.400 --> 01:03:06.120
>> parameter called? Minimum Minimum like

01:03:06.120 --> 01:03:10.240
config yeah, minimum-release-age.

01:03:10.240 --> 01:03:12.160
Oh, cool. Not underscore, but dash, but

01:03:12.160 --> 01:03:14.040
okay, don't matter. And then the second

01:03:14.040 --> 01:03:16.480
thing is you can run NPM audit with

01:03:16.480 --> 01:03:19.240
audit levels. Yeah, NPM audit is really

01:03:19.240 --> 01:03:22.280
good. NPM audit. And the guy like who

01:03:22.280 --> 01:03:24.640
maintains Projen told me he finds it

01:03:24.640 --> 01:03:26.640
good enough. And And honestly, he's also

01:03:26.640 --> 01:03:28.920
the guy I was talking to about doing

01:03:28.920 --> 01:03:31.000
some something else and he says that he

01:03:31.000 --> 01:03:33.320
hasn't been able to work on it because

01:03:33.320 --> 01:03:35.000
and he's on AWS. He says because of

01:03:35.000 --> 01:03:36.320
these supply chain attacks have been

01:03:36.320 --> 01:03:38.240
taking all of his time. Projen is

01:03:38.240 --> 01:03:40.080
heavily used within AWS as far as I

01:03:40.080 --> 01:03:41.880
understand. The way that they maintain

01:03:41.880 --> 01:03:44.120
all of their repository boilerplate is

01:03:44.120 --> 01:03:46.359
with Projen. So all of these features

01:03:46.359 --> 01:03:48.960
allow AWS to very quickly roll out like

01:03:48.960 --> 01:03:50.600
these settings across all of their

01:03:50.600 --> 01:03:52.400
repositories, right? You just update the

01:03:52.400 --> 01:03:54.840
core project and then every repo adopts

01:03:54.840 --> 01:03:56.680
it and and and reconfigures itself

01:03:56.680 --> 01:03:58.640
accordingly. Which is a dream, by the

01:03:58.640 --> 01:04:02.200
way. Like um even here with this CDK

01:04:02.200 --> 01:04:04.040
terrain provider like with the CDK

01:04:04.040 --> 01:04:05.920
terrain setup that I inherited from

01:04:05.920 --> 01:04:09.320
HashiCorp, it's all managed by Projen

01:04:09.320 --> 01:04:11.760
and Terraform for GitHub. So

01:04:11.760 --> 01:04:12.680
>> The tooling that

01:04:12.680 --> 01:04:14.240
The tooling is really good when I think

01:04:14.240 --> 01:04:14.680
about it.

01:04:14.680 --> 01:04:17.000
>> It is really nice. Because I tried to

01:04:17.000 --> 01:04:18.680
roll out Projen within my organization

01:04:18.680 --> 01:04:21.320
as well, but again it's niche, it's

01:04:21.320 --> 01:04:22.359
complicated.

01:04:22.359 --> 01:04:27.920
>> is a is is CDK ecosystem very very very

01:04:27.920 --> 01:04:29.400
I mean, but Have we ever talked about

01:04:29.400 --> 01:04:32.240
Projen? Projen's TypeScript or It's

01:04:32.240 --> 01:04:33.600
written in TypeScript. Oh, it's

01:04:33.600 --> 01:04:35.359
available for Go? What?

01:04:35.359 --> 01:04:38.840
>> Yeah, because it uses JSII, right? Oh.

01:04:38.840 --> 01:04:40.400
How is it You wouldn't You wouldn't want

01:04:40.400 --> 01:04:43.240
to put it in a Go language project. How

01:04:43.240 --> 01:04:44.960
can you say if you never tried? Yeah, I

01:04:44.960 --> 01:04:45.760
I

01:04:45.760 --> 01:04:47.120
I guess I should

01:04:47.120 --> 01:04:49.080
>> It supports Python and they also support

01:04:49.080 --> 01:04:51.640
you UV in it now. Like the good thing

01:04:51.640 --> 01:04:53.960
about Projen is that it's that actually

01:04:53.960 --> 01:04:56.000
quite a lot of people use it even though

01:04:56.000 --> 01:04:57.800
not a lot of people know about it and

01:04:57.800 --> 01:05:00.040
that means that it's been kept up to

01:05:00.040 --> 01:05:02.480
date with like quite recent frameworks.

01:05:02.480 --> 01:05:04.480
We haven't updated the CDK TF supporting

01:05:04.480 --> 01:05:05.920
there yet. We have to switch Projen all

01:05:05.920 --> 01:05:09.560
the way over to CDK terrain. Um but

01:05:09.560 --> 01:05:11.800
it it really gives you like really good

01:05:11.800 --> 01:05:13.200
um

01:05:13.200 --> 01:05:15.920
Okay, so you know how GitHub has this

01:05:15.920 --> 01:05:17.800
amazing feature of creating a template

01:05:17.800 --> 01:05:19.200
repository, right?

01:05:19.200 --> 01:05:21.760
>> Yeah. Absolutely hate it. What happens

01:05:21.760 --> 01:05:22.640
after you create it?

01:05:22.640 --> 01:05:25.880
>> say it's sarcastic uh [snorts]

01:05:25.880 --> 01:05:27.680
amazing. Okay, carry on.

01:05:27.680 --> 01:05:31.359
>> So So And And not just updated. Yeah,

01:05:31.359 --> 01:05:32.880
and then and then people have to come up

01:05:32.880 --> 01:05:35.040
with all kinds of workarounds to

01:05:35.040 --> 01:05:37.840
>> Like at at my last workplace, there was

01:05:37.840 --> 01:05:39.359
uh actually the the workaround that we

01:05:39.359 --> 01:05:41.800
have was pretty good. It was like a pack

01:05:41.800 --> 01:05:44.400
pack new inspired. I don't know if you

01:05:44.400 --> 01:05:46.080
know Arch Linux.

01:05:46.080 --> 01:05:48.000
But basically, we wrote files into

01:05:48.000 --> 01:05:49.920
people's repositories with the suffix

01:05:49.920 --> 01:05:53.760
dot new. Okay. So So but still, you need

01:05:53.760 --> 01:05:55.040
a whole bunch of additional stuff,

01:05:55.040 --> 01:05:57.480
right? Um because the biggest problem of

01:05:57.480 --> 01:06:00.000
these scaffolding framework um libraries

01:06:00.000 --> 01:06:01.960
and cookie cutter and all this is that

01:06:01.960 --> 01:06:04.000
they will bootstrap a file system for

01:06:04.000 --> 01:06:06.320
you once and then that's it, right? How

01:06:06.320 --> 01:06:08.040
do you keep that updated? Most of them

01:06:08.040 --> 01:06:10.400
don't take that into account. So Projen

01:06:10.400 --> 01:06:12.680
is different in a way that it approaches

01:06:12.680 --> 01:06:15.359
the file system exactly the same way as

01:06:15.359 --> 01:06:18.480
it deals with like a construct tree in

01:06:18.480 --> 01:06:21.960
uh AWS CDK. So every file is basically

01:06:21.960 --> 01:06:24.760
an object in memory that you construct

01:06:24.760 --> 01:06:27.120
up into a tree. And then finally, when

01:06:27.120 --> 01:06:28.760
you have defined what the contents of

01:06:28.760 --> 01:06:30.359
the file is to look you know, on that

01:06:30.359 --> 01:06:32.120
particular path, when you when you

01:06:32.120 --> 01:06:34.280
synthesize, it writes everything down to

01:06:34.280 --> 01:06:37.280
disk, okay? So then it creates the whole

01:06:37.280 --> 01:06:39.440
I'm I'm understanding like it keeps your

01:06:39.440 --> 01:06:41.520
answers. It keeps your values. Yeah, and

01:06:41.520 --> 01:06:43.720
it it you can even have a function to

01:06:43.720 --> 01:06:46.600
lazily determine the contents of a file

01:06:46.600 --> 01:06:48.640
at the very end during synthesis, right?

01:06:48.640 --> 01:06:50.400
By you know, collecting all of the other

01:06:50.400 --> 01:06:52.000
files that have been generated and then

01:06:52.000 --> 01:06:54.080
generating the contents of that file. So

01:06:54.080 --> 01:06:55.720
you're able to completely functionally

01:06:55.720 --> 01:06:58.120
define the contents of your of your file

01:06:58.120 --> 01:07:00.440
system. And and and what's good about it

01:07:00.440 --> 01:07:02.040
is that you can also hook into the

01:07:02.040 --> 01:07:05.560
process. Like with with CDK and AWS CDK

01:07:05.560 --> 01:07:08.040
and CDK Terraform, you can before

01:07:08.040 --> 01:07:10.840
synthesis do some activities go on fun

01:07:10.840 --> 01:07:12.760
activities. No, but like what you can do

01:07:12.760 --> 01:07:14.920
is you can programmatically control the

01:07:14.920 --> 01:07:16.800
contents of the file, which also means

01:07:16.800 --> 01:07:19.600
that the consumer can define patches on

01:07:19.600 --> 01:07:21.640
top of the files. So he can go find a

01:07:21.640 --> 01:07:23.800
certain file within the tree and then

01:07:23.800 --> 01:07:25.720
add additional lines or do additional

01:07:25.720 --> 01:07:28.480
things because he in his case

01:07:28.480 --> 01:07:30.840
So that means that if the if the root

01:07:30.840 --> 01:07:33.800
project or the root object that built

01:07:33.800 --> 01:07:36.480
your construct your tree of objects has

01:07:36.480 --> 01:07:39.240
changes they will all just be adopted

01:07:39.240 --> 01:07:41.720
and regenerated your file system on disk

01:07:41.720 --> 01:07:43.680
and then your patches get rerun on top

01:07:43.680 --> 01:07:46.160
of it and get reapplied as well. As long

01:07:46.160 --> 01:07:48.040
as those two don't conflict, you know,

01:07:48.040 --> 01:07:50.120
it will just re-render the file system

01:07:50.120 --> 01:07:51.880
with your patches applied. So you can

01:07:51.880 --> 01:07:54.400
constantly keep updating the the source

01:07:54.400 --> 01:07:55.800
and then roll it out and it's all

01:07:55.800 --> 01:07:57.200
programmatic. So

01:07:57.200 --> 01:07:59.200
>> Yeah. So that's I mean, this is this is

01:07:59.200 --> 01:08:01.480
applicable for for a platform where

01:08:01.480 --> 01:08:03.480
you're supporting lots of products, I

01:08:03.480 --> 01:08:05.120
guess, right? I mean, that's the big

01:08:05.120 --> 01:08:07.480
reason. Yeah, so so where Projen really

01:08:07.480 --> 01:08:09.880
shines is in like a poly repo setup

01:08:09.880 --> 01:08:11.960
because it has really good support for

01:08:11.960 --> 01:08:14.320
like a single repo with just one package

01:08:14.320 --> 01:08:16.040
in it that needs to be published. If you

01:08:16.040 --> 01:08:17.759
start to deal with one repository and

01:08:17.759 --> 01:08:19.600
you need to support publishing multiple

01:08:19.600 --> 01:08:21.719
packages then it becomes a little bit

01:08:21.719 --> 01:08:23.600
more complicated. They have built

01:08:23.600 --> 01:08:25.880
support for that on top of Projen. But

01:08:25.880 --> 01:08:27.600
to be honest, I I never managed to make

01:08:27.600 --> 01:08:29.160
it work because this is the main reason

01:08:29.160 --> 01:08:30.880
I was not able to roll it out at work is

01:08:30.880 --> 01:08:32.799
because we we have a lot of repositories

01:08:32.799 --> 01:08:34.600
that are publishing multiple packages

01:08:34.600 --> 01:08:37.560
and so I had adopted Turbo repo. And you

01:08:37.560 --> 01:08:39.160
know, it's easy to build your own

01:08:39.160 --> 01:08:41.440
project type in Projen because you can

01:08:41.440 --> 01:08:43.759
define your object, which is a project

01:08:43.759 --> 01:08:45.520
type, and you can determine what are

01:08:45.520 --> 01:08:47.359
what is it like and and it works with

01:08:47.359 --> 01:08:49.240
add-ons. So yeah, you can create one

01:08:49.240 --> 01:08:50.600
object, which is like I'm going to take

01:08:50.600 --> 01:08:52.600
care of the GitHub workflows and I'm I'm

01:08:52.600 --> 01:08:54.160
an object that's going to take care of

01:08:54.160 --> 01:08:56.000
the I don't know, your your

01:08:56.000 --> 01:08:58.359
package.json, your your YAML, like your

01:08:58.359 --> 01:09:00.920
your your manifest for your package. So

01:09:00.920 --> 01:09:02.680
So it support all of these in like

01:09:02.680 --> 01:09:04.319
add-ons. So you can say now I have a

01:09:04.319 --> 01:09:06.400
repo, a base project, and I'm going to

01:09:06.400 --> 01:09:08.680
add on the GitHub actions object and

01:09:08.680 --> 01:09:11.160
it's going to generate my actions.

01:09:11.160 --> 01:09:12.839
That sounds really powerful. I mean, it

01:09:12.839 --> 01:09:15.200
sounds really handy for publishers like

01:09:15.200 --> 01:09:17.480
yourself who you know, CDK terrain. I

01:09:17.480 --> 01:09:19.839
mean, you you must be maintaining quite

01:09:19.839 --> 01:09:21.759
a few projects, right?

01:09:21.759 --> 01:09:23.400
>> just one. Yeah, yeah, we are maintaining

01:09:23.400 --> 01:09:25.080
a lot of projects, but we just have one

01:09:25.080 --> 01:09:27.720
project type in Projen that manages all

01:09:27.720 --> 01:09:29.319
of our providers. Yeah, yeah, yeah,

01:09:29.319 --> 01:09:31.319
yeah, yeah. So, so we just define a

01:09:31.319 --> 01:09:34.000
provider within the project is Yeah.

01:09:34.000 --> 01:09:36.359
this. And it comes with like it go it

01:09:36.359 --> 01:09:37.880
comes with jobs to go and check the

01:09:37.880 --> 01:09:39.240
Terraform registry if there's a new

01:09:39.240 --> 01:09:41.040
version of the provider, and then it

01:09:41.040 --> 01:09:43.400
will if there's a a new version, it will

01:09:43.400 --> 01:09:45.240
regenerate the bindings. I know my

01:09:45.240 --> 01:09:47.480
internet just hiccuped, right? But um

01:09:47.480 --> 01:09:49.319
>> It didn't my word. Yeah, but it will be

01:09:49.319 --> 01:09:51.240
fine on the recording. Yeah, yeah, I got

01:09:51.240 --> 01:09:53.920
to edit this. Cool, man. I I I got to

01:09:53.920 --> 01:09:56.000
rush out and buy some bread before this

01:09:56.000 --> 01:09:58.960
the baker sells out and enjoy a few days

01:09:58.960 --> 01:10:01.400
offline, I think. That's my mission. Not

01:10:01.400 --> 01:10:04.520
spend spend some more time with my kids

01:10:04.520 --> 01:10:05.800
since they're not they're on holiday

01:10:05.800 --> 01:10:07.760
now. Yeah. So, I'll I get back to you

01:10:07.760 --> 01:10:09.360
next week. And next week did I I I

01:10:09.360 --> 01:10:10.920
mentioned to you I'm going to this AI

01:10:10.920 --> 01:10:12.840
engineer conference today? Yes, you

01:10:12.840 --> 01:10:13.440
mentioned it.

01:10:13.440 --> 01:10:15.400
>> I'll I'll have some I'll have a lot more

01:10:15.400 --> 01:10:17.920
to say next week. Or the week after.

01:10:17.920 --> 01:10:20.080
I'll send you a link.

01:10:20.080 --> 01:10:21.880
We'll catch up then.

01:10:21.880 --> 01:10:24.000
Okay, see you. All the best. Have a good

01:10:24.000 --> 01:10:26.800
Easter. Yeps. Uh yeah, happy Easter.

01:10:26.800 --> 01:10:28.600
Yeah, are you going to do anything? No?

01:10:28.600 --> 01:10:30.880
No. You don't have Easter in Vietnam.

01:10:30.880 --> 01:10:33.000
You have other things. Yeah, but I'm on

01:10:33.000 --> 01:10:34.640
this very annoying contract where I

01:10:34.640 --> 01:10:37.640
don't have any ability to take leave, so

01:10:37.640 --> 01:10:38.400
I'm not taking any.

01:10:38.400 --> 01:10:41.800
>> of all time. Exactly. See you, man. Bye.

01:10:41.800 --> 01:10:44.480
Okay, bye.