WEBVTT

00:00:00.080 --> 00:00:02.399
Hey, good morning Vincent. So you wanted

00:00:02.399 --> 00:00:06.080
to share a CDK terrain release

00:00:06.080 --> 00:00:09.679
>> and earlier you did like real of a raft

00:00:09.679 --> 00:00:11.840
of features

00:00:11.840 --> 00:00:14.320
>> and I still didn't quite understand what

00:00:14.320 --> 00:00:16.000
it meant. So what does it what does it

00:00:16.000 --> 00:00:19.279
mean to a layman person in the field of

00:00:19.279 --> 00:00:24.560
infrastructure like like our audience?

00:00:24.560 --> 00:00:27.279
Okay, so I don't know where to start now

00:00:27.279 --> 00:00:29.840
and without being like repetitive, but

00:00:29.840 --> 00:00:33.680
basically Terraform allows you to create

00:00:33.680 --> 00:00:36.960
resources against APIs. Um, and those

00:00:36.960 --> 00:00:39.120
can be cloud providers such as AWS or

00:00:39.120 --> 00:00:42.239
Google Cloud or Azure. uh or it could be

00:00:42.239 --> 00:00:45.040
your GitHub API or it could be you know

00:00:45.040 --> 00:00:47.120
to create a repository or it could be

00:00:47.120 --> 00:00:50.320
your data dog to create a a monitor or

00:00:50.320 --> 00:00:52.800
it could be your snowflake

00:00:52.800 --> 00:00:55.675
>> cloud AI management instance in my case

00:00:55.675 --> 00:00:55.680
[clears throat]

00:00:55.680 --> 00:00:56.879
>> no not that

00:00:56.879 --> 00:00:58.000
>> oh [laughter]

00:00:58.000 --> 00:01:00.800
there's no provider for the claw AI okay

00:01:00.800 --> 00:01:01.760
carry on

00:01:01.760 --> 00:01:04.000
>> yeah so but there is one right I I I

00:01:04.000 --> 00:01:05.519
shared there some people made an

00:01:05.519 --> 00:01:08.080
announcement about official not official

00:01:08.080 --> 00:01:10.799
uh like their you did share Right.

00:01:10.799 --> 00:01:12.240
>> So, so that's the great thing about

00:01:12.240 --> 00:01:14.960
about Terraform is that even if there is

00:01:14.960 --> 00:01:18.159
no pro official provider um anyone can

00:01:18.159 --> 00:01:21.520
build one because the the execution

00:01:21.520 --> 00:01:24.640
engine and the providers are

00:01:24.640 --> 00:01:26.159
communicating with each other through a

00:01:26.159 --> 00:01:28.320
protocol. It's called a plug-in system.

00:01:28.320 --> 00:01:30.000
Is is that the best way to go forward?

00:01:30.000 --> 00:01:32.560
Because I we did talk about this before.

00:01:32.560 --> 00:01:35.360
I I created my own provider for

00:01:35.360 --> 00:01:38.000
Snowflake and I found it really hard.

00:01:38.000 --> 00:01:43.360
the snowflake uh missed some features.

00:01:43.360 --> 00:01:47.680
>> I found it really really hard. Uh

00:01:47.680 --> 00:01:48.159
so

00:01:48.159 --> 00:01:51.520
>> there's several versions actually.

00:01:51.520 --> 00:01:54.000
>> Would you suggest like if you were to

00:01:54.000 --> 00:01:56.240
infrastructure as code something is

00:01:56.240 --> 00:01:58.320
terraform provider still the your best

00:01:58.320 --> 00:02:01.520
gateway drug into into nailing down your

00:02:01.520 --> 00:02:03.119
infrastructure?

00:02:03.119 --> 00:02:04.960
I think it comes down to what is the

00:02:04.960 --> 00:02:07.280
most like [gasps]

00:02:07.280 --> 00:02:09.280
accepted way or like what's the most

00:02:09.280 --> 00:02:11.360
integrated with with your existing, you

00:02:11.360 --> 00:02:13.360
know, if you if you're if you're the

00:02:13.360 --> 00:02:15.840
platform team or if you're you're a a

00:02:15.840 --> 00:02:17.920
product team person and you need to get

00:02:17.920 --> 00:02:20.319
buy in from your platform team, it will

00:02:20.319 --> 00:02:21.920
really depend on what they're using,

00:02:21.920 --> 00:02:24.560
right? Because if they are full on in

00:02:24.560 --> 00:02:26.400
Terraform and all of their pipeline

00:02:26.400 --> 00:02:28.319
systems and everything is is is are

00:02:28.319 --> 00:02:30.959
tweaked towards that then you know

00:02:30.959 --> 00:02:33.200
building a uh a little provider for your

00:02:33.200 --> 00:02:36.080
use case will probably be better. If

00:02:36.080 --> 00:02:37.760
they're fully on cloud formation then

00:02:37.760 --> 00:02:40.239
you can use uh their version of custom

00:02:40.239 --> 00:02:43.840
resources. So the ability to define um

00:02:43.840 --> 00:02:45.760
your own

00:02:45.760 --> 00:02:48.319
basically what these things do right

00:02:48.319 --> 00:02:50.080
they create resources and then they

00:02:50.080 --> 00:02:51.920
manage them. So they need create, read,

00:02:51.920 --> 00:02:54.000
update, and delete capabilities. So when

00:02:54.000 --> 00:02:56.959
you create a custom resour outside AWS,

00:02:56.959 --> 00:02:58.160
is Terraform [clears throat] provider

00:02:58.160 --> 00:03:00.319
your best bet then cuz it's got the

00:03:00.319 --> 00:03:01.280
most.

00:03:01.280 --> 00:03:02.879
>> So what are the other options? Right.

00:03:02.879 --> 00:03:06.959
There's a couple of new players like um

00:03:06.959 --> 00:03:09.040
for May. I don't even know. I don't

00:03:09.040 --> 00:03:10.720
think it's it's cheese.

00:03:10.720 --> 00:03:14.080
>> Are they still around? I thought they

00:03:14.080 --> 00:03:16.480
released and then disappeared. Or may

00:03:16.480 --> 00:03:19.360
Yeah. No, I think there's still they had

00:03:19.360 --> 00:03:23.840
initially I'm very skeptic um of any

00:03:23.840 --> 00:03:25.599
new framework being launched because

00:03:25.599 --> 00:03:27.680
you're you're in coming to an ecosystem

00:03:27.680 --> 00:03:29.680
that has been around for almost 10 years

00:03:29.680 --> 00:03:31.920
now and there's a lot of tooling around

00:03:31.920 --> 00:03:34.239
the existing frameworks. So there's a

00:03:34.239 --> 00:03:36.480
lot of tooling around terapform. Um

00:03:36.480 --> 00:03:37.920
there's a lot of tooling around cloud

00:03:37.920 --> 00:03:39.519
formation.

00:03:39.519 --> 00:03:44.159
And um being a new kid on the block, you

00:03:44.159 --> 00:03:47.120
um I I'm I'm 100% like in general all of

00:03:47.120 --> 00:03:48.720
these new frameworks, they believe they

00:03:48.720 --> 00:03:50.159
can do it better, right? They see the

00:03:50.159 --> 00:03:52.560
problems of these existing frameworks.

00:03:52.560 --> 00:03:55.840
Um for example, there's also Chant. C H

00:03:55.840 --> 00:03:57.120
A N T.

00:03:57.120 --> 00:03:59.439
>> I really like that they use Pickle.

00:03:59.439 --> 00:04:02.480
Yeah. Uh Shant.

00:04:02.480 --> 00:04:03.760
>> Yeah.

00:04:03.760 --> 00:04:06.720
But that one is going a little bit. No.

00:04:06.720 --> 00:04:12.799
No. C H A N T like chanting.

00:04:12.799 --> 00:04:14.879
Chant.

00:04:14.879 --> 00:04:18.639
Yeah. Intuent. Intentious. Intensious.

00:04:18.639 --> 00:04:21.040
Alex Artigz is quite active on uh on

00:04:21.040 --> 00:04:25.040
LinkedIn. Um usually responding to

00:04:25.040 --> 00:04:28.880
general threads. There's also

00:04:28.880 --> 00:04:31.600
Brian Grant and Alexis Richardson. they

00:04:31.600 --> 00:04:34.160
created a startup called Config Hub uh

00:04:34.160 --> 00:04:36.080
and and crossplane they're focused on

00:04:36.080 --> 00:04:38.080
like using controllers right constantly

00:04:38.080 --> 00:04:41.360
reconciling um your a your uh resources

00:04:41.360 --> 00:04:44.080
against um a reconciliation loop so

00:04:44.080 --> 00:04:46.800
basically the Kubernetes way so chant

00:04:46.800 --> 00:04:49.280
honestly it's a bit over my head a lot

00:04:49.280 --> 00:04:51.759
of people are a little bit confused

00:04:51.759 --> 00:04:54.880
um because it's quite

00:04:54.880 --> 00:04:58.320
I don't know it's hard to grasp and

00:04:58.320 --> 00:05:00.639
his original comparison to existing

00:05:00.639 --> 00:05:03.759
frameworks was inaccurate. Um, so I told

00:05:03.759 --> 00:05:05.199
him that and then he actually went and

00:05:05.199 --> 00:05:06.800
updated it. So now it's a little bit

00:05:06.800 --> 00:05:08.320
more accurate, but I still don't get why

00:05:08.320 --> 00:05:11.039
what he's building. They say it's it's

00:05:11.039 --> 00:05:13.360
more of like configuration as data. It's

00:05:13.360 --> 00:05:15.520
kind of the same thing as what uh Config

00:05:15.520 --> 00:05:18.800
Hub does. Uh anyway, you ask me like is

00:05:18.800 --> 00:05:20.400
that still the best way to do your

00:05:20.400 --> 00:05:22.880
infrastructure? To be honest, most of

00:05:22.880 --> 00:05:25.039
the organizations I talk to and work

00:05:25.039 --> 00:05:27.360
with, they're either on cloud form or or

00:05:27.360 --> 00:05:30.400
uh or Terraform. Maybe I know very few

00:05:30.400 --> 00:05:32.880
companies on Pulumi and then there's

00:05:32.880 --> 00:05:35.520
this all new kits on the block like uh

00:05:35.520 --> 00:05:37.360
you know crossplane you know based

00:05:37.360 --> 00:05:40.720
Kubernetes controllers config hub is a

00:05:40.720 --> 00:05:43.039
little bit similar but they they take

00:05:43.039 --> 00:05:44.880
some of the ideas of what they've

00:05:44.880 --> 00:05:47.120
learned managing and defining GitHubs

00:05:47.120 --> 00:05:49.600
and and customize because Brian Grant he

00:05:49.600 --> 00:05:51.520
used to be at the Kubernetes uh team at

00:05:51.520 --> 00:05:54.560
Google and he he's been trying to solve

00:05:54.560 --> 00:05:57.199
this problem for like over 20 years like

00:05:57.199 --> 00:05:59.919
even within Google doing it for Borg.

00:05:59.919 --> 00:06:01.919
>> He he saw some patterns and he sees some

00:06:01.919 --> 00:06:04.720
problems. And so what they really do

00:06:04.720 --> 00:06:07.039
what they what they think Conf is

00:06:07.039 --> 00:06:09.840
interesting because they really see like

00:06:09.840 --> 00:06:11.919
we end up building these complicated

00:06:11.919 --> 00:06:14.639
systems of just generating configuration

00:06:14.639 --> 00:06:16.240
and then if you have to make one field

00:06:16.240 --> 00:06:17.600
change you have to push it through this

00:06:17.600 --> 00:06:19.680
whole pipeline of changes that need to

00:06:19.680 --> 00:06:21.600
then ultimately end up in like one or

00:06:21.600 --> 00:06:24.080
two fields to to hit production. So they

00:06:24.080 --> 00:06:26.880
just shortcut that whole loop by just

00:06:26.880 --> 00:06:28.240
you know allowing you to define a

00:06:28.240 --> 00:06:30.800
function that you run against your data

00:06:30.800 --> 00:06:32.960
against your configuration and then you

00:06:32.960 --> 00:06:34.720
can just trigger that function on your

00:06:34.720 --> 00:06:37.120
staging environment and if it's all good

00:06:37.120 --> 00:06:39.199
propagate it and change it onto

00:06:39.199 --> 00:06:42.080
production. This reminds me of Q CLI

00:06:42.080 --> 00:06:47.680
because is it QCLI? I think it's also um

00:06:47.680 --> 00:06:51.759
X it comes out of the Kubernetes.

00:06:51.759 --> 00:06:54.720
Oh no, Q lang you. I think Q lang the

00:06:54.720 --> 00:06:57.680
whole principle. Okay. A

00:06:57.680 --> 00:07:00.720
one of the main guys is ex Kubernetes

00:07:00.720 --> 00:07:03.520
Google team and B the idea is that

00:07:03.520 --> 00:07:05.759
there's a language and would like

00:07:05.759 --> 00:07:08.240
functions to make changes in your

00:07:08.240 --> 00:07:10.160
infrastructure safer and more

00:07:10.160 --> 00:07:11.919
controlled. I mean that's the way I

00:07:11.919 --> 00:07:13.919
understood it probably.

00:07:13.919 --> 00:07:16.639
>> Yeah. I mean there's a lot of like

00:07:16.639 --> 00:07:20.400
languages to manage configuration and

00:07:20.400 --> 00:07:24.800
basically QANG is focused on how do you

00:07:24.800 --> 00:07:26.960
manage and manipulate and mutate your

00:07:26.960 --> 00:07:28.880
configuration across the environments.

00:07:28.880 --> 00:07:31.840
It's not the same as um as your um you

00:07:31.840 --> 00:07:34.560
know software code and I think they're

00:07:34.560 --> 00:07:36.240
all very interesting but not a lot of

00:07:36.240 --> 00:07:40.319
them gain widespread adoption. Right. So

00:07:40.319 --> 00:07:41.199
>> true, true.

00:07:41.199 --> 00:07:43.360
>> The what and and there's so many like

00:07:43.360 --> 00:07:47.520
it's like the famous XK CD comic where

00:07:47.520 --> 00:07:49.599
it's like there's too many standards.

00:07:49.599 --> 00:07:51.840
There's like 11 standards. So I'm going

00:07:51.840 --> 00:07:53.759
to make another I made a new one. Now we

00:07:53.759 --> 00:07:56.400
have 12 standards to try and get rid of

00:07:56.400 --> 00:07:58.800
those 11 standards.

00:07:58.800 --> 00:08:00.800
>> Yeah. Yeah.

00:08:00.800 --> 00:08:04.080
>> That's so common. Actually the the other

00:08:04.080 --> 00:08:06.479
comic I feel that needs to be written is

00:08:06.479 --> 00:08:09.039
it's probably to do with migrations like

00:08:09.039 --> 00:08:11.360
>> if you work with in any big company when

00:08:11.360 --> 00:08:14.240
when they're trying to do a migration it

00:08:14.240 --> 00:08:16.720
very most of the time they never

00:08:16.720 --> 00:08:18.720
completely managed to migrate for one

00:08:18.720 --> 00:08:22.560
reason or another. So now you have uh

00:08:22.560 --> 00:08:25.120
you know X plus one systems purely

00:08:25.120 --> 00:08:26.960
because you couldn't retire the older

00:08:26.960 --> 00:08:28.479
one. Does that make sense?

00:08:28.479 --> 00:08:30.720
>> Yeah. have the same experience. Like

00:08:30.720 --> 00:08:32.159
when you join an organization that's

00:08:32.159 --> 00:08:34.000
been around for a while, you're going to

00:08:34.000 --> 00:08:36.880
see five or six generations of how they

00:08:36.880 --> 00:08:40.159
used to do things and how like the newer

00:08:40.159 --> 00:08:42.240
project has been created with a new way

00:08:42.240 --> 00:08:44.560
of doing things. The old project has

00:08:44.560 --> 00:08:46.399
been like some of the very actively

00:08:46.399 --> 00:08:48.240
maintained projects have been migrated,

00:08:48.240 --> 00:08:50.959
but those that are less maintained stay

00:08:50.959 --> 00:08:53.440
on like older versions of however was

00:08:53.440 --> 00:08:56.160
things being done and um and never get

00:08:56.160 --> 00:08:59.680
ported. I was I was

00:08:59.680 --> 00:09:02.080
I was talking because I was on a call

00:09:02.080 --> 00:09:04.080
like every week checking on them on the

00:09:04.080 --> 00:09:06.160
status and then somebody asked like when

00:09:06.160 --> 00:09:08.320
are we finally going to be done and I

00:09:08.320 --> 00:09:10.206
basically told him we're never done.

00:09:10.206 --> 00:09:12.080
[laughter] That's that's infrastructure

00:09:12.080 --> 00:09:14.560
and operations basically. You're always

00:09:14.560 --> 00:09:18.560
going to have uh something.

00:09:18.560 --> 00:09:18.959
Yeah.

00:09:18.959 --> 00:09:21.680
>> Yeah. It's kind of it's kind of sad and

00:09:21.680 --> 00:09:24.399
like and and then maybe turning back to

00:09:24.399 --> 00:09:27.040
AI like does AI solve any of these

00:09:27.040 --> 00:09:29.839
problems cuz I feel like they don't in a

00:09:29.839 --> 00:09:32.160
way because there's lots of new AI

00:09:32.160 --> 00:09:37.120
initiatives and moving the data from uh

00:09:37.120 --> 00:09:40.000
the old system to the new system doesn't

00:09:40.000 --> 00:09:42.320
seem to be any better solve with AI at

00:09:42.320 --> 00:09:44.720
least

00:09:44.720 --> 00:09:47.120
>> at least I mean this is my own anecdotal

00:09:47.120 --> 00:09:49.680
experience I I mean I I wasn't in these

00:09:49.680 --> 00:09:51.680
projects, but I just feel like okay, now

00:09:51.680 --> 00:09:55.200
we have these new AI projects and

00:09:55.200 --> 00:09:57.200
they're also just plus one plus one plus

00:09:57.200 --> 00:10:00.800
one. So that's this is where I think

00:10:00.800 --> 00:10:02.160
there's two things I want to say about

00:10:02.160 --> 00:10:06.320
that. one um about the flexibility of

00:10:06.320 --> 00:10:10.480
the of the the of Terraform and its

00:10:10.480 --> 00:10:12.800
capability of like refactoring rem like

00:10:12.800 --> 00:10:14.399
when you need to do migrations you need

00:10:14.399 --> 00:10:17.040
to redefine the ownership of certain

00:10:17.040 --> 00:10:19.839
resources. Sometimes you realize that

00:10:19.839 --> 00:10:21.519
something has been created by a team

00:10:21.519 --> 00:10:23.120
then gets reused by another team and

00:10:23.120 --> 00:10:24.560
then actually the responsibility of

00:10:24.560 --> 00:10:26.480
maintaining it needs to be shared needs

00:10:26.480 --> 00:10:28.240
to move to a separate layer. So this

00:10:28.240 --> 00:10:32.959
type of like migrations um are I feel

00:10:32.959 --> 00:10:36.320
very well supported in Terafhone. Maybe

00:10:36.320 --> 00:10:38.640
it's a little bit of uh how you say when

00:10:38.640 --> 00:10:41.680
a hostage falls in love with it's um uh

00:10:41.680 --> 00:10:42.560
>> Stockholm.

00:10:42.560 --> 00:10:44.480
>> Yeah, it's a bit of Stockholm uh that

00:10:44.480 --> 00:10:46.560
I'm like happy with what Terafhone gives

00:10:46.560 --> 00:10:48.079
me. And there's probably people that

00:10:48.079 --> 00:10:49.839
will heavily disagree like no actually

00:10:49.839 --> 00:10:51.440
there's a lot much better way to do

00:10:51.440 --> 00:10:53.839
this. But anyway, so so that's where I

00:10:53.839 --> 00:10:56.800
think with AI agents, they can also help

00:10:56.800 --> 00:10:58.399
you do that much faster. Like something

00:10:58.399 --> 00:11:01.279
that would have taken a long time, um,

00:11:01.279 --> 00:11:03.440
agents can like diligently keep

00:11:03.440 --> 00:11:05.760
hammering at it and slowly chip away and

00:11:05.760 --> 00:11:07.920
and help push through the change. It's

00:11:07.920 --> 00:11:09.360
very funny because I have couple of

00:11:09.360 --> 00:11:12.000
cloud sessions on my laptop and I tell

00:11:12.000 --> 00:11:15.279
it my you know my goals and I ask it to

00:11:15.279 --> 00:11:19.360
create tasks and to arm monitors and um

00:11:19.360 --> 00:11:22.480
and then for example today I wanted to

00:11:22.480 --> 00:11:26.160
release the the CDK terrain version 0.24

00:11:26.160 --> 00:11:28.160
in the morning last night actually but

00:11:28.160 --> 00:11:30.240
then there was a GitHub outage so that

00:11:30.240 --> 00:11:32.079
was blocked and I went to sleep.

00:11:32.079 --> 00:11:33.839
Apparently, there was a 7-hour GitHub

00:11:33.839 --> 00:11:35.839
outage. Thanks. Thank Thankfully, I went

00:11:35.839 --> 00:11:36.800
to sleep.

00:11:36.800 --> 00:11:38.240
>> And [snorts] when I woke up,

00:11:38.240 --> 00:11:40.320
>> yeah, when I woke up, uh, I said, "Okay,

00:11:40.320 --> 00:11:42.079
looks like I saw the posts about the

00:11:42.079 --> 00:11:43.839
seven 7-hour outage and I was like,

00:11:43.839 --> 00:11:45.120
well, looks like everything's healthy

00:11:45.120 --> 00:11:47.120
now." So, I clicked the the release

00:11:47.120 --> 00:11:49.680
button in in GitHub and I went for

00:11:49.680 --> 00:11:51.200
breakfast because it takes like 30

00:11:51.200 --> 00:11:53.200
minutes to for it to go through all of

00:11:53.200 --> 00:11:55.040
the unit tests and integration tests.

00:11:55.040 --> 00:11:56.720
And I totally forgot that I had a cloud

00:11:56.720 --> 00:11:58.560
session running on my laptop that has a

00:11:58.560 --> 00:12:00.640
monitor armed. And that monitor was

00:12:00.640 --> 00:12:02.880
watching when the 0.24 release was

00:12:02.880 --> 00:12:05.680
hitting npmgs and python and and all the

00:12:05.680 --> 00:12:07.680
other package registries. So while I was

00:12:07.680 --> 00:12:09.040
having breakfast, I started getting

00:12:09.040 --> 00:12:11.600
notifications about um you know

00:12:11.600 --> 00:12:13.279
providers getting bumped to the latest

00:12:13.279 --> 00:12:15.440
version. And I was like, "Wow, that's

00:12:15.440 --> 00:12:16.720
pretty cool. I didn't know that was

00:12:16.720 --> 00:12:18.880
automated." And then when I came back to

00:12:18.880 --> 00:12:20.639
my desk, I realized that the cloud

00:12:20.639 --> 00:12:22.720
session that was running had an armed

00:12:22.720 --> 00:12:24.639
monitor and just kicked back to life

00:12:24.639 --> 00:12:26.079
like, "Oh, the release is out. Let me go

00:12:26.079 --> 00:12:27.519
and update all the providers." And it

00:12:27.519 --> 00:12:28.720
started like creating pull requests

00:12:28.720 --> 00:12:30.639
across repos. And I was like, "That's

00:12:30.639 --> 00:12:32.959
crazy cuz I was just I was just having

00:12:32.959 --> 00:12:34.160
breakfast." But but that just

00:12:34.160 --> 00:12:36.240
illustrates a point of having like AI

00:12:36.240 --> 00:12:38.720
push things through, right? I mean, it's

00:12:38.720 --> 00:12:41.440
easier to to define a goal and and then

00:12:41.440 --> 00:12:43.600
let it like push things through. And it

00:12:43.600 --> 00:12:45.120
was pretty secure because every single

00:12:45.120 --> 00:12:47.200
thing gets super validated in this case.

00:12:47.200 --> 00:12:49.040
>> Your comment about own your comment

00:12:49.040 --> 00:12:51.360
about ownership is quite key. I feel

00:12:51.360 --> 00:12:55.440
because in many organizations I feel

00:12:55.440 --> 00:12:57.600
the organ the ownership is distributed.

00:12:57.600 --> 00:13:01.600
So like uh for example

00:13:01.600 --> 00:13:04.240
like maybe there's a a production

00:13:04.240 --> 00:13:06.720
release team, maybe there's like an octa

00:13:06.720 --> 00:13:11.040
or IM team and um I I mean I'm creating

00:13:11.040 --> 00:13:13.040
a hypothetical situation here, but I'm

00:13:13.040 --> 00:13:15.200
I'm sure you'll understand that like you

00:13:15.200 --> 00:13:18.000
are in in control of CDK terrain, but in

00:13:18.000 --> 00:13:23.519
a in a major, you know, big companies

00:13:23.519 --> 00:13:27.839
uh product thing, you have to coordinate

00:13:27.839 --> 00:13:30.959
with human teams and that's where things

00:13:30.959 --> 00:13:35.279
quickly become awkward and uh and and of

00:13:35.279 --> 00:13:38.000
course AI doesn't really help there

00:13:38.000 --> 00:13:38.480
right

00:13:38.480 --> 00:13:40.079
>> it does it does help

00:13:40.079 --> 00:13:43.040
>> because like because of like my

00:13:43.040 --> 00:13:44.480
experience in the last week when I have

00:13:44.480 --> 00:13:46.240
to coordinate I have a pull request that

00:13:46.240 --> 00:13:48.399
has to be reviewed for the the CDK

00:13:48.399 --> 00:13:52.720
terrain release and um literally have

00:13:52.720 --> 00:13:55.200
Claude armed to to wait until the PR is

00:13:55.200 --> 00:13:57.440
ready and then claw just wait.

00:13:57.440 --> 00:13:59.199
>> Okay. I see. I see. I see what you mean.

00:13:59.199 --> 00:14:02.320
Like in your example,

00:14:02.320 --> 00:14:05.440
>> you

00:14:05.440 --> 00:14:09.680
pro you automated and and and uh the AI

00:14:09.680 --> 00:14:12.079
is just waiting for something waiting

00:14:12.079 --> 00:14:13.600
for something to happen and then it

00:14:13.600 --> 00:14:15.360
triggers something. I suppose that's

00:14:15.360 --> 00:14:17.760
that is quite helpful. I suppose

00:14:17.760 --> 00:14:20.160
>> I I I'm at a point and I think there's a

00:14:20.160 --> 00:14:21.680
lot of people that re that have reached

00:14:21.680 --> 00:14:23.040
that point where they have so many

00:14:23.040 --> 00:14:26.720
sessions running on the machine and

00:14:26.720 --> 00:14:28.240
>> be confused to like what the hell's

00:14:28.240 --> 00:14:29.279
going on.

00:14:29.279 --> 00:14:33.839
>> I do I do I use CMX? It's a lip ghosty

00:14:33.839 --> 00:14:34.560
term.

00:14:34.560 --> 00:14:38.800
>> You heard heard is the is the Kool-Aid

00:14:38.800 --> 00:14:40.399
>> now. Now they're super logical from

00:14:40.399 --> 00:14:42.880
Hashi Hashimoto who's who's going to

00:14:42.880 --> 00:14:46.639
rebuild T-Max but like for a ghosty uh

00:14:46.639 --> 00:14:49.680
native you know protocol so much faster

00:14:49.680 --> 00:14:50.079
rendering and

00:14:50.079 --> 00:14:52.639
>> so heard is not really so much a

00:14:52.639 --> 00:14:54.880
terminal multiplexer but it helps keep

00:14:54.880 --> 00:14:57.199
your AI sessions in in one place.

00:14:57.199 --> 00:14:59.440
>> So so CMAX does the same right? I mean I

00:14:59.440 --> 00:15:00.880
found something that works. I haven't

00:15:00.880 --> 00:15:04.079
really had a um you know I just saw

00:15:04.079 --> 00:15:06.240
someone announce his own version of of

00:15:06.240 --> 00:15:08.800
it on on LinkedIn. there's hundreds of

00:15:08.800 --> 00:15:10.959
them. The same with like context

00:15:10.959 --> 00:15:13.680
engineering uh solutions uh spec driven

00:15:13.680 --> 00:15:15.360
solutions. Everyone can make their own

00:15:15.360 --> 00:15:17.760
so easily with with with uh with AI. So,

00:15:17.760 --> 00:15:19.920
so but to come back to the point of like

00:15:19.920 --> 00:15:22.720
coordination and also ownership like I

00:15:22.720 --> 00:15:24.320
think this is one of the biggest uh

00:15:24.320 --> 00:15:26.880
mistakes like I work in a in a very

00:15:26.880 --> 00:15:29.120
restrictive um you know company right

00:15:29.120 --> 00:15:31.279
now and there's like a central team that

00:15:31.279 --> 00:15:33.199
tries to own everything and it is

00:15:33.199 --> 00:15:36.320
creating so many hurdles in in

00:15:36.320 --> 00:15:40.399
coordination issues um where and in this

00:15:40.399 --> 00:15:42.880
case everyone is being blocked by by one

00:15:42.880 --> 00:15:45.760
team um and maybe they are able to to do

00:15:45.760 --> 00:15:49.279
their things. I saw on on a on an annual

00:15:49.279 --> 00:15:51.759
on a quarter review uh slide they showed

00:15:51.759 --> 00:15:53.759
the diagram with the throughput of every

00:15:53.759 --> 00:15:55.920
team and the true of that one team that

00:15:55.920 --> 00:15:58.959
everyone belongs depends on is massive

00:15:58.959 --> 00:16:00.720
they're like double or triple from

00:16:00.720 --> 00:16:03.839
everyone else and then I'm like yeah but

00:16:03.839 --> 00:16:06.160
why do you think that is cuz everyone

00:16:06.160 --> 00:16:08.480
else is waiting for them to be honest

00:16:08.480 --> 00:16:10.000
you could say they're doing such a great

00:16:10.000 --> 00:16:11.759
job they're doing four times what

00:16:11.759 --> 00:16:13.360
everyone else is doing

00:16:13.360 --> 00:16:15.120
>> interesting how you can uh you can spin

00:16:15.120 --> 00:16:17.519
it like that Yeah, you

00:16:17.519 --> 00:16:19.440
one graph shows you the throughput which

00:16:19.440 --> 00:16:21.199
is through the roof and the other and

00:16:21.199 --> 00:16:22.560
then and then if you're a bit more

00:16:22.560 --> 00:16:24.320
careful you show that like everyone's

00:16:24.320 --> 00:16:26.560
blocked by this team. I mean I was

00:16:26.560 --> 00:16:28.079
asking you know when they show this on

00:16:28.079 --> 00:16:30.399
the screen I was asking um in direct

00:16:30.399 --> 00:16:32.079
message to someone and I was saying can

00:16:32.079 --> 00:16:34.000
we can we look that from a different

00:16:34.000 --> 00:16:35.680
angle? [laughter] Yeah,

00:16:35.680 --> 00:16:37.040
>> I don't want to say it publicly here,

00:16:37.040 --> 00:16:38.160
but

00:16:38.160 --> 00:16:39.759
>> yeah, that's that's the trouble. A lot a

00:16:39.759 --> 00:16:43.199
lot of these like big company um

00:16:43.199 --> 00:16:45.839
announcements like when they go like,

00:16:45.839 --> 00:16:49.360
>> "Hey, our team has done so much better

00:16:49.360 --> 00:16:53.120
in the town hall, they have this like

00:16:53.120 --> 00:16:54.880
>> but nobody nobody said the team is doing

00:16:54.880 --> 00:16:56.480
the great job." it just they were at the

00:16:56.480 --> 00:16:58.800
top and and I was just thinking I don't

00:16:58.800 --> 00:17:00.240
think that's very

00:17:00.240 --> 00:17:01.519
>> I just feel like I feel like

00:17:01.519 --> 00:17:04.799
communication I I guess I don't think

00:17:04.799 --> 00:17:07.520
communication sold in any company really

00:17:07.520 --> 00:17:08.400
but

00:17:08.400 --> 00:17:09.280
>> but sometimes

00:17:09.280 --> 00:17:10.640
>> for many reasons right

00:17:10.640 --> 00:17:12.799
>> yeah for many reasons but like sometimes

00:17:12.799 --> 00:17:15.120
you have a team that like wants to show

00:17:15.120 --> 00:17:16.640
the impact because everyone's like has

00:17:16.640 --> 00:17:18.880
to show business value has to show

00:17:18.880 --> 00:17:19.919
impact

00:17:19.919 --> 00:17:21.679
>> and then they they say something which

00:17:21.679 --> 00:17:23.760
is kind of ridiculous

00:17:23.760 --> 00:17:27.199
or doesn't is missing some key context

00:17:27.199 --> 00:17:29.039
and there's just no way to sort of

00:17:29.039 --> 00:17:31.840
contribute in a in a nice way to say

00:17:31.840 --> 00:17:34.320
that h actually the way that you said

00:17:34.320 --> 00:17:37.039
that isn't quite correct because

00:17:37.039 --> 00:17:38.720
um you know it's just too late by then

00:17:38.720 --> 00:17:40.880
or something and everyone's having to

00:17:40.880 --> 00:17:44.240
absorb what one person's point of view

00:17:44.240 --> 00:17:46.799
basically. Yeah. So that was the first

00:17:46.799 --> 00:17:49.120
thing when we talked about like the the

00:17:49.120 --> 00:17:50.559
coordination issues and about

00:17:50.559 --> 00:17:53.200
infrastructure orchestration. Uh but

00:17:53.200 --> 00:17:54.559
then the second thing was you were

00:17:54.559 --> 00:17:57.760
asking if AI can be really useful in

00:17:57.760 --> 00:17:59.600
those cases of how do you know that AI

00:17:59.600 --> 00:18:03.679
is doing a good job and um AWS made a

00:18:03.679 --> 00:18:06.400
repository public called the AWS bench

00:18:06.400 --> 00:18:07.919
AWS-bench

00:18:07.919 --> 00:18:09.280
on GitHub

00:18:09.280 --> 00:18:11.840
>> and I had a play with it because I have

00:18:11.840 --> 00:18:14.640
a hypothesis and I thought you know I

00:18:14.640 --> 00:18:16.880
want to I want to put some numbers

00:18:16.880 --> 00:18:21.600
behind my claims and um while Fable was

00:18:21.600 --> 00:18:24.000
trying to repurposed their benchmarking

00:18:24.000 --> 00:18:26.240
solution to prove my hypothesis or

00:18:26.240 --> 00:18:27.919
disprove my hypothesis. Hopefully not

00:18:27.919 --> 00:18:30.720
disprove, but um I was actually having a

00:18:30.720 --> 00:18:32.559
look at the number of scenarios that are

00:18:32.559 --> 00:18:36.240
in there. Um I mean I don't know if I I

00:18:36.240 --> 00:18:36.720
could share.

00:18:36.720 --> 00:18:38.559
>> Yeah. Yeah. Share your screen. I haven't

00:18:38.559 --> 00:18:40.240
actually jumped into the source code,

00:18:40.240 --> 00:18:45.280
but I do think AWS Bench is a fantastic

00:18:45.280 --> 00:18:48.160
idea because like I've been evaluating

00:18:48.160 --> 00:18:50.320
my models on the on Simon Willis's

00:18:50.320 --> 00:18:52.240
Pelican up until now, which is

00:18:52.240 --> 00:18:53.440
ridiculous. [laughter]

00:18:53.440 --> 00:18:55.039
>> Which is ridiculous because I need a

00:18:55.039 --> 00:18:57.200
model to do infrastructure stuff. I

00:18:57.200 --> 00:18:59.280
don't need it to draw freaking Pelican.

00:18:59.280 --> 00:19:02.320
>> Yeah, this fact's only there. Okay, so

00:19:02.320 --> 00:19:06.160
AWS Bench uh new or on GitHub. I don't

00:19:06.160 --> 00:19:07.840
know if you can still hear me if I go so

00:19:07.840 --> 00:19:12.880
far. So, um, what's interesting is the

00:19:12.880 --> 00:19:14.400
>> Sorry,

00:19:14.400 --> 00:19:16.000
>> I will I will zoom in.

00:19:16.000 --> 00:19:17.840
>> Yeah, let's choose something that maybe

00:19:17.840 --> 00:19:21.120
we all understand like let's go and No,

00:19:21.120 --> 00:19:23.760
let me just find them again first. So,

00:19:23.760 --> 00:19:25.679
wait, I'm in the wrong repo. I think I

00:19:25.679 --> 00:19:28.400
need to go to the data sets. That's

00:19:28.400 --> 00:19:30.559
where I think the scenarios are. So,

00:19:30.559 --> 00:19:33.840
then I think it's under tasks. Yeah. So

00:19:33.840 --> 00:19:36.160
they have these categories and what's

00:19:36.160 --> 00:19:39.840
really interesting is um when I started

00:19:39.840 --> 00:19:42.799
running it on my account actually um it

00:19:42.799 --> 00:19:45.919
needs AWS or it needs an AWS or

00:19:45.919 --> 00:19:50.240
management account and then it uses a um

00:19:50.240 --> 00:19:54.000
an OU operating unit and you you then

00:19:54.000 --> 00:19:57.600
have an anchor AWS account and it finds

00:19:57.600 --> 00:19:59.600
the AWS accounts to test in. Normally it

00:19:59.600 --> 00:20:02.240
will mint a fresh AWS account or reuse

00:20:02.240 --> 00:20:04.080
an AWS account just to run the

00:20:04.080 --> 00:20:06.080
benchmark. Okay. So just to run the

00:20:06.080 --> 00:20:08.799
benchmark it will stand up an isolated

00:20:08.799 --> 00:20:11.919
AWS account deploy an SCP so that the

00:20:11.919 --> 00:20:14.000
agent is completely isolated to work in

00:20:14.000 --> 00:20:16.720
a single region. That's how how it's

00:20:16.720 --> 00:20:19.280
yeah it's pretty cool

00:20:19.280 --> 00:20:22.000
>> and then it time I kind of wish it was

00:20:22.000 --> 00:20:25.039
not requiring live cloud resources. cuz

00:20:25.039 --> 00:20:28.159
I wish you could almost run it offline.

00:20:28.159 --> 00:20:31.440
>> There is there is a version.

00:20:31.440 --> 00:20:33.039
>> Okay, let's not go down this offline

00:20:33.039 --> 00:20:35.679
rabbit hole. So just just show just show

00:20:35.679 --> 00:20:37.679
one of these these um these test

00:20:37.679 --> 00:20:40.799
>> scenarios. Yeah, so there are which one

00:20:40.799 --> 00:20:44.960
was I looking at? I was looking

00:20:44.960 --> 00:20:48.559
like for example diagnose ALB. So, I

00:20:48.559 --> 00:20:51.520
think before it runs,

00:20:51.520 --> 00:20:58.080
um

00:20:58.080 --> 00:21:00.000
I'm a little bit lost now. I I remember

00:21:00.000 --> 00:21:02.720
there being basically an infrastructure

00:21:02.720 --> 00:21:05.919
like the a CDK um stack that gets

00:21:05.919 --> 00:21:09.280
deployed into the AWS account uh before

00:21:09.280 --> 00:21:11.840
the agent gets unleashed and then a

00:21:11.840 --> 00:21:16.640
bunch of um questions a bunch a bunch of

00:21:16.640 --> 00:21:18.640
uh

00:21:18.640 --> 00:21:21.760
troubleshooting scenarios.

00:21:21.760 --> 00:21:24.080
Okay, so this is the the actual CDK app

00:21:24.080 --> 00:21:26.080
with setting up the environment that so

00:21:26.080 --> 00:21:28.080
the scenario sets up the environment for

00:21:28.080 --> 00:21:30.799
the troubleshooting scenario tunnel.

00:21:30.799 --> 00:21:32.720
>> Yeah. So no, that's not very

00:21:32.720 --> 00:21:35.360
interesting. The task So the task run

00:21:35.360 --> 00:21:37.039
inside those scenarios. So I think the

00:21:37.039 --> 00:21:39.039
scenarios set up the AWS environment,

00:21:39.039 --> 00:21:41.440
you know, deploy a bunch of resources

00:21:41.440 --> 00:21:44.320
like an ALB, an API gateway and so on.

00:21:44.320 --> 00:21:46.320
And then it provisions

00:21:46.320 --> 00:21:49.280
>> there's a test uh folder down there if

00:21:49.280 --> 00:21:50.320
you saw that.

00:21:50.320 --> 00:21:52.799
>> This is the the this is it. This is it.

00:21:52.799 --> 00:21:56.240
The instruction one. Um so this is what

00:21:56.240 --> 00:21:59.200
the so it provisions and harness it can

00:21:59.200 --> 00:22:01.760
be Claude Code it can be kiru cli it runs

00:22:01.760 --> 00:22:03.360
inside the docker container completely

00:22:03.360 --> 00:22:05.360
isolated you can decide what is the

00:22:05.360 --> 00:22:06.720
environment that the container gets it

00:22:06.720 --> 00:22:08.799
can have skills or whatever and then you

00:22:08.799 --> 00:22:10.400
give it this is an instruction for this

00:22:10.400 --> 00:22:11.919
particular task in that particular

00:22:11.919 --> 00:22:15.120
scenario my ALB so that gets injected

00:22:15.120 --> 00:22:17.919
started running 5xx what's going on okay

00:22:17.919 --> 00:22:22.080
so all of these you know launch an agent

00:22:22.080 --> 00:22:24.720
with the tools with a certain set of

00:22:24.720 --> 00:22:27.120
tools can be an MCP can be skills insert

00:22:27.120 --> 00:22:29.360
inside a sandbox environment against a

00:22:29.360 --> 00:22:32.400
real AWS account with real AWS resources

00:22:32.400 --> 00:22:34.799
and then a simple prompt and then when

00:22:34.799 --> 00:22:38.080
it the AI is finished it will then u you

00:22:38.080 --> 00:22:40.880
know be judged by by by its performance

00:22:40.880 --> 00:22:43.280
uh against that task and you can see

00:22:43.280 --> 00:22:46.159
that the number of of scenarios are are

00:22:46.159 --> 00:22:47.600
that the number of scenarios and the

00:22:47.600 --> 00:22:49.280
number of tasks against those scenarios

00:22:49.280 --> 00:22:53.360
are are massive really really big right

00:22:53.360 --> 00:22:56.320
Um, it's significant. Very interesting.

00:22:56.320 --> 00:22:58.559
>> It's amazing though at the same time a

00:22:58.559 --> 00:23:00.559
little bit scary because like I've sort

00:23:00.559 --> 00:23:03.520
of prided myself on my troubleshooting

00:23:03.520 --> 00:23:05.039
abilities

00:23:05.039 --> 00:23:06.960
over the years and pride myself

00:23:06.960 --> 00:23:08.799
troubleshooting skills and now this is

00:23:08.799 --> 00:23:10.720
kind of scary because it's automating my

00:23:10.720 --> 00:23:14.400
troubleshooting skills away.

00:23:14.400 --> 00:23:17.919
Uh yeah, I think as a platform engineer,

00:23:17.919 --> 00:23:23.120
if you are not focused on

00:23:23.120 --> 00:23:26.640
finding a way to leverage this type of

00:23:26.640 --> 00:23:29.280
um you know agents and providing these

00:23:29.280 --> 00:23:31.760
as an offer as a service for for for

00:23:31.760 --> 00:23:34.720
your um for your product teams, right?

00:23:34.720 --> 00:23:36.159
Because ultimately what is a platform

00:23:36.159 --> 00:23:38.080
engineer? We are responsible for making

00:23:38.080 --> 00:23:40.080
it easy for products to be delivered to

00:23:40.080 --> 00:23:42.080
production and iterate. So however

00:23:42.080 --> 00:23:44.000
that's done like it's the fact that we

00:23:44.000 --> 00:23:45.600
were doing infras that was just because

00:23:45.600 --> 00:23:47.760
that's the way was what we needed to do

00:23:47.760 --> 00:23:50.000
today product engineers they can write

00:23:50.000 --> 00:23:52.000
the infra all on their own you know they

00:23:52.000 --> 00:23:54.400
can create and define it so actually

00:23:54.400 --> 00:23:56.320
platform engineers always are supposed

00:23:56.320 --> 00:23:58.559
to just be focused on guardrails and

00:23:58.559 --> 00:24:00.880
discoverability and observability

00:24:00.880 --> 00:24:03.840
>> guard rails and discoverability yeah the

00:24:03.840 --> 00:24:07.760
the critical thing here

00:24:07.760 --> 00:24:11.120
is that I hope this AWS bench can be

00:24:11.120 --> 00:24:14.320
extended like for example

00:24:14.320 --> 00:24:17.039
an I mean they they AWS has a whole

00:24:17.039 --> 00:24:18.720
bunch that AWS bench has a whole bunch

00:24:18.720 --> 00:24:21.200
of scenarios but but these are not the

00:24:21.200 --> 00:24:24.320
exact scenarios for uh you know an

00:24:24.320 --> 00:24:27.600
insurance company or a financial company

00:24:27.600 --> 00:24:31.120
or a whatever company they we people

00:24:31.120 --> 00:24:34.080
need to basically contribute or have

00:24:34.080 --> 00:24:37.520
their own internal scenarios here so

00:24:37.520 --> 00:24:40.960
that uh agents can be tested and

00:24:40.960 --> 00:24:44.720
validated that they it can do certain

00:24:44.720 --> 00:24:48.480
day-to-day work in that particular

00:24:48.480 --> 00:24:52.159
uh business environment. So I I I really

00:24:52.159 --> 00:24:54.640
hope that AWS Bench I I'm going to study

00:24:54.640 --> 00:24:56.799
AWS Bench but like you you get the point

00:24:56.799 --> 00:24:58.880
right that you need to to have your own

00:24:58.880 --> 00:25:01.120
benchmark for your own company.

00:25:01.120 --> 00:25:03.200
>> Yes. And and it's also something that

00:25:03.200 --> 00:25:05.600
I've been um because what why did I look

00:25:05.600 --> 00:25:07.679
into AWS bench is because I've had this

00:25:07.679 --> 00:25:11.279
idea and this hypothesis that um if you

00:25:11.279 --> 00:25:14.320
use a higher level language uh like the

00:25:14.320 --> 00:25:16.799
cloud development kit uh allows you to

00:25:16.799 --> 00:25:20.480
build um agents can be more efficient.

00:25:20.480 --> 00:25:22.400
You spend less tokens. They will be

00:25:22.400 --> 00:25:24.880
faster. You spend less time. they will

00:25:24.880 --> 00:25:26.960
be more accurate and the result will be

00:25:26.960 --> 00:25:28.960
more reliable because you will have hit

00:25:28.960 --> 00:25:31.600
less uh issues as as you are maintaining

00:25:31.600 --> 00:25:34.240
it in like long-term deops too. So

00:25:34.240 --> 00:25:36.320
that's why I wanted AWS batch because I

00:25:36.320 --> 00:25:38.080
want to put numbers behind that claim. I

00:25:38.080 --> 00:25:40.400
want to actually evaluate an agent with

00:25:40.400 --> 00:25:41.440
raw terapform.

00:25:41.440 --> 00:25:42.400
>> That's very cool.

00:25:42.400 --> 00:25:44.720
>> An agent with uh raw terraform and some

00:25:44.720 --> 00:25:46.960
skills, an agent with terraform modules

00:25:46.960 --> 00:25:49.600
and some skills versus uh an agent with

00:25:49.600 --> 00:25:52.640
AWS CDK. um you know which one is

00:25:52.640 --> 00:25:55.520
creating a more maintainable solution

00:25:55.520 --> 00:25:59.120
that can be um against

00:25:59.120 --> 00:26:02.159
>> though I mean can I assume that that

00:26:02.159 --> 00:26:03.919
I've got to look at these uh scenarios

00:26:03.919 --> 00:26:06.559
and I've got to look at the the way the

00:26:06.559 --> 00:26:08.960
benchmarks work but like in mo in many

00:26:08.960 --> 00:26:10.960
cases at least on AWS that a lot of

00:26:10.960 --> 00:26:12.720
these

00:26:12.720 --> 00:26:14.960
the chief operating

00:26:14.960 --> 00:26:19.279
mechanism is AWS CLI right

00:26:19.279 --> 00:26:21.200
that's that's the chief

00:26:21.200 --> 00:26:23.679
that that's what the model is expected

00:26:23.679 --> 00:26:27.679
to do, right? to basically or is is the

00:26:27.679 --> 00:26:31.360
model writing typescript?

00:26:31.360 --> 00:26:34.159
I so so this whole thing from AWS is

00:26:34.159 --> 00:26:37.039
built upon an existing framework from uh

00:26:37.039 --> 00:26:40.159
terminal bench uh which is I forgot the

00:26:40.159 --> 00:26:43.360
name for the framework now but um it's

00:26:43.360 --> 00:26:45.600
an existing bench benchmarking framework

00:26:45.600 --> 00:26:48.000
so it can be used to benchmark terminal

00:26:48.000 --> 00:26:50.159
you know proficiency and in this case

00:26:50.159 --> 00:26:53.200
they modified it to to to benchmark um

00:26:53.200 --> 00:26:55.039
troubleshooting capabilities against

00:26:55.039 --> 00:26:58.400
live AWS infrastructure to see if if the

00:26:58.400 --> 00:26:59.840
you know where they can optimize the

00:26:59.840 --> 00:27:02.159
agents either with skills Um so this

00:27:02.159 --> 00:27:04.799
benchmark can be for for anything right

00:27:04.799 --> 00:27:06.799
in my case I'm giving it the task to

00:27:06.799 --> 00:27:10.240
create infrastructure and I've also told

00:27:10.240 --> 00:27:13.600
fable my first you know set of of

00:27:13.600 --> 00:27:15.919
scenarios are focused on creating infra

00:27:15.919 --> 00:27:18.000
my second set of scenarios is mutating

00:27:18.000 --> 00:27:20.000
infra day two ops right we have

00:27:20.000 --> 00:27:21.919
something we need to modify it and then

00:27:21.919 --> 00:27:24.960
we need to make sure that uh because

00:27:24.960 --> 00:27:26.640
sometimes you can create infra but then

00:27:26.640 --> 00:27:27.840
when you have to change it it becomes

00:27:27.840 --> 00:27:29.520
very painful so I want I want to

00:27:29.520 --> 00:27:31.919
validate that as well Right. So, um

00:27:31.919 --> 00:27:33.520
those are the scenarios and the tasks

00:27:33.520 --> 00:27:36.799
that I am I'm looking at. Um the guy

00:27:36.799 --> 00:27:39.919
from Chant that we talked about earlier,

00:27:39.919 --> 00:27:43.039
he he built this version

00:27:43.039 --> 00:27:46.159
uh from you know and what he did is he

00:27:46.159 --> 00:27:48.640
replaced the AWS uh requirements with

00:27:48.640 --> 00:27:51.039
flowy. So flossy I don't know how you

00:27:51.039 --> 00:27:52.640
pronounce it but it's a local stack

00:27:52.640 --> 00:27:54.799
alternative like local stack went they

00:27:54.799 --> 00:27:57.440
added the license requirement and flossy

00:27:57.440 --> 00:27:58.240
what they call

00:27:58.240 --> 00:28:00.080
>> is brilliant because that's that's that

00:28:00.080 --> 00:28:03.039
plays into that offline comment earlier.

00:28:03.039 --> 00:28:05.760
>> Yes. So he built uh he has his own

00:28:05.760 --> 00:28:08.480
container. Um so so you have to build

00:28:08.480 --> 00:28:10.159
the flocky container with all the tools

00:28:10.159 --> 00:28:12.320
in it and then he runs the agents in

00:28:12.320 --> 00:28:14.480
there because he wants to prove that the

00:28:14.480 --> 00:28:16.480
agents are able to do these scenarios

00:28:16.480 --> 00:28:19.039
and these task better with um highle

00:28:19.039 --> 00:28:21.039
solution that is his solution basically

00:28:21.039 --> 00:28:22.880
me and him where are we talking a lot uh

00:28:22.880 --> 00:28:23.440
and and

00:28:23.440 --> 00:28:26.240
>> it's brand

00:28:26.240 --> 00:28:32.159
>> and and he I did not do the flowy or foc

00:28:32.159 --> 00:28:35.039
I I um I was afraid that I would hit um

00:28:35.039 --> 00:28:37.520
limitations of the local stack. And I I

00:28:37.520 --> 00:28:39.039
thought you know let let's just do it

00:28:39.039 --> 00:28:41.279
with the real AWS [laughter]

00:28:41.279 --> 00:28:44.159
for uh for you know first first draft

00:28:44.159 --> 00:28:46.559
and then look at um quick uh sandbox

00:28:46.559 --> 00:28:48.720
like local iterations to improve upon

00:28:48.720 --> 00:28:50.880
that because if I know it works in AWS

00:28:50.880 --> 00:28:53.840
and then I I I put FL Faky in it and it

00:28:53.840 --> 00:28:55.120
starts failing then I know that it's

00:28:55.120 --> 00:28:57.840
Faky and not not uh not my my other

00:28:57.840 --> 00:28:59.120
stuff. Right.

00:28:59.120 --> 00:29:01.279
>> Yeah. This is really cool. This is

00:29:01.279 --> 00:29:06.720
really cool. the the

00:29:06.720 --> 00:29:10.000
thing I wanted to talk to you about um

00:29:10.000 --> 00:29:14.159
is still I think a void for agents I'm

00:29:14.159 --> 00:29:16.480
sorry to bring it up again but like in

00:29:16.480 --> 00:29:18.720
many companies you use something like

00:29:18.720 --> 00:29:22.000
octo or zero whatever and you have

00:29:22.000 --> 00:29:23.440
different vendors right you have your

00:29:23.440 --> 00:29:25.360
page of duty you have your slack you

00:29:25.360 --> 00:29:28.960
have your atlassian you have your uh I

00:29:28.960 --> 00:29:30.320
don't know you have a whole bunch of

00:29:30.320 --> 00:29:33.520
them the the thing that I'm struggling

00:29:33.520 --> 00:29:38.320
with is uh

00:29:38.320 --> 00:29:41.600
like say say um say you want your agent

00:29:41.600 --> 00:29:43.360
or something like that your agent

00:29:43.360 --> 00:29:47.760
identity to to access these various

00:29:47.760 --> 00:29:50.000
services in most cases you're going to

00:29:50.000 --> 00:29:53.120
go for a long live credential right I'm

00:29:53.120 --> 00:29:56.399
just no you can do everything with OIDC

00:29:56.399 --> 00:29:58.000
you can do you can do everything with

00:29:58.000 --> 00:30:00.000
the MC I mean some some things don't

00:30:00.000 --> 00:30:02.080
have MCPS

00:30:02.080 --> 00:30:03.360
some things don't ever

00:30:03.360 --> 00:30:04.880
>> I'm actually having that problem at work

00:30:04.880 --> 00:30:10.880
now as well. Um so I think

00:30:10.880 --> 00:30:12.799
I think you need to have a very like

00:30:12.799 --> 00:30:14.559
mature organization that have proper

00:30:14.559 --> 00:30:16.399
onboarding and offboarding of of of your

00:30:16.399 --> 00:30:18.240
employees and single sign on setup and

00:30:18.240 --> 00:30:20.880
integrated across uh all of your your

00:30:20.880 --> 00:30:22.399
systems.

00:30:22.399 --> 00:30:25.760
um you will I think you're right that if

00:30:25.760 --> 00:30:28.480
you're talking about agents and and um

00:30:28.480 --> 00:30:30.320
basically

00:30:30.320 --> 00:30:33.279
thing things that do not have this um

00:30:33.279 --> 00:30:35.760
device off flow or like that need to

00:30:35.760 --> 00:30:37.679
have a token and then refresh the token

00:30:37.679 --> 00:30:39.919
like with OIDC you have these uh the

00:30:39.919 --> 00:30:42.320
ability to to authenticate it give it a

00:30:42.320 --> 00:30:44.080
token and then a refresh token to keep

00:30:44.080 --> 00:30:45.039
refreshing the

00:30:45.039 --> 00:30:47.760
>> I mean it's also the oorthth flow does

00:30:47.760 --> 00:30:49.600
that right I mean and I I'm always a

00:30:49.600 --> 00:30:52.000
little bit confused I think of oid DC is

00:30:52.000 --> 00:30:54.799
like a like a trust relationship and

00:30:54.799 --> 00:30:56.720
then I think of oorthth as the one that

00:30:56.720 --> 00:30:59.679
with the the token and and the refresh

00:30:59.679 --> 00:31:00.559
rate. But

00:31:00.559 --> 00:31:02.240
>> yeah, you're right. OIDC is a wrapper

00:31:02.240 --> 00:31:03.600
around too.

00:31:03.600 --> 00:31:04.159
>> Yeah.

00:31:04.159 --> 00:31:06.080
>> Um and that's maybe where I'm like

00:31:06.080 --> 00:31:08.320
prompting the AI and guiding it wrong.

00:31:08.320 --> 00:31:09.760
Uh that I should probably be using

00:31:09.760 --> 00:31:11.039
because we just had a discussion about

00:31:11.039 --> 00:31:12.320
that. We had this problem with our

00:31:12.320 --> 00:31:14.559
agents now is like how do we give them

00:31:14.559 --> 00:31:18.640
access um to tools and how do we make

00:31:18.640 --> 00:31:19.840
sure that they don't have lawyers

00:31:19.840 --> 00:31:22.080
>> MCPS is is the right path but there's

00:31:22.080 --> 00:31:24.320
many cases when there when there isn't

00:31:24.320 --> 00:31:26.720
the um

00:31:26.720 --> 00:31:30.000
>> well like for example like say you want

00:31:30.000 --> 00:31:32.080
a service account or something then

00:31:32.080 --> 00:31:34.080
things get weird cuz cuz yeah when you

00:31:34.080 --> 00:31:36.159
run an agent on someone's computer when

00:31:36.159 --> 00:31:38.159
you're when you're authenticated like in

00:31:38.159 --> 00:31:41.200
my Kai Hendry account then I I think

00:31:41.200 --> 00:31:42.880
things become a lot easier but like when

00:31:42.880 --> 00:31:45.840
you want to do machine to machine stuff

00:31:45.840 --> 00:31:49.440
things get weird very quickly and anyway

00:31:49.440 --> 00:31:51.440
I'm just trying to say that like I feel

00:31:51.440 --> 00:31:54.559
like am I'm well this is I'm just

00:31:54.559 --> 00:31:56.640
winging through this problem right now a

00:31:56.640 --> 00:32:00.559
lot of am identity and access management

00:32:00.559 --> 00:32:03.200
stuff seems to be the bottleneck in uh

00:32:03.200 --> 00:32:04.880
in this particular client I'm working

00:32:04.880 --> 00:32:07.600
with right now

00:32:07.600 --> 00:32:12.559
I am it's hard it's really really hard.

00:32:12.559 --> 00:32:14.720
Well, when you run, in our case, we're

00:32:14.720 --> 00:32:16.960
looking at agent core that's integrated.

00:32:16.960 --> 00:32:20.320
They get an IM role and they have the,

00:32:20.320 --> 00:32:23.679
you know, we can use agent core identity

00:32:23.679 --> 00:32:26.240
with the credential provider

00:32:26.240 --> 00:32:28.240
>> identity. What is agent? Oh, Bedrock

00:32:28.240 --> 00:32:30.159
agent core. Yeah, sorry.

00:32:30.159 --> 00:32:31.360
>> No, it's not.

00:32:31.360 --> 00:32:34.000
>> So, Bedrock, they launched agents and

00:32:34.000 --> 00:32:36.799
then they already retired it. So yes,

00:32:36.799 --> 00:32:39.360
it's called bedrock agent core, but the

00:32:39.360 --> 00:32:41.679
messaging from AWS around agent core is

00:32:41.679 --> 00:32:43.600
very weird. Like on one side you have

00:32:43.600 --> 00:32:46.880
bedrock which is all about like models,

00:32:46.880 --> 00:32:50.000
hosting, invoking um you know training,

00:32:50.000 --> 00:32:50.960
tuning

00:32:50.960 --> 00:32:53.039
>> and then you have agent core which is

00:32:53.039 --> 00:32:56.399
all about making agent production ready

00:32:56.399 --> 00:32:59.679
um being able to run agents um with all

00:32:59.679 --> 00:33:02.559
of the requirements around it like in

00:33:02.559 --> 00:33:04.799
terms of obs observability

00:33:04.799 --> 00:33:10.559
um access to MCP tools um identity

00:33:10.559 --> 00:33:12.080
>> yeah eval

00:33:12.080 --> 00:33:14.480
observability

00:33:14.480 --> 00:33:17.440
prompt injection detection or prevention

00:33:17.440 --> 00:33:18.320
rather

00:33:18.320 --> 00:33:20.720
>> and agent core is like it's like your

00:33:20.720 --> 00:33:23.360
ECS for containers you have a container

00:33:23.360 --> 00:33:25.120
you want to run it okay you need to have

00:33:25.120 --> 00:33:27.360
rollout capabilities you need to have

00:33:27.360 --> 00:33:29.279
you know the ability to do ingress with

00:33:29.279 --> 00:33:31.440
load balancers that's what elastic

00:33:31.440 --> 00:33:33.200
container service does for containers

00:33:33.200 --> 00:33:35.120
right so agent core is the same thing

00:33:35.120 --> 00:33:37.279
you have an agent however you built it

00:33:37.279 --> 00:33:38.720
you know ECS doesn't care what you put

00:33:38.720 --> 00:33:41.760
in your container um ECS as uh sorry

00:33:41.760 --> 00:33:43.760
agent core doesn't care which which is

00:33:43.760 --> 00:33:45.760
the SDK that you use to build your

00:33:45.760 --> 00:33:49.600
agent. You can use entropic cloud uh SDK

00:33:49.600 --> 00:33:51.840
or you can use their AWS

00:33:51.840 --> 00:33:53.440
>> like a sandbox, right? It's like

00:33:53.440 --> 00:33:55.279
>> yeah it's it's it's a runtime. It's like

00:33:55.279 --> 00:33:57.200
a container environment. It's it's

00:33:57.200 --> 00:34:00.000
serverless. Um it it can be invoked

00:34:00.000 --> 00:34:01.919
create a session that stays around for a

00:34:01.919 --> 00:34:04.799
while um that can be resumed and then it

00:34:04.799 --> 00:34:07.519
can be you know that's it's a bit like

00:34:07.519 --> 00:34:12.079
lambda. It's a mix between lambda and um

00:34:12.079 --> 00:34:14.399
>> okay but this containers I got that but

00:34:14.399 --> 00:34:17.839
does it solve the identity IM issue?

00:34:17.839 --> 00:34:20.800
>> So agent core has an identity part which

00:34:20.800 --> 00:34:24.879
allows you to register um your your

00:34:24.879 --> 00:34:27.679
sorry wait which you can control the

00:34:27.679 --> 00:34:29.839
incoming user pool and then the agent

00:34:29.839 --> 00:34:32.399
acting on behalf of another user. So you

00:34:32.399 --> 00:34:35.919
can then have an outbo outbound identity

00:34:35.919 --> 00:34:39.359
um mechanism. So how does the agent core

00:34:39.359 --> 00:34:43.760
talk to um other systems and it has like

00:34:43.760 --> 00:34:45.760
I don't know through there's several me

00:34:45.760 --> 00:34:47.520
authentication mechanisms described in

00:34:47.520 --> 00:34:49.839
the outbound identity

00:34:49.839 --> 00:34:51.200
have a look at that.

00:34:51.200 --> 00:34:52.960
>> Yeah. And and then there's a machine to

00:34:52.960 --> 00:34:54.879
machine where where you have an API key

00:34:54.879 --> 00:34:59.440
or a similar for the agent to to use. Um

00:34:59.440 --> 00:35:01.920
identity has an a credential vault so it

00:35:01.920 --> 00:35:04.640
can inject but it doesn't do uh some of

00:35:04.640 --> 00:35:07.280
the things that you would expect. So it

00:35:07.280 --> 00:35:08.800
can be very confusing because it looks

00:35:08.800 --> 00:35:10.160
like it can do it but then when you get

00:35:10.160 --> 00:35:11.359
to the to the point it actually

00:35:11.359 --> 00:35:13.839
>> what about what about state because one

00:35:13.839 --> 00:35:16.240
one one issue that we have at work is

00:35:16.240 --> 00:35:20.000
that we have a lot of data and uh we can

00:35:20.000 --> 00:35:23.119
get agents to grab the data and uh give

00:35:23.119 --> 00:35:24.640
us some insights and everything like

00:35:24.640 --> 00:35:26.880
that but it's so slow to get the data

00:35:26.880 --> 00:35:29.520
out of like you know Jira Workday the

00:35:29.520 --> 00:35:31.920
usual suspects

00:35:31.920 --> 00:35:34.014
>> don't use Jira

00:35:34.014 --> 00:35:35.119
[laughter]

00:35:35.119 --> 00:35:38.160
yeah that's the solution

00:35:38.160 --> 00:35:40.720
But like but like you need a caching

00:35:40.720 --> 00:35:43.119
layer often just otherwise the whole

00:35:43.119 --> 00:35:45.920
solution is not workable. And I wonder

00:35:45.920 --> 00:35:48.880
if someone like I don't think anyone

00:35:48.880 --> 00:35:50.800
would mind me saying that at at my

00:35:50.800 --> 00:35:54.640
employer like GraphQL Apollo the Apollo

00:35:54.640 --> 00:35:58.480
makes a surfaces again. I've used Apollo

00:35:58.480 --> 00:36:01.040
on and off some years. I mean in

00:36:01.040 --> 00:36:03.359
different engagements

00:36:03.359 --> 00:36:05.359
like is there is there a good solution

00:36:05.359 --> 00:36:08.960
for for caching uh

00:36:08.960 --> 00:36:10.560
solution for this was identity actually

00:36:10.560 --> 00:36:12.640
this is something related to MCP

00:36:12.640 --> 00:36:13.599
authentication

00:36:13.599 --> 00:36:15.839
>> client ID yeah that's a good that's a

00:36:15.839 --> 00:36:17.040
good concept

00:36:17.040 --> 00:36:19.280
>> yeah so that one is is the one that that

00:36:19.280 --> 00:36:22.160
this morning um I was I was told that

00:36:22.160 --> 00:36:24.240
this is what we use to to authenticate

00:36:24.240 --> 00:36:27.599
against MCP so I mean I didn't

00:36:27.599 --> 00:36:29.359
understand all the zero but like I do

00:36:29.359 --> 00:36:31.359
feel Sorry for like noobs cuz a lot of

00:36:31.359 --> 00:36:34.720
people don't understand alz

00:36:34.720 --> 00:36:38.640
public private keys client IDs

00:36:38.640 --> 00:36:40.320
secret key. Yeah, there's a lot going

00:36:40.320 --> 00:36:44.240
on. But but do did you have any insight

00:36:44.240 --> 00:36:46.960
to the whole caching besides not using

00:36:46.960 --> 00:36:51.119
uh Jira

00:36:51.119 --> 00:36:52.240
because you because you don't want

00:36:52.240 --> 00:36:54.960
agents to

00:36:54.960 --> 00:36:56.880
uh

00:36:56.880 --> 00:37:00.720
nondeterministically

00:37:00.720 --> 00:37:03.359
fetch

00:37:03.359 --> 00:37:07.520
key data all the time. G

00:37:07.520 --> 00:37:10.480
I think I think this goes to the the

00:37:10.480 --> 00:37:14.000
ongoing you know how how they the

00:37:14.000 --> 00:37:16.079
community goes into fl into cycles and

00:37:16.079 --> 00:37:19.200
and and like waves of technologies and

00:37:19.200 --> 00:37:20.720
you know everyone went all in on

00:37:20.720 --> 00:37:22.320
markdown and then there's this whole

00:37:22.320 --> 00:37:24.320
story about SAS is dead because every

00:37:24.320 --> 00:37:26.720
time an agent needs information it's it

00:37:26.720 --> 00:37:28.160
fetches it and then stores it in

00:37:28.160 --> 00:37:30.160
markdown on disk and then it it grabs it

00:37:30.160 --> 00:37:30.800
quickly.

00:37:30.800 --> 00:37:31.839
>> That's true. Yeah.

00:37:31.839 --> 00:37:32.960
>> Yeah. And then we had this whole

00:37:32.960 --> 00:37:34.720
discussion about markdown is a broken

00:37:34.720 --> 00:37:36.400
mirror because there's like 10 documents

00:37:36.400 --> 00:37:37.760
and they all say something slightly

00:37:37.760 --> 00:37:38.800
different. Um

00:37:38.800 --> 00:37:40.320
>> yeah, we should have

00:37:40.320 --> 00:37:42.560
>> and then we talked about um you know I

00:37:42.560 --> 00:37:45.599
mean the Twitter sphere or uh Xphere

00:37:45.599 --> 00:37:47.599
talks about um you know loop

00:37:47.599 --> 00:37:49.119
engineering, harness engineering,

00:37:49.119 --> 00:37:51.839
context engineering and now they're all

00:37:51.839 --> 00:37:54.160
talking about graph engineering. Yeah.

00:37:54.160 --> 00:37:54.560
And

00:37:54.560 --> 00:37:59.440
>> yeah, that's true. It does we

00:37:59.440 --> 00:38:01.599
Yeah, it does. You're quite right that

00:38:01.599 --> 00:38:04.480
we're going in circles here

00:38:04.480 --> 00:38:05.839
>> engineering. Now we're talking about

00:38:05.839 --> 00:38:07.359
data

00:38:07.359 --> 00:38:09.280
>> about the data. Yeah. Because because

00:38:09.280 --> 00:38:13.200
the you know maintaining that data uh in

00:38:13.200 --> 00:38:16.079
in a consistent way is hard and and

00:38:16.079 --> 00:38:18.240
that's where graph engineer I honestly I

00:38:18.240 --> 00:38:19.760
just read the term. I haven't even

00:38:19.760 --> 00:38:21.280
looked into it but the fact is that we

00:38:21.280 --> 00:38:24.720
are building graphs at work. So I don't

00:38:24.720 --> 00:38:26.079
know what they're talking about when do

00:38:26.079 --> 00:38:27.760
they do graph engineering but we're

00:38:27.760 --> 00:38:29.119
building graphs. So I guess we are doing

00:38:29.119 --> 00:38:30.880
graph engineering. [laughter]

00:38:30.880 --> 00:38:34.079
>> Yeah. I'm not I'm not a huge fan of

00:38:34.079 --> 00:38:35.680
GraphQL though because

00:38:35.680 --> 00:38:37.680
>> it's not GraphQL though. Like we're not

00:38:37.680 --> 00:38:39.440
using Apollo or GraphQL. We're we're

00:38:39.440 --> 00:38:42.640
we're using um we're using

00:38:42.640 --> 00:38:46.320
>> uh Neptune um to to basically like

00:38:46.320 --> 00:38:49.520
Neo4G, you know, graph query language.

00:38:49.520 --> 00:38:52.240
Uh it's not

00:38:52.240 --> 00:38:53.760
>> it's not Apollo. So yeah, we're not

00:38:53.760 --> 00:38:56.640
we're not using uh what's the that's

00:38:56.640 --> 00:38:59.599
like we're not going away from REST.

00:38:59.599 --> 00:39:02.480
>> Yeah. What what is the AWS version of uh

00:39:02.480 --> 00:39:04.400
GraphQL? I think is it Neptune? Yeah, I

00:39:04.400 --> 00:39:05.760
think it's Neptune.

00:39:05.760 --> 00:39:07.280
>> Neptune

00:39:07.280 --> 00:39:09.440
Neptune is a graph database. You store

00:39:09.440 --> 00:39:11.200
entities, you create relationship

00:39:11.200 --> 00:39:13.040
between entities and then you query

00:39:13.040 --> 00:39:14.720
them. You can go from one entity and and

00:39:14.720 --> 00:39:17.440
and do a thinization.

00:39:17.440 --> 00:39:20.320
My probably my wrong opinion about these

00:39:20.320 --> 00:39:22.240
sort of things is that is that people

00:39:22.240 --> 00:39:24.640
tend to

00:39:24.640 --> 00:39:26.960
well graph databases are powerful and

00:39:26.960 --> 00:39:29.680
then people make pretty crazy queries

00:39:29.680 --> 00:39:31.440
with them.

00:39:31.440 --> 00:39:33.359
I think some people argue that it makes

00:39:33.359 --> 00:39:34.960
more efficient queries, but it ends up

00:39:34.960 --> 00:39:36.560
that like it's the opposite is true.

00:39:36.560 --> 00:39:38.880
Like people make wild ass queries and

00:39:38.880 --> 00:39:40.640
then then they're quite difficult to

00:39:40.640 --> 00:39:44.160
tune and u reconcile and operate and

00:39:44.160 --> 00:39:45.839
things like this. But you're probably

00:39:45.839 --> 00:39:48.800
talking out my ass. No, I mean because I

00:39:48.800 --> 00:39:51.200
think it's different when you talk about

00:39:51.200 --> 00:39:53.599
a front end, maybe it's not different,

00:39:53.599 --> 00:39:55.599
but to me when you you talk about a

00:39:55.599 --> 00:39:57.440
front end doing REST endpoint queries

00:39:57.440 --> 00:39:59.040
and those rest endpoint queries go

00:39:59.040 --> 00:40:01.760
against the data repository pattern or

00:40:01.760 --> 00:40:03.839
you know um and then whenever the front

00:40:03.839 --> 00:40:05.520
end needs something new the the back end

00:40:05.520 --> 00:40:07.839
has to build an endpoint or or uh you

00:40:07.839 --> 00:40:10.400
know um and then what GitHub did they

00:40:10.400 --> 00:40:14.640
build this uh GraphQL um uh solution

00:40:14.640 --> 00:40:17.760
which allows front end to to to to send

00:40:17.760 --> 00:40:20.079
a a request of exactly the shape that

00:40:20.079 --> 00:40:22.800
they want that then on the back end

00:40:22.800 --> 00:40:25.839
needs to be resolved uh against um the

00:40:25.839 --> 00:40:29.520
data which to me sounds like an extreme

00:40:29.520 --> 00:40:31.119
>> um expensive operation. You know the

00:40:31.119 --> 00:40:32.960
whole the whole benefit of rest is that

00:40:32.960 --> 00:40:36.240
you can you have like immutable um sorry

00:40:36.240 --> 00:40:38.640
not immutable but you have

00:40:38.640 --> 00:40:41.520
>> input and endpoints and you can um can

00:40:41.520 --> 00:40:44.079
scale it out against a back end but now

00:40:44.079 --> 00:40:45.520
you're putting a query engine in

00:40:45.520 --> 00:40:47.119
between. So now you're increasing the

00:40:47.119 --> 00:40:48.000
compute required.

00:40:48.000 --> 00:40:49.280
>> Exactly. And then there's lots of

00:40:49.280 --> 00:40:53.520
security issues. There's uh

00:40:53.520 --> 00:40:57.280
>> um yeah like schema introspection. Yeah.

00:40:57.280 --> 00:40:59.119
All sorts of weird things can go on. I

00:40:59.119 --> 00:41:01.200
don't I don't know what is the name of

00:41:01.200 --> 00:41:03.839
the AWS service that does equivalent to

00:41:03.839 --> 00:41:07.280
Apollo. But it's not Neptune. Neptune is

00:41:07.280 --> 00:41:09.599
is like RDS. It's like Postgres. It's

00:41:09.599 --> 00:41:12.000
it's Neptune is like Neo4G. you're

00:41:12.000 --> 00:41:15.599
you're hosting a database in a different

00:41:15.599 --> 00:41:18.640
um you know structure. It's it's like

00:41:18.640 --> 00:41:20.400
>> it's Amazon AppSync.

00:41:20.400 --> 00:41:23.200
>> Yeah, AppSync is the GraphQL one uh

00:41:23.200 --> 00:41:25.440
equivalent. Yeah. So when you when you

00:41:25.440 --> 00:41:27.119
move away from that and you you're

00:41:27.119 --> 00:41:29.760
actually moving the data structures to

00:41:29.760 --> 00:41:33.520
something that is you know an a graph of

00:41:33.520 --> 00:41:37.920
object or um that is designed based on

00:41:37.920 --> 00:41:39.599
um objects and their relationship

00:41:39.599 --> 00:41:41.200
between objects and then you can query

00:41:41.200 --> 00:41:43.839
those objects. Uh I can imagine that you

00:41:43.839 --> 00:41:46.240
can actually make much more performant

00:41:46.240 --> 00:41:50.000
data queries. Um, but I have to say it's

00:41:50.000 --> 00:41:53.040
indeed very very hard to to cuz I see

00:41:53.040 --> 00:41:55.680
the demos that we we have and you have

00:41:55.680 --> 00:41:57.680
the MCP tool doing all the queries,

00:41:57.680 --> 00:41:59.760
building all the queries against uh

00:41:59.760 --> 00:42:02.319
against Neptune and it's it's like

00:42:02.319 --> 00:42:04.800
cycling, you know. Um, which is very

00:42:04.800 --> 00:42:08.480
frustrating. Uh, you're looking at cloud

00:42:08.480 --> 00:42:10.640
firing off all these queries to figure

00:42:10.640 --> 00:42:13.119
out the relationship between entities.

00:42:13.119 --> 00:42:15.280
Um, it's not fast. So, there definitely

00:42:15.280 --> 00:42:17.599
must be a faster way. I I think the

00:42:17.599 --> 00:42:22.000
appeal is that you can um represent you

00:42:22.000 --> 00:42:24.079
have to capture the data and then you

00:42:24.079 --> 00:42:26.720
can represent the data from any angle

00:42:26.720 --> 00:42:30.560
like you know materialized view of it

00:42:30.560 --> 00:42:32.880
>> um in a very fast and efficient way. So

00:42:32.880 --> 00:42:35.040
if you're focused on only like you can

00:42:35.040 --> 00:42:36.720
have your whole organization with all of

00:42:36.720 --> 00:42:37.920
the different teams and all of the

00:42:37.920 --> 00:42:39.440
products that they built, but you need

00:42:39.440 --> 00:42:41.520
to add one feature that touches two

00:42:41.520 --> 00:42:43.280
products and then you can focus and

00:42:43.280 --> 00:42:44.880
query only on those and then identify

00:42:44.880 --> 00:42:46.400
the repositories that those products

00:42:46.400 --> 00:42:48.400
have and then identify where you need to

00:42:48.400 --> 00:42:50.560
make modifications and then you can

00:42:50.560 --> 00:42:53.520
create the actual work and then hand it

00:42:53.520 --> 00:42:56.400
off to agents to do that work um

00:42:56.400 --> 00:42:59.119
efficiently without having to render a

00:42:59.119 --> 00:43:01.200
whole markdown plan. And then that

00:43:01.200 --> 00:43:02.800
markdown plan along the way got gets

00:43:02.800 --> 00:43:08.400
stale very quickly. Sorry, this

00:43:08.400 --> 00:43:11.520
old man stares at cloud

00:43:11.520 --> 00:43:14.480
shouts at cloud symptoms. Old man

00:43:14.480 --> 00:43:14.880
shouts.

00:43:14.880 --> 00:43:16.800
>> Sorry, there's a there's a rubbish

00:43:16.800 --> 00:43:18.400
pickup.

00:43:18.400 --> 00:43:21.920
>> I don't hear anything.

00:43:21.920 --> 00:43:23.119
>> You can't hear me?

00:43:23.119 --> 00:43:25.440
>> I can't hear the anything outside. I can

00:43:25.440 --> 00:43:26.400
hear your voice. I can't

00:43:26.400 --> 00:43:29.280
>> Okay. I got I I purposely put noise

00:43:29.280 --> 00:43:32.560
cancellation on. Yeah, I was I was just

00:43:32.560 --> 00:43:34.240
Oh, I got to give this some thought, but

00:43:34.240 --> 00:43:35.920
like

00:43:35.920 --> 00:43:39.200
these GraphQL things

00:43:39.200 --> 00:43:41.839
has come up time and time again on my

00:43:41.839 --> 00:43:45.599
different during my my long career and

00:43:45.599 --> 00:43:47.599
I'm still

00:43:47.599 --> 00:43:53.520
thinking that the the the

00:43:53.520 --> 00:43:56.160
promises of faster development. Okay, I

00:43:56.160 --> 00:43:58.640
I'll get behind greater flexibility

00:43:58.640 --> 00:44:01.040
maybe, but easier data management and

00:44:01.040 --> 00:44:03.520
faster development, I'm not I'm not a

00:44:03.520 --> 00:44:06.720
thousand% sure. I mean, maybe initially,

00:44:06.720 --> 00:44:09.520
but but operationally, I think it's a

00:44:09.520 --> 00:44:12.000
little bit of a nightmare.

00:44:12.000 --> 00:44:14.319
Okay, I think we have we we had a really

00:44:14.319 --> 00:44:16.160
good pod. Maybe we should just end on a

00:44:16.160 --> 00:44:20.000
high before we run out of things to say

00:44:20.000 --> 00:44:20.720
before we start.

00:44:20.720 --> 00:44:22.079
>> You really can't hear any of the stuff.

00:44:22.079 --> 00:44:22.319
What?

00:44:22.319 --> 00:44:26.720
>> Now I just heard one little like like

00:44:26.720 --> 00:44:29.200
somebody fell off. Uh maybe somebody's

00:44:29.200 --> 00:44:33.280
been being kidnapped outside the door.

00:44:33.280 --> 00:44:35.920
We got a rubbish collection.

00:44:35.920 --> 00:44:38.079
We been doing a lot of spring cleaning.

00:44:38.079 --> 00:44:40.160
Well, anyway, I wanted

00:44:40.160 --> 00:44:42.480
>> I don't agree with why you say GraphQL

00:44:42.480 --> 00:44:44.400
all the time. It's not GraphQL, it's

00:44:44.400 --> 00:44:48.319
Graph database queries. It's different.

00:44:48.319 --> 00:44:50.560
>> Um Okay.

00:44:50.560 --> 00:44:53.040
Well, I mean, I'm just thinking

00:44:53.040 --> 00:44:54.319
specifically of the GraphQL

00:44:54.319 --> 00:44:57.040
implementation. And I know there's more

00:44:57.040 --> 00:44:59.200
there's a bigger supererset of these

00:44:59.200 --> 00:45:00.480
sort of stuff but anyway I just

00:45:00.480 --> 00:45:02.240
>> no because because I think it's

00:45:02.240 --> 00:45:04.560
interesting that the problem you face is

00:45:04.560 --> 00:45:06.160
the the way that it's the graph

00:45:06.160 --> 00:45:07.520
engineering or the graph is being

00:45:07.520 --> 00:45:09.920
queried is through graphql

00:45:09.920 --> 00:45:11.680
versus

00:45:11.680 --> 00:45:14.480
um what I was looking at with there's a

00:45:14.480 --> 00:45:16.640
difference between graphql queries where

00:45:16.640 --> 00:45:18.800
you know graphql is from a client to a

00:45:18.800 --> 00:45:21.520
server to fetch data and then define the

00:45:21.520 --> 00:45:25.200
API contract or make sure that the API

00:45:25.200 --> 00:45:28.720
is following the contract and then when

00:45:28.720 --> 00:45:30.800
I talk about Neptune it's a database

00:45:30.800 --> 00:45:32.800
engine and you have a query language to

00:45:32.800 --> 00:45:35.119
query the data in that database with

00:45:35.119 --> 00:45:37.760
deep u data manipulation and

00:45:37.760 --> 00:45:39.839
relationship analysis which is very

00:45:39.839 --> 00:45:42.560
different purpose right one is for a

00:45:42.560 --> 00:45:44.560
front end to render a view another one

00:45:44.560 --> 00:45:49.599
is for um you know a system querying the

00:45:49.599 --> 00:45:51.440
data and I was just saying earlier you

00:45:51.440 --> 00:45:52.800
need to be able to query the data to

00:45:52.800 --> 00:45:55.616
render a view so how is are different.

00:45:55.616 --> 00:45:57.440
[laughter]

00:45:57.440 --> 00:46:01.040
Yeah. And it's cipher and sparkql and

00:46:01.040 --> 00:46:03.520
gremlin.

00:46:03.520 --> 00:46:07.119
Yeah. Well, let's try wrap up here. Like

00:46:07.119 --> 00:46:11.119
I let's try give yourself a challenge of

00:46:11.119 --> 00:46:14.640
writing a benchmark in the AWS bench

00:46:14.640 --> 00:46:15.839
style. [laughter]

00:46:15.839 --> 00:46:19.520
So I can tell you that I had um fable

00:46:19.520 --> 00:46:20.480
>> I think you've already done that,

00:46:20.480 --> 00:46:22.880
haven't you? With your CDK.

00:46:22.880 --> 00:46:25.200
cable has been turnurning on that more

00:46:25.200 --> 00:46:28.640
than a day. Okay. I have no idea why,

00:46:28.640 --> 00:46:31.440
but it has been good.

00:46:31.440 --> 00:46:34.079
>> Yeah. So, basically, I don't your max

00:46:34.079 --> 00:46:36.079
plan has been maxed.

00:46:36.079 --> 00:46:37.440
>> No, not even. [laughter]

00:46:37.440 --> 00:46:40.319
Uh but so basically the full the full

00:46:40.319 --> 00:46:42.240
context is I always postponed creating

00:46:42.240 --> 00:46:43.920
this benchmark because I was always

00:46:43.920 --> 00:46:46.079
afraid of like you know not having

00:46:46.079 --> 00:46:48.720
enough tokens and then I ended up

00:46:48.720 --> 00:46:51.920
without realizing it Wednesday evening.

00:46:51.920 --> 00:46:54.880
Hey you have 18 hours before your um you

00:46:54.880 --> 00:46:56.880
know your weekly budget resets and I had

00:46:56.880 --> 00:47:00.480
like only used 40% of the of the of the

00:47:00.480 --> 00:47:02.400
whole budget. So I said okay now it's

00:47:02.400 --> 00:47:04.400
time to run the benchmark. So, I started

00:47:04.400 --> 00:47:06.880
on Wednesday at 11 p.m.

00:47:06.880 --> 00:47:08.960
>> Yeah. At 11 p.m. I started like, "Okay,

00:47:08.960 --> 00:47:11.920
look at these. I I gave it the AWS bench

00:47:11.920 --> 00:47:16.079
uh repo, the data set repo, and the um

00:47:16.079 --> 00:47:17.920
the the other fork with the flowy

00:47:17.920 --> 00:47:20.400
support and um and some other repos that

00:47:20.400 --> 00:47:22.079
I thought were relevant, but Fable says,

00:47:22.079 --> 00:47:23.920
"No, those are not relevant. Those three

00:47:23.920 --> 00:47:26.000
are really good. I I can use those." It

00:47:26.000 --> 00:47:27.359
asked me a couple of questions for the

00:47:27.359 --> 00:47:29.520
plan and the design, and I said, "Yeah,

00:47:29.520 --> 00:47:31.920
let's go ahead and do this and this."

00:47:31.920 --> 00:47:34.640
And the the way I I prompt fable is

00:47:34.640 --> 00:47:36.960
always you're not doing the work, right?

00:47:36.960 --> 00:47:38.160
Because you're expensive and your

00:47:38.160 --> 00:47:39.920
context needs to stay small. Your

00:47:39.920 --> 00:47:42.319
context I don't tell it that but like

00:47:42.319 --> 00:47:44.400
it's not doing the work, right? It

00:47:44.400 --> 00:47:46.640
always needs to iteratively launch

00:47:46.640 --> 00:47:48.800
dynamic workflows where Sonnet does the

00:47:48.800 --> 00:47:51.119
implementation opens verifies with a

00:47:51.119 --> 00:47:52.640
tight feedback loop and then

00:47:52.640 --> 00:47:54.400
>> you just you just you just tell it to do

00:47:54.400 --> 00:47:55.119
that.

00:47:55.119 --> 00:47:56.880
>> Yeah, I said dynamic. I I don't say it

00:47:56.880 --> 00:47:58.880
with so many words because it knows it

00:47:58.880 --> 00:48:00.400
knows what to do. But I I usually just

00:48:00.400 --> 00:48:03.359
say iterative dynamic workflows

00:48:03.359 --> 00:48:05.280
always override the model otherwise the

00:48:05.280 --> 00:48:06.720
workflow is going to run fable which is

00:48:06.720 --> 00:48:08.560
not what I want. So always override the

00:48:08.560 --> 00:48:08.960
model

00:48:08.960 --> 00:48:11.119
>> that in your cloud MD or you just you

00:48:11.119 --> 00:48:12.000
just buy the

00:48:12.000 --> 00:48:15.359
>> I should have by now right but uh you

00:48:15.359 --> 00:48:18.800
>> so so um I actually re reuse the

00:48:18.800 --> 00:48:20.319
session. I was always the guy who says

00:48:20.319 --> 00:48:22.480
don't compact clear, right? Keep your

00:48:22.480 --> 00:48:24.319
state in in in beats and so on. And now

00:48:24.319 --> 00:48:25.599
I'm the guy like, yeah, I've got a

00:48:25.599 --> 00:48:27.440
longunning fable session that I compact.

00:48:27.440 --> 00:48:28.800
So [laughter]

00:48:28.800 --> 00:48:31.119
yeah, it's dumb. Uh, but yeah, I I

00:48:31.119 --> 00:48:34.240
basically iterative dynamic workflows

00:48:34.240 --> 00:48:36.480
um to

00:48:36.480 --> 00:48:37.839
make sure that Son is the one

00:48:37.839 --> 00:48:40.160
implementing, Opus is the one verifying

00:48:40.160 --> 00:48:42.800
and then when the workflow finishes um

00:48:42.800 --> 00:48:44.000
it Fable is

00:48:44.000 --> 00:48:46.079
>> implementing, Opus is verifying, Fable

00:48:46.079 --> 00:48:47.599
is planning,

00:48:47.599 --> 00:48:50.640
>> Fable is um is basically has the um

00:48:50.640 --> 00:48:53.119
long-term view, right? The vision. So it

00:48:53.119 --> 00:48:54.720
builds the the workflow prompts. This is

00:48:54.720 --> 00:48:56.800
the next slice of what we need to do.

00:48:56.800 --> 00:48:58.559
And Fable said, "Great. We're going to

00:48:58.559 --> 00:49:02.480
go ahead." And it created A B CDE E F G

00:49:02.480 --> 00:49:09.280
E uh sorry, G H. It had H SL H

00:49:09.280 --> 00:49:12.559
slices um to do and and it put that in a

00:49:12.559 --> 00:49:14.880
task list like it keep track of the

00:49:14.880 --> 00:49:17.280
tasks, right? So it has a to-do task

00:49:17.280 --> 00:49:17.680
list.

00:49:17.680 --> 00:49:20.160
>> Slices would be not benchmarks, slices.

00:49:20.160 --> 00:49:22.079
Okay. slices of the whole long-term

00:49:22.079 --> 00:49:24.319
goal, right? The long-term goal. I I had

00:49:24.319 --> 00:49:26.000
speced this out a long time ago about

00:49:26.000 --> 00:49:27.760
what exactly I wanted to benchmark. I

00:49:27.760 --> 00:49:31.359
wanted to prove that an agent in

00:49:31.359 --> 00:49:33.280
different scenarios. And I I basically

00:49:33.280 --> 00:49:35.280
speced this out long time before the AWS

00:49:35.280 --> 00:49:36.559
bench.

00:49:36.559 --> 00:49:38.160
>> Why didn't you just go incrementally?

00:49:38.160 --> 00:49:39.520
Why don't you just come up with one test

00:49:39.520 --> 00:49:41.280
and then build from there? I don't quite

00:49:41.280 --> 00:49:42.800
follow your rationale here.

00:49:42.800 --> 00:49:45.200
>> Yeah. So, I had I had a very detailed

00:49:45.200 --> 00:49:47.680
goal of what I wanted to prove. Again,

00:49:47.680 --> 00:49:49.760
fable is good with goals, right? You

00:49:49.760 --> 00:49:51.599
tell it what you want. It breaks down

00:49:51.599 --> 00:49:52.240
the problem.

00:49:52.240 --> 00:49:54.319
>> Why do you have a golden test? Because

00:49:54.319 --> 00:49:55.920
you sounded like you.

00:49:55.920 --> 00:49:57.839
>> So, so basically what I did a long time

00:49:57.839 --> 00:49:59.440
ago, Kai,

00:49:59.440 --> 00:50:02.319
>> listen for a minute. What I did a long

00:50:02.319 --> 00:50:04.559
time ago was I I I defined exactly what

00:50:04.559 --> 00:50:06.400
I wanted to test, what my goal was, my

00:50:06.400 --> 00:50:08.400
long-term goal. Okay? And then Fable

00:50:08.400 --> 00:50:10.319
does the planning. Fable slices it down.

00:50:10.319 --> 00:50:12.720
Starts with the one test first. Okay? I

00:50:12.720 --> 00:50:14.480
didn't I don't need to tell Fable do one

00:50:14.480 --> 00:50:17.839
test and then, you know, All right? So,

00:50:17.839 --> 00:50:21.119
so basically then I I always postponed

00:50:21.119 --> 00:50:22.720
it. Then AWS Bench came out and I was

00:50:22.720 --> 00:50:24.240
like, okay, now it's, you know, I had,

00:50:24.240 --> 00:50:26.160
you know, 60% of my Fable budget. I have

00:50:26.160 --> 00:50:28.480
to spend it within the next 18 hours. I

00:50:28.480 --> 00:50:30.240
hope, you know, maybe Fable can do it.

00:50:30.240 --> 00:50:33.119
So, um, and this is like a pretty

00:50:33.119 --> 00:50:36.160
substantial plan. It's not like pro or

00:50:36.160 --> 00:50:39.280
it's it's a massive amount cuz I was

00:50:39.280 --> 00:50:43.839
working. Okay. So, from 11:00 p.m. until

00:50:43.839 --> 00:50:47.920
when I I went to sleep and um I always

00:50:47.920 --> 00:50:49.920
tell apparently you need to say to to

00:50:49.920 --> 00:50:52.079
stay caffeinated because caffeine is

00:50:52.079 --> 00:50:54.559
approach is a process that Mac can run

00:50:54.559 --> 00:50:56.880
to to stay awake. So, it launches

00:50:56.880 --> 00:50:59.599
caffeinated. It stays working. Uh, I

00:50:59.599 --> 00:51:01.200
actually put it next to my head when I

00:51:01.200 --> 00:51:03.359
was sleeping. And um, I fell asleep

00:51:03.359 --> 00:51:05.920
while it was working and I woke up and

00:51:05.920 --> 00:51:08.319
it had been turning for like 8 hours and

00:51:08.319 --> 00:51:12.880
it was a slice E B C around D. I think

00:51:12.880 --> 00:51:15.599
it was around four slices in uh, of of

00:51:15.599 --> 00:51:18.160
converting it. And the thing is when it

00:51:18.160 --> 00:51:19.520
writes a dynamic

00:51:19.520 --> 00:51:22.480
>> what do you mean by slice? A slice

00:51:22.480 --> 00:51:23.760
is like a thin slice.

00:51:23.760 --> 00:51:26.160
>> However, Fable decided to break down the

00:51:26.160 --> 00:51:28.800
problem into thin slices.

00:51:28.800 --> 00:51:29.119
Okay.

00:51:29.119 --> 00:51:30.720
>> To reach my end goal,

00:51:30.720 --> 00:51:31.680
>> carry on,

00:51:31.680 --> 00:51:32.480
>> right?

00:51:32.480 --> 00:51:34.559
>> So, so I sent off Fable to do all the

00:51:34.559 --> 00:51:36.400
studies, make the plan. I read it. I

00:51:36.400 --> 00:51:37.839
realized, god damn it, this is a

00:51:37.839 --> 00:51:40.079
complicated uh scenario. This AWS bench

00:51:40.079 --> 00:51:41.280
thing is really complicated. So, I

00:51:41.280 --> 00:51:42.720
started looking at what AWS bench

00:51:42.720 --> 00:51:45.680
actually is. Um, started to understand a

00:51:45.680 --> 00:51:46.880
little bit about what the scenarios are

00:51:46.880 --> 00:51:48.319
in there, the tasks are in there because

00:51:48.319 --> 00:51:50.160
honestly, I first off, I said just I

00:51:50.160 --> 00:51:51.920
want to run a benchmark. Just go figure.

00:51:51.920 --> 00:51:53.920
Um, cuz you got a ton of tokens. Just go

00:51:53.920 --> 00:51:56.640
do it. And and so I I started looking

00:51:56.640 --> 00:51:58.480
into what bench was to try and

00:51:58.480 --> 00:52:00.079
understand the plan. I couldn't really

00:52:00.079 --> 00:52:01.520
understand the plan. So I said just go

00:52:01.520 --> 00:52:04.000
ahead while I I keep reading because I

00:52:04.000 --> 00:52:05.280
was running out of time. I had no more

00:52:05.280 --> 00:52:07.760
time. So So Fable started building.

00:52:07.760 --> 00:52:11.040
>> So you had time before your tokens

00:52:11.040 --> 00:52:12.640
refresh. So you were just like go go go

00:52:12.640 --> 00:52:15.119
go.

00:52:15.119 --> 00:52:17.680
>> So go right go do. And then I woke up

00:52:17.680 --> 00:52:20.240
eight hours later. It had it had like on

00:52:20.240 --> 00:52:21.839
average each dynamic workflow takes

00:52:21.839 --> 00:52:23.839
close to two hours. So I had done about

00:52:23.839 --> 00:52:25.680
four of them. What it does in those

00:52:25.680 --> 00:52:27.359
workflows, it puts a constraint in how

00:52:27.359 --> 00:52:29.599
many loops it can do. So those workflows

00:52:29.599 --> 00:52:32.480
have interesting um um you know concepts

00:52:32.480 --> 00:52:34.400
because they can there can be one son

00:52:34.400 --> 00:52:37.280
implementing then an opus verifies then

00:52:37.280 --> 00:52:39.359
another son needs to fix what opus is

00:52:39.359 --> 00:52:41.599
identified then another opus verifies

00:52:41.599 --> 00:52:43.359
and it can do that up to I don't know

00:52:43.359 --> 00:52:45.280
you can determine how many iterations it

00:52:45.280 --> 00:52:47.760
can do. So, so Fable told me I told them

00:52:47.760 --> 00:52:49.920
max max three times. Like I said, hey,

00:52:49.920 --> 00:52:51.440
why is it not done two hours? It's like,

00:52:51.440 --> 00:52:52.640
oh, we're on we're on the third

00:52:52.640 --> 00:52:53.680
iteration. They're not going to go

00:52:53.680 --> 00:52:56.079
further. Relax. So, [laughter and gasps]

00:52:56.079 --> 00:52:58.480
so, so, so it has this whole this

00:52:58.480 --> 00:53:00.000
dynamic workflow system is quite

00:53:00.000 --> 00:53:02.480
impressive. And then, um, Fable just

00:53:02.480 --> 00:53:04.000
keeps track of the long term and slowly

00:53:04.000 --> 00:53:06.160
the context fills up. I think after four

00:53:06.160 --> 00:53:08.559
after 8 hours it was only at like 30% of

00:53:08.559 --> 00:53:10.240
>> I would love to see how you visualize

00:53:10.240 --> 00:53:12.559
this. How do you know that this is

00:53:12.559 --> 00:53:14.559
actually being carried out as as you

00:53:14.559 --> 00:53:16.720
wish? like you know that so

00:53:16.720 --> 00:53:18.800
>> when usually it only takes 2 hours or 4

00:53:18.800 --> 00:53:20.079
hours and I have the result quite

00:53:20.079 --> 00:53:21.920
quickly okay so when it takes 8 hours

00:53:21.920 --> 00:53:23.359
and you still don't see the result it

00:53:23.359 --> 00:53:24.640
gets frustrating right you don't know

00:53:24.640 --> 00:53:26.079
actually is this going to be exactly

00:53:26.079 --> 00:53:26.880
what I want

00:53:26.880 --> 00:53:28.960
>> but how do you how do you try do you

00:53:28.960 --> 00:53:30.319
actually bother do you see all the

00:53:30.319 --> 00:53:32.640
agents in the claw UI

00:53:32.640 --> 00:53:33.920
>> yeah yeah Claude Code

00:53:33.920 --> 00:53:35.599
>> and you can tell each agent has a

00:53:35.599 --> 00:53:36.880
different model

00:53:36.880 --> 00:53:38.960
>> yeah you you can so basically the moment

00:53:38.960 --> 00:53:40.880
you you you make it launch a dynamic

00:53:40.880 --> 00:53:42.640
workflow it runs in the background and

00:53:42.640 --> 00:53:44.000
the main agent is waiting for the

00:53:44.000 --> 00:53:45.839
workflow So you can go down to that

00:53:45.839 --> 00:53:47.920
workflow. You can see the the different

00:53:47.920 --> 00:53:49.040
phases step one

00:53:49.040 --> 00:53:50.319
>> running right now. Can you share a

00:53:50.319 --> 00:53:54.079
screen? I'm just curious how it look.

00:53:54.079 --> 00:53:56.000
>> This is this is this is two months ago.

00:53:56.000 --> 00:53:57.440
This is uh what what you call this

00:53:57.440 --> 00:53:58.880
again? This is loop engineering. We're

00:53:58.880 --> 00:54:00.640
done with that. We That's solved.

00:54:00.640 --> 00:54:02.000
>> Sorry. [laughter]

00:54:02.000 --> 00:54:03.760
What are we doing today?

00:54:03.760 --> 00:54:06.720
>> Graph engineering.

00:54:06.720 --> 00:54:09.920
>> Uh hold on. I have so many things here.

00:54:09.920 --> 00:54:13.440
Uh it's not this one. So,

00:54:13.440 --> 00:54:14.240
it's

00:54:14.240 --> 00:54:17.359
>> it's funny how we now everyone has a

00:54:17.359 --> 00:54:19.520
million browser tabs and now everyone

00:54:19.520 --> 00:54:21.200
has a million.

00:54:21.200 --> 00:54:23.280
>> I just told you I just told you it's so

00:54:23.280 --> 00:54:25.680
easy for me to get get my session

00:54:25.680 --> 00:54:27.280
because I don't need Herder and then I'm

00:54:27.280 --> 00:54:28.720
spending an hour trying to figure out

00:54:28.720 --> 00:54:30.079
where the session is.

00:54:30.079 --> 00:54:31.680
>> Doesn't even sound like doesn't look

00:54:31.680 --> 00:54:33.599
like you're using CMax. For the love of

00:54:33.599 --> 00:54:36.319
God, this is Semox. It is CMX, but I I

00:54:36.319 --> 00:54:39.760
folded it.

00:54:39.760 --> 00:54:41.119
>> Actually, I don't even know what Cmax

00:54:41.119 --> 00:54:46.160
is. I guess it's a wrap around Ghosty.

00:54:46.160 --> 00:54:48.800
>> So, here it was on on slice G and it

00:54:48.800 --> 00:54:51.200
says slice G fails. This at this point

00:54:51.200 --> 00:54:52.720
it's actually doing live checks against

00:54:52.720 --> 00:54:55.839
my AWS orc and that prompts me. So, when

00:54:55.839 --> 00:54:59.520
I'm not at the computer, I don't nerve.

00:54:59.520 --> 00:55:01.200
>> Yeah. So, I can show you

00:55:01.200 --> 00:55:04.400
>> worried about your bill, your AWS bill.

00:55:04.400 --> 00:55:06.319
So here are all the workflows that ran

00:55:06.319 --> 00:55:08.079
right and you can see this one was 10

00:55:08.079 --> 00:55:10.240
agents. So this is what it looks like.

00:55:10.240 --> 00:55:12.240
This is what while it looks like while

00:55:12.240 --> 00:55:14.400
it's running two except these ones are

00:55:14.400 --> 00:55:16.319
all completed. So I can see that there

00:55:16.319 --> 00:55:18.880
were uh you see this is what what annoys

00:55:18.880 --> 00:55:21.680
me. I have to enter the

00:55:21.680 --> 00:55:23.760
the keychain password and I'm not this

00:55:23.760 --> 00:55:25.599
is my AWS or account so I don't want to

00:55:25.599 --> 00:55:27.520
give it like full access to it. So even

00:55:27.520 --> 00:55:28.640
though I'm not checking the prompt so

00:55:28.640 --> 00:55:30.400
it's a bit ridiculous, right? I'm just

00:55:30.400 --> 00:55:31.680
typing in the password.

00:55:31.680 --> 00:55:33.440
>> You got balls of steel. Well, that's all

00:55:33.440 --> 00:55:34.640
I can say.

00:55:34.640 --> 00:55:37.200
>> Yeah. So, so this is um this is the

00:55:37.200 --> 00:55:39.280
context that Fable designed for this

00:55:39.280 --> 00:55:41.440
first part of the first phase. So, we

00:55:41.440 --> 00:55:42.960
have the phases on the left which is

00:55:42.960 --> 00:55:45.359
scaffold phase and the first task which

00:55:45.359 --> 00:55:48.240
is um scaffold task and um

00:55:48.240 --> 00:55:50.400
>> give this a try. This is this is

00:55:50.400 --> 00:55:50.880
dynamic.

00:55:50.880 --> 00:55:52.880
>> If you never run dynamic workflows,

00:55:52.880 --> 00:55:54.880
you're you're really missing out.

00:55:54.880 --> 00:55:56.720
>> I'm a noob. I'm a total noob.

00:55:56.720 --> 00:55:58.640
>> Didn't say that. But this was the very

00:55:58.640 --> 00:56:00.720
first slice as fault. So this is like

00:56:00.720 --> 00:56:03.599
look at the AWS bench uh layout and re

00:56:03.599 --> 00:56:05.440
recreate it from scratch you know for

00:56:05.440 --> 00:56:09.200
our purpose right. So that was son then

00:56:09.200 --> 00:56:11.440
four more Sonnet agents then here we have

00:56:11.440 --> 00:56:14.079
opus that I guess in parallel verified

00:56:14.079 --> 00:56:16.559
four different layers and then reverify.

00:56:16.559 --> 00:56:18.880
So the reverify means that it went back

00:56:18.880 --> 00:56:20.960
to one of the build agents. I I oh it

00:56:20.960 --> 00:56:23.359
went back to a fix. So here's a fix

00:56:23.359 --> 00:56:25.520
phase round one and then it went back to

00:56:25.520 --> 00:56:27.760
verify and it it's it another opus round

00:56:27.760 --> 00:56:29.440
went off to verify. Now, what I like

00:56:29.440 --> 00:56:31.520
about this is that you see these agents

00:56:31.520 --> 00:56:33.680
never reach a lot of like context

00:56:33.680 --> 00:56:35.200
windows, right? They're they're close to

00:56:35.200 --> 00:56:38.400
200 context uh 200,000 each, which is

00:56:38.400 --> 00:56:40.480
only 20% of the window, right? Which

00:56:40.480 --> 00:56:42.400
makes them very focused and very

00:56:42.400 --> 00:56:45.040
shortlived. So, all of the context

00:56:45.040 --> 00:56:46.400
engineering that you had to do about

00:56:46.400 --> 00:56:48.000
like I need to clear my sessions, I need

00:56:48.000 --> 00:56:49.839
to do the checkpoints, forget about it,

00:56:49.839 --> 00:56:52.400
you know, all taken care of right here.

00:56:52.400 --> 00:56:52.960
>> Okay.

00:56:52.960 --> 00:56:54.799
>> Okay. I got to try dynamic workflows

00:56:54.799 --> 00:56:56.240
now, man. I feel like I'm missing a

00:56:56.240 --> 00:56:57.280
trick.

00:56:57.280 --> 00:57:00.000
>> Okay. So fable. So here I I triggered it

00:57:00.000 --> 00:57:03.920
on a 35% that's 350,000 tokens. So I

00:57:03.920 --> 00:57:07.920
send off a stupid like re relaunch G on

00:57:07.920 --> 00:57:11.119
top of 350,000 tokens. So the price of

00:57:11.119 --> 00:57:12.480
this one sentence is actually quite

00:57:12.480 --> 00:57:15.359
expensive. Um that's where you maybe

00:57:15.359 --> 00:57:18.240
want to like compact uh for Fable to you

00:57:18.240 --> 00:57:20.640
know orchestrate a little bit. So yeah.

00:57:20.640 --> 00:57:20.960
So,

00:57:20.960 --> 00:57:23.119
>> speaking of expenses, I don't see dollar

00:57:23.119 --> 00:57:26.960
signs and I don't see your How do you

00:57:26.960 --> 00:57:29.440
keep track of your your budget? Again,

00:57:29.440 --> 00:57:31.839
>> I had this Codexual.

00:57:31.839 --> 00:57:34.000
>> I had I had like a I had a Codex bar

00:57:34.000 --> 00:57:36.079
here that that that keep track of the

00:57:36.079 --> 00:57:39.920
you know the it has markers. It tells me

00:57:39.920 --> 00:57:42.079
>> in your 5 hour session if you're ahead

00:57:42.079 --> 00:57:43.760
of the marker that means you're going to

00:57:43.760 --> 00:57:45.680
exhaust before the session finish. And

00:57:45.680 --> 00:57:47.760
then also in your weekly budget, if

00:57:47.760 --> 00:57:49.200
you're ahead, you're going to run out

00:57:49.200 --> 00:57:51.040
before the week is done. So here it has

00:57:51.040 --> 00:57:52.799
launched

00:57:52.799 --> 00:57:54.400
>> it has just relaunched it right here.

00:57:54.400 --> 00:57:56.720
Right. So you can see the it has created

00:57:56.720 --> 00:57:58.640
the phases. And now you can see that the

00:57:58.640 --> 00:58:01.280
Sonnet one is running for 18 seconds. And

00:58:01.280 --> 00:58:03.920
if I tap on it, I don't know something's

00:58:03.920 --> 00:58:06.160
wrong with this. Oh, did I zoom in? I

00:58:06.160 --> 00:58:07.839
think because I zoomed in too much.

00:58:07.839 --> 00:58:10.319
Normally you can see the whole context

00:58:10.319 --> 00:58:12.880
that been launched and then you also see

00:58:12.880 --> 00:58:14.640
what's what's it doing at that time. So,

00:58:14.640 --> 00:58:17.520
activity, it's doing 11 tool calls.

00:58:17.520 --> 00:58:18.880
>> This is really exciting. I've got to

00:58:18.880 --> 00:58:20.880
give dynamic workflows a try. Never mind

00:58:20.880 --> 00:58:23.680
bloody AWS bench. I'm giving AWS

00:58:23.680 --> 00:58:25.520
workflows a try.

00:58:25.520 --> 00:58:28.079
>> If anyone's using workflows,

00:58:28.079 --> 00:58:30.000
>> if anyone's using AWS workflow, I mean,

00:58:30.000 --> 00:58:31.760
not

00:58:31.760 --> 00:58:33.440
sorry. If anyone's using clawed

00:58:33.440 --> 00:58:36.000
workflows, please comment below. Please

00:58:36.000 --> 00:58:39.359
like the video regardless

00:58:39.359 --> 00:58:42.720
on please leave us five star reviews on

00:58:42.720 --> 00:58:45.920
all the podcast platforms.

00:58:45.920 --> 00:58:48.079
Please send us some love so we can

00:58:48.079 --> 00:58:50.720
>> read this blog post.

00:58:50.720 --> 00:58:53.680
>> This blog post and Kai disregarded it

00:58:53.680 --> 00:58:56.799
because it was too much words.

00:58:56.799 --> 00:58:58.400
>> Words.

00:58:58.400 --> 00:59:02.240
What? Did you fix the summary?

00:59:02.240 --> 00:59:04.079
>> I know you you will still complain. I

00:59:04.079 --> 00:59:07.359
will not show it. So um so this is this

00:59:07.359 --> 00:59:10.319
is a reusable dynamic workflow. So I

00:59:10.319 --> 00:59:12.079
asked Fable to build a dynamic workflow

00:59:12.079 --> 00:59:14.400
that I could rerun when Fable was no

00:59:14.400 --> 00:59:15.839
longer available because I assumed that

00:59:15.839 --> 00:59:17.760
Fable would not be available or would be

00:59:17.760 --> 00:59:19.680
too expensive. So I asked it to build a

00:59:19.680 --> 00:59:22.079
workflow that I could ask Opus to run

00:59:22.079 --> 00:59:24.319
and so Fable was iterating and in this

00:59:24.319 --> 00:59:26.000
blog post I think the most interesting

00:59:26.000 --> 00:59:28.799
part is that each time I was running the

00:59:28.799 --> 00:59:31.839
workflow I was reducing uh Fable was

00:59:31.839 --> 00:59:33.599
optimizing it based on what it saw the

00:59:33.599 --> 00:59:35.839
agents were having troubles with. So

00:59:35.839 --> 00:59:37.280
each time it became less

00:59:37.280 --> 00:59:38.640
>> this is like this is like prompt

00:59:38.640 --> 00:59:40.079
engineering like where you

00:59:40.079 --> 00:59:41.520
>> this is loop engineering right

00:59:41.520 --> 00:59:44.480
>> loop engineering when you fix your your

00:59:44.480 --> 00:59:47.040
>> so it it it does this this blog post

00:59:47.040 --> 00:59:49.520
basically looks back at how fable

00:59:49.520 --> 00:59:52.000
iterated on the dynamic workflow to make

00:59:52.000 --> 00:59:55.200
it less token uh expensive to make it

00:59:55.200 --> 00:59:56.640
faster and more accurate so that the

00:59:56.640 --> 00:59:58.000
agent wouldn't have errors. So it was

00:59:58.000 --> 00:59:59.920
tuning the prompts, it was tuning the

00:59:59.920 --> 01:00:01.359
phases, it was tuning the way that the

01:00:01.359 --> 01:00:03.920
the work goes into the agents and and it

01:00:03.920 --> 01:00:06.720
and then highlighted that over time it

01:00:06.720 --> 01:00:09.520
went down 20% in token usage. Uh the

01:00:09.520 --> 01:00:11.280
number of agents were reduced, the

01:00:11.280 --> 01:00:13.440
rework was reduced, the total duration

01:00:13.440 --> 01:00:16.240
was reduced. This run four and five are

01:00:16.240 --> 01:00:18.799
two runs on the same task. So I rerun it

01:00:18.799 --> 01:00:20.319
on the same task just to get a

01:00:20.319 --> 01:00:23.280
improvement uh result. So this blog post

01:00:23.280 --> 01:00:25.359
is maybe very dense because I did ask

01:00:25.359 --> 01:00:27.599
Opus to like look at everything and then

01:00:27.599 --> 01:00:30.640
make help me summarize it but it it has

01:00:30.640 --> 01:00:34.559
some very interesting um you know like

01:00:34.559 --> 01:00:36.880
what generalizes. So Opus came up with a

01:00:36.880 --> 01:00:39.760
couple of uh rules but honestly this

01:00:39.760 --> 01:00:42.400
probably not very useful. So but what is

01:00:42.400 --> 01:00:44.480
interesting is the methodology you know

01:00:44.480 --> 01:00:45.280
that's kind of like

01:00:45.280 --> 01:00:47.599
>> this is fascinating. Okay listen I got a

01:00:47.599 --> 01:00:50.240
meeting now got to run. Okay, bye.

01:00:50.240 --> 01:00:52.240
>> Anyway, thanks. Thanks again. Uh, I

01:00:52.240 --> 01:00:53.680
think this is a great part. I really

01:00:53.680 --> 01:00:55.280
enjoyed this one. So, thanks. Thanks.

01:00:55.280 --> 01:00:56.480
Uh,

01:00:56.480 --> 01:00:56.880
>> all right.

01:00:56.880 --> 01:00:59.599
>> Thanks, Vincent. Bye. Thanks. Bye. Have

01:00:59.599 --> 01:01:02.240
a great day.