Episode 40: Benchmarking AI infrastructure

Hey, good morning Vincent. So you wanted
to share a CDK terrain release
>> and earlier you did like real of a raft
of features
>> and I still didn't quite understand what
it meant. So what does it what does it
mean to a layman person in the field of
infrastructure like like our audience?
Okay, so I don't know where to start now
and without being like repetitive, but
basically Terraform allows you to create
resources against APIs. Um, and those
can be cloud providers such as AWS or
Google Cloud or Azure. uh or it could be
your GitHub API or it could be you know
to create a repository or it could be
your data dog to create a a monitor or
it could be your snowflake
>> cloud AI management instance in my case
>> no not that
>> oh there's no provider for the claw AI
okay carry on
>> yeah so but there is one right I I I
shared there some people made an
announcement about official not official
uh like their you did share Right.
>> So, so that's the great thing about
about Terraform is that even if there is
no pro official provider um anyone can
build one because the the execution
engine and the providers are
communicating with each other through a
protocol. It's called a plug-in system.
Is is that the best way to go forward?
Because I we did talk about this before.
I I created my own provider for
Snowflake and I found it really hard.
the snowflake uh missed some features.
>> I found it really really hard. Uh
so
>> there's several versions actually.
>> Would you suggest like if you were to
infrastructure as code something is
terraform provider still the your best
gateway drug into into nailing down your
infrastructure?
I think it comes down to what is the
most like
accepted way or like what's the most
integrated with with your existing, you
know, if you if you're if you're the
platform team or if you're you're a a
product team person and you need to get
buy in from your platform team, it will
really depend on what they're using,
right? Because if they are full on in
Terraform and all of their pipeline
systems and everything is is is are
tweaked towards that then you know
building a uh a little provider for your
use case will probably be better. If
they're fully on cloud formation then
you can use uh their version of custom
resources. So the ability to define um
your own
basically what these things do right
they create resources and then they
manage them. So they need create, read,
update, and delete capabilities. So when
you create a custom resour outside AWS,
is Terraform provider your best bet then
cuz it's got the most.
>> So what are the other options? Right.
There's a couple of new players like um
for May. I don't even know. I don't
think it's it's cheese.
>> Are they still around? I thought they
released and then disappeared. Or may
Yeah. No, I think there's still they had
initially I'm very skeptic um of any
new framework being launched because
you're you're in coming to an ecosystem
that has been around for almost 10 years
now and there's a lot of tooling around
the existing frameworks. So there's a
lot of tooling around terapform. Um
there's a lot of tooling around cloud
formation.
And um being a new kid on the block, you
um I I'm I'm 100% like in general all of
these new frameworks, they believe they
can do it better, right? They see the
problems of these existing frameworks.
Um for example, there's also Chant. C H
A N T.
>> I really like that they use Pickle.
Yeah. Uh Shant.
>> Yeah.
But that one is going a little bit. No.
No. C H A N T like chanting.
Chant.
Yeah. Intuent. Intentious. Intensious.
Alex Artigz is quite active on uh on
LinkedIn. Um usually responding to
general threads. There's also
Brian Grant and Alexis Richardson. they
created a startup called Config Hub uh
and and crossplane they're focused on
like using controllers right constantly
reconciling um your a your uh resources
against um a reconciliation loop so
basically the Kubernetes way so chant
honestly it's a bit over my head a lot
of people are a little bit confused
um because it's quite
I don't know it's hard to grasp and
his original comparison to existing
frameworks was inaccurate. Um, so I told
him that and then he actually went and
updated it. So now it's a little bit
more accurate, but I still don't get why
what he's building. They say it's it's
more of like configuration as data. It's
kind of the same thing as what uh Config
Hub does. Uh anyway, you ask me like is
that still the best way to do your
infrastructure? To be honest, most of
the organizations I talk to and work
with, they're either on cloud form or or
uh or Terraform. Maybe I know very few
companies on Pulumi and then there's
this all new kits on the block like uh
you know crossplane you know based
Kubernetes controllers config hub is a
little bit similar but they they take
some of the ideas of what they've
learned managing and defining GitHubs
and and customize because Brian Grant he
used to be at the Kubernetes uh team at
Google and he he's been trying to solve
this problem for like over 20 years like
even within Google doing it for Borg.
>> He he saw some patterns and he sees some
problems. And so what they really do
what they what they think Conf is
interesting because they really see like
we end up building these complicated
systems of just generating configuration
and then if you have to make one field
change you have to push it through this
whole pipeline of changes that need to
then ultimately end up in like one or
two fields to to hit production. So they
just shortcut that whole loop by just
you know allowing you to define a
function that you run against your data
against your configuration and then you
can just trigger that function on your
staging environment and if it's all good
propagate it and change it onto
production. This reminds me of Q CLI
because is it QCLI? I think it's also um
X it comes out of the Kubernetes.
Oh no, Q lang you. I think Q lang the
whole principle. Okay. A
one of the main guys is ex Kubernetes
Google team and B the idea is that
there's a language and would like
functions to make changes in your
infrastructure safer and more
controlled. I mean that's the way I
understood it probably.
>> Yeah. I mean there's a lot of like
languages to manage configuration and
basically QANG is focused on how do you
manage and manipulate and mutate your
configuration across the environments.
It's not the same as um as your um you
know software code and I think they're
all very interesting but not a lot of
them gain widespread adoption. Right. So
>> true, true.
>> The what and and there's so many like
it's like the famous XK CD comic where
it's like there's too many standards.
There's like 11 standards. So I'm going
to make another I made a new one. Now we
have 12 standards to try and get rid of
those 11 standards.
>> Yeah. Yeah.
>> That's so common. Actually the the other
comic I feel that needs to be written is
it's probably to do with migrations like
>> if you work with in any big company when
when they're trying to do a migration it
very most of the time they never
completely managed to migrate for one
reason or another. So now you have uh
you know X plus one systems purely
because you couldn't retire the older
one. Does that make sense?
>> Yeah. have the same experience. Like
when you join an organization that's
been around for a while, you're going to
see five or six generations of how they
used to do things and how like the newer
project has been created with a new way
of doing things. The old project has
been like some of the very actively
maintained projects have been migrated,
but those that are less maintained stay
on like older versions of however was
things being done and um and never get
ported. I was I was
I was talking because I was on a call
like every week checking on them on the
status and then somebody asked like when
are we finally going to be done and I
basically told him we're never done.
That's that's infrastructure and
operations basically. You're always
going to have uh something.
Yeah.
>> Yeah. It's kind of it's kind of sad and
like and and then maybe turning back to
AI like does AI solve any of these
problems cuz I feel like they don't in a
way because there's lots of new AI
initiatives and moving the data from uh
the old system to the new system doesn't
seem to be any better solve with AI at
least
>> at least I mean this is my own anecdotal
experience I I mean I I wasn't in these
projects, but I just feel like okay, now
we have these new AI projects and
they're also just plus one plus one plus
one. So that's this is where I think
there's two things I want to say about
that. one um about the flexibility of
the of the the of Terraform and its
capability of like refactoring rem like
when you need to do migrations you need
to redefine the ownership of certain
resources. Sometimes you realize that
something has been created by a team
then gets reused by another team and
then actually the responsibility of
maintaining it needs to be shared needs
to move to a separate layer. So this
type of like migrations um are I feel
very well supported in Terafhone. Maybe
it's a little bit of uh how you say when
a hostage falls in love with it's um uh
>> Stockholm.
>> Yeah, it's a bit of Stockholm uh that
I'm like happy with what Terafhone gives
me. And there's probably people that
will heavily disagree like no actually
there's a lot much better way to do
this. But anyway, so so that's where I
think with AI agents, they can also help
you do that much faster. Like something
that would have taken a long time, um,
agents can like diligently keep
hammering at it and slowly chip away and
and help push through the change. It's
very funny because I have couple of
cloud sessions on my laptop and I tell
it my you know my goals and I ask it to
create tasks and to arm monitors and um
and then for example today I wanted to
release the the CDK terrain version 0.24
in the morning last night actually but
then there was a GitHub outage so that
was blocked and I went to sleep.
Apparently, there was a 7-hour GitHub
outage. Thanks. Thank Thankfully, I went
to sleep.
>> And when I woke up,
>> yeah, when I woke up, uh, I said, "Okay,
looks like I saw the posts about the
seven 7-hour outage and I was like,
well, looks like everything's healthy
now." So, I clicked the the release
button in in GitHub and I went for
breakfast because it takes like 30
minutes to for it to go through all of
the unit tests and integration tests.
And I totally forgot that I had a cloud
session running on my laptop that has a
monitor armed. And that monitor was
watching when the 0.24 release was
hitting npmgs and python and and all the
other package registries. So while I was
having breakfast, I started getting
notifications about um you know
providers getting bumped to the latest
version. And I was like, "Wow, that's
pretty cool. I didn't know that was
automated." And then when I came back to
my desk, I realized that the cloud
session that was running had an armed
monitor and just kicked back to life
like, "Oh, the release is out. Let me go
and update all the providers." And it
started like creating pull requests
across repos. And I was like, "That's
crazy cuz I was just I was just having
breakfast." But but that just
illustrates a point of having like AI
push things through, right? I mean, it's
easier to to define a goal and and then
let it like push things through. And it
was pretty secure because every single
thing gets super validated in this case.
>> Your comment about own your comment
about ownership is quite key. I feel
because in many organizations I feel
the organ the ownership is distributed.
So like uh for example
like maybe there's a a production
release team, maybe there's like an octa
or IM team and um I I mean I'm creating
a hypothetical situation here, but I'm
I'm sure you'll understand that like you
are in in control of CDK terrain, but in
a in a major, you know, big companies
uh product thing, you have to coordinate
with human teams and that's where things
quickly become awkward and uh and and of
course AI doesn't really help there
right
>> it does it does help
>> because like because of like my
experience in the last week when I have
to coordinate I have a pull request that
has to be reviewed for the the CDK
terrain release and um literally have
Claude armed to to wait until the PR is
ready and then claw just wait.
>> Okay. I see. I see. I see what you mean.
Like in your example,
>> you
pro you automated and and and uh the AI
is just waiting for something waiting
for something to happen and then it
triggers something. I suppose that's
that is quite helpful. I suppose
>> I I I'm at a point and I think there's a
lot of people that re that have reached
that point where they have so many
sessions running on the machine and
>> be confused to like what the hell's
going on.
>> I do I do I use CMX? It's a lip ghosty
term.
>> You heard heard is the is the Kool-Aid
>> now. Now they're super logical from
Hashi Hashimoto who's who's going to
rebuild T-Max but like for a ghosty uh
native you know protocol so much faster
rendering and
>> so heard is not really so much a
terminal multiplexer but it helps keep
your AI sessions in in one place.
>> So so CMAX does the same right? I mean I
found something that works. I haven't
really had a um you know I just saw
someone announce his own version of of
it on on LinkedIn. there's hundreds of
them. The same with like context
engineering uh solutions uh spec driven
solutions. Everyone can make their own
so easily with with with uh with AI. So,
so but to come back to the point of like
coordination and also ownership like I
think this is one of the biggest uh
mistakes like I work in a in a very
restrictive um you know company right
now and there's like a central team that
tries to own everything and it is
creating so many hurdles in in
coordination issues um where and in this
case everyone is being blocked by by one
team um and maybe they are able to to do
their things. I saw on on a on an annual
on a quarter review uh slide they showed
the diagram with the throughput of every
team and the true of that one team that
everyone belongs depends on is massive
they're like double or triple from
everyone else and then I'm like yeah but
why do you think that is cuz everyone
else is waiting for them to be honest
you could say they're doing such a great
job they're doing four times what
everyone else is doing
>> interesting how you can uh you can spin
it like that Yeah, you
one graph shows you the throughput which
is through the roof and the other and
then and then if you're a bit more
careful you show that like everyone's
blocked by this team. I mean I was
asking you know when they show this on
the screen I was asking um in direct
message to someone and I was saying can
we can we look that from a different
angle? Yeah,
>> I don't want to say it publicly here,
but
>> yeah, that's that's the trouble. A lot a
lot of these like big company um
announcements like when they go like,
>> "Hey, our team has done so much better
in the town hall, they have this like
>> but nobody nobody said the team is doing
the great job." it just they were at the
top and and I was just thinking I don't
think that's very
>> I just feel like I feel like
communication I I guess I don't think
communication sold in any company really
but
>> but sometimes
>> for many reasons right
>> yeah for many reasons but like sometimes
you have a team that like wants to show
the impact because everyone's like has
to show business value has to show
impact
>> and then they they say something which
is kind of ridiculous
or doesn't is missing some key context
and there's just no way to sort of
contribute in a in a nice way to say
that h actually the way that you said
that isn't quite correct because
um you know it's just too late by then
or something and everyone's having to
absorb what one person's point of view
basically. Yeah. So that was the first
thing when we talked about like the the
coordination issues and about
infrastructure orchestration. Uh but
then the second thing was you were
asking if AI can be really useful in
those cases of how do you know that AI
is doing a good job and um AWS made a
repository public called the AWS bench
AWS-bench
on GitHub
>> and I had a play with it because I have
a hypothesis and I thought you know I
want to I want to put some numbers
behind my claims and um while Fable was
trying to repurposed their benchmarking
solution to prove my hypothesis or
disprove my hypothesis. Hopefully not
disprove, but um I was actually having a
look at the number of scenarios that are
in there. Um I mean I don't know if I I
could share.
>> Yeah. Yeah. Share your screen. I haven't
actually jumped into the source code,
but I do think AWS Bench is a fantastic
idea because like I've been evaluating
my models on the on Simon Willis's
Pelican up until now, which is
ridiculous.
>> Which is ridiculous because I need a
model to do infrastructure stuff. I
don't need it to draw freaking Pelican.
>> Yeah, this fact's only there. Okay, so
AWS Bench uh new or on GitHub. I don't
know if you can still hear me if I go so
far. So, um, what's interesting is the
>> Sorry,
>> I will I will zoom in.
>> Yeah, let's choose something that maybe
we all understand like let's go and No,
let me just find them again first. So,
wait, I'm in the wrong repo. I think I
need to go to the data sets. That's
where I think the scenarios are. So,
then I think it's under tasks. Yeah. So
they have these categories and what's
really interesting is um when I started
running it on my account actually um it
needs AWS or it needs an AWS or
management account and then it uses a um
an OU operating unit and you you then
have an anchor AWS account and it finds
the AWS accounts to test in. Normally it
will mint a fresh AWS account or reuse
an AWS account just to run the
benchmark. Okay. So just to run the
benchmark it will stand up an isolated
AWS account deploy an SCP so that the
agent is completely isolated to work in
a single region. That's how how it's
yeah it's pretty cool
>> and then it time I kind of wish it was
not requiring live cloud resources. cuz
I wish you could almost run it offline.
>> There is there is a version.
>> Okay, let's not go down this offline
rabbit hole. So just just show just show
one of these these um these test
>> scenarios. Yeah, so there are which one
was I looking at? I was looking
like for example diagnose ALB. So, I
think before it runs,
um
I'm a little bit lost now. I I remember
there being basically an infrastructure
like the a CDK um stack that gets
deployed into the AWS account uh before
the agent gets unleashed and then a
bunch of um questions a bunch a bunch of
uh
troubleshooting scenarios.
Okay, so this is the the actual CDK app
with setting up the environment that so
the scenario sets up the environment for
the troubleshooting scenario tunnel.
>> Yeah. So no, that's not very
interesting. The task So the task run
inside those scenarios. So I think the
scenarios set up the AWS environment,
you know, deploy a bunch of resources
like an ALB, an API gateway and so on.
And then it provisions
>> there's a test uh folder down there if
you saw that.
>> This is the the this is it. This is it.
The instruction one. Um so this is what
the so it provisions and harness it can
be Claude Code it can be kiru cli it runs
inside the docker container completely
isolated you can decide what is the
environment that the container gets it
can have skills or whatever and then you
give it this is an instruction for this
particular task in that particular
scenario my ALB so that gets injected
started running 5xx what's going on okay
so all of these you know launch an agent
with the tools with a certain set of
tools can be an MCP can be skills insert
inside a sandbox environment against a
real AWS account with real AWS resources
and then a simple prompt and then when
it the AI is finished it will then u you
know be judged by by by its performance
uh against that task and you can see
that the number of of scenarios are are
that the number of scenarios and the
number of tasks against those scenarios
are are massive really really big right
Um, it's significant. Very interesting.
>> It's amazing though at the same time a
little bit scary because like I've sort
of prided myself on my troubleshooting
abilities
over the years and pride myself
troubleshooting skills and now this is
kind of scary because it's automating my
troubleshooting skills away.
Uh yeah, I think as a platform engineer,
if you are not focused on
finding a way to leverage this type of
um you know agents and providing these
as an offer as a service for for for
your um for your product teams, right?
Because ultimately what is a platform
engineer? We are responsible for making
it easy for products to be delivered to
production and iterate. So however
that's done like it's the fact that we
were doing infras that was just because
that's the way was what we needed to do
today product engineers they can write
the infra all on their own you know they
can create and define it so actually
platform engineers always are supposed
to just be focused on guardrails and
discoverability and observability
>> guard rails and discoverability yeah the
the critical thing here
is that I hope this AWS bench can be
extended like for example
an I mean they they AWS has a whole
bunch that AWS bench has a whole bunch
of scenarios but but these are not the
exact scenarios for uh you know an
insurance company or a financial company
or a whatever company they we people
need to basically contribute or have
their own internal scenarios here so
that uh agents can be tested and
validated that they it can do certain
day-to-day work in that particular
uh business environment. So I I I really
hope that AWS Bench I I'm going to study
AWS Bench but like you you get the point
right that you need to to have your own
benchmark for your own company.
>> Yes. And and it's also something that
I've been um because what why did I look
into AWS bench is because I've had this
idea and this hypothesis that um if you
use a higher level language uh like the
cloud development kit uh allows you to
build um agents can be more efficient.
You spend less tokens. They will be
faster. You spend less time. they will
be more accurate and the result will be
more reliable because you will have hit
less uh issues as as you are maintaining
it in like long-term deops too. So
that's why I wanted AWS batch because I
want to put numbers behind that claim. I
want to actually evaluate an agent with
raw terapform.
>> That's very cool.
>> An agent with uh raw terraform and some
skills, an agent with terraform modules
and some skills versus uh an agent with
AWS CDK. um you know which one is
creating a more maintainable solution
that can be um against
>> though I mean can I assume that that
I've got to look at these uh scenarios
and I've got to look at the the way the
benchmarks work but like in mo in many
cases at least on AWS that a lot of
these
the chief operating
mechanism is AWS CLI right
that's that's the chief
that that's what the model is expected
to do, right? to basically or is is the
model writing typescript?
I so so this whole thing from AWS is
built upon an existing framework from uh
terminal bench uh which is I forgot the
name for the framework now but um it's
an existing bench benchmarking framework
so it can be used to benchmark terminal
you know proficiency and in this case
they modified it to to to benchmark um
troubleshooting capabilities against
live AWS infrastructure to see if if the
you know where they can optimize the
agents either with skills Um so this
benchmark can be for for anything right
in my case I'm giving it the task to
create infrastructure and I've also told
fable my first you know set of of
scenarios are focused on creating infra
my second set of scenarios is mutating
infra day two ops right we have
something we need to modify it and then
we need to make sure that uh because
sometimes you can create infra but then
when you have to change it it becomes
very painful so I want I want to
validate that as well Right. So, um
those are the scenarios and the tasks
that I am I'm looking at. Um the guy
from Chant that we talked about earlier,
he he built this version
uh from you know and what he did is he
replaced the AWS uh requirements with
flowy. So flossy I don't know how you
pronounce it but it's a local stack
alternative like local stack went they
added the license requirement and flossy
what they call
>> is brilliant because that's that's that
plays into that offline comment earlier.
>> Yes. So he built uh he has his own
container. Um so so you have to build
the flocky container with all the tools
in it and then he runs the agents in
there because he wants to prove that the
agents are able to do these scenarios
and these task better with um highle
solution that is his solution basically
me and him where are we talking a lot uh
and and
>> it's brand
>> and and he I did not do the flowy or foc
I I um I was afraid that I would hit um
limitations of the local stack. And I I
thought you know let let's just do it
with the real AWS
for uh for you know first first draft
and then look at um quick uh sandbox
like local iterations to improve upon
that because if I know it works in AWS
and then I I I put FL Faky in it and it
starts failing then I know that it's
Faky and not not uh not my my other
stuff. Right.
>> Yeah. This is really cool. This is
really cool. the the
thing I wanted to talk to you about um
is still I think a void for agents I'm
sorry to bring it up again but like in
many companies you use something like
octo or zero whatever and you have
different vendors right you have your
page of duty you have your slack you
have your atlassian you have your uh I
don't know you have a whole bunch of
them the the thing that I'm struggling
with is uh
like say say um say you want your agent
or something like that your agent
identity to to access these various
services in most cases you're going to
go for a long live credential right I'm
just no you can do everything with OIDC
you can do you can do everything with
the MC I mean some some things don't
have MCPS
some things don't ever
>> I'm actually having that problem at work
now as well. Um so I think
I think you need to have a very like
mature organization that have proper
onboarding and offboarding of of of your
employees and single sign on setup and
integrated across uh all of your your
systems.
um you will I think you're right that if
you're talking about agents and and um
basically
thing things that do not have this um
device off flow or like that need to
have a token and then refresh the token
like with OIDC you have these uh the
ability to to authenticate it give it a
token and then a refresh token to keep
refreshing the
>> I mean it's also the oorthth flow does
that right I mean and I I'm always a
little bit confused I think of oid DC is
like a like a trust relationship and
then I think of oorthth as the one that
with the the token and and the refresh
rate. But
>> yeah, you're right. OIDC is a wrapper
around too.
>> Yeah.
>> Um and that's maybe where I'm like
prompting the AI and guiding it wrong.
Uh that I should probably be using
because we just had a discussion about
that. We had this problem with our
agents now is like how do we give them
access um to tools and how do we make
sure that they don't have lawyers
>> MCPS is is the right path but there's
many cases when there when there isn't
the um
>> well like for example like say you want
a service account or something then
things get weird cuz cuz yeah when you
run an agent on someone's computer when
you're when you're authenticated like in
my Kai Hendry account then I I think
things become a lot easier but like when
you want to do machine to machine stuff
things get weird very quickly and anyway
I'm just trying to say that like I feel
like am I'm well this is I'm just
winging through this problem right now a
lot of am identity and access management
stuff seems to be the bottleneck in uh
in this particular client I'm working
with right now
I am it's hard it's really really hard.
Well, when you run, in our case, we're
looking at agent core that's integrated.
They get an IM role and they have the,
you know, we can use agent core identity
with the credential provider
>> identity. What is agent? Oh, Bedrock
agent core. Yeah, sorry.
>> No, it's not.
>> So, Bedrock, they launched agents and
then they already retired it. So yes,
it's called bedrock agent core, but the
messaging from AWS around agent core is
very weird. Like on one side you have
bedrock which is all about like models,
hosting, invoking um you know training,
tuning
>> and then you have agent core which is
all about making agent production ready
um being able to run agents um with all
of the requirements around it like in
terms of obs observability
um access to MCP tools um identity
>> yeah eval
observability
prompt injection detection or prevention
rather
>> and agent core is like it's like your
ECS for containers you have a container
you want to run it okay you need to have
rollout capabilities you need to have
you know the ability to do ingress with
load balancers that's what elastic
container service does for containers
right so agent core is the same thing
you have an agent however you built it
you know ECS doesn't care what you put
in your container um ECS as uh sorry
agent core doesn't care which which is
the SDK that you use to build your
agent. You can use entropic cloud uh SDK
or you can use their AWS
>> like a sandbox, right? It's like
>> yeah it's it's it's a runtime. It's like
a container environment. It's it's
serverless. Um it it can be invoked
create a session that stays around for a
while um that can be resumed and then it
can be you know that's it's a bit like
lambda. It's a mix between lambda and um
>> okay but this containers I got that but
does it solve the identity IM issue?
>> So agent core has an identity part which
allows you to register um your your
sorry wait which you can control the
incoming user pool and then the agent
acting on behalf of another user. So you
can then have an outbo outbound identity
um mechanism. So how does the agent core
talk to um other systems and it has like
I don't know through there's several me
authentication mechanisms described in
the outbound identity
have a look at that.
>> Yeah. And and then there's a machine to
machine where where you have an API key
or a similar for the agent to to use. Um
identity has an a credential vault so it
can inject but it doesn't do uh some of
the things that you would expect. So it
can be very confusing because it looks
like it can do it but then when you get
to the to the point it actually
>> what about what about state because one
one one issue that we have at work is
that we have a lot of data and uh we can
get agents to grab the data and uh give
us some insights and everything like
that but it's so slow to get the data
out of like you know Jira Workday the
usual suspects
>> don't use Jira
yeah that's the solution
But like but like you need a caching
layer often just otherwise the whole
solution is not workable. And I wonder
if someone like I don't think anyone
would mind me saying that at at my
employer like GraphQL Apollo the Apollo
makes a surfaces again. I've used Apollo
on and off some years. I mean in
different engagements
like is there is there a good solution
for for caching uh
solution for this was identity actually
this is something related to MCP
authentication
>> client ID yeah that's a good that's a
good concept
>> yeah so that one is is the one that that
this morning um I was I was told that
this is what we use to to authenticate
against MCP so I mean I didn't
understand all the zero but like I do
feel Sorry for like noobs cuz a lot of
people don't understand alz
public private keys client IDs
secret key. Yeah, there's a lot going
on. But but do did you have any insight
to the whole caching besides not using
uh Jira
because you because you don't want
agents to
uh
nondeterministically
fetch
key data all the time. G
I think I think this goes to the the
ongoing you know how how they the
community goes into fl into cycles and
and and like waves of technologies and
you know everyone went all in on
markdown and then there's this whole
story about SAS is dead because every
time an agent needs information it's it
fetches it and then stores it in
markdown on disk and then it it grabs it
quickly.
>> That's true. Yeah.
>> Yeah. And then we had this whole
discussion about markdown is a broken
mirror because there's like 10 documents
and they all say something slightly
different. Um
>> yeah, we should have
>> and then we talked about um you know I
mean the Twitter sphere or uh Xphere
talks about um you know loop
engineering, harness engineering,
context engineering and now they're all
talking about graph engineering. Yeah.
And
>> yeah, that's true. It does we
Yeah, it does. You're quite right that
we're going in circles here
>> engineering. Now we're talking about
data
>> about the data. Yeah. Because because
the you know maintaining that data uh in
in a consistent way is hard and and
that's where graph engineer I honestly I
just read the term. I haven't even
looked into it but the fact is that we
are building graphs at work. So I don't
know what they're talking about when do
they do graph engineering but we're
building graphs. So I guess we are doing
graph engineering.
>> Yeah. I'm not I'm not a huge fan of
GraphQL though because
>> it's not GraphQL though. Like we're not
using Apollo or GraphQL. We're we're
we're using um we're using
>> uh Neptune um to to basically like
Neo4G, you know, graph query language.
Uh it's not
>> it's not Apollo. So yeah, we're not
we're not using uh what's the that's
like we're not going away from REST.
>> Yeah. What what is the AWS version of uh
GraphQL? I think is it Neptune? Yeah, I
think it's Neptune.
>> Neptune
Neptune is a graph database. You store
entities, you create relationship
between entities and then you query
them. You can go from one entity and and
and do a thinization.
My probably my wrong opinion about these
sort of things is that is that people
tend to
well graph databases are powerful and
then people make pretty crazy queries
with them.
I think some people argue that it makes
more efficient queries, but it ends up
that like it's the opposite is true.
Like people make wild ass queries and
then then they're quite difficult to
tune and u reconcile and operate and
things like this. But you're probably
talking out my ass. No, I mean because I
think it's different when you talk about
a front end, maybe it's not different,
but to me when you you talk about a
front end doing REST endpoint queries
and those rest endpoint queries go
against the data repository pattern or
you know um and then whenever the front
end needs something new the the back end
has to build an endpoint or or uh you
know um and then what GitHub did they
build this uh GraphQL um uh solution
which allows front end to to to to send
a a request of exactly the shape that
they want that then on the back end
needs to be resolved uh against um the
data which to me sounds like an extreme
>> um expensive operation. You know the
whole the whole benefit of rest is that
you can you have like immutable um sorry
not immutable but you have
>> input and endpoints and you can um can
scale it out against a back end but now
you're putting a query engine in
between. So now you're increasing the
compute required.
>> Exactly. And then there's lots of
security issues. There's uh
>> um yeah like schema introspection. Yeah.
All sorts of weird things can go on. I
don't I don't know what is the name of
the AWS service that does equivalent to
Apollo. But it's not Neptune. Neptune is
is like RDS. It's like Postgress. It's
it's Neptune is like Neo4G. you're
you're hosting a database in a different
um you know structure. It's it's like
>> it's Amazon AppSync.
>> Yeah, AppSync is the GraphQL one uh
equivalent. Yeah. So when you when you
move away from that and you you're
actually moving the data structures to
something that is you know an a graph of
object or um that is designed based on
um objects and their relationship
between objects and then you can query
those objects. Uh I can imagine that you
can actually make much more performant
data queries. Um, but I have to say it's
indeed very very hard to to cuz I see
the demos that we we have and you have
the MCP tool doing all the queries,
building all the queries against uh
against Neptune and it's it's like
cycling, you know. Um, which is very
frustrating. Uh, you're looking at cloud
firing off all these queries to figure
out the relationship between entities.
Um, it's not fast. So, there definitely
must be a faster way. I I think the
appeal is that you can um represent you
have to capture the data and then you
can represent the data from any angle
like you know materialized view of it
>> um in a very fast and efficient way. So
if you're focused on only like you can
have your whole organization with all of
the different teams and all of the
products that they built, but you need
to add one feature that touches two
products and then you can focus and
query only on those and then identify
the repositories that those products
have and then identify where you need to
make modifications and then you can
create the actual work and then hand it
off to agents to do that work um
efficiently without having to render a
whole markdown plan. And then that
markdown plan along the way got gets
stale very quickly. Sorry, this
old man stares at cloud
shouts at cloud symptoms. Old man
shouts.
>> Sorry, there's a there's a rubbish
pickup.
>> I don't hear anything.
>> You can't hear me?
>> I can't hear the anything outside. I can
hear your voice. I can't
>> Okay. I got I I purposely put noise
cancellation on. Yeah, I was I was just
Oh, I got to give this some thought, but
like
these GraphQL things
has come up time and time again on my
different during my my long career and
I'm still
thinking that the the the
promises of faster development. Okay, I
I'll get behind greater flexibility
maybe, but easier data management and
faster development, I'm not I'm not a
thousand% sure. I mean, maybe initially,
but but operationally, I think it's a
little bit of a nightmare.
Okay, I think we have we we had a really
good pod. Maybe we should just end on a
high before we run out of things to say
before we start.
>> You really can't hear any of the stuff.
What?
>> Now I just heard one little like like
somebody fell off. Uh maybe somebody's
been being kidnapped outside the door.
We got a rubbish collection.
We been doing a lot of spring cleaning.
Well, anyway, I wanted
>> I don't agree with why you say GraphQL
all the time. It's not GraphQL, it's
Graph database queries. It's different.
>> Um Okay.
Well, I mean, I'm just thinking
specifically of the GraphQL
implementation. And I know there's more
there's a bigger supererset of these
sort of stuff but anyway I just
>> no because because I think it's
interesting that the problem you face is
the the way that it's the graph
engineering or the graph is being
queried is through graphql
versus
um what I was looking at with there's a
difference between graphql queries where
you know graphql is from a client to a
server to fetch data and then define the
API contract or make sure that the API
is following the contract and then when
I talk about Neptune it's a database
engine and you have a query language to
query the data in that database with
deep u data manipulation and
relationship analysis which is very
different purpose right one is for a
front end to render a view another one
is for um you know a system querying the
data and I was just saying earlier you
need to be able to query the data to
render a view so how is are different.
Yeah. And it's cipher and sparkql and
gremlin.
Yeah. Well, let's try wrap up here. Like
I let's try give yourself a challenge of
writing a benchmark in the AWS bench
style. So I can tell you that I had um
fable
>> I think you've already done that,
haven't you? With your CDK.
cable has been turnurning on that more
than a day. Okay. I have no idea why,
but it has been good.
>> Yeah. So, basically, I don't your max
plan has been maxed.
>> No, not even. Uh but so basically the
full the full context is I always
postponed creating this benchmark
because I was always afraid of like you
know not having enough tokens and then I
ended up without realizing it Wednesday
evening. Hey you have 18 hours before
your um you know your weekly budget
resets and I had like only used 40% of
the of the of the whole budget. So I
said okay now it's time to run the
benchmark. So, I started on Wednesday at
11 p.m.
>> Yeah. At 11 p.m. I started like, "Okay,
look at these. I I gave it the AWS bench
uh repo, the data set repo, and the um
the the other fork with the flowy
support and um and some other repos that
I thought were relevant, but Fable says,
"No, those are not relevant. Those three
are really good. I I can use those." It
asked me a couple of questions for the
plan and the design, and I said, "Yeah,
let's go ahead and do this and this."
And the the way I I prompt fable is
always you're not doing the work, right?
Because you're expensive and your
context needs to stay small. Your
context I don't tell it that but like
it's not doing the work, right? It
always needs to iteratively launch
dynamic workflows where Sonnet does the
implementation opens verifies with a
tight feedback loop and then
>> you just you just you just tell it to do
that.
>> Yeah, I said dynamic. I I don't say it
with so many words because it knows it
knows what to do. But I I usually just
say iterative dynamic workflows
always override the model otherwise the
workflow is going to run fable which is
not what I want. So always override the
model
>> that in your cloud MD or you just you
just buy the
>> I should have by now right but uh you
>> so so um I actually re reuse the
session. I was always the guy who says
don't compact clear, right? Keep your
state in in in beats and so on. And now
I'm the guy like, yeah, I've got a
longunning fable session that I compact.
So yeah, it's dumb. Uh, but yeah, I I
basically iterative dynamic workflows
um to
make sure that Son is the one
implementing, Opus is the one verifying
and then when the workflow finishes um
it Fable is
>> implementing, Opus is verifying, Fable
is planning,
>> Fable is um is basically has the um
long-term view, right? The vision. So it
builds the the workflow prompts. This is
the next slice of what we need to do.
And Fable said, "Great. We're going to
go ahead." And it created A B CDE E F G
E uh sorry, G H. It had H SL H
slices um to do and and it put that in a
task list like it keep track of the
tasks, right? So it has a to-do task
list.
>> Slices would be not benchmarks, slices.
Okay. slices of the whole long-term
goal, right? The long-term goal. I I had
speced this out a long time ago about
what exactly I wanted to benchmark. I
wanted to prove that an agent in
different scenarios. And I I basically
speced this out long time before the AWS
bench.
>> Why didn't you just go incrementally?
Why don't you just come up with one test
and then build from there? I don't quite
follow your rationale here.
>> Yeah. So, I had I had a very detailed
goal of what I wanted to prove. Again,
fable is good with goals, right? You
tell it what you want. It breaks down
the problem.
>> Why do you have a golden test? Because
you sounded like you.
>> So, so basically what I did a long time
ago, Kai,
>> listen for a minute. What I did a long
time ago was I I I defined exactly what
I wanted to test, what my goal was, my
long-term goal. Okay? And then Fable
does the planning. Fable slices it down.
Starts with the one test first. Okay? I
didn't I don't need to tell Fable do one
test and then, you know, All right? So,
so basically then I I always postponed
it. Then AWS Bench came out and I was
like, okay, now it's, you know, I had,
you know, 60% of my Fable budget. I have
to spend it within the next 18 hours. I
hope, you know, maybe Fable can do it.
So, um, and this is like a pretty
substantial plan. It's not like pro or
it's it's a massive amount cuz I was
working. Okay. So, from 11:00 p.m. until
when I I went to sleep and um I always
tell apparently you need to say to to
stay caffeinated because caffeine is
approach is a process that Mac can run
to to stay awake. So, it launches
caffeinated. It stays working. Uh, I
actually put it next to my head when I
was sleeping. And um, I fell asleep
while it was working and I woke up and
it had been turning for like 8 hours and
it was a slice E B C around D. I think
it was around four slices in uh, of of
converting it. And the thing is when it
writes a dynamic
>> what do you mean by slice? A slice
is like a thin slice.
>> However, Fable decided to break down the
problem into thin slices.
Okay.
>> To reach my end goal,
>> carry on,
>> right?
>> So, so I sent off Fable to do all the
studies, make the plan. I read it. I
realized, god damn it, this is a
complicated uh scenario. This AWS bench
thing is really complicated. So, I
started looking at what AWS bench
actually is. Um, started to understand a
little bit about what the scenarios are
in there, the tasks are in there because
honestly, I first off, I said just I
want to run a benchmark. Just go figure.
Um, cuz you got a ton of tokens. Just go
do it. And and so I I started looking
into what bench was to try and
understand the plan. I couldn't really
understand the plan. So I said just go
ahead while I I keep reading because I
was running out of time. I had no more
time. So So Fable started building.
>> So you had time before your tokens
refresh. So you were just like go go go
go.
>> So go right go do. And then I woke up
eight hours later. It had it had like on
average each dynamic workflow takes
close to two hours. So I had done about
four of them. What it does in those
workflows, it puts a constraint in how
many loops it can do. So those workflows
have interesting um um you know concepts
because they can there can be one son
implementing then an opus verifies then
another son needs to fix what opus is
identified then another opus verifies
and it can do that up to I don't know
you can determine how many iterations it
can do. So, so Fable told me I told them
max max three times. Like I said, hey,
why is it not done two hours? It's like,
oh, we're on we're on the third
iteration. They're not going to go
further. Relax. So, so, so, so it has
this whole this dynamic workflow system
is quite impressive. And then, um, Fable
just keeps track of the long term and
slowly the context fills up. I think
after four after 8 hours it was only at
like 30% of
>> I would love to see how you visualize
this. How do you know that this is
actually being carried out as as you
wish? like you know that so
>> when usually it only takes 2 hours or 4
hours and I have the result quite
quickly okay so when it takes 8 hours
and you still don't see the result it
gets frustrating right you don't know
actually is this going to be exactly
what I want
>> but how do you how do you try do you
actually bother do you see all the
agents in the claw UI
>> yeah yeah Claude Code
>> and you can tell each agent has a
different model
>> yeah you you can so basically the moment
you you you make it launch a dynamic
workflow it runs in the background and
the main agent is waiting for the
workflow So you can go down to that
workflow. You can see the the different
phases step one
>> running right now. Can you share a
screen? I'm just curious how it look.
>> This is this is this is two months ago.
This is uh what what you call this
again? This is loop engineering. We're
done with that. We That's solved.
>> Sorry. What are we doing today?
>> Graph engineering.
>> Uh hold on. I have so many things here.
Uh it's not this one. So,
it's
>> it's funny how we now everyone has a
million browser tabs and now everyone
has a million.
>> I just told you I just told you it's so
easy for me to get get my session
because I don't need Herder and then I'm
spending an hour trying to figure out
where the session is.
>> Doesn't even sound like doesn't look
like you're using CMax. For the love of
God, this is Semox. It is CMX, but I I
folded it.
>> Actually, I don't even know what Cmax
is. I guess it's a wrap around Ghosty.
>> So, here it was on on slice G and it
says slice G fails. This at this point
it's actually doing live checks against
my AWS orc and that prompts me. So, when
I'm not at the computer, I don't nerve.
>> Yeah. So, I can show you
>> worried about your bill, your AWS bill.
So here are all the workflows that ran
right and you can see this one was 10
agents. So this is what it looks like.
This is what while it looks like while
it's running two except these ones are
all completed. So I can see that there
were uh you see this is what what annoys
me. I have to enter the
the keychain password and I'm not this
is my AWS or account so I don't want to
give it like full access to it. So even
though I'm not checking the prompt so
it's a bit ridiculous, right? I'm just
typing in the password.
>> You got balls of steel. Well, that's all
I can say.
>> Yeah. So, so this is um this is the
context that Fable designed for this
first part of the first phase. So, we
have the phases on the left which is
scaffold phase and the first task which
is um scaffold task and um
>> give this a try. This is this is
dynamic.
>> If you never run dynamic workflows,
you're you're really missing out.
>> I'm a noob. I'm a total noob.
>> Didn't say that. But this was the very
first slice as fault. So this is like
look at the AWS bench uh layout and re
recreate it from scratch you know for
our purpose right. So that was son then
four more Sonnet agents then here we have
opus that I guess in parallel verified
four different layers and then reverify.
So the reverify means that it went back
to one of the build agents. I I oh it
went back to a fix. So here's a fix
phase round one and then it went back to
verify and it it's it another opus round
went off to verify. Now, what I like
about this is that you see these agents
never reach a lot of like context
windows, right? They're they're close to
200 context uh 200,000 each, which is
only 20% of the window, right? Which
makes them very focused and very
shortlived. So, all of the context
engineering that you had to do about
like I need to clear my sessions, I need
to do the checkpoints, forget about it,
you know, all taken care of right here.
>> Okay.
>> Okay. I got to try dynamic workflows
now, man. I feel like I'm missing a
trick.
>> Okay. So fable. So here I I triggered it
on a 35% that's 350,000 tokens. So I
send off a stupid like re relaunch G on
top of 350,000 tokens. So the price of
this one sentence is actually quite
expensive. Um that's where you maybe
want to like compact uh for Fable to you
know orchestrate a little bit. So yeah.
So,
>> speaking of expenses, I don't see dollar
signs and I don't see your How do you
keep track of your your budget? Again,
>> I had this codeexual.
>> I had I had like a I had a codeex bar
here that that that keep track of the
you know the it has markers. It tells me
>> in your 5 hour session if you're ahead
of the marker that means you're going to
exhaust before the session finish. And
then also in your weekly budget, if
you're ahead, you're going to run out
before the week is done. So here it has
launched
>> it has just relaunched it right here.
Right. So you can see the it has created
the phases. And now you can see that the
Sonnet one is running for 18 seconds. And
if I tap on it, I don't know something's
wrong with this. Oh, did I zoom in? I
think because I zoomed in too much.
Normally you can see the whole context
that been launched and then you also see
what's what's it doing at that time. So,
activity, it's doing 11 tool calls.
>> This is really exciting. I've got to
give dynamic workflows a try. Never mind
bloody AWS bench. I'm giving AWS
workflows a try.
>> If anyone's using workflows,
>> if anyone's using AWS workflow, I mean,
not
sorry. If anyone's using clawed
workflows, please comment below. Please
like the video regardless
on please leave us five star reviews on
all the podcast platforms.
Please send us some love so we can
>> read this blog post.
>> This blog post and Kai disregarded it
because it was too much words.
>> Words.
What? Did you fix the summary?
>> I know you you will still complain. I
will not show it. So um so this is this
is a reusable dynamic workflow. So I
asked Fable to build a dynamic workflow
that I could rerun when Fable was no
longer available because I assumed that
Fable would not be available or would be
too expensive. So I asked it to build a
workflow that I could ask Opus to run
and so Fable was iterating and in this
blog post I think the most interesting
part is that each time I was running the
workflow I was reducing uh Fable was
optimizing it based on what it saw the
agents were having troubles with. So
each time it became less
>> this is like this is like prompt
engineering like where you
>> this is loop engineering right
>> loop engineering when you fix your your
>> so it it it does this this blog post
basically looks back at how fable
iterated on the dynamic workflow to make
it less token uh expensive to make it
faster and more accurate so that the
agent wouldn't have errors. So it was
tuning the prompts, it was tuning the
phases, it was tuning the way that the
the work goes into the agents and and it
and then highlighted that over time it
went down 20% in token usage. Uh the
number of agents were reduced, the
rework was reduced, the total duration
was reduced. This run four and five are
two runs on the same task. So I rerun it
on the same task just to get a
improvement uh result. So this blog post
is maybe very dense because I did ask
Opus to like look at everything and then
make help me summarize it but it it has
some very interesting um you know like
what generalizes. So Opus came up with a
couple of uh rules but honestly this
probably not very useful. So but what is
interesting is the methodology you know
that's kind of like
>> this is fascinating. Okay listen I got a
meeting now got to run. Okay, bye.
>> Anyway, thanks. Thanks again. Uh, I
think this is a great part. I really
enjoyed this one. So, thanks. Thanks.
Uh,
>> all right.
>> Thanks, Vincent. Bye. Thanks. Bye. Have
a great day.