Episode 42: Agent IDE

Welcome to episode 42
of the AI infrastructure podcast with
Vincent Dismat and myself Kai Hendry
talking about AI workflows in the
context of provisioning infrastructure.
Today's topic is about agent idees. It's
kind of like the next step over using
claw code or Codex in your terminal. I
think it has some potential. I've just
been using it only for a couple of days
and we're talking about that amongst
other things. I hope you enjoy it.
Please like, please rate the podcast on
Spotify, Apple, Music, uh YouTube,
comment, you know the drill. Thanks and
enjoy.
>> I have been using AI and Orca and
>> Oh, share about Orca. Sorry, I couldn't
see your uh breakdown, but you've used
Herder. I've used Semox. So maybe you
can show Orca and explain how you like
it compared to Herder.
>> Well, to be honest, I haven't really
used Herder in anger in all honesty.
>> I saw that you were trying to do a
comparison between two agents. Kind of
boring. Just show me like something
interesting.
>> Oh
Okay. Well, let me just share my screen.
I mean, I've only really used it for the
like two days solidly. solidly two days
is that a lot that's not much is it? So
for example,
what do I like about it? Okay, so one
thing I like about it is is that
it's supports all the the different
agents easily. I like how it's like kind
of terminal based in a way like the tabs
are are kind of terminals.
>> Looks like CMX
>> and then it's easy to start a new work
tree with a different thing
>> and a different configuration. It has
has mobile so that's that's a killer
feature for me. So that so that so now I
can you know check what's going on my
mobile
>> but that means you have to pay their cl
for their cloud service. I I use the the
local one, but the trouble is
>> But it must still have like some type of
control plane for your mobile app to
connect to your to your MacBook, right?
Some type of reverse tunnel.
>> Well, I mean, I I manage my own LAN and
I use Tail Scale, so I don't use their
their stuff.
>> Oh, so your so so is there requirement
that your mobile phone and the Mac are
on the same network?
>> Exactly. on LAN.
>> Yes. So, basically I just use the LAN
and it works.
>> Oh, perfect. So, you can set up a VPN
and then you can be out of home and they
can still be on a network on a virtual
network.
>> Yeah. The there is a there is a big
downside that I haven't really figured
out is that um
>> so somebody Well, when they replay it,
this code will be will no longer be
valid, right?
>> I hope so.
>> Sorry. One big downside you were saying
>> one big downside is that uh this is like
a what do you call it? first world
problem. But like I use Orca on this
machine and I use and then when I'm in
my in my home which is it's just like 20
m that way. I use another MacBook Pro.
So the trouble is is that is that like
I've paired this with my I've paired my
phone with the other Orca. I haven't
paired it with this Orca. So I don't
know if I don't know. I I guess I could
just set up a new one.
>> It's a paid feature. I
>> I'm sure this will work to be honest.
They must have a business model and some
type of paid
>> I can see two hosts. Oh, it does work.
It does work. I can see two hosts now.
But but the problem is is that is that
well I mean it's it's I'm just confusing
things a bit. But like I basically work
from two laptops is what I'm trying to
say. So okay. So the the mobile
integration is great. Awesome. The other
thing that I that's quite nice is that
for example actually I haven't tested
this thing just I'm just working with
Can you see?
Oh no I can't see I I'm using SSH on one
of my hosts here. I was hoping that it
would do the image pushing stuff. It
doesn't seem to work.
Oh that sucks. Okay that's another
negative point. I'm just finding all the
negatives here.
like the SSH doesn't work to send images
across and then other than that it's
just it's just quite tasteful like you
can see the files
the sessions the git changes yeah the g
I haven't got any git changes to show
you but it's quite the uh the pager to
to to view uh diffs is quite nice and
you can you can comment on them so you
so you can do things
>> direct the agent that's really nice I Is
this the future of an IDE? Then
>> I think this is getting damn close to
it. I mean, this is this is something I
wish for like a month or two ago and I
should have just got off my ass in some
ways to it's I guess everything like
this is in hindsight. It's like I wanted
this a couple of months ago and now it's
here and um is this the future? Well, it
doesn't it certainly has a lot of
features that are better than I mean a
lot more than CMOX. CMOX is quite old
and I haven't really been paying
attention to new capabilities.
>> It does have like like one thing that
impressed me is that like when I when I
create a new work tree for example here
on one of my projects it it sort of
automatically you know gets the git
>> you call that
>> the the
automatically gets the git issues. Sorry
I'm really slow. And then for example,
yeah, I want to work on issue here with
say Codex for a change and then booms.
>> That's nice.
>> And it it doesn't like do too much
trickery in a way. Like for example, it
just pre-populates it with the the issue
number and then I go doink. So it's just
little
>> that sent a message. Mhm.
>> It's just just a tiny bit of polish. You
know what I mean? It's just like does
what you expect it to do. It's just
tasteful. Does it support like u reviews
of markdown files or like in a rendered
markdown view? Does it have artifact
support that you can easily
maybe get an like an overview of a
composition of modules that you can
review?
>> I haven't tried that but I mean but it
it's just it is just as you can see a
clawed code. I mean I could I could just
run artifacts here, couldn't I? or what?
>> Yeah, but I'm interested in like what
happens when the artifact
uh is available uh like created by the
session. Does it open an external
browser? I would imagine and then um I
guess
giving feedback on because we were
looking at a while ago people are trying
to break down work with planning
artifacts.
Markdown isn't cutting it anymore. We're
moving more and more towards HTML. Those
HTML documents need to receive comments
in line.
>> Oh, wow.
>> And that needs to go back to the agent,
right?
>> Oh, wow. Yeah. Yeah, I think I know what
you mean. No, I haven't I haven't got I
haven't got as far as doing multiplayer
HTML,
if that's what you'd say.
>> Yeah, I haven't played multiplayer. I I
think multiplayer HTML or or MDX, which
is Markdown with React components. So
you can build reusable component library
that intuitively fit within a markdown
file but that get rendered interactively
within a browser window. I really look
forward to this type of interactivity.
Imagine that you have a an ontology or a
glossery of a sequence diagram component
model and then you just tell the agent
build it and it uses the you know like a
React component
>> inside the markdown and then when you
view it in in a browser the React
renders as an actually interactive
sequence diagram
>> that allows you to plan uh nicely,
right?
>> Yeah. Okay. Get get alignment and things
like that. Yeah, that sounds quite cool.
>> So, in in a way, I think Orcai looks
interesting compared to more like
personal agents like Kiro Crew that we
looked at a while ago. I think I do not
sure if I was able to show it back then,
but with Kiro Crew, the way that the
artifacts is integrated. It's not a
terminal shell, but Kiro Crew under the
under the hood can run Codex well if if
or can run Kiro terminal and you can
also get access to the terminal and it
all runs in a remote machine. So
currently I I'm able to use my Kir crew
that's running on my uh desktop at home
>> and I'm able to tunnel to it from work
through a mic or like a nano instance
running in AWS that is running tail
scale as well. So, I'm actually using it
as a jump host. So, I'm sshing out of
work u or wherever I am.
>> You can set up Orca, I believe in the
documentation, on a on a on a remote VPS
if you wanted to, by the way.
>> But I I don't want to set it up on a
remote VPS like I have a powerful
hardware at home. It doesn't really
matter like cuz ultimately I just told
cloud look uh I want the jump host in
the cloud that has a fixed IP. So, I pay
for what is it? how much US dollar you
pay for a for um
>> elastic IP in AWS and then you you just
jump towards it
>> I don't see the point I mean my my home
internet is I've got like static IP it's
very easy for me to jump back into my
home internet so
>> if you have a static IP yeah for me
>> I'm hash or tail scale even
>> yeah well that's a problem if your
device is um is is controlled by by um
you know network, it's easier to SSH
out. Funny thing is when I told cloud to
set this up, I was talking about working
from hotel rooms and working from coffee
shops and and it was like and working
from office and it it actually set up a
whole bunch of like constraints because
it it runs SSH on 443. So it's an it
looks like an HTTPS session, but within
there's a tunnel actually because a lot
of offices will block port 22.
>> Yeah. But but tail scale gets around all
that stuff.
>> No, but that again if you listen to what
I said at the beginning which is of that
sentence which is a device that's
controlled you cannot just install any
networking on top of
>> see okay right
>> if I try to install the tail scale
client
immediately gets blocked by
>> Oh really?
>> Yeah. So if you can create an SSH tunnel
it's pretty simple. That's that's one
issue that I have with with work and and
clients is that is that uh the mix of
devices and because like I want to
leverage my home infrastructure to do my
work and I do for the most part like
like you know I do get a laptop from
work but like the monitors, the mobile
phone, the ISP, all that stuff is is in
my domain, right? And why can't I
leverage it? It's it's insane if I can't
leverage my own network and my own
my own just to basically I just want to
make sure that I can just I can just
work from my mobile phone is what I'm
trying to say. Yes. I think I think
still I mean Orca can turn your your
personal laptop into a harness for your
like leveraging those Claude Code or
codecs or PI whatever harness you choose
via terminal but I think there's a
couple of tools that go a level above
that that turn a device into kind of an
assistant that has its own harness on it
and that you talk to like you the
original
claw Kiro Crew.
>> No, that you talk to like it it's not
really a terminal. It's more like you
give it instructions and it gives you
overviews of the plans. I don't know if
like if if the if the layer should be at
the terminal or it should be at like a
higher level assistant like when you
work with Hermes or with open claw or
with Kira crew um the agent has full
authority over its machine. You can give
it a VM and uh and then from there it
can install any binaries. It can set up
any project workspace. It can the same
thing that you give your Claude Code the
ability to manage like you know work
trees within your repositories on your
machine. Uh and it acts with your
authority. I guess that's the
difference. I guess it's it's a it's a
tool that runs on a machine that's like
set up for you to work with that you
were acts under your authority versus an
agent that has its own authority and
that you uh you know interact with and
you know you give it tasks so it can run
on its own it can have a chron to pick
up work you know like the is radar and
this type
>> I see I see yeah this is like level
yeah okay yeah that's yeah like I think
I think I know you're saying, Vincent,
you're saying that like you you want a
host, you want infrastructure so that
you can have like a whole pipeline of
agents working while you sleep. That's
that's the huge problem I have. It's
like every every time I go to bed, I
basically shut down my laptop and
everything stops. And uh and it's the
same way for work. Like for example, you
work 9 to5, you have eight billable
hours. Do you want your agents running
in the middle of the night so that you
maintain them while you brush your teeth
as you go to bed? That sounds kind of
crazy, right?
>> For me also, when I have to move from
office to home and I I spend 30 minutes,
I have to close my laptop and I I
usually drive and if there's dynamic
workflows running on my machine, if
there's network connection issues or the
machine goes to sleep, then it
completely messes up the thing. So
having all of that run remotely is kind
of the dream, right? Um
>> yeah, that's I I see what you're saying.
Yeah, I I I I've yet to solve that
problem myself really. The the other
problem it's this is more to do with my
work and my clients is that now that I'm
using Orca, I see the power of switching
between uh Codex and and and Claude and
I'm actually blown away how good Codex
is. I'm blown away. It's so much faster
than Claude.
>> Is it still open source? I thought the
open
>> No, it's not open source. It's the least
>> used to be right.
>> Is it?
>> It used to be open source. It used to be
like Claude Code is s closed source and
codex is open source.
>> I think at some point well yeah it's a
patch 2 license. Interesting.
>> It's kind of maybe that's why people are
like more people are able to contribute
and focus on performance. Well,
nice.
>> Oh, GitHub checks are always just messy,
aren't they? The
>> you know, Hero Crew, you can't get
anything merged without
>> Okay. Okay. Going back to my point here
is that most
like like my current client, they they
give every employee a clawed
subscription, which is nice,
but you don't get Codex.
So basically I feel like my world, you
know, personally speaking, I have Claude
and Codex and I've got other ones too.
My I feel like my mind is expanding. I
feel like my internet has just got
higher bandwidth because it's really
interesting to see what different models
are doing with the same problem.
>> And it's very easy to do in Orca. And
I'm like I'm actually blown away with it
with open AI. I feel like Theo,
what's his name? Theo GG3 T3 GG
>> and I'm like I'm like I'm dissing Claude
now. But it's it's just interesting. So
>> I think it I think it's never good to
just make an absolute blanket statement.
It we we've covered this that they're
constantly trading blows. One will be
ahead. Yeah.
>> It's very uh problem specific. It's very
time time sensitive. It's very
>> um you know personal feeling anecdotes.
Some people will have a really good
experience with Opus. Other people will
say Sona is really a lot better
depending on the on on the context
depending on and this is I think a hard
bit as you work with Claude Code you it
creates a lot of memories on outside of
your repository. So even if I was I was
asking someone you know I
I I asked it to make a lot of memory. So
I want to see I I I I was on a call and
because they asked me how do we do this
and I said look I took the question that
they put to me I put it straight into
the Claude Code while sharing screen and
I said uh it created the PR it
automatically figured out a bunch of
stuff and I said you know I assume it's
because I built the cloud MD properly
but I'm not sure if it's because of
memories on my machine. So I would like
you to repeat the same thing. Yeah,
>> do not like let it go like maybe I don't
push the branch. I already opened the
PR, but do not let it like see the the
result of of of work that it did from my
machine. Uh and and and ask the same
question and let's see if this is like
because of memories on my machine versus
>> Yeah. Yeah. I've noticed this memory
thing like I I think when I saw memory
in Claude, I turned it off, but now it's
somehow back again. And to be honest,
>> oh yes,
>> with with with Hermes, I've noticed that
you just get it by default and that's
that, right? So,
>> what what's Hermes have to do with this?
Because Hermes is an agent that works on
its own. It's very different, right?
>> Well, it's not the same.
>> It's it's not the same, but I've noticed
that uh that when when I've noticed with
Hermes that it knows about me, it
>> agents agents, they are not just a
harness, right? They are a harness plus
a memory system plus a gateway plus a
completely isolated like some people run
an agent on their machine which I don't
but like I believe an agent should be
able should have its own identity it
should have its memory system it should
have a gateway connectivity and a
dashboard like once you add a dashboard
on top of an agent system it's pretty
cool like what what Kiro Crew did there
is really nice I haven't been able to
>> what do you mean by a dashboard like a
>> yeah I I think that's what I missed from
Hermes because like I was using it only
via gateway like I was only using it via
Discord or you know you're using it via
WhatsApp but if you use the the
dashboard and you you remember the Ko
dashboard that I showed you I I can show
it to you again what I mean by a
dashboard.
>> Okay,
>> I know that I have a lot of tabs open
but this is Discord, right? I have a
bunch of servers and then I have my
Hermes agent server
>> different channels.
>> I have the channels.
>> Yeah. So I have it's a little it's it's
clunky right I have a general channel I
can you know mention the bot and then it
automatically creates a thread and then
in there it shows that the process and
sometimes shows an interactive card for
permissions like um here then I need to
approve and if I don't approve it within
the timeout then it just uh proceeds and
and the agent then tries to work around
it or something silly like
So this is what a dashboard looks like,
right? In this case, I'm connected to
the via the jump host in So here are the
the the gateways, right? The um remote
crew. So this one is trying to connect
over local LAN and this one is
connecting via a wire guard through a
jump host. So it's an SSH tunnel to and
then into the the desktop. So desktop is
connected to the you know wire guard
network and then I SSH into the remote
like the the hub and then the rest is
the the other nodes are are spoke right
hub and spoke setup. So, so this is a
what a dashboard looks like. Right. I
have sessions. It's very similar to what
you saw in in the um in the Discord
channel. So, you can have your threads,
right? But you get more interactive
things like um you when I generate an
artifact, where is it?
>> Yeah, I guess Ok doesn't have these
features when I think about it.
>> Yeah. So, here was a little markdown and
markdown is not interesting. At least it
gets rendered, but then when I publish
it, so I did publish it, uh, it goes
into CloudFront, but the markdown
doesn't get rendered into HTML. So
that's kind of a bug. U, so more
interesting is if you ask it to to build
like H is there an artifact here? Create
an artifact for Yeah, the I'm just
thinking aloud here that like Orca is
definitely focused on development. It
even has this camb board that I didn't
really show you. Uh, and then when it
comes to chats, I basically just jump
into into Gemini because I find it fast
and free or or I jump into claw.ai. I I
basically jump into a web browser for
chats. But I can see here that your your
chat your chats are probably a bit a
little bit better organized in Kira.
>> Yeah, you can you can see active
sessions. you can search, you can
probably uh clean up like I haven't used
a lot of capabilities. Like I said, I
used it heavily until the beginning of
last week and then I was in heavy
planning session for the rest of the
week and then even the weekend and then
I haven't been able to use it. I still
have to contribute my PR that uses
Codex instead of um Kiro. So I'm using
my chbpt subscription here.
>> I need to contribute that so and get
that merged.
One thing at work, I'm just thinking
with my with my the multiplayer team
aspect is that is that uh at work at the
client, we're building a lot of tools to
essentially do what you're doing here
>> and have the agents set up for for the
organizational uh context, right? Have
the organ have have skills set up, have
uh different connectors set up. And
that's probably that's probably the
future. We're probably going to see a
lot of development in this sort of like
guys like Kira like Orca where where
where
essentially it boots up at the at the at
said organization and it's it's it's
just basically got the organizational
guardrails, the organizational context
ready for you to go to do your job.
And and at the moment we're we're we're
basically just trying to figure it out,
you and I. uh trying different tools,
different approaches. We're basically
>> Isn't this really cool?
>> Pioneers.
>> You get the artifact in line right here.
>> The I'm a big fan of artifacts, the HTML
artifact thing.
>> But this is like even cooler than the
terminal experience, right? Yeah, it is.
>> It right here.
>> And and this is why I think you kind of
need Orca or some electron. I I mean I'm
no I'm no fan of
>> But this is what Hashimoto is working
on, right?
>> Well, it's going to render HTML in the
terminal.
>> He's building a new terminal multiplexer
that leverages ghost ty the TTY
libraries with a a different protocol
like a network protocol that doesn't
depend on SSH anymore. So they are
rendering back to the terminal and he
did write ghosty to to render like 3D
graphics in the terminal like he has
these animations right with super high
FPS. I I wonder if that will you know be
able to render this type of of of
>> interaction. I imagine it being a really
good multiplexer so that you can connect
back to your session but I don't I doubt
it's going to be like the electron. I
think what I'm seeing with Orca is
already well it doesn't have all the
features like like rendering stuff but
but it's it's definitely on the right
path. Yeah. I I see a a mismatch a
mismatch a m a mix match is that the
right term of of kirao features and orca
and we'll get pretty close to what we
need in most companies in most
development environments I'm starting to
think. So this is the artifact that was
created from a chat session and when I
click publish it goes into a public URL
on my AWS. So it's using CloudFront. Um,
so this is actually running on my
desktop at home and you just have to
>> and can you comment on the
>> Yeah, you can you you can you I think
normally you can
>> but not on the on the publish one can
you
>> I can leave comments here but it's a bit
janky if you create like an interactive
slide.
>> Yeah, exactly. Can imagine this if it
has if it has state. Yeah, because no
that and you can edit, you can review,
you can res Oh yeah, you can actually uh
stage several comments into one new
session to to address them and and and
build uh you know build on on the
comments that are coming in. So but on
the cloud front, I don't think you can
comment. I mean it's it's publishing.
Let's look at the artifacts.
>> Do you know would be really cool if that
you could publish an artifact. It could
be interactive and then you can record
decisions from your teammates just then
and there. That would be cool. Like, you
know, have like a trade-off slider and
then vote with your colleagues like, oh
yeah, we should go here or we should go
there and get and and get agreement and
and then
>> get buy in and just move on to the next
thing.
>> So here,
>> this is so challenging at work actually.
So challenging. This this is currently
deploying every single artifact under a
new cloudfront distribution.
>> So, um I don't think that's cheap if I
keep them running. Okay. So, it's
>> Yeah, that's that's my concern about
using any AWS thing with Bedrock is like
there's no easy way to budget. Oh, by
the way, I tried open router. Very nice
product. Budgeting works really well.
budgeting on bedrock
doesn't work well. So, so I'm I'm just
too nervous to try AWS features because
there's no proper guard rails about
expenditure and things like this. You
know, dude, I I'm using my my credit
card and my wife is going to see I
really spend 4050 uh US dollars a month
on AWS.
I can't try I can't afford to try
Bedrock. I haven't I haven't used
anyway. Bedrock is indeed not not uh
cheap. What's the difference between
open router and light LLM? I've never
tried light LLM, but open but I but try
open router. It's amazing. It's just the
the on boarding flow so polished like
like the way it asks you for your credit
card but like it says like you know I'm
just testing for
>> at least they do it nicely. That's a
nice way to say it's so smooth. That's
why they got how much money did they did
were they acquired for
>> the product is so good like the other
thing that really blew my mind was that
uh it had an MCP that you authenticate
very smoothly in Clawude so you don't
even have to put any credit like my my
balance in open router is zero right now
but but you can use the MCP and you can
and with the MCP you can ask questions
like what is the best model for image
generation at the moment and what's your
best price. You can ask questions like
this to the open router MCP working out
which model, which price to do a certain
task and it will tell you and the
information is like totally up to date
and it it's not like hugging phase where
where these rate limits all over the
place and it's very difficult to work
out which model you should use. So,
open router very polished. It doesn't
surprise me that it would got it went
from zero to hero. I mean, I don't know.
I don't know who's using it, but like
whatever. The the product you can tell
is just very smooth,
>> very tasteful, just like Walker.
>> Tasteful. I I recognize good taste
>> when I see it.
>> H another
>> strange. The the publish didn't really
work well. It shows, but without any of
the CSS. So, interesting to know because
this is like a widget. I don't know
what's the difference between widget and
other things. Anyway, uh we were talking
about Orca and then um the terminal
experience. You're talking about um open
router and I I think it I don't know
actually the difference with lightm LLM
being something to control budgets
around
LLM calls which within the organization
I work we're also looking at light LLM
to control uh budgets per team and and
model
>> say anything about it. Oh, can I just
steal the the screen share? I I showed I
showed the killer crew again.
>> Well, one thing that I I think really
that Oh my god. One thing that I think
that is really good about the orca is
the documentation is quite well done.
It's like for example these recipes it
it's just like only five of them but but
they basically
really hit hit what do you call it? You
know they they hit hit the spot for me.
Like for example, brace three agents on
the same task. I did that my I did that
and my mind was blown.
>> But why?
>> Review AI diff. Once I discovered the
diff view, I was like, "Yes, this is
good."
>> Jump between work trees. So, command J.
Really, really nice. I haven't tried
this design mode. But then last but not
least, SSH to a remote host and working
from that remote host. Yay. I loved it.
Well, that's exactly what I just showed
a beefier dev box. So, this is what I I
use C crew for, uh, which is it it's a
dev box, but managed by an agent and it
runs its own scheduled tasks. It's a bit
similar, but it seems the the focus here
is different in in that you are the one
doing the work assisted by a harness
with with Orca and with Kiro Crew,
you're the one fully delegating work to
an agent that you trust to work on its
own, find new work uh and and and and
submit work. You're not really like I
think when when you show me in Orca that
you are able to look at the kit locally
and review it, it's it's like you and
and and the the the hardness are working
as one like you one is a pilot, the
other one is a driver or how what you
say.
>> Yeah, I think I know
>> you you submit in your name. So you're
the one reviewing it locally before you
PR it under your name. Whereas the
agent, you delegate it and it submits a
PR for you to review. So you're not
actually reviewing it before it submits
the PR. It it's working on its own. Then
you delegate tasks like hey go and
update this uh do this differently and
things like that.
>> I guess it all depends what what you're
comfortable with and what what what
level of maturity you're on. I think
Orca is the right is the right uh level
for me at this point.
>> I think it depends on the tasks and the
work. uh some things you can delegate
some things you have to be hands-on and
this is kind of also like uh some of the
difficulty that I had so I've been now
one month working on one month and a
half working on almost two months a
project and a lot of it was like p
showing that it works a lot of it was
like dynamic workflows to prove that um
that it would work and and literally
just me alone building highly complex
proof of concepts of a of like an an
agentic durable workflow solution that
can be triggered off uh Slack or uh
version control events. So it's like
event source uh and also multi-tenant.
So we we have to be able to onboard uh
tenants to to basically route based on
their version control repository. um
what is the uh you know scope to the
token then it's scoped down to their
permissions and and things like that
very very interesting very exciting um
but now it's moving from P towards
actual product and
>> it needs to be a lot more hands-on it
needs to be a lot a lot less like the
agents can build and prove this but it
needs to be now we need
>> so it needs to go from from your high
level where you delegate to agents back
to my level which is I mean you it
sounds like you went to level seven and
then I went but you need as you as
you're trying to explain I think that
you go you need to go back to level five
to sort of productionize it or something
like that. Yeah, you need to be more um
hands-on or you need to take back
control. And I like I let go of a lot of
the upfront
like spec driven uh workflows where we
are upfront identifying the the user
user stories and and and breaking down
the phases and the MVP deliverables and
and phasing the work so that we can you
know build a larger amount of work in
into smaller slices that are uh
controlled and reviewed and grown in a
in a maintainable way versus you're kick
taking off a couple of agents to go and
build a PC and yes you can it it works
but it's a immense mess to to maintain I
know and one thing is you get journals u
inside every comment it just keeps
growing
>> hey maybe that that that is a segue into
another one where uh you know remember
that I was working on the on the
benchmarking
>> yeah I I wanted to show you my
benchmarking journey uh because I I was
using AWS Bench
Oh, so you played with AWS Bench.
Interesting.
>> And
>> what is it that you were trying to
benchmark first? Start with that.
>> Well, let me just let me just show you
and then I feel it will be uh
>> Can you describe it before you show it?
>> I think it's let me just show you the
demo and then and then I think it's
easier to talk about it. I'm a picture
and then talk kind of guy.
>> Oh, so you were trying to use the local.
>> Yeah. So, to be honest, it was it was
non-trivial to set up and I had to
get the agents to patch the hell out of
Flocky. But,
>> and did you did you mine the other
pre-existing work that was using Flocky?
>> I was trying to point the agents to it.
I didn't really understand what he did.
He did.
>> No, nobody does it. But like the the
thing like for example so essentially
like
I think I have it set up here. It's it's
passing but then I set I set it up so
that it's broken right and then I I I
verify that it's that it's broken. It's
failing. And then and then I have the
task here which I sort of like you know
I I tell Claude to fix or something like
that you know pipe it to Claude whatever
I can just run
>> do the task first
>> and and then Claude is able to do it and
and and run verify and things like that.
So what's my point here? So I I was
actually um surprised by this
>> because
>> that's not a benchmark though. So it's
it's a single task in a single scenario
against a local
>> Yeah. and and and and that's as far as I
can tell how it it it it works for the
for the most part. It I think it's
hopefully fixed it.
>> Yeah, it's passing. So
>> Mhm.
>> This is what this is what kind of
surprised me with AWS Bench because
when you look around it's it's like a
lot of broken things and it's testing. I
mean, we talked about it and
there should be no surprises, but like I
understand that it has all these
scenarios for for debugging things, but
to be honest, I was thinking that it
would have more scenarios for like uh
>> generating or building.
>> Yeah, bu generating building things. And
as far as I can tell, there's not many
of those cases. There's lots of cases
for for debugging broken setups.
>> Yeah. I think what AWS really wants to
know is what are the tools they provide
because ultimately AWS as a service
provider uh wants to know how easily and
how correct and accurately people can
use coding harnesses to um work with the
cloud right so they're providing AWS doc
MCPS they're providing skill
repositories and they want to know and
evaluate if those things actually work
>> but
>> and that's exactly why you need that
>> but the trouble is the trouble is it's
like a it's It's like curing the disease
instead of preventing the the disease.
>> No, no, it's a different target like
when what they are trying to solve for
is for for people that are running on
AWS to make their experience easier and
to make sure that the tools that they're
using can troubleshoot and help them.
>> You know, when you talk to like I think
that's one thing that focuses on, right?
>> It's legit like
>> they want people not just they want to
increase your usage of their services.
They're trying to help you save money so
you can, you know, use more of their
services for a longer time. They want
success stories for the humans.
>> Well, that's the trouble with AWS Bench.
I don't think it has any of that uh
value. Like I don't think I mean I
haven't looked at every scenario, but I
don't think it's it's saving you money.
>> No, it's not it's not about that, right?
I mean, it's about making the
customer experience as good as possible,
right? Whether that is by saving you
money or by making sure that if you use
Claude Code uh that that if that it can
really quickly help you troubleshoot any
issues with your website if you're
running or or your web services on top
of them. Right.
>> Yeah. Okay. Fair fair. But I'm
definitely more in the whole best
practice
guard rails
mindset of course but you know I get it.
So, so what I did use AWS bench for is
to test how good
uh how easy different infrastructure as
code frameworks
can be used and how token efficient they
are uh compared like I'm only testing
Claude Code as a harness I'm testing son
uh and I am giving it tasks to generate
code right um you need to build an API
gateway with lambda here is the paths
and the the you know here's there's a
couple of lambda like function bodies
and you need to hook them up for
implementing this rest API um and so it
go and generates terapform configuration
so on one I I have three arms right arm
number one it's terafform
I want to have two versions of it one is
raw no modules the other one is with
modules and ideally with also skills so
that maybe Anton Babeno's terraform
module skills so that that actually uses
those.
>> Okay.
>> Right.
>> Uh so I I currently
the second arm is AWS CDK and the third
arm is my Terara constructs library
which is CDK.
>> If you can compare them that that would
be amazing.
>> Yeah. So so this been already three
weeks right and every time I have a bit
of spare time I I go and prompt it. So
I'm not actually directing much. I I
sorry I'm not actually reviewing very
carefully just reading what happened and
what is the next phase of scenario
writing and things like that. So I can
tell you some of my observations so far
which is really really cool and really
interesting and completely against my
expectations.
>> This sounds good. This sounds good. Hold
on. Let me just double check. I'm
expecting someone to come around at some
point. Okay. Go. You have my undivided
attention.
>> Okay.
>> Surprise me. So of course I had to came
up with some uh scenarios actually in
AWS bench they the scenario is what what
this how you um build the AWS account
right you have one scenario that sets up
a broken um load balancer and all that
um scenarios in my case don't really
apply because I didn't have um you know
pre-existing infra tests although that I
added them now it's called brown field
so initially it was green field right
nothing is in the account it's a brand
brand new account here's your task so
there was called one scenario called
anchor which had nothing in it like it's
an empty account. The second uh
component within AWS bench is a task
right within a scenario you have couple
of tasks.
>> Exactly.
>> So so what what um Fable built was a
task generator that would take one
template task build API gateway uh and
then that would duplicate it with some
injected additional information like
build AWS gateway with Terraform raw.
And here is your environment build AWS
gateway with AWS CDK and here's your
environment for that. So each one would
be basically it generates all the tasks.
So if I have if I have three tasks it
would generate three * 3 nine tasks
right
>> then it needs to run the task
>> but
generating the task I mean the the tasks
should be
>> using the template it generates nine
tasks.
>> Okay. one for Terraform raw, one for AWS
CDK and one for Terra Constructs. Okay.
>> And then it runs the task which is
called a trial. So when it runs a trial,
it provisions the account. It runs uh
Claude Code and it it tells it like exe
it pro it invokes Claude Code with dashp
to
>> so provisioning the account is is it all
isolated somehow?
>> Yeah. So
>> when branch runs
>> it it it runs against my AWS
organization like I I I have an orc
management account and then it creates
like under an OU it can um leverage AWS
accounts. It has one anchor um scenario
account that it reuses for the tasks. So
it does have like a wipe mechanism. It
does determine if it's like if it's read
only like for example it writes the
configuration and it runs the plan but
it doesn't deploy which is not what I
want. I wanted to deploy but if it's
read only then it will it will run
multiple in parallel because not nothing
can conflict with each other right
because uh anyway so so it created this
really complicated verification which is
called oracles so once the the cloud
code reads the and generates
configuration it then runs OPA policies
over that to validate that it it match
matches certain criteria so so it has a
task and then it has criteria right you
need to build an API gateway these are
the routes that need to be in there and
then it runs the OPA policy as the
oracle to say like yes uh it's kind of
like a judge
>> you use OPA that sounds very
overengineered but like all right I mean
why do you use OPA is it is that
something use at work
>> right so so anyway that's the setup
right so let's talk about what it is
observed so far which is very very
unintuitive I would expect that if you
give it a very powerful uh L2 construct
library like Terra constructs it would
work better than raw, right? Actually,
raw terraform on a green field task is
way more token efficient. It generates a
ter. So and and the reason why it's
interesting the the model Sonnet has been
trained on so much like you know data
from the internet that within the the
weights of the model that is massive um
it can generate terapform very
comfortably very confident you know
green feed is easy right it it just from
nothing so it just starts writing almost
instantly you give it a task it
generates the code it validates it boom
it passes right you give it AWS CDK It's
spending a little bit more time setting
up TypeScript and all that doing the
synthesis
>> and then it generates the code very very
fast. Now give it Terra constructs which
is a niche library that is not in a lot
of the you know it's not used across the
internet. There's not a lot of like
adoption of it.
>> And guess what happens? The model spends
out of 52 turns to complete the task 29
reading the terracon library reading the
typescript interface reading and
understanding how things fit together.
So most of the time the model spends
reading the abstraction. So it's
actually um the my hypothesis was that
it's more efficient to have a li a
higher level abstraction as long as an
abstraction is you know able to cater
for all the use cases because of course
the the quality of the abstraction like
a module versus an actual uh object that
can be you know that that can bind and
dynamically uh you know generate more
scenarios than a than a very fixed
terapform module. My hypothesis is that
the modules are not good and and the
object uh oriented approach is better.
I'm not there because I haven't tested
modules against I'm testing raw against
modules which is kind of not fair. Turns
out the raw is better. Um at green
field, right? So so that was not what I
wanted to see obviously.
>> Yeah, of course. How come you call it
ter constructs? Don't you call it CDK
terrain?
>> CDK terrain is the L1 uh binding
generating library and L2s are basically
working integration patterns. So if you
write terraform
you if maybe apples to apples would be
comparing raw terraform against CDK
terrain because the raw terraform is the
L1 resources within the provider AWS. So
Terraform is using the raw resources and
generating all the config and CDK
terrain provider AWS bindings would give
you the raw resources. I'm actually
asking it to use terra constructs which
is a a library of like integration
patterns. So if you are trying to build
an API gateway that invokes a lambda
then you need to also set up IM
policies. You need to give permissions.
So you need to do a lot of glue of
different resources and and and the L2
take care of that.
>> So there's two two thoughts here like I
think with AI has shown itself to be
kind of bespoke software easy to make
your own software. So like in a way you
don't need it you don't need modules so
much because you can just generate it
from scratch with your exacting
requirements. You can get to this was an
interesting discussion on LinkedIn
yesterday which is you can get to day
one which is like day zero is you have
nothing. Day one you have built your
first uh you know deployment. You can
get to day one with raw terraform sorry
like having an AI generate the the the
code the config you can get to day one
very quickly. But the day two which is
where you need to modify an existing
deployment and you need to adapt to
changes like coming in requirements um
you know compliance web application
firewalls need to be added on top of
things type right.
>> This is when you want to lean on modules
and libraries don't you?
>> Exactly. Because modules and libraries
have been built with with like um common
use cases in in mind and have been
optimized for operational type of
activities like handling change and
that's why you build abstractions right
ultimately
>> you want to encode best practices but
not just best practices you want to
encode um maybe sometimes you will set
create before destroy because you want
to make
>> guardrails you want to create
uh
>> there's guard you can add guardrails on
top of that but u but a lot of times
it's also because you want to encode
patterns, usage patterns of how these
things com, you know, bind together to
to to build your cloud.
>> Something this is it's so nuanced, isn't
it?
>> I think.
>> Yeah. So, so so so basically the first
results were surprising. Uh so I decided
like actually this is not really what I
want to measure, right? I want to
measure how maintainable something is
that you've built. I don't want to
measure how it how quick it is to to to
build the first version of it. I want to
measure how does it deal with change. So
AWS bench uses harbor under the hood
which is a terminal bench like basically
the whole concept of scenarios, tasks,
oracles that's all built into harbor
which is the framework that AWS bench
builds upon and it only uses one
capability which is called a singlestep
task. So that means your task has one
step generate this thing solve this
problem and Harbor supports multistep
that means maybe step one build the op
API gateway step two we want to modify
the function body and trigger a redeploy
and this is a very tricky question
because I know that if you write naive
terraform and you then modify the
function body it doesn't trigger a
redeploy like you must set up a trigger
on your resource and you must use some
type of input like a check sum on the
function body to make sure that when you
change the function body there's a new
deployment being generated and that type
of of of knowledge operational knowledge
is encoded within AWS CDK so if you use
AWS CDK to build an API gateway and you
use the lambda function L2 construct
when you bind them together it will take
a checksum of the body of the of the
lambda that means if you change it it
will automatically trigger a roll out
and redeployment of your API gateway
something that you don't get if you
write if you're not aware and you write
raw terraform.
>> Exactly. Exactly.
>> That's the type of thing I want to
measure. Right.
>> Yeah. Those those sort of like gotchas
that just come with experience.
>> Yeah. And and and that are encoded in
these abstractions. So these these uh so
what I then did was first off I I
embedded uh the multi-step trials. So
not just single step trial but
multi-step. I removed because uh also
notice that when Opus wrote the task it
wrote right you're going to build an API
gateway and you must make sure that when
we deploy a new lambda like when we
change the lambda that it redeploys uh
correctly uh so that that you know it it
kind of like led on you know some
knowledge that sometimes you don't know
right if you give it to someone they
don't know that so I I I I make it
multi-step so the first step is hey
built this API gateway then the second
step is here's your the thing that was
built before so it's kind of like brown
field uh some of it's already deployed I
want you to change the lambda function
and then redeploy it and then we do the
oracle verification did it really serve
the new lambda bundy like is it properly
done and and so they will verify and
then realize it's not and then it may
take extra turns to to encode that that
things
>> yeah I guess what you're trying to
measure here the operational nuances I'm
sorry you got my head in a spin there so
I'm just thinking of other cases
>> that's what I need I need other cases
other well I'm just thinking of other
cases I mean I've had I mean I'm like a
48 year old
>> so do you know what I did to get other
cases
>> that's been working infrastructure for
decades and this there's a ton in a way
but like
>> yeah so you know what I did to get other
cases
>> okay what did you do
>> so I I spun up through uh two sona
agents to uh to go across terapform
provider AWS issues and look for u open
issues or maybe issues with a lot of
reactions uh basically from a users
usage perspective right what are some of
the common pitfalls so I asked you to go
through the Hashi Corp community forums
through the Terraform provider AWS
GitHub repository issues and of course
to be fair to look at the AWS CDK issues
as well cuz maybe there's some
abstractions that don't really work the
way people expect and and it came up
with 225 scenarios which then I had to
go through.
>> Yeah. But I I'm sure a lot of many of
the scenarios are are probably outdated
because of
fixing things and
>> yeah so but no the funny thing is that
sometimes terapform provider AWS
maintainers will say works as expected
and they close the the GitHub issue
right or they tell you you need to put
in a trigger with a check summ on the
function body that's what I was looking
for like where is this things that you
find out
>> like when you've already built your
infrastructure and then you go and
search on the Google and then find where
you find this thing. So I was like,
where are those things? Where is this
operational knowledge? Well, it's in the
GitHub issues, isn't it? So that's where
I I mined them and then I sorted them.
>> So deep.
>> Yeah, I sorted them by reactions like
how many comments are there because some
of them are like, you know, like, hey
guys, we really need this and then they
close it anyway. And and then also when
was the latest activity because
sometimes it's a capability that was
added and then merged and released in
version five of the Terraform ABF
provider for example. Uh so there is a
way to do it. For example, there's
default tags. Apparently, there was a
bug with the way that the defa the
provider default tags uh were not
merging correctly like in version five
and I think they fixed it in version
six. So so finding that scenario is kind
of like useless, right? Because I know
it's fixed in version six unless the
unless cloud uses Terraform version 5
pin somehow and then it still, you know,
gets that issue. So So I got 225
scenarios. I think I went and picked 30
of them sorted by last activity and
number of reactions like signal. It's
funny because when you talk to fable it
comes up with these words, right? Oh,
this one has high signal because a lot
of comments and then when was the last
one?
>> This is awesome.
>> Yeah, it's really awesome because how do
you go through 20 225 scenarios?
Markdown isn't it, right? So then I
said, "Hey,
>> and I think I sent you that message,
right? turn this into an artifact, an
HTML that I can see the scenario, I can
leave some comments, and then I can
click a button and get all of the
comments and then feed that back into
Fable like, "Hey, actually, I got 30
scenarios right there that I want you to
have a a deeper look at."
>> Yeah.
>> Um,
>> yeah.
>> I'm just I'm just thinking aloud. Is it
is it actually worth doing all this
stuff? Because there's a part of me that
that thinks that like that
>> it it depends on what you
>> some anthropic engineer is just going to
train the LLM on all these
>> these things. Oh, wait. I I think I need
to go. There's someone.
>> Yeah. So, so why do I do this is because
I have a library that I want to I want
to put some actual data behind behind my
words, right? So, yeah, more to more to
follow. definitely will share when I
actually have. But the first finding was
already super interesting.
>> Just unexpected.
>> I'm glad you're looking. Well, in
hindsight, you could argue not, but
anyway.
>> Oh, yeah. Of course. Of course you knew
it.
>> I I knew it. Yeah. Anyway, really
interesting stuff, Vincent, as always.
Thanks again for having a chit chat. See
you.
>> Yeah. Yeah. See you.