Episode 36: Agent loops

And we're starting. Well, it's been a
while since we last talked. I I kind of
made an agenda on that AI infrastructure
podcast thingy, the the Google Doc.
Remember that one?
>> Yeah, it's great. I didn't read it.
>> I'm sorry. I don't know if you
I don't know if you want to talk about I
mean, I just rambled a a lot a lot of
bad stuff, like maybe it's worth talking
about the most obvious thing in the
room. I mean, what what do you What was
your take? What what the elephant? What
I I was using Fable. I was enjoying it,
and then it got pulled. And now what
what's your What's your hot take?
Aliens?
>> Aliens. What did aliens do wrong this
time? They keep getting, you know, stray
bullets, catching bullets stray.
Catching strays. Geez, we're very bad
with like, um, what do you call them?
Idioms?
Um, what was
No aliens?
Uh, I have
heard some theories. I I think some of
them are far overblown. Um
I mean, I hear the the
the standard mocking. My initial
response was like, I want to like, right
after it happened, I went home. I I I
read the news. Um, it's was evening
Friday in in US, but it was Saturday
morning in Vietnam. I had just finished
my breakfast. I see a post 25 minutes
ago.
Um, they're pulling Fable.
Um, I was eating breakfast with my
family. I got home. I go to my machine,
and I had a session open with Fable, and
I said, "Can you just spin up a dynamic
workflow real quick?" Hoping that maybe
it wasn't revoked yet, and I would still
get a dynamic workflow out of it. And it
says, "Model not available." And I was
like, "God."
Ah.
Yeah. Then, funny enough, not long after
it asked me, "How is your experience
with with Claude?" And I went, "Bad."
>> [laughter]
>> And that's how far it went with me,
because the next day I was already using
Opus again. Yeah. I mean, it's my
experience with AI can be a bit weird.
It's like, I have these good sessions,
and I have these these sort of myriad of
like bad labyrinth confusing sessions.
And
on Friday afternoon, I I I vibe coded
a tool to
help me take
help me annotate screenshots. I did that
with Fable and I one shot at it and then
you know these experiences become
become legendary in in your mind, right?
Yeah, Swift. I like the problem was
pretty simple and less than a thousand
lines of Swift. Uh
create
Uh
you actually prompted less than a
thousand lines of Swift?
>> Yeah, exactly.
Exactly.
I'm I'm I'm big on that's my technique.
I'm I'm patenting patenting
>> So there's this one thing
>> I tell it to lock of code
key to lock of code lines of code down.
>> There was this one
um
hype skill recently which was ponytail.
Have you heard of it?
>> So ponytail
>> and and it has like almost like this
meme type of drawing. It's a bit
negative in the way it describes the
skill. It's about every team has this
experienced season dev when everyone is
like creating these over engineered
solutions. He just looks at it and cuts
it down to like just one or two lines of
code and then they have like this
ponytail do we like uh you know
type of behavior. So the skill is is
exactly in alignment for the model to to
focus on reducing lines of code and
getting rid of like over engineered
solutions and has a whole bunch of like
rules around that. And I have some very
interesting experience because I run a
Hermes agent to review pull requests and
I spend quite a lot of time building
pull requests and then having the agent
review them running simulations. I find
it amazing. I can be again I can be
outside
>> You mean simulations like it's deploying
it and testing it or something?
>> Yeah, like so the heart so the the
danger is to get an agent to just rerun
the CI/CD test, right? That's kind of
useless unless there's a CI/CD test
failing and you need the agent catches
it because he's able to reproduce the
failure locally.
Or he can detect that locally it passes
and the remote CI fails, then he can
identify something wrong in the CI/CD
workflow. So there's there's some
benefit of it of it running, but it
shouldn't be just always run all the
CI/CD locally because that doesn't make
I mean a lot of make a lot of sense,
right? But in terms of simulations is
because this is a Hermes agent and I
don't know is my connection active?
>> I can't see your screen if that's what
you're saying.
>> It says trying to reconnect, but I mean
it will be buffering my video feed and
it will come back, right? It's a bit
weird.
>> Yeah, I can see it fine. I can see it
fine.
>> Am I on a VPN? No, I am on the normal. I
have two Wi-Fi's, one is VPN, one is
without. I'm on the normal Wi-Fi. So,
the the the Hermes agent What is What is
the real benefit? My god.
>> Who opens the banana from the other way
around or is it just me? It's really
hard to open up a from here. Maybe it's
just the
kind of banana I'm I'm eating nowadays.
Okay, but whilst we wait for Vincent, I
can maybe share my screen and show you
and show you what this annotation tool
does. So bounded to hyperkey A,
it just
it creates this little window with the
screenshot and then I'm able to press A
for arrow, B for box, or T for words.
Hello.
Of course you're like, "Oh, how do you
change the colors?" Oh, you just you
just change the source code.
And and then when I dismiss it, it's in
my clipboard and ready to be pasted and
you might be wondering what ones if you
make a mistake and you have to move
things around.
You just control Z. Simple. And the
really good thing about it is also that
if you take a screenshot, you can just
paste it into the same
um annotate window. Um you can just
paste it in here. I think it's I think
it's brilliant. Brilliant. I dare say so
myself.
I dare say so myself.
>> [sighs]
>> What else did I want to share?
>> Pinsel, you're back.
>> Yeah, but I'm on a hotspot. I don't want
to do that. I don't think the data's
going to be good enough anyway. Um but
What?
>> What is going on with the internet? It's
some Someone bust your supply or
something?
>> I don't I don't know. The The other
machine is on a on a stand-up. So, it's
works fine. I could Anyway,
>> You're You're on a You're on a You're on
a stand-up at the same time.
>> Because I These stand-ups take too long,
you know? Keep them to 15 minutes. I
tell you let's meet I think a stand-up
should be 15 minutes. I agreed with you
to meet. I'm already 5 minutes late.
We're already 5 minutes over. Stand-ups
literally finished right now. So, I've
I've said my piece. Uh there's nothing
else for me to say.
>> The The The worst about stand-ups is
like when there's no real updates, you
know, there's no real progress, but no
one ever that seems to admit it.
>> Yeah, that's the I mean, I was I was for
I was waiting for the completion of the
sentence because there's no real
updates, but we're still spending 30
minutes talking about no real updates.
>> Yeah, let's get back to work sort of
thing.
>> Yeah. No, what I was saying is that the
the the simulations, for example, I I
introduced release-please bot, which is
a Google GitHub action and an and a and
a local CLI that you can run that
that does a standing release pull
request that tracks main and then keeps
updating it as new features land on main
and then uses conventional commit to
generate a change log and it then allows
you to just click merge and then boom it
detects that the the the standing
release PR was merged and it then means
a new release and creates your change
log, which is I think a really nice way
of
A lot of libraries work that way, right?
Main is is is not constantly releasing.
Some Some are always releasing. Every
Every merge to main releases, which
which can be okay, but then if you have
but dependable merging a lot of fixes,
then you get releases non-stop, which is
also annoying. So, release-please is
pretty nice. And there was this manual
process
>> Release-please, maybe you should share a
screen, though.
I don't need Are you
So, this is this is the this is the this
is the latest in tech for for what? Node
modules or
>> No, no, release please is really old and
it's from Google. It's It supports
Golang, it supports Python, it supports
Terraform modules if you want.
>> Go projects I'm using Go releaser, old
school.
>> Yeah, Go releaser, but Go releaser works
really well with release please. So, so
you can combine those. Because Go
releaser maybe you will release oops
uh on every PR. So, basically on the I I
noticed I set up an agent and I noticed
manual activity and of course manual
activity and and and how do you call
those in the SRE um toil? Toil is what
you aim to reduce, right? Every cycle
you should spend a certain percentage on
reducing toil. That's one of the SRE
tenants. And um so, there's manual
release PRs being created. This is an
example of it being integrated. I don't
like that it's um it's collapsing the
release notes. But, as you can see that
this is a standing PR that constantly
gets
um updated each time a new
>> can you define standing PR? Something
that's a bit long-lived? What does
standing PR mean?
>> It's It's a PR that stays open until you
want to cut the release and it keeps
tracking head. So, it gets rebased each
time new features land on head and then
it updates the the features that are
going to land, uh bug fixes,
miscellaneous. So, that's what you
>> do you It's like a way of having like a
uh a change log or
really It's like you're doing your
release notes in an issue, right? And
then you then when you hit you smash
merge, that becomes your release notes.
Ah, I don't think I've ever done it that
way.
>> So, I've used I use release please in
quite a few repositories and I really
like it. So, so this was a manual toil
and my initial reaction what was to say,
"Hey, um my my agent can do this, right?
My my my beautiful AI agent can can
create um help you create these release
PRs until I remember of course there's a
tool for it. Why would I let AI do
something that's very deterministic and
a tool exists and it's really well
established tool. It's a really nice
tool. So I said let's me let me create a
pull request to add release please. As
you can see this branch was created by
Claude so I was on the Claude code app
when I said I want to integrate release
please into this into this project and
Claude created this pull request. And
and then
>> Wait how do you know it's created by
Claude again sorry? Uh I just lost that
bit.
>> I mean
it just shows here that the branch is
Claude slash
>> It's only like a small branch name.
>> Yeah.
>> How did you know Claude did that? Did
you I guess you must have told Claude to
do that okay.
>> And then another thing though uh this is
Claude code. So you can actually see the
session here. So you And this is this
doesn't link to like your question was
like I want to see the session that
created this PR. Claude code does that
but it's not public. It's only me and
only I can see it.
>> Is this Is this a new thing? I've never
seen this before.
>> No. Yeah yeah I've never I hadn't seen
this before. So here I go. I just uh I
opened the app on my phone and I said as
you can see it's very short so because I
was on my phone I typed this manually
like a caveman. Uh everyone left the
stand.
So I said set up release please. Ground
yourself in latest release please
version best practice uh and this is an
NX learner yarn mono repo. It actually
found out very interesting stuff because
normally release please in a mono repo
it will version each package
individually but this particular
repository has a has a standard version
at the root that stays at zero and then
it uses a special CLI to align the
versions and it then also mints every
package to the same version when it
releases which is again not something
that you usually do out of a mono repo.
So So there's something I didn't think
about and Claude code um surfaced that
early and I said I I think it prompted
me for some questions uh so I said I
want additive coexist. The existing
workflows called to century blah blah
blah. Oh I wanted it to be aware that
about some things things that this thing
does. It's it's a little bit weird. We
inherited this from HashiCorp. And
I just gave it a bit of hints here. And
then after I was happy with the PR, it
went ahead and created the PR. Now,
where does the agent come in, right? So,
the first thing that I found what the
agent did really well was
Claude Code didn't pin the
the checksums of the GitHub workflows.
And so, I asked we did not I didn't want
>> You can get deterministic tools to do
that, but carry on.
>> Yeah, so that's exactly what I do here.
And I did not want to integrate this
more into this repo because I want to I
don't want to keep adding on more and
more tools around it. Not yet, at least
not in this PR, right? So, so this is
one of the things that that the agent
was able to do for me while I was having
a coffee. I told it I don't know if I
can find a Discord thread. I could I
could probably go find a Discord thread.
>> Okay, I just Okay, I'm just I'll just
want to replay this. So, you're asking
uh you just you just prompted the AI to
do the Sismore
uh
what do you call it? Lock in the on the
GitHub action. I'm still a little bit
puzzled why you why you even want to do
that, but I mean it looks a bit
overkill, but carry on. You there,
Vincent?
>> It's not overkill.
>> [laughter]
>> Um basically basically a code agent is
not very deterministic and I don't trust
it to put in the right checksums. So, I
told Cloud to leave it at the tags
mutable tags. And I told my agent to
pull out the branch and run Sismore. And
Sismore found a ton of issues with the
GitHub workflows, right? So, it ignores
every other finding. And I told it to
only deterministically pin the checksum
of these mutable tags. That's all I
asked it to do. And so, the agent was
very very simple. It it gave a a diff a
patch. And then I went back to Cloud and
I said, "Hey, there's a common common
this is the actual deterministic
determined checksum for you to pin.
Okay, so don't I don't trust you. It's
funny I trusted to do a lot of things
but I didn't trust it to do that, okay?
Um but okay, this is not the only
scenario where I find an agent and being
able to just execute whatever you want
because again, Claude Codes sandboxes do
not download binaries and do not uh let
you do whatever you want, right?
>> Mhm.
Sorry, um I may be I may be I'm just
being pedantic here but like you you
you're doing this
this pinning just just for the release.
This
uh you're you're generating the
>> the repo.
And Dependabot is configured to update
them. So, we don't own them anyway. So,
the only thing I wanted and I think
Cloud delivered a PR where the workflows
weren't pinned. Uh in any case, I think
I I've made it a habit um to ask the
agent the review agent that basically
runs on any uh pull request that is
whitelist sorry, allow listed. Um I ask
it anyway to always do a baseline versus
PR head check if the PR is um touching
files and introducing significant new
regressions against security. So, the
agent is always going to do this now. Um
I think at this stage it didn't do this
by like automatically but I made it a
rule that from now on the agent will
always do this.
Um but then, you know, Claude Code picked
it up.
>> I mean let me stop it. So, you're doing
this so that you you you can improve the
security posture of your of your GitHub
workflow by making sure that the actions
can't be injected and they're they're
more dialed in and locked in so that
they're they're less susceptible to
attacks. That's why you're doing this.
>> Yeah, in this day and age of supply
chain attacks, we we really have to
improve our CICD security stance
definitely on GitHub workflows. So,
the biggest issue with a public project
like that is if when you do PRs like
that, they get they get to they start to
introduce too much change. And this is
one of the things I want to touch upon
because AI agents are creating massive
PRs. And one of the things I learned
with agents as well is to really find a
way to to make PRs like this ties into
your earlier comment about reducing the
the number of lines and this ponytail
skill which I want to get to eventually
if you let me get to that point.
Um, right? So so the point is I'm trying
to explain to you that the agent is able
to run simulations that I cannot do in
Claude Code because Claude Code is a
provided sandbox by Entropic that
doesn't download binaries. It doesn't
install uh things. It doesn't run a
whole bunch of things. And I can
delegate those type of simulations to my
um agent that's running on a box that I
control. That's the main thing I'm
trying to say.
>> Okay.
>> Aside from all of this pedantic, you
know, narrowing into this particular
case, I'm just giving some examples and
this is just one of them and I haven't
even come to the second one.
>> Can you show the Can you actually show
the simulation run? I'm still a little
bit
>> Let's focus on the second one because
running this more
uh on on a branch and then you focusing
on the actual diff is is is not that
interesting as what I actually the
second simulation does. Um, so the the
the agent is able to run a bunch of
simulations
and it found some other issues, but then
ultimately I think what was the most
interesting one was this one.
Where I ask it to merge locally the PR
into main and run release, please to
actually see what the effect is. Which
this is something as a maintainer that
you sometimes would need to do, right? I
mean, if you if you're getting PRs, if
the code looks good, but um until you
actually check out the branch and maybe
run the tests, you don't know some of
the like changes work the way you expect
them to work. So again,
um I would really love to show you like
the Discord uh if I have Discord here,
but
>> So
>> I don't have it here.
>> understanding you right, Vincent, that
your your simulation is like replace
Well, not replacing, but augmenting a
human
to do to do some um to do some smoke
tests and to prod around and do things
that like a a deterministic test
couldn't hope to do, right?
Yes. It's probably named
it's I think it's called exploratory
testing, isn't it?
Yes. So so I want to show you the thread
of of So this is all of the PR reviews
that my bot's doing. And at the time it
didn't have nice threads per PR. Now it
does. I I I failing to find the find the
search button cuz I actually want to
show you the prompt that I sent from my
phone
to for it to do basically do the
simulation that I asked it to do, right?
This this particular one right here.
I asked it merge this PR in in a like a
temporary directory, merge this branch
into main, and run release, please.
And it actually highlighted a whole
bunch of things. Like first of the title
didn't meet the the the the convention,
but it has a PR title linter in this
repository, right? It has a regular
expression that runs across PR titles,
and the release please configuration
created a title that did not meet our
expectations or our our requirements.
Like the description of the PR said it
will create a PR like this, but the
actual created dry run output was
different. Right? Yeah, and another
thing it found was
Yeah, the this is where it says the
format's supposed to be type scope
message, and this doesn't match that.
What we expect is we should probably
have something like that. And then
another thing was
um
a problem. There was another problem, I
think. I mean, this still seems a little
bit like pedantic territory, but but
okay. pedantic? This is this is Okay,
I'm taking offense to this because this
is exactly the type of thing as a
maintainer you need to do. And I want to
get this stuff through because this is I
want to get as much validation around
the pull request as I can uh, without
having to leave whatever I'm doing, go
to my machine, and and go and and, you
know, check out the branch. Yes, you can
say like, of course you can go to your
laptop and set up a temp directory,
check out the the the branch, merge it,
run the dry run, look at it. And and it
would have definitely had values because
I there was actually something that came
up after this PR was merged, which
proves that no matter how much
validation you do, we'll still have ex
surprises.
Um, but I don't think this is pedantic
at all. This is literally like, um,
changing my life in me not being, you
know, required to go to a machine and do
the checkout, right?
>> Yeah, I guess I guess you're right. For
for maintainer doing releases, if you
can if you can sort of automate
>> on this particular example. Do the
release please part is the is the minor
part, okay? The fact that you can get a
pull request from a contributor and run
a simulation, no matter which type of
simulation, is the interesting part, not
the fact that I'm doing a a release
please merge and then running the dry
run of release please. That's just a
minor thing.
>> I'm just I mean, yeah, I'm I'm I'm
playing devil's advocate. I I I I am
curious why you call why you call it a
simulation. By the way, which agent are
you
>> How is this not a simulation? Explain it
to me.
>> I I know it's I just think it's like an
interesting word, like simulation. It's
it's sounds
>> Give me a better I'm not English native.
Give me a better word for what it's
doing here.
>> Um,
>> A dry run?
>> Well, I
>> A simulation, right?
>> I mean, I was I was thinking exploratory
testing earlier, but you
>> What is exploratory testing? I don't
know this
>> It's when
I think it's a I think it's a thing.
>> Sounds like freezing.
>> Exploratory,
if I can ex- if I could spell
exploratory testing. Hold on, let me
share the screen.
>> Okay. I don't think the term matters
much. The the thing is, I
>> well, I mean, you Exploratory testing is
is a big part of what we do. Hands-on
software testing where we test is
actively explore an application without
predefined test cases.
>> Okay. And what is the problem with the
term uh, simulation? Like what what what
is exactly
that that that is the wrong use of that
word. I I don't really see how that's a
wrong
>> Well, I mean simulation it's it's
interesting. I just I just think of the
matrix, you know, I'm just thinking like
now you have agents simulating your life
and
I know, it's it's a bit broad. Anyway,
it Okay, it's interesting, but I got two
questions for you. Which agent are you
using with Hermes? Second, on this new
gig, to my surprise with GitHub Actions,
I just wanted to note that like they
they disallow anything that doesn't
start with action slash. So, any like
third-party actions are not allowed.
You're only allowed to use the action
slash, which I guess is supported by
GitHub only.
I just thought I'd mention
>> GitHub scripts? Yeah, that starts with
action slash. That just gives you a a
major gaping hole in that.
>> Yeah, because
>> That's just hackliest security to me.
>> Yeah, that that's it's interesting
because like I couldn't install I think
it was pre-commit and I couldn't install
UV. So, what what you end up having to
do is basically using actions Python and
then and then pip or or you don't even
need actions Python, but you you end up
just using pip install and your your
your actions become become harder to
maintain because because you're not
using the proper action.
Okay, but yeah, what what agent are you
using with your Hermes?
>> Why would I not want to say? I use
because with ChatGPT, okay, Anthropic
does not allow you to use its its
subscription plan with with these
agents, right? Open Open Claw and
Hermes, you have to use the API key. If
the Claude Code harness detects activity
like nested calling and and things like
that, it will automatically use API key
wallet instead of your actual
subscription usage. Like they put a lot
of things around it. Yeah. Uh so, some
people got a surprise I think I got this
from reading the Hermes Reddit threads
where people were saying like how do I
why am I getting paid like charged why
is my wallet going down when when I'm
using 12 C on Hermes. It detects these
things, right? Initially it
automatically banned your account. I
guess they went back back from that and
now they are just trying to make sure
you pay.
>> Well, yeah. I I'm always in favor of
monetary incentives or penalties the way
Singapore does it.
>> Yeah. So so what I really wanted for
these
Oh my now it's saying my phone is
running out of battery. So so basically
what I what I really wanted for this
experiment was to leverage the chat GPT
Codex budget with a simple plus
subscription you get a certain amount
and so Codex GPT 5.5 on the first week
it ran out after 3 days and then I have
a a fallback. So with with these agents
you can you have your main harness that
does most of the work like tool use and
and reasoning and so on and then you
have auxiliary calls such as summarizing
the conversation which I think is the
one of the things of
>> The hardest Your internet is going to
hell.
>> Yeah, the hardest thing to get around
>> If if
>> It's it's using my phone. My phone isn't
that it's just charging. My audio's
going to come through fine. It's
recording from my laptop anyway.
>> Yeah, yeah. Your audio did just stop for
a second or two. Don't worry. You could
continue
>> recorded on my my laptop so so it's not
a not not not a concern. So what I was
saying is that with these agents they
use different API different LLM
endpoints based on what they need to do.
So auxiliary functionality such as
summarizing sessions can use a different
endpoint. I'm using Deep Seek Flash for
that and main agent is Codex GPT 5.5 on
the chat GPT plus so you get about 3
days out of a week with Hermes agent or
moderate use and then it falls back. So
the moment the chat GPT API returns I
don't know what is the HTTP the HTTP
code for you you your limit. At that
point do still hear me? At at that point
it falls back to DPC.
>> So, I'm I'm trying not to I'm trying not
to interrupt you so that the the flow is
>> DeepSeek 4 Pro.
>> intelligible while listening.
>> DeepSeek, okay.
Yeah, it works.
>> Oh, yeah, you mentioned the the
>> Let me try and and I don't know if I
should reconnect, but
I don't know why my phone is now super
slow. Maybe my data is
>> can try switch back. This still
work.
>> Okay, I might drop out in a bit.
>> it will work if you want to come back.
Try Try and switch back to your fiber or
something. Just try.
>> Test.
>> Okay, you're back. Yeah, it looks a
little bit more stable.
>> I'm on the Wi-Fi now. There's no other
laptop on the stand-up anymore.
Hopefully it should be better.
>> You don't I guess you don't have wired
in your in your home for your
>> wired?
>> Wired, like like a actual physical
network.
>> I do. I do. I do have a switch on this
desk, but with Apple you need to then
like connect a an add-on with Apple.
It should be fine now. The connection is
stable.
>> I I use my monitors for like network and
powering, you know, like one cable.
>> One cable. Yeah, so I think the most the
the most difficult part of switching to
agents and then you have Discord where
you send a message or you mentioned it
and then it creates a thread is to map
that back to the session. Because if
you're on a Claude Code, you're in a
session, you see your context usage, you
can you use sub-agents, you can see how
much context you're using. When I have
Discord and I ask the agent to do
something, I can't see none of that. And
I'm also using it more like like a cloud
code in in Discord rather than than an
actual independent agent that finds work
to do and does its own work.
>> That that cloud chat, the thing you
shared, show show all the tool calls. I
don't think I don't think it does, does
it?
>> You want me to share the screen?
>> Yeah, show me your your Claude trace.
>> It said run command.
Run tool command user tool check status
run read tool.
>> If there's if there's any takeaway that
I've learned from talking with you this
morning is that the
the Claude AI session
um transcript there is a lot better than
I remember.
>> And I never saw it before and I asked
someone who has Claude's Enterprise and
I said, "So, if it's Enterprise, does it
allow the team to see other people's
session?" And he says, "No."
So, I thought that's a little bit That's
sad.
>> That's really cool.
>> It's all the the the questions and
everything.
I'm I'm sure they'll make this better.
>> was only um It was only the other day I
saw the GitHub Enterprise uh control
panel for the first time. I think it I'm
going to maybe sign up for a GitHub
Enterprise just on a personal basis just
so I can I can get an idea of all the
changes that are happening on the
Enterprise panel because I I didn't know
about all the controls that you have and
I need to know because I'm a I'm a AI
guardrail
engineer right now.
>> Yeah. Forcing people to use actions is
such a stupid security rule rule. I
mean, I think what you should do is to
have a white list and an allow list of
actions that are supported and then a
process of requesting um a new action to
be used in a workflow cuz like you said,
you're going to end up with like
workarounds, unmaintainable workflows,
and then even if you use this script
which is runs JavaScript, you can do
whatever you want and this isn't actions
one. So, security
>> Yeah, you can do what whatever you want
with actions / Python really or you
know, you can There's a ton of ways to
get around it. Yeah, I mean, I'm not I'm
not supporting this whole actions prefix
thing. I just think it's interesting
that it's even done that way.
>> Yeah, and it doesn't work very well. And
this is something I I I learned a lot
about and you see a lot of solutions now
about not just these GitHub actions that
can be used, but also what skills you
can use or what MCP service you can use
>> Yeah.
>> in an organization and
>> that that's what I'm I'm currently
working on right now is is is allow
listing MCPs, allow listing the tools,
whether the tools are blocked, whether
they
require approval, or whether they always
allow. I'm doing This is my This is my
day job right now, seriously.
>> Yeah.
>> And it the the the magical I I mentioned
it on a tweet, and I think you replied
to the tweet, but the the thing that
clicked for me with MCPs is that which
it and it we know we've been
bad-mouthing MCPs for a while, but it
really clicked for me that that there's
all the support in the enterprise for
it, so that the user experience is
great. You log into
uh
into your Claude session, and you're
logged into Google Drive, you're logged
into Atlassian, you're logged into
Figma, Miro, all that stuff. It is
amazing.
>> Absolutely.
>> And it it is actually like a marvel of
technology. It almost sent chills up my
spine cuz like in the Google enterprise,
that that that that connector, when you
connect your your Claude to your, you
know, your your Slack workspace, or your
your Google workspace, it's amazing.
It's amazing that that this that this
cross-authentication thing works. I
mean, I I still need to wrap my head
around it because like there's a
connection between Claude and your and
your your asset there, and then there's
also the user authentication and the
users' scopes that also come into play.
It's It's amazing that it all works
until you get an internal server error,
but it's amazing when it works. And MCPs
allow that, and and this is where I
think it will probably tear it will
These are the These are the pain points
that it gets rid of, which is really
powerful. Because with skills, you you
there's there's definitely a lot of
people, including my own my own
colleagues, who would say like "Kai, you
can't have You can't have a Claude just
calling an API key on your on your dot
That's That's a That's You've divulged
the key to Claude." And like, not sure
you're right about that that You kind of
need to have dot M access to get things
done. And of course MCB's just solved
that problem big time.
>> Yeah, I I I think another thing where
MCP's really solved the problem aside
from the authentication and like maybe
secret management. I think I've got a
couple of situations where I really felt
a skill couldn't do what it what what
needed to be done. One of them is the
ability to I mean let's look at the AWS
docs MCP for example, right? It it's
it exposes a it uses a search endpoint
and then it
sources fresh docs from their server
which it's pretty cool and I wanted to
>> That's actually number almost number one
for me.
>> Yeah, but but I wanted to to get the
same thing for another like a vendor
product not AWS. It's a gravitational
teleport which is a remote access
solution or privilege access management
solution. And in this case, what I
wanted was to to have these docs
available to the LLM to make sure cuz I
noticed that the the LLM uses web
search, finds the teleport docs, goes
through them and I found that that's not
efficient and I was wondering if Google
was really giving it like the right or
whatever web search is using, right? I
don't even know Anthropic is using
Google as as a as a web search engine.
It's maybe using something else and and
or maybe it's its own like search engine
because
>> Yeah,
it's a tool or something.
>> Yeah, so so what I really liked about
this when I tried to do this as a skill.
So I I I thought hey, the the AWS docs
MCP is pretty cool, but I want to do it
as a skill and I wanted to to like fetch
the go teleport data like in an
efficient way. So the first approach was
apparently the go docs the the docs they
use like these LLM.txt. So every human
document which is HTML nicely rendered
diagrams has a sibling .md markdown
version that an LLM can use. So I wasn't
even sure if the if the agent was
finding it, but I think this is a
protocol that these agents are adopting.
So, I think that the if they do a
request, I am an agent, they will get
the markdown instead of the HTML. I
think that that might be already in
place. So, they use something called
Inkeep to do that. But then the second
problem was I ingested all of these
markdown into like an index, and I did a
local lexical search. So, to find the
most relevant documents very quickly
instead of having to go to Google web
search. And and then I was like, hold on
a minute, this doesn't take into account
if the term that the that is being
searched for is actually on the page
itself, right? And if I go into the
Teleport Docs and I enter search, it's
actually doing a semantic search. So, if
I talk about like privilege escalation,
it will automatically surface articles
about requests like access requests and
approval workflows, right? Which is like
semantically related, but not exactly
lexically matching. So, so that's where
an MCP can do really nice things because
if I want to do that content-based
semantic search, now I have to like ship
a Chroma DB or a vector DB, and I have
to also be able to embed on the client.
That means there must be an LLM call to
for the embedding model, which again I
cannot do as a skill. So, I worked
around it, but ultimately I did an eval
comparing Google results against my
final skill using Python and using term
frequency a search index, very
complicated like 4 MB JSON file search
index shipped with the skill. And then I
said, "Now eval compared to Google." And
the only difference was the speed,
right? It was just a few milliseconds to
get to the document versus else it would
go to Google
>> How did you do the review? Did you use
that Anthropic review skill?
>> Like I said, because Fable is gone, I I
and and I think they reset the credits.
I just went massive on dynamic
workflows. I said, "Run a full dynamic
workflows." It ran like 15 agents, but
it does use it has a memory and it uses
Sonnet for these evals. And it did a
complete comparison like running the
agent without any skill and and actually
telling it like use web search and make
sure to ground yourself and then it says
and it came back it says and and another
thing I did was I asked go through all
of the documents and in the content make
sure you query something that's in the
content that's not in the title and
something that is semantically like
similar but not exactly lexical match. I
added those evals in so it's like really
tricky and then I asked now one agent
uses Google one agent uses my advanced
non-semantic but a search index based on
content and end result both are surfing
selling the same information and the
Google one is just a little bit slower
and the other one took like took like
you know maybe a minute to build a
search index and maybe 30 minutes to
build a code to to do to build a search
index and then you have to think about
it because if you do term frequency if
you have a change log entry which has
the term repeatedly then you have a
document that has the term mentioned a
lot but it's not dense it's not actually
relevant so then you get all different
problems and you're like
>> Yeah man.
>> MCV solved the problem instantly.
>> Yeah. Yeah I mean it's so I got I I was
my thought was like that whole knowledge
base thing I feel is is like a big
product
it's a big market opportunity right like
there's a lot of people working on this.
You just for context you were working on
this whole knowledge knowledge local
knowledge thing because of some research
you were doing right that's that's the
reason why you did it. So you did all
that work Vincent but then you compared
it to Google and it's about the same
right?
>> It's about the same and the pros and
cons was like it's just a little bit
faster it's offline only but it does the
cons are you need to rebuild the search
index you know it becomes stale and the
shipping 4 megabyte a search index JSON
file with your skill is kind of like
what's the point?
>> Yeah the
the just to play devil's devil's
advocate
what I found with the Google is that
sometimes it's I'm not too sure what the
the metadata you get back from it, but
sometimes you you don't really know why
that result is top. You don't know how
old it is and things like this. I mean,
Google does a pretty damn good job if
you just trust it, but there's probably
going to come a point where
well, especially for like internal data,
then you
you just can't you can't do that, right?
This is Yeah, this is why I mentioned
the product the market opportunity
because like internal data, the stuff
that you need to make your private
business work has to be private, right?
>> Yeah, absolutely. Yeah.
I'm sorry. I just got scolded for I
turned off my camera and I should have
just left the call because there was
nothing else for me to say there and
then apparently they asked and I didn't
respond.
>> I'm on this like new new remote first
job and the funny thing is they have a
lot of they have lots of blocks for
focus time and it is amazing to have
this focus time because now I can
actually do some deep work, which I
don't usually do because I'm I guess I'm
like you in in in previous
gigs where I'm like I have back-to-back
calls pretty much all day and it's like
impossible to actually get anything
done.
>> Yeah, I don't have that many calls, but
I mean, I
I I wouldn't have done this like maybe
earlier, but now I'm like I'm off I'm
I'm handing over and and so on and I
feel like
>> Okay.
>> Yeah, it's I feel it was less important,
but I I got scolded, so I feel guilty
and I'm wrong. I mean, I should have
just
>> So, you you you basically were on a call
the same time you were doing this
podcast.
>> I had finished. I should have just
dropped. It
Yeah.
>> You should have dropped. You should have
dropped.
>> Yeah, I should have said, "Guys, I need
to go."
>> Important stuff.
>> I should have time box this better. I
need to be more Communication is key,
right? Setting expectation is is what my
What was the mistake here?
>> Setting expectations, also working out
what Yeah, what the expectations are of
the person on the other side.
>> It's just that it happened twice two
days in a row now. It happened yesterday
and happened today again.
>> But you're you're rolling off, so but I
mean you should always keep keep in very
good standing
with your colleagues. Yeah, you're
messing up Vincent. I mean, do you
do you want to should we end it now or
did you want to carry on or
>> There's nothing else. I apologized and
there's nothing else I can do. But in
you know, to come to the like internal
internal knowledge bases is is actually
the very interesting use case there,
right? But you wouldn't scale you
wouldn't ship a skill and an MCP is the
right answer. I think I I've also
learned so many use cases where MCPs are
a much better fit. And because I'm
working on a on learning more about
agent core and its capabilities, it has
this really interesting agent registry
and agents registry policies, which
allows you to control who has access to
what and which agents can use what tools
depending on the identity of the of the
user. So if the user is prompting the
with an agent, then
back to what tools the agent can use. So
it shouldn't be able to to query the
financials of a database or knowledge
base when the user is shouldn't have
access to it. So things like that are
are possible with the with the policies
around and the user identity like the
identity system around it. And then you
can do that with with MCPs. It becomes
really interesting and I think that's
definitely something that when you're
looking at deploying agents across
across enterprises, you need there's a
lot more than I've got I I've run rag
vectorized and added all of the
knowledge and my agent has it and I'm
good to go.
>> I I talked about CQ
to you before, but the CQ is one that I
I I came across when I went to this this
uh AI conference in London. And and it I
wonder if the the MCP you like like if
you just think if you just is this the
right Yeah, I think this is what I want
to talk about. If you just think about
like what every say say you're rolling
out in cloud enterprise to your thousand
employees, you want an MCP so you can
query your knowledge base. Okay, that's
that's like number one. And then number
two is that you want your employees to
propose new knowledge
items. You got that. Um but what I think
that a lot of knowledge base systems
miss out on, which CQ does really really
well, I don't know if your knowledge
base
does that, is that it allows you to give
feedback on that knowledge unit. So, you
can say like, "Hey, this knowledge unit
unit is actually good and useful."
Or or flag that it's actually crap. And
you can you can even see
um statistics around that the usage of
that knowledge. And that's that that
that feedback design in CQ is absolutely
next level compared to anything else
I've seen.
>> Right. I think this is what I also
learned about like when I did vector
embeddings and and and retrieval
augmented which is very like type of
memory system
versus what I run recently for a local
knowledge base that I was playing with.
It's more like a memory system, but it's
those dimensions that that that play a
part in like recency of the of the
memory, relevancy, conflicting memory
facts. That's what like proper memory
systems actually provide now, not just
like you can embed it, there's a vector,
you can do a similarity search, and
that's it. There's so many more
dimensions to it. Like what you said,
like usage of it.
>> Yeah. The feed The feedback loop is is
absolutely critical.
Cuz like that's that's another reason
why skills almost just fall down is
because there's just there's just one
guy that maintains a skill, and then he
probably publishes it and maybe works on
it a little bit to get a couple of
stars, and then it's just basically it
might not it might that that skill is
specific for a thing, and it doesn't get
the opportunity to to evolve, and people
Yeah, I mean I'm I'm looking I'm I'm
thinking of a couple of enterprise skill
repos that I've seen. They're all just
become dead spaces right now, as far as
I can tell.
>> Yeah, I still
I think that's right that skill quality
and how well maintained it is and
security scanning around skills like
that's what some of the new security
products are about, right? Versel or was
it another company just announced I mean
Versel skill registry made a public API
to query the registry now and you you
use OIDC and there's rate limiting and
they also skill with SSH usually. Yeah,
so skill with SSH but that's a public
registry, right? They made a public API
because I was using it and API wasn't
really like a contract and there was no
no proper control around it so they make
it they make it a version API now and
what I was saying like they have space
scanning like snake and other partners
can post updates about the skill has
been scanned
there seem to be no injection
but I guess what you like like you said
recency like has it been kept up to
date? Is it still relevant? Those are
all aspects of skills and and if you're
looking at the local skill registry, I
built something very simple which uses a
get mono repo. I think skills have their
place. I think skills they are very
useful in in a lot of cases
where an MCP might pollute the context
might you know, not make sense because
now an MCP you have to run something
like a lot of the MCPs you require you
to have something running and and maybe
our an offering like
>> Yeah.
>> for example Passion has an MCP. I think
they called made their MCP GA.
>> fill a gap but they if they don't evolve
then they're pretty much dead in the
water, aren't they really?
>> Okay. It it depends on the skill. It
depends on how it needs to evolve. I
think some skills are I think very
useful.
>> Yeah, I agree. Some skills are extremely
useful.
I I wanted to maybe tangentially talk
about a security
the uh
I was I was just thinking
I was just going to ask you if you had
any opinions about So, this is a bit of
a huge leap and change of topic about
humans in the loop like You Can I assume
you're using auto mode at this point,
right?
>> Yeah. Yeah, I mean, it's crazy now. I'm
just I'm just I feel like I've I've I've
I've devolved in this complete cognitive
offload of just handing my brain off
today and to the engine going all the
way into auto mode and then just looking
at the details.
>> So, human in the loop is pretty much
dead if you're using auto mode, aren't
you, really?
>> I don't I don't think so.
>> I mean, do you do you have any prompts
where you have to like a up
>> Yeah, yeah, yeah. I think that's what
they did really well. I mean, for it to
be successfully used um you must first
trust it, right? We talked about that
that we've seen it prompt us. And it it
stops and it does that really well as
well. Like Claude stops and says, "I
found uh an important divergence from
your statement, which I believe I need
to align with you on." And it stops. And
it and it and and it basically grabs
your attention now more when it's
needed. I think that's one of the quotes
that I really liked somebody told me. In
the age of AI, it's all about finding
like reducing noise, finding signal and
and and and capturing human focus for
the limited time it's available because
humans are only focused and and and
actually paying attention for a very
small amount of time compared to this.
>> Yeah, that's true. That's true. Like if
if if if a human is doing work and
you're basically wasting it by asking
the human to make decisions about, you
know, do I do this move or do I do this
copy or do I do this echo into into a in
a hair doc or something, you're you're
you're wasting people's precious
cognitive energy.
>> But, I think one one big mental shift
that I'm not have made yet with the
Hermes agent is when I send the prompt
equals off. And like in Claude, I can
press escape. I mean, I think in Hermes
3 out of the Discord gateway, there's
also a way to send a signal to to to
stop the and and interrupt the model. I
just haven't figured it out yet. And
maybe I shouldn't figure it out because
I I I need to treat this more as an
autonomous entity. And what I think I'm
failing failing to do is for it to to
find work and do and do things on its
own, which is something I haven't
grasped yet. I'm I'm still treating it
like a remote prompt and and a sandbox
environment that I can have some control
over which runs on my hardware.
>> that solves a lot of permission solves a
lot of security issues.
The The going back to auto mode, do do
you configure auto mode? Like cuz you
can like create your own rules and you
can critique your own. Do you ever do
that?
>> No, I've been I mean, I think this goes
back to like one of of earlier calls
where I was like, oh, I'm really not
going to bother with like a lot of
harness engineering or or anything like
that because I trust the when they
release a model and they release a
harness around it that it is fine-tuned
to each other. So, I don't want to make
too much effort
>> Yeah, that's a good point. That's a good
point like Cuz
I get
>> a balance, right? I mean, a lot of that
has changed. Um I think originally I I
was very much like, oh, I don't even
know Codex or Gemini or or or cloud
code. I'm just, you know, have a little
budget everywhere. I trust I'm not going
to spend time to like build my own
pi.dev harness. It's It's such a hard
balance and it changes all the time
depending on capabilities.
>> I suppose, but like but if you if you if
you're all in on Anthropic Claude, you
you should just trust Claude and and and
not have all these pointless discussions
almost saying that like I don't trust
Claude, you know what I mean? Like you
got to register the fact that you trust
Claude to do the right thing.
>> I think I think that's where having an
agent an agent with its own authority
and identity helps a lot. Cuz if you're
running Claude on your machine, it acts
upon like on your authority and it has
your permissions and it can do a lot of
things which you maybe like you you want
to have a control over. Whereas if you
dedicate it to an individual like an an
entity an agent that runs on its own and
has its own identity, you can control a
lot more like like how you would, you
know, separate like what do you call
separation of of of
>> duties.
>> duties, you you would be able to control
a lot more of what it can do based on
its permissions at at of course the cost
of like having to jump in when when it
gets stuck. Uh so, but I just feel it's
it's different if you run a a coding
agent on your machine and it uses your
credentials versus if you have like an
entity or a separate
>> You just hit on You just hit the nail on
the head here because like imagine you
are working in a company like I am with
like thousands of people, non-technical
people running Claude code. They
basically have Claude code desktop
installed or something like this or or
Claude code. In fact, I think they was
other way around in this company. They
they they rolled out Claude code before
the Claude desktop. But, what I'm trying
to say is that is that non-technical
people are basically running Claude in
their home directory and they're not
properly
able to sandbox it and that skill of
sandboxing, as you say with the Hermes
agent or as as us developers know how to
do, that's extremely powerful security
like that's a that's extremely powerful
security
standard capability to do that. But, but
most people don't know how to sandbox
and hence you're you have to fall back
on like this is why like people say you
have to have human in the loop and then
and then the human loop doesn't work
because people just smash enter. So,
anyway, that's
That's why
>> So, so another thing that I learned from
building a a a pull request review agent
related to the human in the loop like
where does the the human attention gets
grabbed versus like having a shell
running on your under your authority and
it figuring out when something doesn't
align or where it needs to grab your
attention and stops only when it's
really needed. What I learned about
Hermes and having an individual like an
AI acting like a PR reviewer is that
first of trust is
is built up and it's broken down very
quickly. So, the moment that agent is
posting comments on PRs and they are not
relevant, maybe simple things like just
the way it it it posts the comment
immediately triggers people, "Oh, this
is an AI." And and it's like when we're
watching websites and we are very good
at filtering out advertisement. When we
identify something is AI, we're going to
end up filtering it out very quickly and
it it signal becomes like
>> What?
>> useless.
>> you you're running Hermes agent. I'm
running Open Claw and I had like again I
had these like really amazing
experiences with Open Claw in the
beginning, but now it says it it it
can't run my quiz correctly. Every time
one of my family members answers the
quiz it often just randomly misses out
the answer from from random members of
my family. So, so trust is broken, but
the the worst thing about Open Claw is
that I don't know why it's failing. I
don't know what what what went wrong. I
don't know how to give it feedback. I
don't know how to fix my Open Claw
instance. I basically got this this
crippled sick Open Claw instance running
for months now and it's doing a few
things like doing my calendaring. It's
doing my the family quiz, but it it's
just not been operating 100% and I don't
know how to fix it and I don't want to
completely reset it at the same time
either.
>> So, I I very early on with with the
Hermes agent when it was running on EC2
even when it was just accessible over
SSH by Claude Code. Um at about a one
week milestone I went and asked Claude
to actually ask my son, "Like tell
Claude to ship all the logs and whatever
logs it can find, session logs,
gateway logs, whatever. Zip it, tar it,
stream it over the the SSH tunnel into
your machine and run a dynamic workflow
locally to go look at failure modes and
issues that we we may improve.
>> Wow. Okay.
>> And and it it did that really well. As
soon as my son had done the basics, I
basically took over his laptop again.
This is kind of the worst part of about
me giving my son during summer time jobs
to hopefully him learn things where as
soon as he had done the most basic
thing, I just had to take over his
laptop and start doing it myself because
it's just too exciting. And and I
actually asked ChatGPT to help me about
like giving him assignments and then
also framing it in a way that it like it
better so that it this doesn't happen.
Uh and ChatGPT scolded me for for my
behavior. But anyway, what I found
really interesting with these Sonnet
logs even even like and my probably Opus
but like dynamic workflow so like five
agents go out, look at all the session
logs, find common failure issues. It's
so it surfaced so many misconfigurations
like the auxiliary LLM configuration.
It's saying you you don't have an
auxiliary LLM endpoint configured and
they're failing most of the time. So you
like your summarized sessions is always
failing because of it. Um it identified
other issues and
>> And these logs are are Hermes logs or
are they or they Claude Claude traces or
>> So I have several No, they they So
Hermes and probably Open Claude as well
has like several session sources. For
example, in an SQLite database where it
keeps track of every session that
happens and and it can find the very
detailed information there. It can also
find more high-level summarized session
logs. It can also look at the gateway
like Discord interactions. For example,
it has these interactive cards that it
sends over Discord that you need to
approve or has multiple choice and they
have a a timeout like maybe maybe
600 seconds timeout on it. So sometimes
these session like these interactive
cards weren't reaching me because they
were not running on a gateway thread
because of the way I had set up some
cron jobs. So the cron job wasn't
running on a gateway integration path
and it wasn't surfacing those and it was
timing out every time, and it was like
there's several of them there that's
being timed out. I haven't tried
upgrading Hermes yet, but I found doing
this, and I've done it now it's already
third week. I've done it two consecutive
weeks, and it each time it found a whole
bunch of information that allows me to
fine-tune the agent and to optimize the
way it works.
>> Yeah.
>> And that I think would help you with
your open cloud.
>> Yeah, you you've inspired me to do that.
I don't know why I didn't think about
doing that, but thanks for the
inspiration. I guess maybe it's
the tweets that I was reading in the
last few days. It's like everyone's
talking about these loops, and and it's
interesting that you you can run an It's
interesting that you you run an agent,
you emit events and logs, and you put
them somewhere, and then you analyze
that log to basically improve the the
agent run. And you you go in that that
loop.
I think
>> that's it's the dreaming concept, right?
You you dream and you clean up your
memory, and you you you evaluate while
you're behaving, and you you build the
rules like Boris hi says this that you
should do that a lot. Um they also have
this like on team onboarding command to
allow you to share what you learned in
your local setup against with your team
to onboard other team members.
>> Uh /team onboarding, I've seen
that.
>> Yeah, but like when I tried it was it
just came out. It was really not
interesting. It didn't really say much.
Like, "Hey, your team is using uh cloud
code, and they want you to help you
onboard." And then it was like a very
stupid. It wasn't even It was like,
"Vincent has been using this." And it
was like, "How is that relevant?" But
anyway, like like this is what I was
telling you like on the last podcast,
right? I say, "Check the Twittersphere.
Check what's being on going on about
like you engineer the loop, not the
prompt. You're no longer prompting the
agent. Focus on how you engineer the
loop." I was telling you that last time,
and and that's how when when that all
came out.
>> This is very inspiring. Like, the
trouble is with my current employer,
I I wanted to see the the Claude logs
inside DataDog, but but they've deemed
to be highly sensitive. So, like only
like a couple of people in the whole
organization can see cloud logs in data
dog. And now, on the basis of our of our
conversation, I'm thinking that like the
team almost should have access to the to
the team uh cloud logs so that they can
have a retrospective that makes sense
for the for the agent usage. This is all
quite inspiring stuff.
>> this is a This when you say this this
brings me back to exactly the role of
DevOps engineer or platform engineering,
right? We're supposed to give people
like we're supposed to surface the tools
for them to do their job without giving
them the access. For example, we not
giving them access SSH to a node, but we
have to surface all of the tools for
them to troubleshoot the issue so that
they can, you know, use a dashboard and
and correlate events. And and it's very
similar to like what we used to do when
we when we do Kubernetes, right? Again,
we don't want to give them Cube CTL
access to a production cluster, but we
have to give them all of the tools,
knobs, and
dials so that they are able to fine-tune
their service and they're able to
troubleshoot and identify the issue
without SSH or, you know, admin access,
which is similar to to now, like we need
to do the same for the cloud agent
sessions.
>> Yeah, and and and I need to get I need
to ask for DataDog access. Have you ever
got DataDog pup working? This is CLI
tool called
>> Yeah, I built it. I I was So, I was
having some some some effort at work.
This is reminding me I have to follow up
on something. But,
for ex- It's very funny because you
design the whole thing and it goes to
production and then opens randomly goes
like, "Oh, by the way,
um I noticed something. Should we do
this or that?" And I'm like, "Damn it,
this is going to production. It's a bit
late." So, then go ahead and make a
dashboard to track this if this is a
real issue in production. And then it
just goes off. It builds a whole
dashboard, you know, to watch this
particular type of like failure scenario
if this is really a concern. Uh and then
so that we could get better
>> get it in DataDog.
>> Yeah, yeah. So, I I I I used pub. You
can go to the repo. You can see from me
like a month ago I reported an issue and
they fixed it by rolling back something
they had released, which I think it was
still in beta. I don't know. I think
Opus surfaced it somehow. I was I was
like looking at the DataDog uh
docs to to how to build and pub came up
very quickly. It was very early in the
>> Yeah, I I tried pub just now a couple of
days ago and it didn't work. There's
some sort of up the issue or something
like that, but
>> I had the same experience.
Huh?
I had the same. So, when I tried pub, it
was using a token that didn't have
permission to do certain things. And I
raised a I raised a request. I I
actually had a separate Claude Code
session go through the whole thing cuz
when you try to log in, it failed. And
then and then it DataDog uh sorry, Cloud
did like a whole binary search of like
it's failing on a certain scope request
against the token. And then it went
like, "Okay, we're going to do the first
50 scopes." And then, "Oh, we're going
to do the second 50 scopes." "Okay, it's
failing in that batch." And then it
narrowed it down. It says, "Okay, this
particular scope that's being requested
in the login is causing the failure."
And then it created a ticket and it it
shared all the information. And then the
response was, "I rolled back a feature
and we fixed it." And I was like, "In
the meantime, I had a working
set up because instead of like doing a
an authentication and it it doing the
default scope where it was always
failing, it was actually doing a
dedicated like subset of list of scopes
that I needed and it worked."
>> Wait a minute. If I understand
correctly, you you you managed to create
a ticket complaining about how pub
wasn't working to the admins for the
admins to fix, right?
>> Yes, exactly. Like for for for the
DataDog to fix because they had when
they did their authentication, so pub is
using some token from DataDog
authentication.
>> I need to update and try again. The
>> It was a it was a month ago. So, I maybe
they're just keep
recreating this bug, but it's a problem
with the scope request on the token. And
and I I spent I spent quite a while to
figure out what the scope was.
>> The thing I wanted to mention, but I
need to run off to to take my kids to
school, is that like it's really
frustrating that the Cloud Enterprise,
you know, it there's an admin panel for
a couple of people who have access.
There There's things that they can touch
inside there. There's There's new rows
that appear all the time. There should
be Terraform to control um that Cloud
Enterprise settings. I mean, it's so
critical that this should be under under
source control, right? That the settings
are controlled. Right now, people log
into these admin panels, they go, "What
is this new OAuth scope thing? I don't
know what it is. I'm turning it off or
something." And then it breaks the flow,
and then you're you're you're left
scratching your head for like days.
>> On Anthropic?
>> Well,
>> Is it an Anthropic admin, or is it the
Datadog admin panel?
>> Well, I I I
It's Look,
I'm just making an example here that
like these these admin panels are
extremely important to control, but they
there isn't a Terraform provider for it.
I'm like, "What?"
>> So, this is what I I I struggle with in
every organization I worked is these
ITSM tickets flows and ITSM teams, which
often are like ex- accepted because we
are not programmers. We don't know how
to do this. And then they end up like
click-opsing everything. And
>> Oh, yeah. And then they take like a week
to respond to a ticket. It's like,
"Okay."
>> Yeah, but I mean I mean, I can
understand, obviously, that like
terraforming the whole situation is also
not a solution. I agree with that. So,
what happened recently is like a lot of
people left on on a on on a contract I'm
on. And some people new people reach out
to me, and they asked me like we have we
have this ITSM ticket we have to deal
with. And I was like I thought that the
it was around AWS permission sets in AWS
SS. And I was like, "Oh, yeah, I know.
We we we we Terraform the permission
sets. It's just the membership like
that's managed in the in the identity
platform. So, the the user to like that
this identity provider user to group
membership, that's there. That's that's
onboarding, that's HR. I'm not touching
that. I have learned my lesson. So, so
on the actual permission sets against
AWS accounts, that's definitely
Terraform. Like I've I've got that in
Terraform for ages, right? And then I
was like
I talking with with them, the new people
taking over from people that have been
doing it behind the scenes shadowy ish
for a while, and Cloud exploring along.
And then no, none of these permission
sets for production are in Terraform.
And I was like this is horrible. This is
terrible. Why is it like that? You know,
the the these people keep up saying
behind it that keep doing this.
>> I mean I've joined I've joined clients
and they have like an an IAM team and I
should have asked them, "Do are you
using Terraform to maintain your I am?"
I bet you they're not.
>> Yeah, so for permission sets I think
having Terraform is definitely a must. I
said in this intermediate phase where
we're not like because Cloud immediately
went, "Let's go import everything." I
was like, "Yeah, we're not we're not
going to do that right now because we
got a ticket to close, right?" So, I
told people in lieu of proper processes,
let's just because everyone is adopting
LLMs, I created a folder because LLMs
now are clickops, right? Because they're
running the API commands directly,
they're modifying configurations,
they're they're modifying the settings.
And are you getting the same problem? It
happens behind the back, nobody knows
who did it, when what when it happened.
And you need to have that for our
principles, somebody has to review it.
So, I said to those people, I I approve
I'm I I implore on you that you use AI
to do this because AI is going to be way
more grounded than you who just came
into this position and have to solve
this problem. But as long as we capture
the context properly, and also I I don't
want you to trust AI. So, in this
particular case, you cannot make the
decision because you don't know about
the system and you don't know what
should be the way. Until we have either
this Terraform or we have proper
context, you are going to get the LLM
plan and commit it into this repo on
this position and you're going to tag us
for review so we have a four eye view
>> Yeah. I'm
>> So it's kind of like the intermediate
ClickOps solution. Without Terraform,
you get an LLM plan that needs to be
reviewed. You have a Git log of when
this was changed. You know exactly what
the LLM was.
>> It's a it's a total smell when there's
no
source source It's a total smell in an
organization that you have these IT ITSM
flows and there doesn't seem to be any
uh
you know, Git commits behind them. It's
it's a total
>> like a compliance issue like
In a lot of cases, this will be a
complete failure on compliance.
>> Yeah. Yeah. Okay. Thanks again, Vincent.
I got to run.
>> of those compliances are like paper are
just checklists. They're just like yeah,
we have someone treated ticket and
someone reviewed the ticket.
>> Or they're like yeah, totally manual and
they think it's manual.
The worst is when they think it's like
it's manual and and it's a good thing
that it's manual or something. That's
That's the thing that bothers me a bit.
>> Okay. See you. Cheers, everyone. Bye.
>> Bye.