WEBVTT

00:00:00.000 --> 00:00:02.360
And we're starting. Well, it's been a

00:00:02.360 --> 00:00:03.960
while since we last talked. I I kind of

00:00:03.960 --> 00:00:06.440
made an agenda on that AI infrastructure

00:00:06.440 --> 00:00:10.120
podcast thingy, the the Google Doc.

00:00:10.120 --> 00:00:10.880
Remember that one?

00:00:10.880 --> 00:00:13.040
>> Yeah, it's great. I didn't read it.

00:00:13.040 --> 00:00:14.360
>> I'm sorry. I don't know if you

00:00:14.360 --> 00:00:15.480
I don't know if you want to talk about I

00:00:15.480 --> 00:00:17.600
mean, I just rambled a a lot a lot of

00:00:17.600 --> 00:00:20.200
bad stuff, like maybe it's worth talking

00:00:20.200 --> 00:00:21.720
about the most obvious thing in the

00:00:21.720 --> 00:00:23.800
room. I mean, what what do you What was

00:00:23.800 --> 00:00:26.520
your take? What what the elephant? What

00:00:26.520 --> 00:00:28.480
I I was using Fable. I was enjoying it,

00:00:28.480 --> 00:00:31.320
and then it got pulled. And now what

00:00:31.320 --> 00:00:32.920
what's your What's your hot take?

00:00:32.920 --> 00:00:33.640
Aliens?

00:00:33.640 --> 00:00:35.680
>> Aliens. What did aliens do wrong this

00:00:35.680 --> 00:00:37.840
time? They keep getting, you know, stray

00:00:37.840 --> 00:00:40.560
bullets, catching bullets stray.

00:00:40.560 --> 00:00:42.880
Catching strays. Geez, we're very bad

00:00:42.880 --> 00:00:44.680
with like, um, what do you call them?

00:00:44.680 --> 00:00:46.000
Idioms?

00:00:46.000 --> 00:00:47.480
Um, what was

00:00:47.480 --> 00:00:48.960
No aliens?

00:00:48.960 --> 00:00:50.400
Uh, I have

00:00:50.400 --> 00:00:52.920
heard some theories. I I think some of

00:00:52.920 --> 00:00:56.720
them are far overblown. Um

00:00:56.720 --> 00:00:59.560
I mean, I hear the the

00:00:59.560 --> 00:01:01.240
the standard mocking. My initial

00:01:01.240 --> 00:01:04.760
response was like, I want to like, right

00:01:04.760 --> 00:01:06.720
after it happened, I went home. I I I

00:01:06.720 --> 00:01:08.720
read the news. Um, it's was evening

00:01:08.720 --> 00:01:10.960
Friday in in US, but it was Saturday

00:01:10.960 --> 00:01:12.600
morning in Vietnam. I had just finished

00:01:12.600 --> 00:01:14.760
my breakfast. I see a post 25 minutes

00:01:14.760 --> 00:01:15.760
ago.

00:01:15.760 --> 00:01:17.480
Um, they're pulling Fable.

00:01:17.480 --> 00:01:18.840
Um, I was eating breakfast with my

00:01:18.840 --> 00:01:21.400
family. I got home. I go to my machine,

00:01:21.400 --> 00:01:23.320
and I had a session open with Fable, and

00:01:23.320 --> 00:01:25.440
I said, "Can you just spin up a dynamic

00:01:25.440 --> 00:01:27.320
workflow real quick?" Hoping that maybe

00:01:27.320 --> 00:01:28.680
it wasn't revoked yet, and I would still

00:01:28.680 --> 00:01:30.800
get a dynamic workflow out of it. And it

00:01:30.800 --> 00:01:32.640
says, "Model not available." And I was

00:01:32.640 --> 00:01:33.440
like, "God."

00:01:33.440 --> 00:01:35.160
Ah.

00:01:35.160 --> 00:01:38.080
Yeah. Then, funny enough, not long after

00:01:38.080 --> 00:01:39.880
it asked me, "How is your experience

00:01:39.880 --> 00:01:42.836
with with Claude?" And I went, "Bad."

00:01:42.836 --> 00:01:45.840
>> [laughter]

00:01:45.840 --> 00:01:47.200
>> And that's how far it went with me,

00:01:47.200 --> 00:01:48.800
because the next day I was already using

00:01:48.800 --> 00:01:51.520
Opus again. Yeah. I mean, it's my

00:01:51.520 --> 00:01:53.240
experience with AI can be a bit weird.

00:01:53.240 --> 00:01:55.200
It's like, I have these good sessions,

00:01:55.200 --> 00:01:57.520
and I have these these sort of myriad of

00:01:57.520 --> 00:02:01.800
like bad labyrinth confusing sessions.

00:02:01.800 --> 00:02:02.960
And

00:02:02.960 --> 00:02:07.040
on Friday afternoon, I I I vibe coded

00:02:07.040 --> 00:02:08.880
a tool to

00:02:08.880 --> 00:02:10.160
help me take

00:02:10.160 --> 00:02:12.680
help me annotate screenshots. I did that

00:02:12.680 --> 00:02:15.480
with Fable and I one shot at it and then

00:02:15.480 --> 00:02:18.360
you know these experiences become

00:02:18.360 --> 00:02:20.720
become legendary in in your mind, right?

00:02:20.720 --> 00:02:22.560
Yeah, Swift. I like the problem was

00:02:22.560 --> 00:02:24.240
pretty simple and less than a thousand

00:02:24.240 --> 00:02:26.680
lines of Swift. Uh

00:02:26.680 --> 00:02:27.800
create

00:02:27.800 --> 00:02:28.440
Uh

00:02:28.440 --> 00:02:29.920
you actually prompted less than a

00:02:29.920 --> 00:02:31.280
thousand lines of Swift?

00:02:31.280 --> 00:02:32.720
>> Yeah, exactly.

00:02:32.720 --> 00:02:34.520
Exactly.

00:02:34.520 --> 00:02:37.360
I'm I'm I'm big on that's my technique.

00:02:37.360 --> 00:02:41.160
I'm I'm patenting patenting

00:02:41.160 --> 00:02:42.640
>> So there's this one thing

00:02:42.640 --> 00:02:44.800
>> I tell it to lock of code

00:02:44.800 --> 00:02:47.360
key to lock of code lines of code down.

00:02:47.360 --> 00:02:49.160
>> There was this one

00:02:49.160 --> 00:02:50.160
um

00:02:50.160 --> 00:02:53.600
hype skill recently which was ponytail.

00:02:53.600 --> 00:02:54.320
Have you heard of it?

00:02:54.320 --> 00:02:55.120
>> So ponytail

00:02:55.120 --> 00:02:57.520
>> and and it has like almost like this

00:02:57.520 --> 00:03:00.320
meme type of drawing. It's a bit

00:03:00.320 --> 00:03:01.840
negative in the way it describes the

00:03:01.840 --> 00:03:04.360
skill. It's about every team has this

00:03:04.360 --> 00:03:06.400
experienced season dev when everyone is

00:03:06.400 --> 00:03:08.480
like creating these over engineered

00:03:08.480 --> 00:03:11.239
solutions. He just looks at it and cuts

00:03:11.239 --> 00:03:13.120
it down to like just one or two lines of

00:03:13.120 --> 00:03:14.519
code and then they have like this

00:03:14.519 --> 00:03:17.480
ponytail do we like uh you know

00:03:17.480 --> 00:03:20.040
type of behavior. So the skill is is

00:03:20.040 --> 00:03:22.480
exactly in alignment for the model to to

00:03:22.480 --> 00:03:25.320
focus on reducing lines of code and

00:03:25.320 --> 00:03:26.920
getting rid of like over engineered

00:03:26.920 --> 00:03:28.920
solutions and has a whole bunch of like

00:03:28.920 --> 00:03:31.080
rules around that. And I have some very

00:03:31.080 --> 00:03:33.800
interesting experience because I run a

00:03:33.800 --> 00:03:37.080
Hermes agent to review pull requests and

00:03:37.080 --> 00:03:38.959
I spend quite a lot of time building

00:03:38.959 --> 00:03:40.400
pull requests and then having the agent

00:03:40.400 --> 00:03:42.200
review them running simulations. I find

00:03:42.200 --> 00:03:44.519
it amazing. I can be again I can be

00:03:44.519 --> 00:03:45.239
outside

00:03:45.239 --> 00:03:47.800
>> You mean simulations like it's deploying

00:03:47.800 --> 00:03:49.080
it and testing it or something?

00:03:49.080 --> 00:03:51.800
>> Yeah, like so the heart so the the

00:03:51.800 --> 00:03:53.560
danger is to get an agent to just rerun

00:03:53.560 --> 00:03:55.400
the CI/CD test, right? That's kind of

00:03:55.400 --> 00:03:57.240
useless unless there's a CI/CD test

00:03:57.240 --> 00:03:58.600
failing and you need the agent catches

00:03:58.600 --> 00:03:59.960
it because he's able to reproduce the

00:03:59.960 --> 00:04:01.400
failure locally.

00:04:01.400 --> 00:04:03.120
Or he can detect that locally it passes

00:04:03.120 --> 00:04:04.840
and the remote CI fails, then he can

00:04:04.840 --> 00:04:06.560
identify something wrong in the CI/CD

00:04:06.560 --> 00:04:08.200
workflow. So there's there's some

00:04:08.200 --> 00:04:09.720
benefit of it of it running, but it

00:04:09.720 --> 00:04:11.200
shouldn't be just always run all the

00:04:11.200 --> 00:04:13.440
CI/CD locally because that doesn't make

00:04:13.440 --> 00:04:15.080
I mean a lot of make a lot of sense,

00:04:15.080 --> 00:04:17.519
right? But in terms of simulations is

00:04:17.519 --> 00:04:19.079
because this is a Hermes agent and I

00:04:19.079 --> 00:04:21.040
don't know is my connection active?

00:04:21.040 --> 00:04:22.720
>> I can't see your screen if that's what

00:04:22.720 --> 00:04:23.800
you're saying.

00:04:23.800 --> 00:04:25.800
>> It says trying to reconnect, but I mean

00:04:25.800 --> 00:04:27.520
it will be buffering my video feed and

00:04:27.520 --> 00:04:29.040
it will come back, right? It's a bit

00:04:29.040 --> 00:04:29.280
weird.

00:04:29.280 --> 00:04:30.520
>> Yeah, I can see it fine. I can see it

00:04:30.520 --> 00:04:30.960
fine.

00:04:30.960 --> 00:04:34.160
>> Am I on a VPN? No, I am on the normal. I

00:04:34.160 --> 00:04:35.919
have two Wi-Fi's, one is VPN, one is

00:04:35.919 --> 00:04:38.840
without. I'm on the normal Wi-Fi. So,

00:04:38.840 --> 00:04:41.200
the the the Hermes agent What is What is

00:04:41.200 --> 00:04:42.840
the real benefit? My god.

00:04:42.840 --> 00:04:44.520
>> Who opens the banana from the other way

00:04:44.520 --> 00:04:46.240
around or is it just me? It's really

00:04:46.240 --> 00:04:48.120
hard to open up a from here. Maybe it's

00:04:48.120 --> 00:04:49.240
just the

00:04:49.240 --> 00:04:51.240
kind of banana I'm I'm eating nowadays.

00:04:51.240 --> 00:04:54.040
Okay, but whilst we wait for Vincent, I

00:04:54.040 --> 00:04:56.160
can maybe share my screen and show you

00:04:56.160 --> 00:04:58.160
and show you what this annotation tool

00:04:58.160 --> 00:05:02.120
does. So bounded to hyperkey A,

00:05:02.120 --> 00:05:03.520
it just

00:05:03.520 --> 00:05:05.680
it creates this little window with the

00:05:05.680 --> 00:05:07.760
screenshot and then I'm able to press A

00:05:07.760 --> 00:05:11.120
for arrow, B for box, or T for words.

00:05:11.120 --> 00:05:13.200
Hello.

00:05:13.200 --> 00:05:14.360
Of course you're like, "Oh, how do you

00:05:14.360 --> 00:05:15.760
change the colors?" Oh, you just you

00:05:15.760 --> 00:05:18.280
just change the source code.

00:05:18.280 --> 00:05:20.320
And and then when I dismiss it, it's in

00:05:20.320 --> 00:05:23.040
my clipboard and ready to be pasted and

00:05:23.040 --> 00:05:24.440
you might be wondering what ones if you

00:05:24.440 --> 00:05:26.440
make a mistake and you have to move

00:05:26.440 --> 00:05:28.160
things around.

00:05:28.160 --> 00:05:30.320
You just control Z. Simple. And the

00:05:30.320 --> 00:05:32.320
really good thing about it is also that

00:05:32.320 --> 00:05:33.760
if you take a screenshot, you can just

00:05:33.760 --> 00:05:35.840
paste it into the same

00:05:35.840 --> 00:05:38.280
um annotate window. Um you can just

00:05:38.280 --> 00:05:40.040
paste it in here. I think it's I think

00:05:40.040 --> 00:05:42.160
it's brilliant. Brilliant. I dare say so

00:05:42.160 --> 00:05:43.840
myself.

00:05:43.840 --> 00:05:45.843
I dare say so myself.

00:05:45.843 --> 00:05:47.120
>> [sighs]

00:05:47.120 --> 00:05:48.440
>> What else did I want to share?

00:05:48.440 --> 00:05:49.960
>> Pinsel, you're back.

00:05:49.960 --> 00:05:51.800
>> Yeah, but I'm on a hotspot. I don't want

00:05:51.800 --> 00:05:53.240
to do that. I don't think the data's

00:05:53.240 --> 00:05:57.120
going to be good enough anyway. Um but

00:05:57.120 --> 00:05:57.560
What?

00:05:57.560 --> 00:05:59.240
>> What is going on with the internet? It's

00:05:59.240 --> 00:06:01.400
some Someone bust your supply or

00:06:01.400 --> 00:06:01.920
something?

00:06:01.920 --> 00:06:03.240
>> I don't I don't know. The The other

00:06:03.240 --> 00:06:05.440
machine is on a on a stand-up. So, it's

00:06:05.440 --> 00:06:07.480
works fine. I could Anyway,

00:06:07.480 --> 00:06:08.920
>> You're You're on a You're on a You're on

00:06:08.920 --> 00:06:10.280
a stand-up at the same time.

00:06:10.280 --> 00:06:12.320
>> Because I These stand-ups take too long,

00:06:12.320 --> 00:06:13.840
you know? Keep them to 15 minutes. I

00:06:13.840 --> 00:06:15.800
tell you let's meet I think a stand-up

00:06:15.800 --> 00:06:17.480
should be 15 minutes. I agreed with you

00:06:17.480 --> 00:06:19.480
to meet. I'm already 5 minutes late.

00:06:19.480 --> 00:06:21.320
We're already 5 minutes over. Stand-ups

00:06:21.320 --> 00:06:23.160
literally finished right now. So, I've

00:06:23.160 --> 00:06:24.600
I've said my piece. Uh there's nothing

00:06:24.600 --> 00:06:25.720
else for me to say.

00:06:25.720 --> 00:06:27.560
>> The The The worst about stand-ups is

00:06:27.560 --> 00:06:29.400
like when there's no real updates, you

00:06:29.400 --> 00:06:30.760
know, there's no real progress, but no

00:06:30.760 --> 00:06:33.080
one ever that seems to admit it.

00:06:33.080 --> 00:06:35.120
>> Yeah, that's the I mean, I was I was for

00:06:35.120 --> 00:06:36.440
I was waiting for the completion of the

00:06:36.440 --> 00:06:38.520
sentence because there's no real

00:06:38.520 --> 00:06:40.120
updates, but we're still spending 30

00:06:40.120 --> 00:06:43.160
minutes talking about no real updates.

00:06:43.160 --> 00:06:44.600
>> Yeah, let's get back to work sort of

00:06:44.600 --> 00:06:46.120
thing.

00:06:46.120 --> 00:06:48.240
>> Yeah. No, what I was saying is that the

00:06:48.240 --> 00:06:50.919
the the simulations, for example, I I

00:06:50.919 --> 00:06:53.640
introduced release-please bot, which is

00:06:53.640 --> 00:06:55.840
a Google GitHub action and an and a and

00:06:55.840 --> 00:06:58.800
a local CLI that you can run that

00:06:58.800 --> 00:07:00.919
that does a standing release pull

00:07:00.919 --> 00:07:03.600
request that tracks main and then keeps

00:07:03.600 --> 00:07:05.840
updating it as new features land on main

00:07:05.840 --> 00:07:07.240
and then uses conventional commit to

00:07:07.240 --> 00:07:10.400
generate a change log and it then allows

00:07:10.400 --> 00:07:12.600
you to just click merge and then boom it

00:07:12.600 --> 00:07:14.280
detects that the the the standing

00:07:14.280 --> 00:07:16.360
release PR was merged and it then means

00:07:16.360 --> 00:07:17.720
a new release and creates your change

00:07:17.720 --> 00:07:19.760
log, which is I think a really nice way

00:07:19.760 --> 00:07:20.720
of

00:07:20.720 --> 00:07:22.760
A lot of libraries work that way, right?

00:07:22.760 --> 00:07:25.120
Main is is is not constantly releasing.

00:07:25.120 --> 00:07:26.680
Some Some are always releasing. Every

00:07:26.680 --> 00:07:28.960
Every merge to main releases, which

00:07:28.960 --> 00:07:30.400
which can be okay, but then if you have

00:07:30.400 --> 00:07:33.080
but dependable merging a lot of fixes,

00:07:33.080 --> 00:07:34.760
then you get releases non-stop, which is

00:07:34.760 --> 00:07:36.240
also annoying. So, release-please is

00:07:36.240 --> 00:07:38.720
pretty nice. And there was this manual

00:07:38.720 --> 00:07:39.240
process

00:07:39.240 --> 00:07:40.800
>> Release-please, maybe you should share a

00:07:40.800 --> 00:07:42.000
screen, though.

00:07:42.000 --> 00:07:43.520
I don't need Are you

00:07:43.520 --> 00:07:45.200
So, this is this is the this is the this

00:07:45.200 --> 00:07:48.000
is the latest in tech for for what? Node

00:07:48.000 --> 00:07:48.800
modules or

00:07:48.800 --> 00:07:50.960
>> No, no, release please is really old and

00:07:50.960 --> 00:07:52.600
it's from Google. It's It supports

00:07:52.600 --> 00:07:54.960
Golang, it supports Python, it supports

00:07:54.960 --> 00:07:56.680
Terraform modules if you want.

00:07:56.680 --> 00:07:59.200
>> Go projects I'm using Go releaser, old

00:07:59.200 --> 00:07:59.600
school.

00:07:59.600 --> 00:08:01.440
>> Yeah, Go releaser, but Go releaser works

00:08:01.440 --> 00:08:03.440
really well with release please. So, so

00:08:03.440 --> 00:08:05.000
you can combine those. Because Go

00:08:05.000 --> 00:08:07.960
releaser maybe you will release oops

00:08:07.960 --> 00:08:11.760
uh on every PR. So, basically on the I I

00:08:11.760 --> 00:08:14.760
noticed I set up an agent and I noticed

00:08:14.760 --> 00:08:16.600
manual activity and of course manual

00:08:16.600 --> 00:08:18.840
activity and and and how do you call

00:08:18.840 --> 00:08:22.080
those in the SRE um toil? Toil is what

00:08:22.080 --> 00:08:25.000
you aim to reduce, right? Every cycle

00:08:25.000 --> 00:08:26.680
you should spend a certain percentage on

00:08:26.680 --> 00:08:28.640
reducing toil. That's one of the SRE

00:08:28.640 --> 00:08:31.520
tenants. And um so, there's manual

00:08:31.520 --> 00:08:33.840
release PRs being created. This is an

00:08:33.840 --> 00:08:36.479
example of it being integrated. I don't

00:08:36.479 --> 00:08:38.800
like that it's um it's collapsing the

00:08:38.800 --> 00:08:40.800
release notes. But, as you can see that

00:08:40.800 --> 00:08:43.120
this is a standing PR that constantly

00:08:43.120 --> 00:08:44.280
gets

00:08:44.280 --> 00:08:46.520
um updated each time a new

00:08:46.520 --> 00:08:48.480
>> can you define standing PR? Something

00:08:48.480 --> 00:08:50.480
that's a bit long-lived? What does

00:08:50.480 --> 00:08:51.560
standing PR mean?

00:08:51.560 --> 00:08:55.400
>> It's It's a PR that stays open until you

00:08:55.400 --> 00:08:57.680
want to cut the release and it keeps

00:08:57.680 --> 00:08:59.760
tracking head. So, it gets rebased each

00:08:59.760 --> 00:09:02.240
time new features land on head and then

00:09:02.240 --> 00:09:03.839
it updates the the features that are

00:09:03.839 --> 00:09:06.040
going to land, uh bug fixes,

00:09:06.040 --> 00:09:07.720
miscellaneous. So, that's what you

00:09:07.720 --> 00:09:11.080
>> do you It's like a way of having like a

00:09:11.080 --> 00:09:13.080
uh a change log or

00:09:13.080 --> 00:09:14.240
really It's like you're doing your

00:09:14.240 --> 00:09:16.839
release notes in an issue, right? And

00:09:16.839 --> 00:09:18.800
then you then when you hit you smash

00:09:18.800 --> 00:09:22.000
merge, that becomes your release notes.

00:09:22.000 --> 00:09:23.800
Ah, I don't think I've ever done it that

00:09:23.800 --> 00:09:24.200
way.

00:09:24.200 --> 00:09:25.680
>> So, I've used I use release please in

00:09:25.680 --> 00:09:27.240
quite a few repositories and I really

00:09:27.240 --> 00:09:29.920
like it. So, so this was a manual toil

00:09:29.920 --> 00:09:32.440
and my initial reaction what was to say,

00:09:32.440 --> 00:09:35.120
"Hey, um my my agent can do this, right?

00:09:35.120 --> 00:09:37.760
My my my beautiful AI agent can can

00:09:37.760 --> 00:09:40.000
create um help you create these release

00:09:40.000 --> 00:09:41.720
PRs until I remember of course there's a

00:09:41.720 --> 00:09:43.680
tool for it. Why would I let AI do

00:09:43.680 --> 00:09:45.120
something that's very deterministic and

00:09:45.120 --> 00:09:46.680
a tool exists and it's really well

00:09:46.680 --> 00:09:47.880
established tool. It's a really nice

00:09:47.880 --> 00:09:50.120
tool. So I said let's me let me create a

00:09:50.120 --> 00:09:51.880
pull request to add release please. As

00:09:51.880 --> 00:09:53.400
you can see this branch was created by

00:09:53.400 --> 00:09:55.960
Claude so I was on the Claude code app

00:09:55.960 --> 00:09:57.440
when I said I want to integrate release

00:09:57.440 --> 00:09:59.800
please into this into this project and

00:09:59.800 --> 00:10:02.200
Claude created this pull request. And

00:10:02.200 --> 00:10:02.720
and then

00:10:02.720 --> 00:10:04.320
>> Wait how do you know it's created by

00:10:04.320 --> 00:10:06.240
Claude again sorry? Uh I just lost that

00:10:06.240 --> 00:10:06.560
bit.

00:10:06.560 --> 00:10:07.560
>> I mean

00:10:07.560 --> 00:10:09.280
it just shows here that the branch is

00:10:09.280 --> 00:10:10.600
Claude slash

00:10:10.600 --> 00:10:12.600
>> It's only like a small branch name.

00:10:12.600 --> 00:10:12.880
>> Yeah.

00:10:12.880 --> 00:10:14.440
>> How did you know Claude did that? Did

00:10:14.440 --> 00:10:15.800
you I guess you must have told Claude to

00:10:15.800 --> 00:10:16.400
do that okay.

00:10:16.400 --> 00:10:18.280
>> And then another thing though uh this is

00:10:18.280 --> 00:10:20.120
Claude code. So you can actually see the

00:10:20.120 --> 00:10:22.360
session here. So you And this is this

00:10:22.360 --> 00:10:24.400
doesn't link to like your question was

00:10:24.400 --> 00:10:25.640
like I want to see the session that

00:10:25.640 --> 00:10:27.960
created this PR. Claude code does that

00:10:27.960 --> 00:10:29.720
but it's not public. It's only me and

00:10:29.720 --> 00:10:31.480
only I can see it.

00:10:31.480 --> 00:10:33.240
>> Is this Is this a new thing? I've never

00:10:33.240 --> 00:10:34.040
seen this before.

00:10:34.040 --> 00:10:35.680
>> No. Yeah yeah I've never I hadn't seen

00:10:35.680 --> 00:10:38.320
this before. So here I go. I just uh I

00:10:38.320 --> 00:10:40.240
opened the app on my phone and I said as

00:10:40.240 --> 00:10:41.720
you can see it's very short so because I

00:10:41.720 --> 00:10:43.280
was on my phone I typed this manually

00:10:43.280 --> 00:10:45.280
like a caveman. Uh everyone left the

00:10:45.280 --> 00:10:46.800
stand.

00:10:46.800 --> 00:10:48.280
So I said set up release please. Ground

00:10:48.280 --> 00:10:49.400
yourself in latest release please

00:10:49.400 --> 00:10:51.520
version best practice uh and this is an

00:10:51.520 --> 00:10:53.839
NX learner yarn mono repo. It actually

00:10:53.839 --> 00:10:56.080
found out very interesting stuff because

00:10:56.080 --> 00:10:58.240
normally release please in a mono repo

00:10:58.240 --> 00:10:59.560
it will version each package

00:10:59.560 --> 00:11:01.400
individually but this particular

00:11:01.400 --> 00:11:04.160
repository has a has a standard version

00:11:04.160 --> 00:11:06.640
at the root that stays at zero and then

00:11:06.640 --> 00:11:08.520
it uses a special CLI to align the

00:11:08.520 --> 00:11:11.480
versions and it then also mints every

00:11:11.480 --> 00:11:13.280
package to the same version when it

00:11:13.280 --> 00:11:15.520
releases which is again not something

00:11:15.520 --> 00:11:17.080
that you usually do out of a mono repo.

00:11:17.080 --> 00:11:18.880
So So there's something I didn't think

00:11:18.880 --> 00:11:21.440
about and Claude code um surfaced that

00:11:21.440 --> 00:11:23.680
early and I said I I think it prompted

00:11:23.680 --> 00:11:25.720
me for some questions uh so I said I

00:11:25.720 --> 00:11:27.600
want additive coexist. The existing

00:11:27.600 --> 00:11:28.960
workflows called to century blah blah

00:11:28.960 --> 00:11:31.040
blah. Oh I wanted it to be aware that

00:11:31.040 --> 00:11:32.840
about some things things that this thing

00:11:32.840 --> 00:11:34.600
does. It's it's a little bit weird. We

00:11:34.600 --> 00:11:38.000
inherited this from HashiCorp. And

00:11:38.000 --> 00:11:39.880
I just gave it a bit of hints here. And

00:11:39.880 --> 00:11:42.400
then after I was happy with the PR, it

00:11:42.400 --> 00:11:44.120
went ahead and created the PR. Now,

00:11:44.120 --> 00:11:46.600
where does the agent come in, right? So,

00:11:46.600 --> 00:11:48.200
the first thing that I found what the

00:11:48.200 --> 00:11:50.840
agent did really well was

00:11:50.840 --> 00:11:54.200
Claude Code didn't pin the

00:11:54.200 --> 00:11:57.000
the checksums of the GitHub workflows.

00:11:57.000 --> 00:12:00.000
And so, I asked we did not I didn't want

00:12:00.000 --> 00:12:01.720
>> You can get deterministic tools to do

00:12:01.720 --> 00:12:02.760
that, but carry on.

00:12:02.760 --> 00:12:04.760
>> Yeah, so that's exactly what I do here.

00:12:04.760 --> 00:12:06.800
And I did not want to integrate this

00:12:06.800 --> 00:12:08.480
more into this repo because I want to I

00:12:08.480 --> 00:12:09.760
don't want to keep adding on more and

00:12:09.760 --> 00:12:11.560
more tools around it. Not yet, at least

00:12:11.560 --> 00:12:14.000
not in this PR, right? So, so this is

00:12:14.000 --> 00:12:15.480
one of the things that that the agent

00:12:15.480 --> 00:12:17.040
was able to do for me while I was having

00:12:17.040 --> 00:12:19.240
a coffee. I told it I don't know if I

00:12:19.240 --> 00:12:21.160
can find a Discord thread. I could I

00:12:21.160 --> 00:12:23.120
could probably go find a Discord thread.

00:12:23.120 --> 00:12:25.120
>> Okay, I just Okay, I'm just I'll just

00:12:25.120 --> 00:12:27.920
want to replay this. So, you're asking

00:12:27.920 --> 00:12:30.760
uh you just you just prompted the AI to

00:12:30.760 --> 00:12:32.160
do the Sismore

00:12:32.160 --> 00:12:33.920
uh

00:12:33.920 --> 00:12:36.120
what do you call it? Lock in the on the

00:12:36.120 --> 00:12:38.400
GitHub action. I'm still a little bit

00:12:38.400 --> 00:12:40.120
puzzled why you why you even want to do

00:12:40.120 --> 00:12:42.280
that, but I mean it looks a bit

00:12:42.280 --> 00:12:44.320
overkill, but carry on. You there,

00:12:44.320 --> 00:12:45.040
Vincent?

00:12:45.040 --> 00:12:47.013
>> It's not overkill.

00:12:47.013 --> 00:12:47.720
>> [laughter]

00:12:47.720 --> 00:12:51.400
>> Um basically basically a code agent is

00:12:51.400 --> 00:12:53.560
not very deterministic and I don't trust

00:12:53.560 --> 00:12:55.839
it to put in the right checksums. So, I

00:12:55.839 --> 00:12:58.600
told Cloud to leave it at the tags

00:12:58.600 --> 00:13:01.600
mutable tags. And I told my agent to

00:13:01.600 --> 00:13:04.320
pull out the branch and run Sismore. And

00:13:04.320 --> 00:13:06.640
Sismore found a ton of issues with the

00:13:06.640 --> 00:13:09.160
GitHub workflows, right? So, it ignores

00:13:09.160 --> 00:13:11.560
every other finding. And I told it to

00:13:11.560 --> 00:13:14.360
only deterministically pin the checksum

00:13:14.360 --> 00:13:16.280
of these mutable tags. That's all I

00:13:16.280 --> 00:13:18.160
asked it to do. And so, the agent was

00:13:18.160 --> 00:13:21.000
very very simple. It it gave a a diff a

00:13:21.000 --> 00:13:22.920
patch. And then I went back to Cloud and

00:13:22.920 --> 00:13:24.520
I said, "Hey, there's a common common

00:13:24.520 --> 00:13:26.120
this is the actual deterministic

00:13:26.120 --> 00:13:28.560
determined checksum for you to pin.

00:13:28.560 --> 00:13:30.360
Okay, so don't I don't trust you. It's

00:13:30.360 --> 00:13:31.680
funny I trusted to do a lot of things

00:13:31.680 --> 00:13:33.760
but I didn't trust it to do that, okay?

00:13:33.760 --> 00:13:35.720
Um but okay, this is not the only

00:13:35.720 --> 00:13:38.320
scenario where I find an agent and being

00:13:38.320 --> 00:13:40.080
able to just execute whatever you want

00:13:40.080 --> 00:13:42.840
because again, Claude Codes sandboxes do

00:13:42.840 --> 00:13:45.360
not download binaries and do not uh let

00:13:45.360 --> 00:13:47.040
you do whatever you want, right?

00:13:47.040 --> 00:13:47.880
>> Mhm.

00:13:47.880 --> 00:13:50.120
Sorry, um I may be I may be I'm just

00:13:50.120 --> 00:13:51.920
being pedantic here but like you you

00:13:51.920 --> 00:13:53.480
you're doing this

00:13:53.480 --> 00:13:55.560
this pinning just just for the release.

00:13:55.560 --> 00:13:56.440
This

00:13:56.440 --> 00:13:57.560
uh you're you're generating the

00:13:57.560 --> 00:13:59.600
>> the repo.

00:13:59.600 --> 00:14:01.280
And Dependabot is configured to update

00:14:01.280 --> 00:14:03.920
them. So, we don't own them anyway. So,

00:14:03.920 --> 00:14:05.560
the only thing I wanted and I think

00:14:05.560 --> 00:14:07.480
Cloud delivered a PR where the workflows

00:14:07.480 --> 00:14:10.160
weren't pinned. Uh in any case, I think

00:14:10.160 --> 00:14:12.720
I I've made it a habit um to ask the

00:14:12.720 --> 00:14:14.640
agent the review agent that basically

00:14:14.640 --> 00:14:16.520
runs on any uh pull request that is

00:14:16.520 --> 00:14:19.280
whitelist sorry, allow listed. Um I ask

00:14:19.280 --> 00:14:21.280
it anyway to always do a baseline versus

00:14:21.280 --> 00:14:24.240
PR head check if the PR is um touching

00:14:24.240 --> 00:14:26.360
files and introducing significant new

00:14:26.360 --> 00:14:28.360
regressions against security. So, the

00:14:28.360 --> 00:14:30.360
agent is always going to do this now. Um

00:14:30.360 --> 00:14:32.440
I think at this stage it didn't do this

00:14:32.440 --> 00:14:34.960
by like automatically but I made it a

00:14:34.960 --> 00:14:36.640
rule that from now on the agent will

00:14:36.640 --> 00:14:38.000
always do this.

00:14:38.000 --> 00:14:40.240
Um but then, you know, Claude Code picked

00:14:40.240 --> 00:14:40.640
it up.

00:14:40.640 --> 00:14:42.520
>> I mean let me stop it. So, you're doing

00:14:42.520 --> 00:14:44.680
this so that you you you can improve the

00:14:44.680 --> 00:14:46.880
security posture of your of your GitHub

00:14:46.880 --> 00:14:49.640
workflow by making sure that the actions

00:14:49.640 --> 00:14:51.640
can't be injected and they're they're

00:14:51.640 --> 00:14:53.839
more dialed in and locked in so that

00:14:53.839 --> 00:14:55.360
they're they're less susceptible to

00:14:55.360 --> 00:14:57.240
attacks. That's why you're doing this.

00:14:57.240 --> 00:14:59.600
>> Yeah, in this day and age of supply

00:14:59.600 --> 00:15:01.720
chain attacks, we we really have to

00:15:01.720 --> 00:15:04.240
improve our CICD security stance

00:15:04.240 --> 00:15:06.800
definitely on GitHub workflows. So,

00:15:06.800 --> 00:15:08.560
the biggest issue with a public project

00:15:08.560 --> 00:15:10.080
like that is if when you do PRs like

00:15:10.080 --> 00:15:12.200
that, they get they get to they start to

00:15:12.200 --> 00:15:14.040
introduce too much change. And this is

00:15:14.040 --> 00:15:15.280
one of the things I want to touch upon

00:15:15.280 --> 00:15:18.000
because AI agents are creating massive

00:15:18.000 --> 00:15:20.240
PRs. And one of the things I learned

00:15:20.240 --> 00:15:22.720
with agents as well is to really find a

00:15:22.720 --> 00:15:25.520
way to to make PRs like this ties into

00:15:25.520 --> 00:15:27.760
your earlier comment about reducing the

00:15:27.760 --> 00:15:29.480
the number of lines and this ponytail

00:15:29.480 --> 00:15:31.640
skill which I want to get to eventually

00:15:31.640 --> 00:15:33.840
if you let me get to that point.

00:15:33.840 --> 00:15:36.640
Um, right? So so the point is I'm trying

00:15:36.640 --> 00:15:39.040
to explain to you that the agent is able

00:15:39.040 --> 00:15:41.120
to run simulations that I cannot do in

00:15:41.120 --> 00:15:43.080
Claude Code because Claude Code is a

00:15:43.080 --> 00:15:45.320
provided sandbox by Entropic that

00:15:45.320 --> 00:15:47.400
doesn't download binaries. It doesn't

00:15:47.400 --> 00:15:49.560
install uh things. It doesn't run a

00:15:49.560 --> 00:15:51.320
whole bunch of things. And I can

00:15:51.320 --> 00:15:54.200
delegate those type of simulations to my

00:15:54.200 --> 00:15:56.040
um agent that's running on a box that I

00:15:56.040 --> 00:15:57.600
control. That's the main thing I'm

00:15:57.600 --> 00:15:58.360
trying to say.

00:15:58.360 --> 00:15:58.800
>> Okay.

00:15:58.800 --> 00:16:00.440
>> Aside from all of this pedantic, you

00:16:00.440 --> 00:16:01.880
know, narrowing into this particular

00:16:01.880 --> 00:16:04.080
case, I'm just giving some examples and

00:16:04.080 --> 00:16:05.320
this is just one of them and I haven't

00:16:05.320 --> 00:16:06.560
even come to the second one.

00:16:06.560 --> 00:16:08.040
>> Can you show the Can you actually show

00:16:08.040 --> 00:16:10.280
the simulation run? I'm still a little

00:16:10.280 --> 00:16:10.600
bit

00:16:10.600 --> 00:16:12.840
>> Let's focus on the second one because

00:16:12.840 --> 00:16:14.280
running this more

00:16:14.280 --> 00:16:16.920
uh on on a branch and then you focusing

00:16:16.920 --> 00:16:19.040
on the actual diff is is is not that

00:16:19.040 --> 00:16:20.520
interesting as what I actually the

00:16:20.520 --> 00:16:23.920
second simulation does. Um, so the the

00:16:23.920 --> 00:16:26.360
the agent is able to run a bunch of

00:16:26.360 --> 00:16:27.880
simulations

00:16:27.880 --> 00:16:30.760
and it found some other issues, but then

00:16:30.760 --> 00:16:32.480
ultimately I think what was the most

00:16:32.480 --> 00:16:36.040
interesting one was this one.

00:16:36.040 --> 00:16:39.320
Where I ask it to merge locally the PR

00:16:39.320 --> 00:16:41.600
into main and run release, please to

00:16:41.600 --> 00:16:43.720
actually see what the effect is. Which

00:16:43.720 --> 00:16:45.400
this is something as a maintainer that

00:16:45.400 --> 00:16:47.000
you sometimes would need to do, right? I

00:16:47.000 --> 00:16:49.160
mean, if you if you're getting PRs, if

00:16:49.160 --> 00:16:52.440
the code looks good, but um until you

00:16:52.440 --> 00:16:54.960
actually check out the branch and maybe

00:16:54.960 --> 00:16:57.400
run the tests, you don't know some of

00:16:57.400 --> 00:16:59.560
the like changes work the way you expect

00:16:59.560 --> 00:17:01.440
them to work. So again,

00:17:01.440 --> 00:17:03.000
um I would really love to show you like

00:17:03.000 --> 00:17:05.400
the Discord uh if I have Discord here,

00:17:05.400 --> 00:17:05.640
but

00:17:05.640 --> 00:17:06.480
>> So

00:17:06.480 --> 00:17:07.480
>> I don't have it here.

00:17:07.480 --> 00:17:09.520
>> understanding you right, Vincent, that

00:17:09.520 --> 00:17:12.280
your your simulation is like replace

00:17:12.280 --> 00:17:14.079
Well, not replacing, but augmenting a

00:17:14.079 --> 00:17:15.480
human

00:17:15.480 --> 00:17:18.880
to do to do some um to do some smoke

00:17:18.880 --> 00:17:22.280
tests and to prod around and do things

00:17:22.280 --> 00:17:24.920
that like a a deterministic test

00:17:24.920 --> 00:17:27.400
couldn't hope to do, right?

00:17:27.400 --> 00:17:29.160
Yes. It's probably named

00:17:29.160 --> 00:17:30.760
it's I think it's called exploratory

00:17:30.760 --> 00:17:32.800
testing, isn't it?

00:17:32.800 --> 00:17:36.200
Yes. So so I want to show you the thread

00:17:36.200 --> 00:17:38.640
of of So this is all of the PR reviews

00:17:38.640 --> 00:17:40.880
that my bot's doing. And at the time it

00:17:40.880 --> 00:17:43.600
didn't have nice threads per PR. Now it

00:17:43.600 --> 00:17:47.840
does. I I I failing to find the find the

00:17:47.840 --> 00:17:49.640
search button cuz I actually want to

00:17:49.640 --> 00:17:52.240
show you the prompt that I sent from my

00:17:52.240 --> 00:17:53.320
phone

00:17:53.320 --> 00:17:55.160
to for it to do basically do the

00:17:55.160 --> 00:17:57.000
simulation that I asked it to do, right?

00:17:57.000 --> 00:17:59.520
This this particular one right here.

00:17:59.520 --> 00:18:02.520
I asked it merge this PR in in a like a

00:18:02.520 --> 00:18:05.120
temporary directory, merge this branch

00:18:05.120 --> 00:18:07.760
into main, and run release, please.

00:18:07.760 --> 00:18:10.080
And it actually highlighted a whole

00:18:10.080 --> 00:18:12.160
bunch of things. Like first of the title

00:18:12.160 --> 00:18:15.280
didn't meet the the the the convention,

00:18:15.280 --> 00:18:17.440
but it has a PR title linter in this

00:18:17.440 --> 00:18:19.120
repository, right? It has a regular

00:18:19.120 --> 00:18:21.840
expression that runs across PR titles,

00:18:21.840 --> 00:18:23.920
and the release please configuration

00:18:23.920 --> 00:18:26.440
created a title that did not meet our

00:18:26.440 --> 00:18:28.440
expectations or our our requirements.

00:18:28.440 --> 00:18:30.240
Like the description of the PR said it

00:18:30.240 --> 00:18:31.840
will create a PR like this, but the

00:18:31.840 --> 00:18:33.600
actual created dry run output was

00:18:33.600 --> 00:18:35.720
different. Right? Yeah, and another

00:18:35.720 --> 00:18:37.680
thing it found was

00:18:37.680 --> 00:18:38.960
Yeah, the this is where it says the

00:18:38.960 --> 00:18:40.520
format's supposed to be type scope

00:18:40.520 --> 00:18:42.840
message, and this doesn't match that.

00:18:42.840 --> 00:18:44.520
What we expect is we should probably

00:18:44.520 --> 00:18:45.960
have something like that. And then

00:18:45.960 --> 00:18:48.000
another thing was

00:18:48.000 --> 00:18:49.200
um

00:18:49.200 --> 00:18:51.200
a problem. There was another problem, I

00:18:51.200 --> 00:18:52.360
think. I mean, this still seems a little

00:18:52.360 --> 00:18:54.800
bit like pedantic territory, but but

00:18:54.800 --> 00:18:57.800
okay. pedantic? This is this is Okay,

00:18:57.800 --> 00:18:59.760
I'm taking offense to this because this

00:18:59.760 --> 00:19:01.520
is exactly the type of thing as a

00:19:01.520 --> 00:19:04.000
maintainer you need to do. And I want to

00:19:04.000 --> 00:19:06.920
get this stuff through because this is I

00:19:06.920 --> 00:19:09.000
want to get as much validation around

00:19:09.000 --> 00:19:11.440
the pull request as I can uh, without

00:19:11.440 --> 00:19:13.680
having to leave whatever I'm doing, go

00:19:13.680 --> 00:19:16.000
to my machine, and and go and and, you

00:19:16.000 --> 00:19:17.320
know, check out the branch. Yes, you can

00:19:17.320 --> 00:19:18.760
say like, of course you can go to your

00:19:18.760 --> 00:19:21.240
laptop and set up a temp directory,

00:19:21.240 --> 00:19:23.880
check out the the the branch, merge it,

00:19:23.880 --> 00:19:26.400
run the dry run, look at it. And and it

00:19:26.400 --> 00:19:27.720
would have definitely had values because

00:19:27.720 --> 00:19:29.400
I there was actually something that came

00:19:29.400 --> 00:19:31.440
up after this PR was merged, which

00:19:31.440 --> 00:19:32.640
proves that no matter how much

00:19:32.640 --> 00:19:34.240
validation you do, we'll still have ex

00:19:34.240 --> 00:19:35.680
surprises.

00:19:35.680 --> 00:19:37.040
Um, but I don't think this is pedantic

00:19:37.040 --> 00:19:39.720
at all. This is literally like, um,

00:19:39.720 --> 00:19:43.280
changing my life in me not being, you

00:19:43.280 --> 00:19:44.920
know, required to go to a machine and do

00:19:44.920 --> 00:19:45.920
the checkout, right?

00:19:45.920 --> 00:19:47.480
>> Yeah, I guess I guess you're right. For

00:19:47.480 --> 00:19:49.840
for maintainer doing releases, if you

00:19:49.840 --> 00:19:52.080
can if you can sort of automate

00:19:52.080 --> 00:19:53.880
>> on this particular example. Do the

00:19:53.880 --> 00:19:56.040
release please part is the is the minor

00:19:56.040 --> 00:19:58.960
part, okay? The fact that you can get a

00:19:58.960 --> 00:20:01.040
pull request from a contributor and run

00:20:01.040 --> 00:20:03.040
a simulation, no matter which type of

00:20:03.040 --> 00:20:05.200
simulation, is the interesting part, not

00:20:05.200 --> 00:20:06.880
the fact that I'm doing a a release

00:20:06.880 --> 00:20:08.200
please merge and then running the dry

00:20:08.200 --> 00:20:09.360
run of release please. That's just a

00:20:09.360 --> 00:20:10.040
minor thing.

00:20:10.040 --> 00:20:12.200
>> I'm just I mean, yeah, I'm I'm I'm

00:20:12.200 --> 00:20:13.720
playing devil's advocate. I I I I am

00:20:13.720 --> 00:20:15.480
curious why you call why you call it a

00:20:15.480 --> 00:20:18.160
simulation. By the way, which agent are

00:20:18.160 --> 00:20:18.320
you

00:20:18.320 --> 00:20:19.840
>> How is this not a simulation? Explain it

00:20:19.840 --> 00:20:20.240
to me.

00:20:20.240 --> 00:20:22.360
>> I I know it's I just think it's like an

00:20:22.360 --> 00:20:24.440
interesting word, like simulation. It's

00:20:24.440 --> 00:20:25.240
it's sounds

00:20:25.240 --> 00:20:26.800
>> Give me a better I'm not English native.

00:20:26.800 --> 00:20:28.080
Give me a better word for what it's

00:20:28.080 --> 00:20:28.720
doing here.

00:20:28.720 --> 00:20:29.680
>> Um,

00:20:29.680 --> 00:20:30.360
>> A dry run?

00:20:30.360 --> 00:20:31.120
>> Well, I

00:20:31.120 --> 00:20:33.360
>> A simulation, right?

00:20:33.360 --> 00:20:35.640
>> I mean, I was I was thinking exploratory

00:20:35.640 --> 00:20:38.080
testing earlier, but you

00:20:38.080 --> 00:20:39.640
>> What is exploratory testing? I don't

00:20:39.640 --> 00:20:40.280
know this

00:20:40.280 --> 00:20:41.600
>> It's when

00:20:41.600 --> 00:20:43.920
I think it's a I think it's a thing.

00:20:43.920 --> 00:20:44.840
>> Sounds like freezing.

00:20:44.840 --> 00:20:46.880
>> Exploratory,

00:20:46.880 --> 00:20:48.440
if I can ex- if I could spell

00:20:48.440 --> 00:20:50.320
exploratory testing. Hold on, let me

00:20:50.320 --> 00:20:51.040
share the screen.

00:20:51.040 --> 00:20:52.560
>> Okay. I don't think the term matters

00:20:52.560 --> 00:20:55.520
much. The the thing is, I

00:20:55.520 --> 00:20:57.960
>> well, I mean, you Exploratory testing is

00:20:57.960 --> 00:21:00.360
is a big part of what we do. Hands-on

00:21:00.360 --> 00:21:01.680
software testing where we test is

00:21:01.680 --> 00:21:04.640
actively explore an application without

00:21:04.640 --> 00:21:06.360
predefined test cases.

00:21:06.360 --> 00:21:07.800
>> Okay. And what is the problem with the

00:21:07.800 --> 00:21:10.080
term uh, simulation? Like what what what

00:21:10.080 --> 00:21:11.440
is exactly

00:21:11.440 --> 00:21:13.800
that that that is the wrong use of that

00:21:13.800 --> 00:21:16.040
word. I I don't really see how that's a

00:21:16.040 --> 00:21:16.160
wrong

00:21:16.160 --> 00:21:18.120
>> Well, I mean simulation it's it's

00:21:18.120 --> 00:21:19.600
interesting. I just I just think of the

00:21:19.600 --> 00:21:21.880
matrix, you know, I'm just thinking like

00:21:21.880 --> 00:21:24.520
now you have agents simulating your life

00:21:24.520 --> 00:21:25.840
and

00:21:25.840 --> 00:21:28.320
I know, it's it's a bit broad. Anyway,

00:21:28.320 --> 00:21:30.880
it Okay, it's interesting, but I got two

00:21:30.880 --> 00:21:32.680
questions for you. Which agent are you

00:21:32.680 --> 00:21:35.120
using with Hermes? Second, on this new

00:21:35.120 --> 00:21:37.400
gig, to my surprise with GitHub Actions,

00:21:37.400 --> 00:21:39.200
I just wanted to note that like they

00:21:39.200 --> 00:21:41.000
they disallow anything that doesn't

00:21:41.000 --> 00:21:43.800
start with action slash. So, any like

00:21:43.800 --> 00:21:45.640
third-party actions are not allowed.

00:21:45.640 --> 00:21:47.480
You're only allowed to use the action

00:21:47.480 --> 00:21:49.080
slash, which I guess is supported by

00:21:49.080 --> 00:21:50.760
GitHub only.

00:21:50.760 --> 00:21:51.520
I just thought I'd mention

00:21:51.520 --> 00:21:53.200
>> GitHub scripts? Yeah, that starts with

00:21:53.200 --> 00:21:55.120
action slash. That just gives you a a

00:21:55.120 --> 00:21:57.280
major gaping hole in that.

00:21:57.280 --> 00:21:58.240
>> Yeah, because

00:21:58.240 --> 00:22:00.240
>> That's just hackliest security to me.

00:22:00.240 --> 00:22:01.680
>> Yeah, that that's it's interesting

00:22:01.680 --> 00:22:04.680
because like I couldn't install I think

00:22:04.680 --> 00:22:07.360
it was pre-commit and I couldn't install

00:22:07.360 --> 00:22:09.680
UV. So, what what you end up having to

00:22:09.680 --> 00:22:11.960
do is basically using actions Python and

00:22:11.960 --> 00:22:14.080
then and then pip or or you don't even

00:22:14.080 --> 00:22:15.960
need actions Python, but you you end up

00:22:15.960 --> 00:22:18.560
just using pip install and your your

00:22:18.560 --> 00:22:21.080
your actions become become harder to

00:22:21.080 --> 00:22:23.000
maintain because because you're not

00:22:23.000 --> 00:22:24.720
using the proper action.

00:22:24.720 --> 00:22:26.920
Okay, but yeah, what what agent are you

00:22:26.920 --> 00:22:28.320
using with your Hermes?

00:22:28.320 --> 00:22:30.240
>> Why would I not want to say? I use

00:22:30.240 --> 00:22:32.520
because with ChatGPT, okay, Anthropic

00:22:32.520 --> 00:22:34.920
does not allow you to use its its

00:22:34.920 --> 00:22:36.640
subscription plan with with these

00:22:36.640 --> 00:22:39.240
agents, right? Open Open Claw and

00:22:39.240 --> 00:22:42.040
Hermes, you have to use the API key. If

00:22:42.040 --> 00:22:44.680
the Claude Code harness detects activity

00:22:44.680 --> 00:22:47.120
like nested calling and and things like

00:22:47.120 --> 00:22:49.960
that, it will automatically use API key

00:22:49.960 --> 00:22:52.160
wallet instead of your actual

00:22:52.160 --> 00:22:54.280
subscription usage. Like they put a lot

00:22:54.280 --> 00:22:56.520
of things around it. Yeah. Uh so, some

00:22:56.520 --> 00:22:57.960
people got a surprise I think I got this

00:22:57.960 --> 00:23:00.560
from reading the Hermes Reddit threads

00:23:00.560 --> 00:23:02.240
where people were saying like how do I

00:23:02.240 --> 00:23:04.720
why am I getting paid like charged why

00:23:04.720 --> 00:23:06.440
is my wallet going down when when I'm

00:23:06.440 --> 00:23:08.600
using 12 C on Hermes. It detects these

00:23:08.600 --> 00:23:09.720
things, right? Initially it

00:23:09.720 --> 00:23:10.960
automatically banned your account. I

00:23:10.960 --> 00:23:12.640
guess they went back back from that and

00:23:12.640 --> 00:23:14.160
now they are just trying to make sure

00:23:14.160 --> 00:23:14.520
you pay.

00:23:14.520 --> 00:23:17.240
>> Well, yeah. I I'm always in favor of

00:23:17.240 --> 00:23:19.960
monetary incentives or penalties the way

00:23:19.960 --> 00:23:20.960
Singapore does it.

00:23:20.960 --> 00:23:23.800
>> Yeah. So so what I really wanted for

00:23:23.800 --> 00:23:24.800
these

00:23:24.800 --> 00:23:26.320
Oh my now it's saying my phone is

00:23:26.320 --> 00:23:28.160
running out of battery. So so basically

00:23:28.160 --> 00:23:30.040
what I what I really wanted for this

00:23:30.040 --> 00:23:33.560
experiment was to leverage the chat GPT

00:23:33.560 --> 00:23:36.520
Codex budget with a simple plus

00:23:36.520 --> 00:23:38.440
subscription you get a certain amount

00:23:38.440 --> 00:23:41.320
and so Codex GPT 5.5 on the first week

00:23:41.320 --> 00:23:43.280
it ran out after 3 days and then I have

00:23:43.280 --> 00:23:46.720
a a fallback. So with with these agents

00:23:46.720 --> 00:23:49.200
you can you have your main harness that

00:23:49.200 --> 00:23:51.960
does most of the work like tool use and

00:23:51.960 --> 00:23:53.760
and reasoning and so on and then you

00:23:53.760 --> 00:23:56.640
have auxiliary calls such as summarizing

00:23:56.640 --> 00:23:59.160
the conversation which I think is the

00:23:59.160 --> 00:24:00.040
one of the things of

00:24:00.040 --> 00:24:01.880
>> The hardest Your internet is going to

00:24:01.880 --> 00:24:02.440
hell.

00:24:02.440 --> 00:24:04.200
>> Yeah, the hardest thing to get around

00:24:04.200 --> 00:24:05.240
>> If if

00:24:05.240 --> 00:24:07.160
>> It's it's using my phone. My phone isn't

00:24:07.160 --> 00:24:09.040
that it's just charging. My audio's

00:24:09.040 --> 00:24:10.160
going to come through fine. It's

00:24:10.160 --> 00:24:11.760
recording from my laptop anyway.

00:24:11.760 --> 00:24:14.560
>> Yeah, yeah. Your audio did just stop for

00:24:14.560 --> 00:24:16.240
a second or two. Don't worry. You could

00:24:16.240 --> 00:24:16.640
continue

00:24:16.640 --> 00:24:19.280
>> recorded on my my laptop so so it's not

00:24:19.280 --> 00:24:22.080
a not not not a concern. So what I was

00:24:22.080 --> 00:24:24.240
saying is that with these agents they

00:24:24.240 --> 00:24:26.560
use different API different LLM

00:24:26.560 --> 00:24:29.440
endpoints based on what they need to do.

00:24:29.440 --> 00:24:32.000
So auxiliary functionality such as

00:24:32.000 --> 00:24:34.160
summarizing sessions can use a different

00:24:34.160 --> 00:24:36.440
endpoint. I'm using Deep Seek Flash for

00:24:36.440 --> 00:24:41.520
that and main agent is Codex GPT 5.5 on

00:24:41.520 --> 00:24:44.000
the chat GPT plus so you get about 3

00:24:44.000 --> 00:24:46.320
days out of a week with Hermes agent or

00:24:46.320 --> 00:24:48.600
moderate use and then it falls back. So

00:24:48.600 --> 00:24:51.800
the moment the chat GPT API returns I

00:24:51.800 --> 00:24:53.760
don't know what is the HTTP the HTTP

00:24:53.760 --> 00:24:56.200
code for you you your limit. At that

00:24:56.200 --> 00:24:58.280
point do still hear me? At at that point

00:24:58.280 --> 00:24:59.560
it falls back to DPC.

00:24:59.560 --> 00:25:01.400
>> So, I'm I'm trying not to I'm trying not

00:25:01.400 --> 00:25:03.600
to interrupt you so that the the flow is

00:25:03.600 --> 00:25:04.720
>> DeepSeek 4 Pro.

00:25:04.720 --> 00:25:06.240
>> intelligible while listening.

00:25:06.240 --> 00:25:08.120
>> DeepSeek, okay.

00:25:08.120 --> 00:25:09.160
Yeah, it works.

00:25:09.160 --> 00:25:10.960
>> Oh, yeah, you mentioned the the

00:25:10.960 --> 00:25:12.960
>> Let me try and and I don't know if I

00:25:12.960 --> 00:25:14.920
should reconnect, but

00:25:14.920 --> 00:25:16.560
I don't know why my phone is now super

00:25:16.560 --> 00:25:17.920
slow. Maybe my data is

00:25:17.920 --> 00:25:19.240
>> can try switch back. This still

00:25:19.240 --> 00:25:19.760
work.

00:25:19.760 --> 00:25:21.000
>> Okay, I might drop out in a bit.

00:25:21.000 --> 00:25:22.320
>> it will work if you want to come back.

00:25:22.320 --> 00:25:24.000
Try Try and switch back to your fiber or

00:25:24.000 --> 00:25:25.360
something. Just try.

00:25:25.360 --> 00:25:25.920
>> Test.

00:25:25.920 --> 00:25:27.480
>> Okay, you're back. Yeah, it looks a

00:25:27.480 --> 00:25:28.520
little bit more stable.

00:25:28.520 --> 00:25:30.440
>> I'm on the Wi-Fi now. There's no other

00:25:30.440 --> 00:25:32.120
laptop on the stand-up anymore.

00:25:32.120 --> 00:25:33.280
Hopefully it should be better.

00:25:33.280 --> 00:25:34.920
>> You don't I guess you don't have wired

00:25:34.920 --> 00:25:37.080
in your in your home for your

00:25:37.080 --> 00:25:38.280
>> wired?

00:25:38.280 --> 00:25:41.160
>> Wired, like like a actual physical

00:25:41.160 --> 00:25:41.800
network.

00:25:41.800 --> 00:25:43.960
>> I do. I do. I do have a switch on this

00:25:43.960 --> 00:25:46.280
desk, but with Apple you need to then

00:25:46.280 --> 00:25:49.000
like connect a an add-on with Apple.

00:25:49.000 --> 00:25:50.480
It should be fine now. The connection is

00:25:50.480 --> 00:25:51.480
stable.

00:25:51.480 --> 00:25:54.560
>> I I use my monitors for like network and

00:25:54.560 --> 00:25:56.600
powering, you know, like one cable.

00:25:56.600 --> 00:25:59.360
>> One cable. Yeah, so I think the most the

00:25:59.360 --> 00:26:01.120
the most difficult part of switching to

00:26:01.120 --> 00:26:03.120
agents and then you have Discord where

00:26:03.120 --> 00:26:04.640
you send a message or you mentioned it

00:26:04.640 --> 00:26:07.000
and then it creates a thread is to map

00:26:07.000 --> 00:26:09.000
that back to the session. Because if

00:26:09.000 --> 00:26:10.120
you're on a Claude Code, you're in a

00:26:10.120 --> 00:26:12.240
session, you see your context usage, you

00:26:12.240 --> 00:26:14.720
can you use sub-agents, you can see how

00:26:14.720 --> 00:26:17.000
much context you're using. When I have

00:26:17.000 --> 00:26:18.520
Discord and I ask the agent to do

00:26:18.520 --> 00:26:21.520
something, I can't see none of that. And

00:26:21.520 --> 00:26:23.920
I'm also using it more like like a cloud

00:26:23.920 --> 00:26:25.880
code in in Discord rather than than an

00:26:25.880 --> 00:26:27.800
actual independent agent that finds work

00:26:27.800 --> 00:26:28.840
to do and does its own work.

00:26:28.840 --> 00:26:30.200
>> That that cloud chat, the thing you

00:26:30.200 --> 00:26:32.640
shared, show show all the tool calls. I

00:26:32.640 --> 00:26:33.920
don't think I don't think it does, does

00:26:33.920 --> 00:26:34.240
it?

00:26:34.240 --> 00:26:35.600
>> You want me to share the screen?

00:26:35.600 --> 00:26:38.600
>> Yeah, show me your your Claude trace.

00:26:38.600 --> 00:26:41.080
>> It said run command.

00:26:41.080 --> 00:26:43.720
Run tool command user tool check status

00:26:43.720 --> 00:26:44.800
run read tool.

00:26:44.800 --> 00:26:46.520
>> If there's if there's any takeaway that

00:26:46.520 --> 00:26:48.280
I've learned from talking with you this

00:26:48.280 --> 00:26:50.040
morning is that the

00:26:50.040 --> 00:26:53.400
the Claude AI session

00:26:53.400 --> 00:26:56.200
um transcript there is a lot better than

00:26:56.200 --> 00:26:56.920
I remember.

00:26:56.920 --> 00:26:58.840
>> And I never saw it before and I asked

00:26:58.840 --> 00:27:00.880
someone who has Claude's Enterprise and

00:27:00.880 --> 00:27:02.520
I said, "So, if it's Enterprise, does it

00:27:02.520 --> 00:27:04.120
allow the team to see other people's

00:27:04.120 --> 00:27:06.040
session?" And he says, "No."

00:27:06.040 --> 00:27:07.880
So, I thought that's a little bit That's

00:27:07.880 --> 00:27:08.400
sad.

00:27:08.400 --> 00:27:09.400
>> That's really cool.

00:27:09.400 --> 00:27:11.280
>> It's all the the the questions and

00:27:11.280 --> 00:27:13.040
everything.

00:27:13.040 --> 00:27:14.680
I'm I'm sure they'll make this better.

00:27:14.680 --> 00:27:16.480
>> was only um It was only the other day I

00:27:16.480 --> 00:27:19.560
saw the GitHub Enterprise uh control

00:27:19.560 --> 00:27:21.680
panel for the first time. I think it I'm

00:27:21.680 --> 00:27:23.840
going to maybe sign up for a GitHub

00:27:23.840 --> 00:27:26.360
Enterprise just on a personal basis just

00:27:26.360 --> 00:27:29.040
so I can I can get an idea of all the

00:27:29.040 --> 00:27:30.480
changes that are happening on the

00:27:30.480 --> 00:27:32.680
Enterprise panel because I I didn't know

00:27:32.680 --> 00:27:34.880
about all the controls that you have and

00:27:34.880 --> 00:27:38.000
I need to know because I'm a I'm a AI

00:27:38.000 --> 00:27:39.640
guardrail

00:27:39.640 --> 00:27:40.800
engineer right now.

00:27:40.800 --> 00:27:43.680
>> Yeah. Forcing people to use actions is

00:27:43.680 --> 00:27:47.240
such a stupid security rule rule. I

00:27:47.240 --> 00:27:49.920
mean, I think what you should do is to

00:27:49.920 --> 00:27:52.960
have a white list and an allow list of

00:27:52.960 --> 00:27:55.000
actions that are supported and then a

00:27:55.000 --> 00:27:57.880
process of requesting um a new action to

00:27:57.880 --> 00:27:59.800
be used in a workflow cuz like you said,

00:27:59.800 --> 00:28:01.160
you're going to end up with like

00:28:01.160 --> 00:28:03.680
workarounds, unmaintainable workflows,

00:28:03.680 --> 00:28:05.600
and then even if you use this script

00:28:05.600 --> 00:28:07.560
which is runs JavaScript, you can do

00:28:07.560 --> 00:28:09.680
whatever you want and this isn't actions

00:28:09.680 --> 00:28:11.240
one. So, security

00:28:11.240 --> 00:28:13.000
>> Yeah, you can do what whatever you want

00:28:13.000 --> 00:28:16.640
with actions / Python really or you

00:28:16.640 --> 00:28:18.840
know, you can There's a ton of ways to

00:28:18.840 --> 00:28:20.560
get around it. Yeah, I mean, I'm not I'm

00:28:20.560 --> 00:28:22.800
not supporting this whole actions prefix

00:28:22.800 --> 00:28:24.160
thing. I just think it's interesting

00:28:24.160 --> 00:28:26.080
that it's even done that way.

00:28:26.080 --> 00:28:28.560
>> Yeah, and it doesn't work very well. And

00:28:28.560 --> 00:28:30.240
this is something I I I learned a lot

00:28:30.240 --> 00:28:32.000
about and you see a lot of solutions now

00:28:32.000 --> 00:28:33.800
about not just these GitHub actions that

00:28:33.800 --> 00:28:35.720
can be used, but also what skills you

00:28:35.720 --> 00:28:38.440
can use or what MCP service you can use

00:28:38.440 --> 00:28:38.920
>> Yeah.

00:28:38.920 --> 00:28:40.880
>> in an organization and

00:28:40.880 --> 00:28:42.560
>> that that's what I'm I'm currently

00:28:42.560 --> 00:28:45.160
working on right now is is is allow

00:28:45.160 --> 00:28:48.800
listing MCPs, allow listing the tools,

00:28:48.800 --> 00:28:50.880
whether the tools are blocked, whether

00:28:50.880 --> 00:28:51.960
they

00:28:51.960 --> 00:28:54.440
require approval, or whether they always

00:28:54.440 --> 00:28:56.480
allow. I'm doing This is my This is my

00:28:56.480 --> 00:28:58.880
day job right now, seriously.

00:28:58.880 --> 00:28:59.200
>> Yeah.

00:28:59.200 --> 00:29:01.600
>> And it the the the magical I I mentioned

00:29:01.600 --> 00:29:03.080
it on a tweet, and I think you replied

00:29:03.080 --> 00:29:04.640
to the tweet, but the the thing that

00:29:04.640 --> 00:29:06.640
clicked for me with MCPs is that which

00:29:06.640 --> 00:29:07.960
it and it we know we've been

00:29:07.960 --> 00:29:11.200
bad-mouthing MCPs for a while, but it

00:29:11.200 --> 00:29:12.920
really clicked for me that that there's

00:29:12.920 --> 00:29:15.400
all the support in the enterprise for

00:29:15.400 --> 00:29:17.160
it, so that the user experience is

00:29:17.160 --> 00:29:18.680
great. You log into

00:29:18.680 --> 00:29:19.280
uh

00:29:19.280 --> 00:29:21.160
into your Claude session, and you're

00:29:21.160 --> 00:29:23.000
logged into Google Drive, you're logged

00:29:23.000 --> 00:29:24.800
into Atlassian, you're logged into

00:29:24.800 --> 00:29:26.720
Figma, Miro, all that stuff. It is

00:29:26.720 --> 00:29:27.800
amazing.

00:29:27.800 --> 00:29:28.880
>> Absolutely.

00:29:28.880 --> 00:29:30.960
>> And it it is actually like a marvel of

00:29:30.960 --> 00:29:33.320
technology. It almost sent chills up my

00:29:33.320 --> 00:29:35.800
spine cuz like in the Google enterprise,

00:29:35.800 --> 00:29:38.320
that that that that connector, when you

00:29:38.320 --> 00:29:42.040
connect your your Claude to your, you

00:29:42.040 --> 00:29:44.520
know, your your Slack workspace, or your

00:29:44.520 --> 00:29:47.240
your Google workspace, it's amazing.

00:29:47.240 --> 00:29:49.320
It's amazing that that this that this

00:29:49.320 --> 00:29:52.360
cross-authentication thing works. I

00:29:52.360 --> 00:29:53.760
mean, I I still need to wrap my head

00:29:53.760 --> 00:29:55.240
around it because like there's a

00:29:55.240 --> 00:29:57.840
connection between Claude and your and

00:29:57.840 --> 00:30:00.120
your your asset there, and then there's

00:30:00.120 --> 00:30:02.400
also the user authentication and the

00:30:02.400 --> 00:30:05.080
users' scopes that also come into play.

00:30:05.080 --> 00:30:07.200
It's It's amazing that it all works

00:30:07.200 --> 00:30:08.680
until you get an internal server error,

00:30:08.680 --> 00:30:11.200
but it's amazing when it works. And MCPs

00:30:11.200 --> 00:30:13.320
allow that, and and this is where I

00:30:13.320 --> 00:30:15.760
think it will probably tear it will

00:30:15.760 --> 00:30:18.120
These are the These are the pain points

00:30:18.120 --> 00:30:19.840
that it gets rid of, which is really

00:30:19.840 --> 00:30:22.680
powerful. Because with skills, you you

00:30:22.680 --> 00:30:23.960
there's there's definitely a lot of

00:30:23.960 --> 00:30:25.880
people, including my own my own

00:30:25.880 --> 00:30:27.880
colleagues, who would say like "Kai, you

00:30:27.880 --> 00:30:30.840
can't have You can't have a Claude just

00:30:30.840 --> 00:30:33.920
calling an API key on your on your dot

00:30:33.920 --> 00:30:36.360
That's That's a That's You've divulged

00:30:36.360 --> 00:30:38.880
the key to Claude." And like, not sure

00:30:38.880 --> 00:30:40.720
you're right about that that You kind of

00:30:40.720 --> 00:30:43.360
need to have dot M access to get things

00:30:43.360 --> 00:30:45.680
done. And of course MCB's just solved

00:30:45.680 --> 00:30:47.280
that problem big time.

00:30:47.280 --> 00:30:49.040
>> Yeah, I I I think another thing where

00:30:49.040 --> 00:30:50.840
MCP's really solved the problem aside

00:30:50.840 --> 00:30:52.600
from the authentication and like maybe

00:30:52.600 --> 00:30:55.920
secret management. I think I've got a

00:30:55.920 --> 00:30:58.360
couple of situations where I really felt

00:30:58.360 --> 00:30:59.800
a skill couldn't do what it what what

00:30:59.800 --> 00:31:02.160
needed to be done. One of them is the

00:31:02.160 --> 00:31:04.800
ability to I mean let's look at the AWS

00:31:04.800 --> 00:31:07.840
docs MCP for example, right? It it's

00:31:07.840 --> 00:31:10.760
it exposes a it uses a search endpoint

00:31:10.760 --> 00:31:11.720
and then it

00:31:11.720 --> 00:31:14.840
sources fresh docs from their server

00:31:14.840 --> 00:31:17.480
which it's pretty cool and I wanted to

00:31:17.480 --> 00:31:19.200
>> That's actually number almost number one

00:31:19.200 --> 00:31:19.720
for me.

00:31:19.720 --> 00:31:21.320
>> Yeah, but but I wanted to to get the

00:31:21.320 --> 00:31:22.880
same thing for another like a vendor

00:31:22.880 --> 00:31:25.520
product not AWS. It's a gravitational

00:31:25.520 --> 00:31:27.640
teleport which is a remote access

00:31:27.640 --> 00:31:29.360
solution or privilege access management

00:31:29.360 --> 00:31:31.880
solution. And in this case, what I

00:31:31.880 --> 00:31:34.800
wanted was to to have these docs

00:31:34.800 --> 00:31:37.480
available to the LLM to make sure cuz I

00:31:37.480 --> 00:31:39.640
noticed that the the LLM uses web

00:31:39.640 --> 00:31:42.080
search, finds the teleport docs, goes

00:31:42.080 --> 00:31:44.040
through them and I found that that's not

00:31:44.040 --> 00:31:46.640
efficient and I was wondering if Google

00:31:46.640 --> 00:31:49.000
was really giving it like the right or

00:31:49.000 --> 00:31:50.360
whatever web search is using, right? I

00:31:50.360 --> 00:31:51.960
don't even know Anthropic is using

00:31:51.960 --> 00:31:54.080
Google as as a as a web search engine.

00:31:54.080 --> 00:31:57.760
It's maybe using something else and and

00:31:57.760 --> 00:32:00.440
or maybe it's its own like search engine

00:32:00.440 --> 00:32:00.680
because

00:32:00.680 --> 00:32:01.560
>> Yeah,

00:32:01.560 --> 00:32:03.320
it's a tool or something.

00:32:03.320 --> 00:32:05.520
>> Yeah, so so what I really liked about

00:32:05.520 --> 00:32:07.800
this when I tried to do this as a skill.

00:32:07.800 --> 00:32:10.520
So I I I thought hey, the the AWS docs

00:32:10.520 --> 00:32:12.760
MCP is pretty cool, but I want to do it

00:32:12.760 --> 00:32:15.480
as a skill and I wanted to to like fetch

00:32:15.480 --> 00:32:18.960
the go teleport data like in an

00:32:18.960 --> 00:32:21.320
efficient way. So the first approach was

00:32:21.320 --> 00:32:23.800
apparently the go docs the the docs they

00:32:23.800 --> 00:32:27.800
use like these LLM.txt. So every human

00:32:27.800 --> 00:32:29.880
document which is HTML nicely rendered

00:32:29.880 --> 00:32:33.600
diagrams has a sibling .md markdown

00:32:33.600 --> 00:32:36.320
version that an LLM can use. So I wasn't

00:32:36.320 --> 00:32:38.040
even sure if the if the agent was

00:32:38.040 --> 00:32:39.160
finding it, but I think this is a

00:32:39.160 --> 00:32:40.920
protocol that these agents are adopting.

00:32:40.920 --> 00:32:42.960
So, I think that the if they do a

00:32:42.960 --> 00:32:44.840
request, I am an agent, they will get

00:32:44.840 --> 00:32:46.440
the markdown instead of the HTML. I

00:32:46.440 --> 00:32:47.720
think that that might be already in

00:32:47.720 --> 00:32:49.360
place. So, they use something called

00:32:49.360 --> 00:32:51.440
Inkeep to do that. But then the second

00:32:51.440 --> 00:32:54.080
problem was I ingested all of these

00:32:54.080 --> 00:32:56.840
markdown into like an index, and I did a

00:32:56.840 --> 00:32:59.480
local lexical search. So, to find the

00:32:59.480 --> 00:33:02.040
most relevant documents very quickly

00:33:02.040 --> 00:33:03.400
instead of having to go to Google web

00:33:03.400 --> 00:33:05.600
search. And and then I was like, hold on

00:33:05.600 --> 00:33:07.080
a minute, this doesn't take into account

00:33:07.080 --> 00:33:08.960
if the term that the that is being

00:33:08.960 --> 00:33:10.560
searched for is actually on the page

00:33:10.560 --> 00:33:13.080
itself, right? And if I go into the

00:33:13.080 --> 00:33:16.160
Teleport Docs and I enter search, it's

00:33:16.160 --> 00:33:18.280
actually doing a semantic search. So, if

00:33:18.280 --> 00:33:20.840
I talk about like privilege escalation,

00:33:20.840 --> 00:33:23.120
it will automatically surface articles

00:33:23.120 --> 00:33:25.400
about requests like access requests and

00:33:25.400 --> 00:33:27.040
approval workflows, right? Which is like

00:33:27.040 --> 00:33:28.640
semantically related, but not exactly

00:33:28.640 --> 00:33:31.000
lexically matching. So, so that's where

00:33:31.000 --> 00:33:32.920
an MCP can do really nice things because

00:33:32.920 --> 00:33:35.000
if I want to do that content-based

00:33:35.000 --> 00:33:37.720
semantic search, now I have to like ship

00:33:37.720 --> 00:33:39.960
a Chroma DB or a vector DB, and I have

00:33:39.960 --> 00:33:42.560
to also be able to embed on the client.

00:33:42.560 --> 00:33:44.320
That means there must be an LLM call to

00:33:44.320 --> 00:33:46.200
for the embedding model, which again I

00:33:46.200 --> 00:33:48.680
cannot do as a skill. So, I worked

00:33:48.680 --> 00:33:51.280
around it, but ultimately I did an eval

00:33:51.280 --> 00:33:52.880
comparing Google results against my

00:33:52.880 --> 00:33:55.480
final skill using Python and using term

00:33:55.480 --> 00:33:57.400
frequency a search index, very

00:33:57.400 --> 00:33:59.480
complicated like 4 MB JSON file search

00:33:59.480 --> 00:34:01.560
index shipped with the skill. And then I

00:34:01.560 --> 00:34:04.000
said, "Now eval compared to Google." And

00:34:04.000 --> 00:34:05.840
the only difference was the speed,

00:34:05.840 --> 00:34:07.520
right? It was just a few milliseconds to

00:34:07.520 --> 00:34:09.320
get to the document versus else it would

00:34:09.320 --> 00:34:09.720
go to Google

00:34:09.720 --> 00:34:10.840
>> How did you do the review? Did you use

00:34:10.840 --> 00:34:12.800
that Anthropic review skill?

00:34:12.800 --> 00:34:15.480
>> Like I said, because Fable is gone, I I

00:34:15.480 --> 00:34:17.320
and and I think they reset the credits.

00:34:17.320 --> 00:34:18.919
I just went massive on dynamic

00:34:18.919 --> 00:34:21.360
workflows. I said, "Run a full dynamic

00:34:21.360 --> 00:34:23.919
workflows." It ran like 15 agents, but

00:34:23.919 --> 00:34:25.600
it does use it has a memory and it uses

00:34:25.600 --> 00:34:27.600
Sonnet for these evals. And it did a

00:34:27.600 --> 00:34:29.639
complete comparison like running the

00:34:29.639 --> 00:34:31.800
agent without any skill and and actually

00:34:31.800 --> 00:34:33.639
telling it like use web search and make

00:34:33.639 --> 00:34:35.800
sure to ground yourself and then it says

00:34:35.800 --> 00:34:38.080
and it came back it says and and another

00:34:38.080 --> 00:34:40.360
thing I did was I asked go through all

00:34:40.360 --> 00:34:42.800
of the documents and in the content make

00:34:42.800 --> 00:34:44.159
sure you query something that's in the

00:34:44.159 --> 00:34:45.960
content that's not in the title and

00:34:45.960 --> 00:34:47.720
something that is semantically like

00:34:47.720 --> 00:34:49.720
similar but not exactly lexical match. I

00:34:49.720 --> 00:34:51.679
added those evals in so it's like really

00:34:51.679 --> 00:34:54.159
tricky and then I asked now one agent

00:34:54.159 --> 00:34:57.560
uses Google one agent uses my advanced

00:34:57.560 --> 00:35:00.400
non-semantic but a search index based on

00:35:00.400 --> 00:35:03.280
content and end result both are surfing

00:35:03.280 --> 00:35:05.200
selling the same information and the

00:35:05.200 --> 00:35:06.960
Google one is just a little bit slower

00:35:06.960 --> 00:35:09.560
and the other one took like took like

00:35:09.560 --> 00:35:11.120
you know maybe a minute to build a

00:35:11.120 --> 00:35:13.440
search index and maybe 30 minutes to

00:35:13.440 --> 00:35:15.560
build a code to to do to build a search

00:35:15.560 --> 00:35:17.280
index and then you have to think about

00:35:17.280 --> 00:35:19.520
it because if you do term frequency if

00:35:19.520 --> 00:35:21.120
you have a change log entry which has

00:35:21.120 --> 00:35:22.920
the term repeatedly then you have a

00:35:22.920 --> 00:35:24.520
document that has the term mentioned a

00:35:24.520 --> 00:35:26.960
lot but it's not dense it's not actually

00:35:26.960 --> 00:35:28.800
relevant so then you get all different

00:35:28.800 --> 00:35:30.000
problems and you're like

00:35:30.000 --> 00:35:32.160
>> Yeah man.

00:35:32.160 --> 00:35:34.040
>> MCV solved the problem instantly.

00:35:34.040 --> 00:35:37.320
>> Yeah. Yeah I mean it's so I got I I was

00:35:37.320 --> 00:35:39.240
my thought was like that whole knowledge

00:35:39.240 --> 00:35:42.200
base thing I feel is is like a big

00:35:42.200 --> 00:35:43.560
product

00:35:43.560 --> 00:35:45.680
it's a big market opportunity right like

00:35:45.680 --> 00:35:48.280
there's a lot of people working on this.

00:35:48.280 --> 00:35:50.600
You just for context you were working on

00:35:50.600 --> 00:35:52.840
this whole knowledge knowledge local

00:35:52.840 --> 00:35:55.080
knowledge thing because of some research

00:35:55.080 --> 00:35:57.040
you were doing right that's that's the

00:35:57.040 --> 00:35:59.280
reason why you did it. So you did all

00:35:59.280 --> 00:36:01.520
that work Vincent but then you compared

00:36:01.520 --> 00:36:03.240
it to Google and it's about the same

00:36:03.240 --> 00:36:04.960
right?

00:36:04.960 --> 00:36:07.080
>> It's about the same and the pros and

00:36:07.080 --> 00:36:09.080
cons was like it's just a little bit

00:36:09.080 --> 00:36:11.760
faster it's offline only but it does the

00:36:11.760 --> 00:36:13.480
cons are you need to rebuild the search

00:36:13.480 --> 00:36:16.840
index you know it becomes stale and the

00:36:16.840 --> 00:36:19.600
shipping 4 megabyte a search index JSON

00:36:19.600 --> 00:36:21.800
file with your skill is kind of like

00:36:21.800 --> 00:36:22.920
what's the point?

00:36:22.920 --> 00:36:24.200
>> Yeah the

00:36:24.200 --> 00:36:25.760
the just to play devil's devil's

00:36:25.760 --> 00:36:26.880
advocate

00:36:26.880 --> 00:36:28.280
what I found with the Google is that

00:36:28.280 --> 00:36:31.080
sometimes it's I'm not too sure what the

00:36:31.080 --> 00:36:33.040
the metadata you get back from it, but

00:36:33.040 --> 00:36:36.440
sometimes you you don't really know why

00:36:36.440 --> 00:36:38.680
that result is top. You don't know how

00:36:38.680 --> 00:36:40.720
old it is and things like this. I mean,

00:36:40.720 --> 00:36:42.760
Google does a pretty damn good job if

00:36:42.760 --> 00:36:45.040
you just trust it, but there's probably

00:36:45.040 --> 00:36:46.840
going to come a point where

00:36:46.840 --> 00:36:48.600
well, especially for like internal data,

00:36:48.600 --> 00:36:49.480
then you

00:36:49.480 --> 00:36:51.800
you just can't you can't do that, right?

00:36:51.800 --> 00:36:54.320
This is Yeah, this is why I mentioned

00:36:54.320 --> 00:36:55.720
the product the market opportunity

00:36:55.720 --> 00:36:58.080
because like internal data, the stuff

00:36:58.080 --> 00:36:59.880
that you need to make your private

00:36:59.880 --> 00:37:02.680
business work has to be private, right?

00:37:02.680 --> 00:37:04.160
>> Yeah, absolutely. Yeah.

00:37:04.160 --> 00:37:05.800
I'm sorry. I just got scolded for I

00:37:05.800 --> 00:37:07.280
turned off my camera and I should have

00:37:07.280 --> 00:37:08.800
just left the call because there was

00:37:08.800 --> 00:37:10.200
nothing else for me to say there and

00:37:10.200 --> 00:37:11.880
then apparently they asked and I didn't

00:37:11.880 --> 00:37:13.120
respond.

00:37:13.120 --> 00:37:15.600
>> I'm on this like new new remote first

00:37:15.600 --> 00:37:18.000
job and the funny thing is they have a

00:37:18.000 --> 00:37:20.000
lot of they have lots of blocks for

00:37:20.000 --> 00:37:22.600
focus time and it is amazing to have

00:37:22.600 --> 00:37:24.400
this focus time because now I can

00:37:24.400 --> 00:37:26.800
actually do some deep work, which I

00:37:26.800 --> 00:37:28.680
don't usually do because I'm I guess I'm

00:37:28.680 --> 00:37:31.160
like you in in in previous

00:37:31.160 --> 00:37:33.360
gigs where I'm like I have back-to-back

00:37:33.360 --> 00:37:35.200
calls pretty much all day and it's like

00:37:35.200 --> 00:37:36.640
impossible to actually get anything

00:37:36.640 --> 00:37:37.120
done.

00:37:37.120 --> 00:37:39.040
>> Yeah, I don't have that many calls, but

00:37:39.040 --> 00:37:40.560
I mean, I

00:37:40.560 --> 00:37:43.120
I I wouldn't have done this like maybe

00:37:43.120 --> 00:37:45.320
earlier, but now I'm like I'm off I'm

00:37:45.320 --> 00:37:46.960
I'm handing over and and so on and I

00:37:46.960 --> 00:37:47.600
feel like

00:37:47.600 --> 00:37:48.000
>> Okay.

00:37:48.000 --> 00:37:51.400
>> Yeah, it's I feel it was less important,

00:37:51.400 --> 00:37:53.560
but I I got scolded, so I feel guilty

00:37:53.560 --> 00:37:54.840
and I'm wrong. I mean, I should have

00:37:54.840 --> 00:37:55.080
just

00:37:55.080 --> 00:37:57.040
>> So, you you you basically were on a call

00:37:57.040 --> 00:37:58.200
the same time you were doing this

00:37:58.200 --> 00:37:58.960
podcast.

00:37:58.960 --> 00:38:00.720
>> I had finished. I should have just

00:38:00.720 --> 00:38:02.320
dropped. It

00:38:02.320 --> 00:38:02.800
Yeah.

00:38:02.800 --> 00:38:03.840
>> You should have dropped. You should have

00:38:03.840 --> 00:38:04.240
dropped.

00:38:04.240 --> 00:38:05.880
>> Yeah, I should have said, "Guys, I need

00:38:05.880 --> 00:38:06.760
to go."

00:38:06.760 --> 00:38:07.360
>> Important stuff.

00:38:07.360 --> 00:38:09.120
>> I should have time box this better. I

00:38:09.120 --> 00:38:11.280
need to be more Communication is key,

00:38:11.280 --> 00:38:13.200
right? Setting expectation is is what my

00:38:13.200 --> 00:38:14.320
What was the mistake here?

00:38:14.320 --> 00:38:16.880
>> Setting expectations, also working out

00:38:16.880 --> 00:38:19.520
what Yeah, what the expectations are of

00:38:19.520 --> 00:38:22.400
the person on the other side.

00:38:22.400 --> 00:38:23.680
>> It's just that it happened twice two

00:38:23.680 --> 00:38:25.400
days in a row now. It happened yesterday

00:38:25.400 --> 00:38:26.920
and happened today again.

00:38:26.920 --> 00:38:29.000
>> But you're you're rolling off, so but I

00:38:29.000 --> 00:38:31.880
mean you should always keep keep in very

00:38:31.880 --> 00:38:33.400
good standing

00:38:33.400 --> 00:38:34.680
with your colleagues. Yeah, you're

00:38:34.680 --> 00:38:37.160
messing up Vincent. I mean, do you

00:38:37.160 --> 00:38:38.760
do you want to should we end it now or

00:38:38.760 --> 00:38:40.440
did you want to carry on or

00:38:40.440 --> 00:38:42.400
>> There's nothing else. I apologized and

00:38:42.400 --> 00:38:44.240
there's nothing else I can do. But in

00:38:44.240 --> 00:38:46.320
you know, to come to the like internal

00:38:46.320 --> 00:38:48.120
internal knowledge bases is is actually

00:38:48.120 --> 00:38:49.520
the very interesting use case there,

00:38:49.520 --> 00:38:50.880
right? But you wouldn't scale you

00:38:50.880 --> 00:38:52.480
wouldn't ship a skill and an MCP is the

00:38:52.480 --> 00:38:54.760
right answer. I think I I've also

00:38:54.760 --> 00:38:57.400
learned so many use cases where MCPs are

00:38:57.400 --> 00:38:59.520
a much better fit. And because I'm

00:38:59.520 --> 00:39:01.680
working on a on learning more about

00:39:01.680 --> 00:39:04.080
agent core and its capabilities, it has

00:39:04.080 --> 00:39:06.080
this really interesting agent registry

00:39:06.080 --> 00:39:08.280
and agents registry policies, which

00:39:08.280 --> 00:39:10.440
allows you to control who has access to

00:39:10.440 --> 00:39:14.320
what and which agents can use what tools

00:39:14.320 --> 00:39:16.560
depending on the identity of the of the

00:39:16.560 --> 00:39:19.640
user. So if the user is prompting the

00:39:19.640 --> 00:39:22.200
with an agent, then

00:39:22.200 --> 00:39:24.680
back to what tools the agent can use. So

00:39:24.680 --> 00:39:26.480
it shouldn't be able to to query the

00:39:26.480 --> 00:39:28.440
financials of a database or knowledge

00:39:28.440 --> 00:39:30.040
base when the user is shouldn't have

00:39:30.040 --> 00:39:32.040
access to it. So things like that are

00:39:32.040 --> 00:39:33.960
are possible with the with the policies

00:39:33.960 --> 00:39:36.080
around and the user identity like the

00:39:36.080 --> 00:39:37.920
identity system around it. And then you

00:39:37.920 --> 00:39:39.720
can do that with with MCPs. It becomes

00:39:39.720 --> 00:39:41.360
really interesting and I think that's

00:39:41.360 --> 00:39:43.080
definitely something that when you're

00:39:43.080 --> 00:39:44.760
looking at deploying agents across

00:39:44.760 --> 00:39:46.760
across enterprises, you need there's a

00:39:46.760 --> 00:39:49.960
lot more than I've got I I've run rag

00:39:49.960 --> 00:39:51.440
vectorized and added all of the

00:39:51.440 --> 00:39:53.400
knowledge and my agent has it and I'm

00:39:53.400 --> 00:39:54.440
good to go.

00:39:54.440 --> 00:39:56.880
>> I I talked about CQ

00:39:56.880 --> 00:40:00.240
to you before, but the CQ is one that I

00:40:00.240 --> 00:40:02.880
I I came across when I went to this this

00:40:02.880 --> 00:40:05.880
uh AI conference in London. And and it I

00:40:05.880 --> 00:40:09.200
wonder if the the MCP you like like if

00:40:09.200 --> 00:40:11.160
you just think if you just is this the

00:40:11.160 --> 00:40:12.800
right Yeah, I think this is what I want

00:40:12.800 --> 00:40:14.400
to talk about. If you just think about

00:40:14.400 --> 00:40:17.000
like what every say say you're rolling

00:40:17.000 --> 00:40:19.920
out in cloud enterprise to your thousand

00:40:19.920 --> 00:40:22.680
employees, you want an MCP so you can

00:40:22.680 --> 00:40:25.520
query your knowledge base. Okay, that's

00:40:25.520 --> 00:40:27.560
that's like number one. And then number

00:40:27.560 --> 00:40:29.840
two is that you want your employees to

00:40:29.840 --> 00:40:31.880
propose new knowledge

00:40:31.880 --> 00:40:35.120
items. You got that. Um but what I think

00:40:35.120 --> 00:40:37.840
that a lot of knowledge base systems

00:40:37.840 --> 00:40:40.600
miss out on, which CQ does really really

00:40:40.600 --> 00:40:41.800
well, I don't know if your knowledge

00:40:41.800 --> 00:40:42.920
base

00:40:42.920 --> 00:40:45.840
does that, is that it allows you to give

00:40:45.840 --> 00:40:47.840
feedback on that knowledge unit. So, you

00:40:47.840 --> 00:40:49.480
can say like, "Hey, this knowledge unit

00:40:49.480 --> 00:40:53.080
unit is actually good and useful."

00:40:53.080 --> 00:40:56.040
Or or flag that it's actually crap. And

00:40:56.040 --> 00:40:58.600
you can you can even see

00:40:58.600 --> 00:41:01.400
um statistics around that the usage of

00:41:01.400 --> 00:41:03.320
that knowledge. And that's that that

00:41:03.320 --> 00:41:07.080
that feedback design in CQ is absolutely

00:41:07.080 --> 00:41:08.840
next level compared to anything else

00:41:08.840 --> 00:41:09.400
I've seen.

00:41:09.400 --> 00:41:11.080
>> Right. I think this is what I also

00:41:11.080 --> 00:41:13.320
learned about like when I did vector

00:41:13.320 --> 00:41:15.320
embeddings and and and retrieval

00:41:15.320 --> 00:41:17.760
augmented which is very like type of

00:41:17.760 --> 00:41:19.360
memory system

00:41:19.360 --> 00:41:21.920
versus what I run recently for a local

00:41:21.920 --> 00:41:23.640
knowledge base that I was playing with.

00:41:23.640 --> 00:41:25.560
It's more like a memory system, but it's

00:41:25.560 --> 00:41:27.920
those dimensions that that that play a

00:41:27.920 --> 00:41:30.080
part in like recency of the of the

00:41:30.080 --> 00:41:32.480
memory, relevancy, conflicting memory

00:41:32.480 --> 00:41:34.600
facts. That's what like proper memory

00:41:34.600 --> 00:41:36.760
systems actually provide now, not just

00:41:36.760 --> 00:41:38.960
like you can embed it, there's a vector,

00:41:38.960 --> 00:41:40.280
you can do a similarity search, and

00:41:40.280 --> 00:41:41.600
that's it. There's so many more

00:41:41.600 --> 00:41:43.040
dimensions to it. Like what you said,

00:41:43.040 --> 00:41:44.600
like usage of it.

00:41:44.600 --> 00:41:47.040
>> Yeah. The feed The feedback loop is is

00:41:47.040 --> 00:41:49.080
absolutely critical.

00:41:49.080 --> 00:41:51.200
Cuz like that's that's another reason

00:41:51.200 --> 00:41:53.600
why skills almost just fall down is

00:41:53.600 --> 00:41:55.840
because there's just there's just one

00:41:55.840 --> 00:41:58.480
guy that maintains a skill, and then he

00:41:58.480 --> 00:42:02.040
probably publishes it and maybe works on

00:42:02.040 --> 00:42:03.640
it a little bit to get a couple of

00:42:03.640 --> 00:42:06.160
stars, and then it's just basically it

00:42:06.160 --> 00:42:08.320
might not it might that that skill is

00:42:08.320 --> 00:42:10.440
specific for a thing, and it doesn't get

00:42:10.440 --> 00:42:13.400
the opportunity to to evolve, and people

00:42:13.400 --> 00:42:15.280
Yeah, I mean I'm I'm looking I'm I'm

00:42:15.280 --> 00:42:17.440
thinking of a couple of enterprise skill

00:42:17.440 --> 00:42:19.000
repos that I've seen. They're all just

00:42:19.000 --> 00:42:21.160
become dead spaces right now, as far as

00:42:21.160 --> 00:42:21.800
I can tell.

00:42:21.800 --> 00:42:23.160
>> Yeah, I still

00:42:23.160 --> 00:42:25.160
I think that's right that skill quality

00:42:25.160 --> 00:42:27.960
and how well maintained it is and

00:42:27.960 --> 00:42:29.840
security scanning around skills like

00:42:29.840 --> 00:42:31.560
that's what some of the new security

00:42:31.560 --> 00:42:33.720
products are about, right? Versel or was

00:42:33.720 --> 00:42:36.360
it another company just announced I mean

00:42:36.360 --> 00:42:38.760
Versel skill registry made a public API

00:42:38.760 --> 00:42:40.400
to query the registry now and you you

00:42:40.400 --> 00:42:42.400
use OIDC and there's rate limiting and

00:42:42.400 --> 00:42:45.280
they also skill with SSH usually. Yeah,

00:42:45.280 --> 00:42:46.960
so skill with SSH but that's a public

00:42:46.960 --> 00:42:48.560
registry, right? They made a public API

00:42:48.560 --> 00:42:50.560
because I was using it and API wasn't

00:42:50.560 --> 00:42:52.800
really like a contract and there was no

00:42:52.800 --> 00:42:54.600
no proper control around it so they make

00:42:54.600 --> 00:42:57.720
it they make it a version API now and

00:42:57.720 --> 00:43:00.520
what I was saying like they have space

00:43:00.520 --> 00:43:03.160
scanning like snake and other partners

00:43:03.160 --> 00:43:05.120
can post updates about the skill has

00:43:05.120 --> 00:43:06.440
been scanned

00:43:06.440 --> 00:43:08.640
there seem to be no injection

00:43:08.640 --> 00:43:10.600
but I guess what you like like you said

00:43:10.600 --> 00:43:12.280
recency like has it been kept up to

00:43:12.280 --> 00:43:14.640
date? Is it still relevant? Those are

00:43:14.640 --> 00:43:16.640
all aspects of skills and and if you're

00:43:16.640 --> 00:43:18.400
looking at the local skill registry, I

00:43:18.400 --> 00:43:20.040
built something very simple which uses a

00:43:20.040 --> 00:43:22.160
get mono repo. I think skills have their

00:43:22.160 --> 00:43:24.360
place. I think skills they are very

00:43:24.360 --> 00:43:26.920
useful in in a lot of cases

00:43:26.920 --> 00:43:29.560
where an MCP might pollute the context

00:43:29.560 --> 00:43:32.560
might you know, not make sense because

00:43:32.560 --> 00:43:34.320
now an MCP you have to run something

00:43:34.320 --> 00:43:36.120
like a lot of the MCPs you require you

00:43:36.120 --> 00:43:38.000
to have something running and and maybe

00:43:38.000 --> 00:43:38.960
our an offering like

00:43:38.960 --> 00:43:39.440
>> Yeah.

00:43:39.440 --> 00:43:41.960
>> for example Passion has an MCP. I think

00:43:41.960 --> 00:43:43.600
they called made their MCP GA.

00:43:43.600 --> 00:43:45.960
>> fill a gap but they if they don't evolve

00:43:45.960 --> 00:43:47.520
then they're pretty much dead in the

00:43:47.520 --> 00:43:48.760
water, aren't they really?

00:43:48.760 --> 00:43:50.840
>> Okay. It it depends on the skill. It

00:43:50.840 --> 00:43:52.720
depends on how it needs to evolve. I

00:43:52.720 --> 00:43:55.440
think some skills are I think very

00:43:55.440 --> 00:43:56.560
useful.

00:43:56.560 --> 00:43:58.920
>> Yeah, I agree. Some skills are extremely

00:43:58.920 --> 00:43:59.920
useful.

00:43:59.920 --> 00:44:02.600
I I wanted to maybe tangentially talk

00:44:02.600 --> 00:44:04.640
about a security

00:44:04.640 --> 00:44:06.040
the uh

00:44:06.040 --> 00:44:08.440
I was I was just thinking

00:44:08.440 --> 00:44:09.880
I was just going to ask you if you had

00:44:09.880 --> 00:44:13.760
any opinions about So, this is a bit of

00:44:13.760 --> 00:44:16.120
a huge leap and change of topic about

00:44:16.120 --> 00:44:19.160
humans in the loop like You Can I assume

00:44:19.160 --> 00:44:20.800
you're using auto mode at this point,

00:44:20.800 --> 00:44:21.600
right?

00:44:21.600 --> 00:44:23.720
>> Yeah. Yeah, I mean, it's crazy now. I'm

00:44:23.720 --> 00:44:26.240
just I'm just I feel like I've I've I've

00:44:26.240 --> 00:44:27.960
I've devolved in this complete cognitive

00:44:27.960 --> 00:44:30.200
offload of just handing my brain off

00:44:30.200 --> 00:44:31.840
today and to the engine going all the

00:44:31.840 --> 00:44:33.880
way into auto mode and then just looking

00:44:33.880 --> 00:44:34.760
at the details.

00:44:34.760 --> 00:44:36.040
>> So, human in the loop is pretty much

00:44:36.040 --> 00:44:37.720
dead if you're using auto mode, aren't

00:44:37.720 --> 00:44:38.400
you, really?

00:44:38.400 --> 00:44:40.040
>> I don't I don't think so.

00:44:40.040 --> 00:44:41.520
>> I mean, do you do you have any prompts

00:44:41.520 --> 00:44:42.640
where you have to like a up

00:44:42.640 --> 00:44:43.760
>> Yeah, yeah, yeah. I think that's what

00:44:43.760 --> 00:44:45.520
they did really well. I mean, for it to

00:44:45.520 --> 00:44:48.320
be successfully used um you must first

00:44:48.320 --> 00:44:49.760
trust it, right? We talked about that

00:44:49.760 --> 00:44:52.680
that we've seen it prompt us. And it it

00:44:52.680 --> 00:44:54.600
stops and it does that really well as

00:44:54.600 --> 00:44:57.040
well. Like Claude stops and says, "I

00:44:57.040 --> 00:44:59.520
found uh an important divergence from

00:44:59.520 --> 00:45:01.360
your statement, which I believe I need

00:45:01.360 --> 00:45:03.520
to align with you on." And it stops. And

00:45:03.520 --> 00:45:05.320
it and it and and it basically grabs

00:45:05.320 --> 00:45:06.640
your attention now more when it's

00:45:06.640 --> 00:45:08.520
needed. I think that's one of the quotes

00:45:08.520 --> 00:45:09.920
that I really liked somebody told me. In

00:45:09.920 --> 00:45:12.720
the age of AI, it's all about finding

00:45:12.720 --> 00:45:15.200
like reducing noise, finding signal and

00:45:15.200 --> 00:45:18.360
and and and capturing human focus for

00:45:18.360 --> 00:45:19.840
the limited time it's available because

00:45:19.840 --> 00:45:21.760
humans are only focused and and and

00:45:21.760 --> 00:45:23.080
actually paying attention for a very

00:45:23.080 --> 00:45:24.560
small amount of time compared to this.

00:45:24.560 --> 00:45:26.200
>> Yeah, that's true. That's true. Like if

00:45:26.200 --> 00:45:28.440
if if if a human is doing work and

00:45:28.440 --> 00:45:31.200
you're basically wasting it by asking

00:45:31.200 --> 00:45:33.320
the human to make decisions about, you

00:45:33.320 --> 00:45:36.240
know, do I do this move or do I do this

00:45:36.240 --> 00:45:40.120
copy or do I do this echo into into a in

00:45:40.120 --> 00:45:42.240
a hair doc or something, you're you're

00:45:42.240 --> 00:45:44.120
you're wasting people's precious

00:45:44.120 --> 00:45:45.720
cognitive energy.

00:45:45.720 --> 00:45:48.000
>> But, I think one one big mental shift

00:45:48.000 --> 00:45:49.440
that I'm not have made yet with the

00:45:49.440 --> 00:45:52.000
Hermes agent is when I send the prompt

00:45:52.000 --> 00:45:53.880
equals off. And like in Claude, I can

00:45:53.880 --> 00:45:55.920
press escape. I mean, I think in Hermes

00:45:55.920 --> 00:45:57.280
3 out of the Discord gateway, there's

00:45:57.280 --> 00:45:59.400
also a way to send a signal to to to

00:45:59.400 --> 00:46:01.040
stop the and and interrupt the model. I

00:46:01.040 --> 00:46:02.760
just haven't figured it out yet. And

00:46:02.760 --> 00:46:05.040
maybe I shouldn't figure it out because

00:46:05.040 --> 00:46:07.160
I I I need to treat this more as an

00:46:07.160 --> 00:46:10.320
autonomous entity. And what I think I'm

00:46:10.320 --> 00:46:12.360
failing failing to do is for it to to

00:46:12.360 --> 00:46:14.200
find work and do and do things on its

00:46:14.200 --> 00:46:16.120
own, which is something I haven't

00:46:16.120 --> 00:46:18.000
grasped yet. I'm I'm still treating it

00:46:18.000 --> 00:46:20.720
like a remote prompt and and a sandbox

00:46:20.720 --> 00:46:22.680
environment that I can have some control

00:46:22.680 --> 00:46:24.280
over which runs on my hardware.

00:46:24.280 --> 00:46:26.880
>> that solves a lot of permission solves a

00:46:26.880 --> 00:46:28.960
lot of security issues.

00:46:28.960 --> 00:46:31.120
The The going back to auto mode, do do

00:46:31.120 --> 00:46:33.280
you configure auto mode? Like cuz you

00:46:33.280 --> 00:46:35.040
can like create your own rules and you

00:46:35.040 --> 00:46:36.560
can critique your own. Do you ever do

00:46:36.560 --> 00:46:37.120
that?

00:46:37.120 --> 00:46:39.200
>> No, I've been I mean, I think this goes

00:46:39.200 --> 00:46:41.080
back to like one of of earlier calls

00:46:41.080 --> 00:46:42.600
where I was like, oh, I'm really not

00:46:42.600 --> 00:46:43.800
going to bother with like a lot of

00:46:43.800 --> 00:46:46.080
harness engineering or or anything like

00:46:46.080 --> 00:46:48.880
that because I trust the when they

00:46:48.880 --> 00:46:50.280
release a model and they release a

00:46:50.280 --> 00:46:52.360
harness around it that it is fine-tuned

00:46:52.360 --> 00:46:53.720
to each other. So, I don't want to make

00:46:53.720 --> 00:46:54.800
too much effort

00:46:54.800 --> 00:46:56.560
>> Yeah, that's a good point. That's a good

00:46:56.560 --> 00:46:58.240
point like Cuz

00:46:58.240 --> 00:46:58.600
I get

00:46:58.600 --> 00:47:00.240
>> a balance, right? I mean, a lot of that

00:47:00.240 --> 00:47:03.000
has changed. Um I think originally I I

00:47:03.000 --> 00:47:04.920
was very much like, oh, I don't even

00:47:04.920 --> 00:47:07.280
know Codex or Gemini or or or cloud

00:47:07.280 --> 00:47:09.240
code. I'm just, you know, have a little

00:47:09.240 --> 00:47:11.480
budget everywhere. I trust I'm not going

00:47:11.480 --> 00:47:13.480
to spend time to like build my own

00:47:13.480 --> 00:47:16.040
pi.dev harness. It's It's such a hard

00:47:16.040 --> 00:47:18.000
balance and it changes all the time

00:47:18.000 --> 00:47:19.920
depending on capabilities.

00:47:19.920 --> 00:47:21.840
>> I suppose, but like but if you if you if

00:47:21.840 --> 00:47:24.120
you're all in on Anthropic Claude, you

00:47:24.120 --> 00:47:26.920
you should just trust Claude and and and

00:47:26.920 --> 00:47:28.920
not have all these pointless discussions

00:47:28.920 --> 00:47:31.040
almost saying that like I don't trust

00:47:31.040 --> 00:47:32.800
Claude, you know what I mean? Like you

00:47:32.800 --> 00:47:35.040
got to register the fact that you trust

00:47:35.040 --> 00:47:36.920
Claude to do the right thing.

00:47:36.920 --> 00:47:38.720
>> I think I think that's where having an

00:47:38.720 --> 00:47:41.960
agent an agent with its own authority

00:47:41.960 --> 00:47:45.000
and identity helps a lot. Cuz if you're

00:47:45.000 --> 00:47:47.200
running Claude on your machine, it acts

00:47:47.200 --> 00:47:49.600
upon like on your authority and it has

00:47:49.600 --> 00:47:51.560
your permissions and it can do a lot of

00:47:51.560 --> 00:47:54.840
things which you maybe like you you want

00:47:54.840 --> 00:47:57.720
to have a control over. Whereas if you

00:47:57.720 --> 00:47:59.880
dedicate it to an individual like an an

00:47:59.880 --> 00:48:01.600
entity an agent that runs on its own and

00:48:01.600 --> 00:48:03.640
has its own identity, you can control a

00:48:03.640 --> 00:48:05.800
lot more like like how you would, you

00:48:05.800 --> 00:48:07.440
know, separate like what do you call

00:48:07.440 --> 00:48:08.920
separation of of of

00:48:08.920 --> 00:48:09.680
>> duties.

00:48:09.680 --> 00:48:11.920
>> duties, you you would be able to control

00:48:11.920 --> 00:48:13.440
a lot more of what it can do based on

00:48:13.440 --> 00:48:15.440
its permissions at at of course the cost

00:48:15.440 --> 00:48:18.000
of like having to jump in when when it

00:48:18.000 --> 00:48:21.280
gets stuck. Uh so, but I just feel it's

00:48:21.280 --> 00:48:22.920
it's different if you run a a coding

00:48:22.920 --> 00:48:24.520
agent on your machine and it uses your

00:48:24.520 --> 00:48:26.760
credentials versus if you have like an

00:48:26.760 --> 00:48:27.720
entity or a separate

00:48:27.720 --> 00:48:29.400
>> You just hit on You just hit the nail on

00:48:29.400 --> 00:48:32.440
the head here because like imagine you

00:48:32.440 --> 00:48:34.440
are working in a company like I am with

00:48:34.440 --> 00:48:37.240
like thousands of people, non-technical

00:48:37.240 --> 00:48:39.400
people running Claude code. They

00:48:39.400 --> 00:48:41.840
basically have Claude code desktop

00:48:41.840 --> 00:48:44.040
installed or something like this or or

00:48:44.040 --> 00:48:46.360
Claude code. In fact, I think they was

00:48:46.360 --> 00:48:47.720
other way around in this company. They

00:48:47.720 --> 00:48:49.840
they they rolled out Claude code before

00:48:49.840 --> 00:48:51.400
the Claude desktop. But, what I'm trying

00:48:51.400 --> 00:48:53.360
to say is that is that non-technical

00:48:53.360 --> 00:48:55.440
people are basically running Claude in

00:48:55.440 --> 00:48:57.560
their home directory and they're not

00:48:57.560 --> 00:48:58.760
properly

00:48:58.760 --> 00:49:01.360
able to sandbox it and that skill of

00:49:01.360 --> 00:49:03.520
sandboxing, as you say with the Hermes

00:49:03.520 --> 00:49:07.000
agent or as as us developers know how to

00:49:07.000 --> 00:49:10.880
do, that's extremely powerful security

00:49:10.880 --> 00:49:12.800
like that's a that's extremely powerful

00:49:12.800 --> 00:49:14.120
security

00:49:14.120 --> 00:49:16.720
standard capability to do that. But, but

00:49:16.720 --> 00:49:18.960
most people don't know how to sandbox

00:49:18.960 --> 00:49:21.320
and hence you're you have to fall back

00:49:21.320 --> 00:49:23.480
on like this is why like people say you

00:49:23.480 --> 00:49:25.280
have to have human in the loop and then

00:49:25.280 --> 00:49:27.160
and then the human loop doesn't work

00:49:27.160 --> 00:49:28.960
because people just smash enter. So,

00:49:28.960 --> 00:49:30.280
anyway, that's

00:49:30.280 --> 00:49:31.160
That's why

00:49:31.160 --> 00:49:33.280
>> So, so another thing that I learned from

00:49:33.280 --> 00:49:35.840
building a a a pull request review agent

00:49:35.840 --> 00:49:37.680
related to the human in the loop like

00:49:37.680 --> 00:49:40.480
where does the the human attention gets

00:49:40.480 --> 00:49:42.520
grabbed versus like having a shell

00:49:42.520 --> 00:49:44.840
running on your under your authority and

00:49:44.840 --> 00:49:46.960
it figuring out when something doesn't

00:49:46.960 --> 00:49:48.560
align or where it needs to grab your

00:49:48.560 --> 00:49:50.400
attention and stops only when it's

00:49:50.400 --> 00:49:52.160
really needed. What I learned about

00:49:52.160 --> 00:49:54.560
Hermes and having an individual like an

00:49:54.560 --> 00:49:57.640
AI acting like a PR reviewer is that

00:49:57.640 --> 00:49:59.440
first of trust is

00:49:59.440 --> 00:50:02.040
is built up and it's broken down very

00:50:02.040 --> 00:50:05.480
quickly. So, the moment that agent is

00:50:05.480 --> 00:50:07.840
posting comments on PRs and they are not

00:50:07.840 --> 00:50:10.720
relevant, maybe simple things like just

00:50:10.720 --> 00:50:12.840
the way it it it posts the comment

00:50:12.840 --> 00:50:14.560
immediately triggers people, "Oh, this

00:50:14.560 --> 00:50:16.600
is an AI." And and it's like when we're

00:50:16.600 --> 00:50:18.320
watching websites and we are very good

00:50:18.320 --> 00:50:20.120
at filtering out advertisement. When we

00:50:20.120 --> 00:50:22.440
identify something is AI, we're going to

00:50:22.440 --> 00:50:24.600
end up filtering it out very quickly and

00:50:24.600 --> 00:50:27.240
it it signal becomes like

00:50:27.240 --> 00:50:28.200
>> What?

00:50:28.200 --> 00:50:28.920
>> useless.

00:50:28.920 --> 00:50:30.560
>> you you're running Hermes agent. I'm

00:50:30.560 --> 00:50:33.560
running Open Claw and I had like again I

00:50:33.560 --> 00:50:35.040
had these like really amazing

00:50:35.040 --> 00:50:36.720
experiences with Open Claw in the

00:50:36.720 --> 00:50:39.840
beginning, but now it says it it it

00:50:39.840 --> 00:50:42.160
can't run my quiz correctly. Every time

00:50:42.160 --> 00:50:43.680
one of my family members answers the

00:50:43.680 --> 00:50:46.600
quiz it often just randomly misses out

00:50:46.600 --> 00:50:49.280
the answer from from random members of

00:50:49.280 --> 00:50:51.440
my family. So, so trust is broken, but

00:50:51.440 --> 00:50:53.920
the the worst thing about Open Claw is

00:50:53.920 --> 00:50:56.520
that I don't know why it's failing. I

00:50:56.520 --> 00:50:59.240
don't know what what what went wrong. I

00:50:59.240 --> 00:51:01.000
don't know how to give it feedback. I

00:51:01.000 --> 00:51:02.960
don't know how to fix my Open Claw

00:51:02.960 --> 00:51:05.600
instance. I basically got this this

00:51:05.600 --> 00:51:08.960
crippled sick Open Claw instance running

00:51:08.960 --> 00:51:11.240
for months now and it's doing a few

00:51:11.240 --> 00:51:13.080
things like doing my calendaring. It's

00:51:13.080 --> 00:51:15.680
doing my the family quiz, but it it's

00:51:15.680 --> 00:51:18.360
just not been operating 100% and I don't

00:51:18.360 --> 00:51:20.640
know how to fix it and I don't want to

00:51:20.640 --> 00:51:23.200
completely reset it at the same time

00:51:23.200 --> 00:51:23.680
either.

00:51:23.680 --> 00:51:26.320
>> So, I I very early on with with the

00:51:26.320 --> 00:51:28.080
Hermes agent when it was running on EC2

00:51:28.080 --> 00:51:29.480
even when it was just accessible over

00:51:29.480 --> 00:51:32.480
SSH by Claude Code. Um at about a one

00:51:32.480 --> 00:51:35.040
week milestone I went and asked Claude

00:51:35.040 --> 00:51:37.360
to actually ask my son, "Like tell

00:51:37.360 --> 00:51:39.320
Claude to ship all the logs and whatever

00:51:39.320 --> 00:51:41.520
logs it can find, session logs,

00:51:41.520 --> 00:51:43.920
gateway logs, whatever. Zip it, tar it,

00:51:43.920 --> 00:51:46.600
stream it over the the SSH tunnel into

00:51:46.600 --> 00:51:48.600
your machine and run a dynamic workflow

00:51:48.600 --> 00:51:51.120
locally to go look at failure modes and

00:51:51.120 --> 00:51:53.160
issues that we we may improve.

00:51:53.160 --> 00:51:53.680
>> Wow. Okay.

00:51:53.680 --> 00:51:55.760
>> And and it it did that really well. As

00:51:55.760 --> 00:51:57.240
soon as my son had done the basics, I

00:51:57.240 --> 00:51:59.320
basically took over his laptop again.

00:51:59.320 --> 00:52:01.320
This is kind of the worst part of about

00:52:01.320 --> 00:52:04.080
me giving my son during summer time jobs

00:52:04.080 --> 00:52:06.320
to hopefully him learn things where as

00:52:06.320 --> 00:52:07.640
soon as he had done the most basic

00:52:07.640 --> 00:52:09.040
thing, I just had to take over his

00:52:09.040 --> 00:52:10.600
laptop and start doing it myself because

00:52:10.600 --> 00:52:12.560
it's just too exciting. And and I

00:52:12.560 --> 00:52:14.440
actually asked ChatGPT to help me about

00:52:14.440 --> 00:52:16.120
like giving him assignments and then

00:52:16.120 --> 00:52:18.080
also framing it in a way that it like it

00:52:18.080 --> 00:52:20.200
better so that it this doesn't happen.

00:52:20.200 --> 00:52:22.400
Uh and ChatGPT scolded me for for my

00:52:22.400 --> 00:52:24.120
behavior. But anyway, what I found

00:52:24.120 --> 00:52:26.120
really interesting with these Sonnet

00:52:26.120 --> 00:52:29.120
logs even even like and my probably Opus

00:52:29.120 --> 00:52:31.200
but like dynamic workflow so like five

00:52:31.200 --> 00:52:32.760
agents go out, look at all the session

00:52:32.760 --> 00:52:35.200
logs, find common failure issues. It's

00:52:35.200 --> 00:52:37.680
so it surfaced so many misconfigurations

00:52:37.680 --> 00:52:39.840
like the auxiliary LLM configuration.

00:52:39.840 --> 00:52:41.000
It's saying you you don't have an

00:52:41.000 --> 00:52:43.560
auxiliary LLM endpoint configured and

00:52:43.560 --> 00:52:44.960
they're failing most of the time. So you

00:52:44.960 --> 00:52:46.440
like your summarized sessions is always

00:52:46.440 --> 00:52:49.240
failing because of it. Um it identified

00:52:49.240 --> 00:52:50.400
other issues and

00:52:50.400 --> 00:52:52.280
>> And these logs are are Hermes logs or

00:52:52.280 --> 00:52:55.000
are they or they Claude Claude traces or

00:52:55.000 --> 00:52:57.280
>> So I have several No, they they So

00:52:57.280 --> 00:52:59.440
Hermes and probably Open Claude as well

00:52:59.440 --> 00:53:01.880
has like several session sources. For

00:53:01.880 --> 00:53:03.760
example, in an SQLite database where it

00:53:03.760 --> 00:53:05.360
keeps track of every session that

00:53:05.360 --> 00:53:07.400
happens and and it can find the very

00:53:07.400 --> 00:53:09.840
detailed information there. It can also

00:53:09.840 --> 00:53:12.240
find more high-level summarized session

00:53:12.240 --> 00:53:14.080
logs. It can also look at the gateway

00:53:14.080 --> 00:53:16.600
like Discord interactions. For example,

00:53:16.600 --> 00:53:18.600
it has these interactive cards that it

00:53:18.600 --> 00:53:19.840
sends over Discord that you need to

00:53:19.840 --> 00:53:21.960
approve or has multiple choice and they

00:53:21.960 --> 00:53:24.920
have a a timeout like maybe maybe

00:53:24.920 --> 00:53:28.200
600 seconds timeout on it. So sometimes

00:53:28.200 --> 00:53:29.760
these session like these interactive

00:53:29.760 --> 00:53:31.480
cards weren't reaching me because they

00:53:31.480 --> 00:53:34.000
were not running on a gateway thread

00:53:34.000 --> 00:53:35.480
because of the way I had set up some

00:53:35.480 --> 00:53:37.120
cron jobs. So the cron job wasn't

00:53:37.120 --> 00:53:39.920
running on a gateway integration path

00:53:39.920 --> 00:53:41.600
and it wasn't surfacing those and it was

00:53:41.600 --> 00:53:43.280
timing out every time, and it was like

00:53:43.280 --> 00:53:44.720
there's several of them there that's

00:53:44.720 --> 00:53:46.120
being timed out. I haven't tried

00:53:46.120 --> 00:53:48.080
upgrading Hermes yet, but I found doing

00:53:48.080 --> 00:53:49.960
this, and I've done it now it's already

00:53:49.960 --> 00:53:52.200
third week. I've done it two consecutive

00:53:52.200 --> 00:53:54.440
weeks, and it each time it found a whole

00:53:54.440 --> 00:53:56.040
bunch of information that allows me to

00:53:56.040 --> 00:53:57.800
fine-tune the agent and to optimize the

00:53:57.800 --> 00:53:58.760
way it works.

00:53:58.760 --> 00:53:59.760
>> Yeah.

00:53:59.760 --> 00:54:01.000
>> And that I think would help you with

00:54:01.000 --> 00:54:01.680
your open cloud.

00:54:01.680 --> 00:54:03.960
>> Yeah, you you've inspired me to do that.

00:54:03.960 --> 00:54:05.200
I don't know why I didn't think about

00:54:05.200 --> 00:54:06.840
doing that, but thanks for the

00:54:06.840 --> 00:54:10.000
inspiration. I guess maybe it's

00:54:10.000 --> 00:54:12.000
the tweets that I was reading in the

00:54:12.000 --> 00:54:13.640
last few days. It's like everyone's

00:54:13.640 --> 00:54:16.360
talking about these loops, and and it's

00:54:16.360 --> 00:54:18.520
interesting that you you can run an It's

00:54:18.520 --> 00:54:20.440
interesting that you you run an agent,

00:54:20.440 --> 00:54:23.680
you emit events and logs, and you put

00:54:23.680 --> 00:54:25.840
them somewhere, and then you analyze

00:54:25.840 --> 00:54:29.440
that log to basically improve the the

00:54:29.440 --> 00:54:32.520
agent run. And you you go in that that

00:54:32.520 --> 00:54:34.040
loop.

00:54:34.040 --> 00:54:34.520
I think

00:54:34.520 --> 00:54:36.760
>> that's it's the dreaming concept, right?

00:54:36.760 --> 00:54:38.360
You you dream and you clean up your

00:54:38.360 --> 00:54:41.000
memory, and you you you evaluate while

00:54:41.000 --> 00:54:42.440
you're behaving, and you you build the

00:54:42.440 --> 00:54:45.160
rules like Boris hi says this that you

00:54:45.160 --> 00:54:47.120
should do that a lot. Um they also have

00:54:47.120 --> 00:54:49.680
this like on team onboarding command to

00:54:49.680 --> 00:54:51.960
allow you to share what you learned in

00:54:51.960 --> 00:54:53.600
your local setup against with your team

00:54:53.600 --> 00:54:55.400
to onboard other team members.

00:54:55.400 --> 00:54:56.920
>> Uh /team onboarding, I've seen

00:54:56.920 --> 00:54:57.240
that.

00:54:57.240 --> 00:54:59.240
>> Yeah, but like when I tried it was it

00:54:59.240 --> 00:55:00.600
just came out. It was really not

00:55:00.600 --> 00:55:02.160
interesting. It didn't really say much.

00:55:02.160 --> 00:55:03.840
Like, "Hey, your team is using uh cloud

00:55:03.840 --> 00:55:05.200
code, and they want you to help you

00:55:05.200 --> 00:55:06.640
onboard." And then it was like a very

00:55:06.640 --> 00:55:08.560
stupid. It wasn't even It was like,

00:55:08.560 --> 00:55:10.160
"Vincent has been using this." And it

00:55:10.160 --> 00:55:11.800
was like, "How is that relevant?" But

00:55:11.800 --> 00:55:13.400
anyway, like like this is what I was

00:55:13.400 --> 00:55:15.560
telling you like on the last podcast,

00:55:15.560 --> 00:55:17.560
right? I say, "Check the Twittersphere.

00:55:17.560 --> 00:55:19.360
Check what's being on going on about

00:55:19.360 --> 00:55:20.720
like you engineer the loop, not the

00:55:20.720 --> 00:55:21.960
prompt. You're no longer prompting the

00:55:21.960 --> 00:55:24.280
agent. Focus on how you engineer the

00:55:24.280 --> 00:55:26.200
loop." I was telling you that last time,

00:55:26.200 --> 00:55:28.120
and and that's how when when that all

00:55:28.120 --> 00:55:29.080
came out.

00:55:29.080 --> 00:55:30.640
>> This is very inspiring. Like, the

00:55:30.640 --> 00:55:33.280
trouble is with my current employer,

00:55:33.280 --> 00:55:36.120
I I wanted to see the the Claude logs

00:55:36.120 --> 00:55:38.440
inside DataDog, but but they've deemed

00:55:38.440 --> 00:55:41.640
to be highly sensitive. So, like only

00:55:41.640 --> 00:55:43.320
like a couple of people in the whole

00:55:43.320 --> 00:55:45.600
organization can see cloud logs in data

00:55:45.600 --> 00:55:48.400
dog. And now, on the basis of our of our

00:55:48.400 --> 00:55:50.520
conversation, I'm thinking that like the

00:55:50.520 --> 00:55:53.320
team almost should have access to the to

00:55:53.320 --> 00:55:56.400
the team uh cloud logs so that they can

00:55:56.400 --> 00:55:58.480
have a retrospective that makes sense

00:55:58.480 --> 00:56:01.280
for the for the agent usage. This is all

00:56:01.280 --> 00:56:03.160
quite inspiring stuff.

00:56:03.160 --> 00:56:05.320
>> this is a This when you say this this

00:56:05.320 --> 00:56:07.520
brings me back to exactly the role of

00:56:07.520 --> 00:56:09.400
DevOps engineer or platform engineering,

00:56:09.400 --> 00:56:11.840
right? We're supposed to give people

00:56:11.840 --> 00:56:13.960
like we're supposed to surface the tools

00:56:13.960 --> 00:56:16.880
for them to do their job without giving

00:56:16.880 --> 00:56:18.760
them the access. For example, we not

00:56:18.760 --> 00:56:20.960
giving them access SSH to a node, but we

00:56:20.960 --> 00:56:22.400
have to surface all of the tools for

00:56:22.400 --> 00:56:24.160
them to troubleshoot the issue so that

00:56:24.160 --> 00:56:26.000
they can, you know, use a dashboard and

00:56:26.000 --> 00:56:28.480
and correlate events. And and it's very

00:56:28.480 --> 00:56:30.160
similar to like what we used to do when

00:56:30.160 --> 00:56:32.160
we when we do Kubernetes, right? Again,

00:56:32.160 --> 00:56:33.520
we don't want to give them Cube CTL

00:56:33.520 --> 00:56:35.440
access to a production cluster, but we

00:56:35.440 --> 00:56:37.200
have to give them all of the tools,

00:56:37.200 --> 00:56:38.360
knobs, and

00:56:38.360 --> 00:56:40.120
dials so that they are able to fine-tune

00:56:40.120 --> 00:56:41.160
their service and they're able to

00:56:41.160 --> 00:56:42.920
troubleshoot and identify the issue

00:56:42.920 --> 00:56:45.480
without SSH or, you know, admin access,

00:56:45.480 --> 00:56:48.240
which is similar to to now, like we need

00:56:48.240 --> 00:56:49.840
to do the same for the cloud agent

00:56:49.840 --> 00:56:50.360
sessions.

00:56:50.360 --> 00:56:52.640
>> Yeah, and and and I need to get I need

00:56:52.640 --> 00:56:54.800
to ask for DataDog access. Have you ever

00:56:54.800 --> 00:56:57.800
got DataDog pup working? This is CLI

00:56:57.800 --> 00:56:58.880
tool called

00:56:58.880 --> 00:57:00.640
>> Yeah, I built it. I I was So, I was

00:57:00.640 --> 00:57:03.360
having some some some effort at work.

00:57:03.360 --> 00:57:04.800
This is reminding me I have to follow up

00:57:04.800 --> 00:57:06.280
on something. But,

00:57:06.280 --> 00:57:08.320
for ex- It's very funny because you

00:57:08.320 --> 00:57:10.080
design the whole thing and it goes to

00:57:10.080 --> 00:57:12.720
production and then opens randomly goes

00:57:12.720 --> 00:57:14.440
like, "Oh, by the way,

00:57:14.440 --> 00:57:16.560
um I noticed something. Should we do

00:57:16.560 --> 00:57:19.080
this or that?" And I'm like, "Damn it,

00:57:19.080 --> 00:57:20.840
this is going to production. It's a bit

00:57:20.840 --> 00:57:22.520
late." So, then go ahead and make a

00:57:22.520 --> 00:57:24.080
dashboard to track this if this is a

00:57:24.080 --> 00:57:25.640
real issue in production. And then it

00:57:25.640 --> 00:57:27.040
just goes off. It builds a whole

00:57:27.040 --> 00:57:28.640
dashboard, you know, to watch this

00:57:28.640 --> 00:57:31.080
particular type of like failure scenario

00:57:31.080 --> 00:57:33.400
if this is really a concern. Uh and then

00:57:33.400 --> 00:57:34.600
so that we could get better

00:57:34.600 --> 00:57:36.520
>> get it in DataDog.

00:57:36.520 --> 00:57:39.040
>> Yeah, yeah. So, I I I I used pub. You

00:57:39.040 --> 00:57:40.720
can go to the repo. You can see from me

00:57:40.720 --> 00:57:42.720
like a month ago I reported an issue and

00:57:42.720 --> 00:57:44.080
they fixed it by rolling back something

00:57:44.080 --> 00:57:45.680
they had released, which I think it was

00:57:45.680 --> 00:57:47.200
still in beta. I don't know. I think

00:57:47.200 --> 00:57:49.240
Opus surfaced it somehow. I was I was

00:57:49.240 --> 00:57:51.080
like looking at the DataDog uh

00:57:51.080 --> 00:57:53.760
docs to to how to build and pub came up

00:57:53.760 --> 00:57:55.320
very quickly. It was very early in the

00:57:55.320 --> 00:57:57.720
>> Yeah, I I tried pub just now a couple of

00:57:57.720 --> 00:57:58.960
days ago and it didn't work. There's

00:57:58.960 --> 00:58:00.720
some sort of up the issue or something

00:58:00.720 --> 00:58:01.480
like that, but

00:58:01.480 --> 00:58:03.520
>> I had the same experience.

00:58:03.520 --> 00:58:04.480
Huh?

00:58:04.480 --> 00:58:06.320
I had the same. So, when I tried pub, it

00:58:06.320 --> 00:58:08.200
was using a token that didn't have

00:58:08.200 --> 00:58:10.880
permission to do certain things. And I

00:58:10.880 --> 00:58:12.680
raised a I raised a request. I I

00:58:12.680 --> 00:58:14.280
actually had a separate Claude Code

00:58:14.280 --> 00:58:16.120
session go through the whole thing cuz

00:58:16.120 --> 00:58:18.240
when you try to log in, it failed. And

00:58:18.240 --> 00:58:21.320
then and then it DataDog uh sorry, Cloud

00:58:21.320 --> 00:58:23.360
did like a whole binary search of like

00:58:23.360 --> 00:58:24.920
it's failing on a certain scope request

00:58:24.920 --> 00:58:26.640
against the token. And then it went

00:58:26.640 --> 00:58:28.160
like, "Okay, we're going to do the first

00:58:28.160 --> 00:58:29.680
50 scopes." And then, "Oh, we're going

00:58:29.680 --> 00:58:31.480
to do the second 50 scopes." "Okay, it's

00:58:31.480 --> 00:58:32.960
failing in that batch." And then it

00:58:32.960 --> 00:58:34.440
narrowed it down. It says, "Okay, this

00:58:34.440 --> 00:58:36.360
particular scope that's being requested

00:58:36.360 --> 00:58:38.080
in the login is causing the failure."

00:58:38.080 --> 00:58:40.120
And then it created a ticket and it it

00:58:40.120 --> 00:58:41.760
shared all the information. And then the

00:58:41.760 --> 00:58:44.360
response was, "I rolled back a feature

00:58:44.360 --> 00:58:46.120
and we fixed it." And I was like, "In

00:58:46.120 --> 00:58:48.000
the meantime, I had a working

00:58:48.000 --> 00:58:50.000
set up because instead of like doing a

00:58:50.000 --> 00:58:51.680
an authentication and it it doing the

00:58:51.680 --> 00:58:52.960
default scope where it was always

00:58:52.960 --> 00:58:54.680
failing, it was actually doing a

00:58:54.680 --> 00:58:56.720
dedicated like subset of list of scopes

00:58:56.720 --> 00:58:58.360
that I needed and it worked."

00:58:58.360 --> 00:58:59.400
>> Wait a minute. If I understand

00:58:59.400 --> 00:59:01.840
correctly, you you you managed to create

00:59:01.840 --> 00:59:04.000
a ticket complaining about how pub

00:59:04.000 --> 00:59:06.400
wasn't working to the admins for the

00:59:06.400 --> 00:59:08.680
admins to fix, right?

00:59:08.680 --> 00:59:10.640
>> Yes, exactly. Like for for for the

00:59:10.640 --> 00:59:13.080
DataDog to fix because they had when

00:59:13.080 --> 00:59:15.360
they did their authentication, so pub is

00:59:15.360 --> 00:59:17.560
using some token from DataDog

00:59:17.560 --> 00:59:18.440
authentication.

00:59:18.440 --> 00:59:21.520
>> I need to update and try again. The

00:59:21.520 --> 00:59:23.600
>> It was a it was a month ago. So, I maybe

00:59:23.600 --> 00:59:25.120
they're just keep

00:59:25.120 --> 00:59:27.200
recreating this bug, but it's a problem

00:59:27.200 --> 00:59:29.560
with the scope request on the token. And

00:59:29.560 --> 00:59:31.840
and I I spent I spent quite a while to

00:59:31.840 --> 00:59:33.080
figure out what the scope was.

00:59:33.080 --> 00:59:34.680
>> The thing I wanted to mention, but I

00:59:34.680 --> 00:59:36.440
need to run off to to take my kids to

00:59:36.440 --> 00:59:38.240
school, is that like it's really

00:59:38.240 --> 00:59:40.400
frustrating that the Cloud Enterprise,

00:59:40.400 --> 00:59:43.200
you know, it there's an admin panel for

00:59:43.200 --> 00:59:45.560
a couple of people who have access.

00:59:45.560 --> 00:59:47.680
There There's things that they can touch

00:59:47.680 --> 00:59:49.640
inside there. There's There's new rows

00:59:49.640 --> 00:59:51.640
that appear all the time. There should

00:59:51.640 --> 00:59:55.640
be Terraform to control um that Cloud

00:59:55.640 --> 00:59:57.920
Enterprise settings. I mean, it's so

00:59:57.920 --> 01:00:00.080
critical that this should be under under

01:00:00.080 --> 01:00:02.320
source control, right? That the settings

01:00:02.320 --> 01:00:04.320
are controlled. Right now, people log

01:00:04.320 --> 01:00:05.840
into these admin panels, they go, "What

01:00:05.840 --> 01:00:08.920
is this new OAuth scope thing? I don't

01:00:08.920 --> 01:00:10.280
know what it is. I'm turning it off or

01:00:10.280 --> 01:00:12.440
something." And then it breaks the flow,

01:00:12.440 --> 01:00:13.920
and then you're you're you're left

01:00:13.920 --> 01:00:15.680
scratching your head for like days.

01:00:15.680 --> 01:00:16.920
>> On Anthropic?

01:00:16.920 --> 01:00:17.480
>> Well,

01:00:17.480 --> 01:00:19.320
>> Is it an Anthropic admin, or is it the

01:00:19.320 --> 01:00:20.520
Datadog admin panel?

01:00:20.520 --> 01:00:22.320
>> Well, I I I

01:00:22.320 --> 01:00:23.680
It's Look,

01:00:23.680 --> 01:00:25.080
I'm just making an example here that

01:00:25.080 --> 01:00:27.520
like these these admin panels are

01:00:27.520 --> 01:00:30.600
extremely important to control, but they

01:00:30.600 --> 01:00:32.520
there isn't a Terraform provider for it.

01:00:32.520 --> 01:00:33.720
I'm like, "What?"

01:00:33.720 --> 01:00:35.720
>> So, this is what I I I struggle with in

01:00:35.720 --> 01:00:37.640
every organization I worked is these

01:00:37.640 --> 01:00:40.840
ITSM tickets flows and ITSM teams, which

01:00:40.840 --> 01:00:43.600
often are like ex- accepted because we

01:00:43.600 --> 01:00:44.920
are not programmers. We don't know how

01:00:44.920 --> 01:00:46.920
to do this. And then they end up like

01:00:46.920 --> 01:00:49.280
click-opsing everything. And

01:00:49.280 --> 01:00:50.720
>> Oh, yeah. And then they take like a week

01:00:50.720 --> 01:00:52.120
to respond to a ticket. It's like,

01:00:52.120 --> 01:00:53.320
"Okay."

01:00:53.320 --> 01:00:55.000
>> Yeah, but I mean I mean, I can

01:00:55.000 --> 01:00:56.640
understand, obviously, that like

01:00:56.640 --> 01:00:58.400
terraforming the whole situation is also

01:00:58.400 --> 01:01:00.160
not a solution. I agree with that. So,

01:01:00.160 --> 01:01:01.720
what happened recently is like a lot of

01:01:01.720 --> 01:01:04.360
people left on on a on on a contract I'm

01:01:04.360 --> 01:01:07.040
on. And some people new people reach out

01:01:07.040 --> 01:01:08.680
to me, and they asked me like we have we

01:01:08.680 --> 01:01:10.440
have this ITSM ticket we have to deal

01:01:10.440 --> 01:01:13.640
with. And I was like I thought that the

01:01:13.640 --> 01:01:16.120
it was around AWS permission sets in AWS

01:01:16.120 --> 01:01:17.800
SS. And I was like, "Oh, yeah, I know.

01:01:17.800 --> 01:01:19.560
We we we we Terraform the permission

01:01:19.560 --> 01:01:22.280
sets. It's just the membership like

01:01:22.280 --> 01:01:24.880
that's managed in the in the identity

01:01:24.880 --> 01:01:27.320
platform. So, the the user to like that

01:01:27.320 --> 01:01:29.400
this identity provider user to group

01:01:29.400 --> 01:01:31.120
membership, that's there. That's that's

01:01:31.120 --> 01:01:33.360
onboarding, that's HR. I'm not touching

01:01:33.360 --> 01:01:35.320
that. I have learned my lesson. So, so

01:01:35.320 --> 01:01:37.080
on the actual permission sets against

01:01:37.080 --> 01:01:38.720
AWS accounts, that's definitely

01:01:38.720 --> 01:01:40.280
Terraform. Like I've I've got that in

01:01:40.280 --> 01:01:42.440
Terraform for ages, right? And then I

01:01:42.440 --> 01:01:43.520
was like

01:01:43.520 --> 01:01:45.840
I talking with with them, the new people

01:01:45.840 --> 01:01:47.120
taking over from people that have been

01:01:47.120 --> 01:01:49.360
doing it behind the scenes shadowy ish

01:01:49.360 --> 01:01:52.560
for a while, and Cloud exploring along.

01:01:52.560 --> 01:01:54.880
And then no, none of these permission

01:01:54.880 --> 01:01:57.280
sets for production are in Terraform.

01:01:57.280 --> 01:01:59.080
And I was like this is horrible. This is

01:01:59.080 --> 01:02:00.760
terrible. Why is it like that? You know,

01:02:00.760 --> 01:02:02.320
the the these people keep up saying

01:02:02.320 --> 01:02:04.200
behind it that keep doing this.

01:02:04.200 --> 01:02:05.960
>> I mean I've joined I've joined clients

01:02:05.960 --> 01:02:08.920
and they have like an an IAM team and I

01:02:08.920 --> 01:02:10.040
should have asked them, "Do are you

01:02:10.040 --> 01:02:11.800
using Terraform to maintain your I am?"

01:02:11.800 --> 01:02:13.120
I bet you they're not.

01:02:13.120 --> 01:02:14.480
>> Yeah, so for permission sets I think

01:02:14.480 --> 01:02:16.840
having Terraform is definitely a must. I

01:02:16.840 --> 01:02:18.840
said in this intermediate phase where

01:02:18.840 --> 01:02:20.520
we're not like because Cloud immediately

01:02:20.520 --> 01:02:21.760
went, "Let's go import everything." I

01:02:21.760 --> 01:02:22.640
was like, "Yeah, we're not we're not

01:02:22.640 --> 01:02:24.040
going to do that right now because we

01:02:24.040 --> 01:02:26.360
got a ticket to close, right?" So, I

01:02:26.360 --> 01:02:29.600
told people in lieu of proper processes,

01:02:29.600 --> 01:02:31.520
let's just because everyone is adopting

01:02:31.520 --> 01:02:34.280
LLMs, I created a folder because LLMs

01:02:34.280 --> 01:02:36.000
now are clickops, right? Because they're

01:02:36.000 --> 01:02:37.720
running the API commands directly,

01:02:37.720 --> 01:02:39.240
they're modifying configurations,

01:02:39.240 --> 01:02:40.880
they're they're modifying the settings.

01:02:40.880 --> 01:02:42.840
And are you getting the same problem? It

01:02:42.840 --> 01:02:44.600
happens behind the back, nobody knows

01:02:44.600 --> 01:02:46.600
who did it, when what when it happened.

01:02:46.600 --> 01:02:48.440
And you need to have that for our

01:02:48.440 --> 01:02:50.520
principles, somebody has to review it.

01:02:50.520 --> 01:02:52.960
So, I said to those people, I I approve

01:02:52.960 --> 01:02:55.320
I'm I I implore on you that you use AI

01:02:55.320 --> 01:02:57.080
to do this because AI is going to be way

01:02:57.080 --> 01:02:58.960
more grounded than you who just came

01:02:58.960 --> 01:03:00.520
into this position and have to solve

01:03:00.520 --> 01:03:02.800
this problem. But as long as we capture

01:03:02.800 --> 01:03:05.880
the context properly, and also I I don't

01:03:05.880 --> 01:03:07.320
want you to trust AI. So, in this

01:03:07.320 --> 01:03:08.960
particular case, you cannot make the

01:03:08.960 --> 01:03:10.400
decision because you don't know about

01:03:10.400 --> 01:03:11.640
the system and you don't know what

01:03:11.640 --> 01:03:13.800
should be the way. Until we have either

01:03:13.800 --> 01:03:15.440
this Terraform or we have proper

01:03:15.440 --> 01:03:17.800
context, you are going to get the LLM

01:03:17.800 --> 01:03:20.120
plan and commit it into this repo on

01:03:20.120 --> 01:03:21.880
this position and you're going to tag us

01:03:21.880 --> 01:03:23.600
for review so we have a four eye view

01:03:23.600 --> 01:03:24.360
>> Yeah. I'm

01:03:24.360 --> 01:03:25.720
>> So it's kind of like the intermediate

01:03:25.720 --> 01:03:28.080
ClickOps solution. Without Terraform,

01:03:28.080 --> 01:03:29.800
you get an LLM plan that needs to be

01:03:29.800 --> 01:03:31.760
reviewed. You have a Git log of when

01:03:31.760 --> 01:03:33.360
this was changed. You know exactly what

01:03:33.360 --> 01:03:33.920
the LLM was.

01:03:33.920 --> 01:03:35.560
>> It's a it's a total smell when there's

01:03:35.560 --> 01:03:36.680
no

01:03:36.680 --> 01:03:38.920
source source It's a total smell in an

01:03:38.920 --> 01:03:41.280
organization that you have these IT ITSM

01:03:41.280 --> 01:03:43.840
flows and there doesn't seem to be any

01:03:43.840 --> 01:03:44.440
uh

01:03:44.440 --> 01:03:47.040
you know, Git commits behind them. It's

01:03:47.040 --> 01:03:47.760
it's a total

01:03:47.760 --> 01:03:49.920
>> like a compliance issue like

01:03:49.920 --> 01:03:51.680
In a lot of cases, this will be a

01:03:51.680 --> 01:03:53.200
complete failure on compliance.

01:03:53.200 --> 01:03:55.760
>> Yeah. Yeah. Okay. Thanks again, Vincent.

01:03:55.760 --> 01:03:56.640
I got to run.

01:03:56.640 --> 01:03:58.800
>> of those compliances are like paper are

01:03:58.800 --> 01:04:00.720
just checklists. They're just like yeah,

01:04:00.720 --> 01:04:02.160
we have someone treated ticket and

01:04:02.160 --> 01:04:03.280
someone reviewed the ticket.

01:04:03.280 --> 01:04:06.000
>> Or they're like yeah, totally manual and

01:04:06.000 --> 01:04:07.880
they think it's manual.

01:04:07.880 --> 01:04:09.320
The worst is when they think it's like

01:04:09.320 --> 01:04:11.400
it's manual and and it's a good thing

01:04:11.400 --> 01:04:13.200
that it's manual or something. That's

01:04:13.200 --> 01:04:15.920
That's the thing that bothers me a bit.

01:04:15.920 --> 01:04:18.640
>> Okay. See you. Cheers, everyone. Bye.

01:04:18.640 --> 01:04:20.840
>> Bye.