Episode 35: AI is transforming Security

Hey, I also shaved.
Oh. Oh. Oh. Oh.
Oh, there's a bit of an echo.
Maybe it's my headset feedback. Test.
Test.
Better. Um. Oh, damn. What have I done
with my AirPods?
H.
I should have this this Na'vi signed
jersey more in in in view.
>> What is that about?
>> This is Oh, this thing. I I don't know.
This just Zeus
>> the goat Na'vi team
>> Guardian Edward
>> talking about things I have no idea
about.
>> Counter Strike Global Offensive lineup
from 2011 or something. 2013 maybe.
Yeah. the first uh okay so
I've just taken a more uh what do you
call it security thing and like I get
asked things about things that usually I
would probably frankly not care so much
about like is this for real should I be
worried about it an active attack is
happening claw right now
>> yeah I've seen that
uh what was it again you There's so much
packages
>> with security stuff. So much like what
do you call it?
>> Well,
it's just like guys,
>> it's just continuation of team PCP,
right? Isn't that like the North Korean
linked hacker group that
>> did the shy hulut and they basically
made the the worm public and there's
been several worms similarly and
apparently one of the redhead employee
GitHub login was compromised and some
packages were published
but but has it I mean like what isn't
this the responsibility of the clawed
code maintainers to fix why is it
blurted out to the wider community. You
know what I mean?
>> It's not Claude Code specific though.
It's just like the problem is that these
packages are often propagating through
popular frameworks like in this case
it's like a spring uh or Java type of
framework and also often via VS code
plugins because VS code plugins are or
the market base is also like not really
properly
>> so is a theory that like uh Claude code
would just install these things while
vibing.
>> Yeah. Yeah. Sorry, I'm putting my legs
up because I'm I want to chill. So,
yeah, that's that's one val that's one
thing though, but cloud has been trained
on it. Like opus 4.8 when I was trying
to do a hindsight uh setup immediately
said like hey this package that I'm
trying to download for pi from Python I
cannot verify that it is the official uh
Python package for this framework that
you want to work with and then it um it
blocked it. I don't know if it was auto
mode, but I think the the model itself
kind of had this
>> incentive to not just randomly like it
it it it suspected uh name squatting.
But yeah, name squatting is scary,
right? That's been people have been
warning about name squatting with LLMs
for a long time. Basically, how do you
know the LLM isn't just like
hallucinating a package name and then
somebody has has, you know, published
that package and then the LLM just goes
ahead and downloads and installs it.
>> Yeah, it's it's interesting. It's
interesting to think that
>> but I think overall
>> attackers are going for the uh the the
tra the train models um sort of allow
and it takes some time for them to
update
>> but it's also because of the ecosystem
stance like for example VS code plugins
have um way too many permissions they
nothing in VS Code prevents a plug-in
from reading whatever it wants on your
disk and and maybe exploring
>> I guess it's the same with it's the same
sort of black hole of uh browser
extensions, they also had like a poor
security model for the longest time,
right?
>> For the longest time. Yeah. So now that
becomes more apparent. Same with like
all these packet manager managers for
npm. Like PNPM is now very like by
default does not execute hooks which
breaks some package installations. So uh
but they're they're becoming very
aggressive. I think PNPM is probably the
most aggressive in disabling these
things for security reasons which is
also why a lot of frameworks are moving
towards PNPM because they're very
aggressively blocking this type of
exploits and works.
>> Cool. Cool. Cool.
>> Yeah. I mean I guess this stuff is all
all entirely predictable now. Um I I
messaged you on WhatsApp about like an
inter do you think that you use AWS KOD
factory or co coder whatever was it
called? Kodi
Ko.
>> No, not Kira, but like code. Um, yeah, I
use code art. Is it code artifact or
>> Yeah, I think it's code artifact. God
damn. Now you you put this in my head
and I can't remember.
>> Code artifact. Yes.
>> But does that have
>> artifactory?
>> Yeah, code code artifact. So in light of
PMPM doing its thing,
is there I don't actually quite
understand what the AWS code um
what the the AWS code artifact does. It
just gives a local copy of known good
stuff, right? That's in your that's in
your organization. For me, code artifact
um came into play maybe five years ago
when um I had to inherit like somebody
left and there was this shadow IT uh
what's it called? It's not the JROG one,
but oh Nexus. Somebody had set up a
private Nexus instance on a Kubernet,
you know, like Helm install Nexus,
right? The classic Helm false confidence
of yes, just install this package and
now I've got a whole service on my
>> a huge stack.
>> Yeah, but I mean it's not huge. it just
it's it's not production ready like it
doesn't provide SSO people were sharing
and like the CI/CD was using um you know
user credentials username and password
and these credentials were shared all
across the the teams and then it started
failing
um because it started falling flat on
its face uh due to CPU memory and disk
restrictions because this was also
stateful and and and you know this is
the classic somebody can't be bothered
with waiting on um you know a platform
team to do this properly and just goes
ahead and and Helms installs it into one
of the clusters and then it comes back
to platform anyway. Um but the migration
to AWS coder effect was just so smooth
because it all piggybacks off the IM
authentication and we were using AWS SSO
which is tied into our uh entra ID or
octa um so it's very to be honest
>> you don't like what
>> I I I love I am it just it's my
>> it's the perk of being in the AWS
ecosystem right the moment that you have
a service uh and it's properly
integrated with IM now you you can
resolve a lot of the like um access
permission roles. So the roll out
becomes super simple like you I set up a
dedicated I have a shared services AWS
account set up code artifact do a
crossorg trust relationship read only so
that all of the other like environments
like dev can download it and also the
developers they do not get like push
permission. uh you can with IM policies
very clearly say this guy can
authenticate. He can you know set up his
local npm to use this private uh coded
defect hosted registry but he cannot
push and uh we only have like GitHub
workflows with AWS configuration and
this GitHub workflow IM role is the one
that is able to publish to these
particular repositories. You can even
cross-link it from g this GitHub repo to
this code artifact um repository. So
your whole arbback all of the stuff that
you have to start paying for with Nexus
um just instantly gets solved. And yes,
you can set up complete upstreams to do
a complete proxy. So so you can force
global registry setup with code
artifact. means that every developer uh
will download all of its dependencies
through your AWS account and and and
your AWS um code artifact registry is a
full pull through and basically caches
and protects against npmgs outages and
things like that. Um
>> so so the so the main risk that it
solves is like a denial of service on a
public uh package. Yeah, it could. But
you can also have your complete um
privately scoped uh packages, right?
With npm or python or others. I you you
can say this scope maps to my private
codified registry and that you of course
have to make sure that you also own the
public scope because if somebody an
engineer doesn't authenticate or doesn't
set it up properly they don't want to
accidentally uh end up downloading the
somebody else uh packages put on the on
I guess
>> so we do always register make sure that
we own the npgs scope the public scope
even though we don't put anything in
there and then when somebody onboards
Um, we have like the AWS SSO register.
It's a make target or whatever is fast
as you know in your package or JSON
script to to get the code artifact token
which is valid for a day and it handles
your offboarding because it's all tied
into your SSO. So somebody's offboarded
their tokens expired. They can't request
a new one. So you don't have to worry
about that either. Uh, and you have your
full like GitHub actions IDC integration
with IM authentication.
>> It sounds like code co code uh artifact
is a must. Does it support go to or is
it it seems targeted at Node.js if I was
reading the
>> Go I'm not sure. So So I'm I used goat
artifact for Maven, Nugat. Uh I no I'm
not sure about Nougat but definitely for
Java Maven Ruby Nougat. What is Nougat?
>> Is the net uh package manager package
system?
>> Uh but I have not tried it with Golang.
I mean honestly but overall Golang with
private GitHub repos is quite horrible
right I mean you have to set up this
like
>> private proxy or let's say
authentication
>> if you have poly repo or multiple it
becomes a nightmare
>> uh I've had nothing but good experiences
with go package management
>> yeah anyways it's been a while since we
caught up I guess I've been busy with a
new job you've been busy with a new job
>> not yet no but um preparing for it. Um
so I've been having fun. But back to
because one thing that
>> new role you you were like asking me to
guess your new title.
>> So one thing that code edifact doesn't
do that if you were to look at JROG or
the other like uh enterprise offerings.
Um I think there's less of this full
integration within with like
vulnerability scanning and um like coded
effect is very basic like it's just
npmgs uh private host with IM off
authentication and it does a lot already
but if you want to have like super fancy
enterprise type of stuff on top of your
registry then I know that art code sorry
artifactory from JROG and probably
others they do uh a lot more there but
they also cost I I think more money. So
anyway, yes, in terms of the the new
role,
>> I I didn't even know what a super fancy
enterprise feature is.
>> Well, there's definitely a few where I
I've been asked like, can we do this?
And then I look into it and no, you
can't really do that with code artifact.
Like code artifact effect is really nice
mirror pull through.
>> Can you give an example? I mean, I can't
even I can't even think of one.
>> Can you give an example what of a super
fancy enterprise?
I guess I I see code artifact and AWS
ECR very similar like OCI and one of the
common requests there is whenever you
publish uh an ECR image you want to have
like a full scan on the file system you
want to do vulnerability scan I mean you
can probably do all of that but you're
going to have to build like events uh
maybe with uh I know that ECR has like a
hook to automatically scan like there's
a ECR registry scanner I'm not I don't
think code artifact has it but you can
also build on top it like at a point you
got to wonder do I build in house or do
I go for something if I need all of that
I might as well go for this
>> actually while I'm thinking about it
have you ever had like discussions where
you introduce a security you know
layered uh feature like a scan and then
it slows down your pipeline by let's
just say 30% and then people are
complaining.
Do you have you ever like had that sort
of like uh discussion where like yeah
this new security thing isn't really
working out for us because it makes us
so much slower or something like that.
Yeah. So I think the um the that's the
classic problem with security right
security has always to find the balance
between making sure that things are done
securely and um and and you know not
slowing things down not being a
bottleneck not not causing people to
find workarounds and go um you know
through the back of of security because
of they are blocked by it. So yeah um
and and yeah slowdowns like on the CI/CD
I mean sec we had a very funny um you
know when somebody new a new CTO came in
and then uh there was like a double down
because it's a fintech double down on
sec on on like auditability and
compliance and then we were looking at
the costs I mean for a full sem
ingestion of all of these events across
the enterprise uh I mean the cost
balloon amazing like even in in a recent
role where I was like Echo, everyone was
running Terraform off their laptop.
There's this thing called Tacos. Um,
they provide cool stuff, but you know,
they have all of these initiatives
around security that cost a lot of
money. And then I'm like, okay, I can
run this on a $5, you know, AWS instance
or maybe $20 if it's a slightly larger
instance and I I get P like pull request
plans so everyone can like coordinate
basically very simple tackles with
Atlantis. I don't have to spend a lot of
time setting up the GitHub workflows. I
get a nice, you know, plan, approve,
apply workflow with with Atlantis. Works
really decent. Doesn't cost a lot of
money. And then I'm like looking at the
costs of security, you know, in putting
up like IDS, IDP across all of your
hosts, setting up the full wiz like
cloud uh security posture management,
CSPM, from the data dog.
>> The costs are
>> Yeah. I mean data dog alone I've just
seen in countless clients now has been
an insane cost.
>> Uh wait until they then also need
security
>> the security
>> because like because data log on its own
is just like people start with just um
you know maybe infrastructure. I mean
originally data was pure infra only APM
came later right and then now they have
this whole security suite cloud you know
security poster management the whole
traces
>> uh setup the synthetics like they have
absorbed so many additional products
that you know on top of the original
basic like in monitoring that it was
>> well
>> it's insane
>> thanks to AI I like to think
security the whole the whole profession
the whole the whole security um area I
feel is being kind of re re looked at
with a new lens like one thing I I I um
I heard through uh work was that clients
are now realizing that they can't hire
the typical security people that you
know are not engineer mind who don't
have an engineer mindset you know they
can't hire people who sit in a
spreadsheet, you know, checking off
boxes and things like that anymore.
>> You need different types of security
capability.
>> That's a problem because when we were
hiring security and like people, we were
looking for engineers because I I
absolutely did not want the security guy
that would just be there uh you know
pushing checklists through. But the
reality is that we hired two guys very
strong technical engineers and one of
them slightly more uh senior was just
stuck going through endless list of
compliance on spreadsheets identifying
proof that this was implemented like
that's what the job divulges into
unfortunately that's the reality like
that's what the
>> comp in the age of AI surely that that
sort of stuff can be automated no
>> yeah this was four years ago now AWS is
like ramping up crazily in terms of well
architectured framework and security
advisories as well on your AWS account
leveraging AI and providing all the
context
>> required. This is this is a big change I
feel in in our industry.
Um
>> but still security is has got a new
lens.
>> Yeah. But like security moves super
slow. Like often they are related to
very like traditional industries and
they will require you like if you need
to get licensed to get access to a
certain data set, you're going to have
to spend months going through
checklists. Yes. Today I can give that
checklist to uh an AI and it can go
through all of my AWS setup and identify
or even my Git repos and identify we are
compliant or we are not compliant with
this particular checklist item. uh and
he can find the source of where we are
compliant and maybe what is the
remediation and then we can go back to
the auditors.
>> What if I said to you that the checklist
will now become like a unit test like
for example one thing that I'm hoping to
roll out is you know clawed auto mode
you basically have all these rules and
prompts to say that you know you can
trust these domains and you shouldn't do
that you shouldn't do that. I'm I'm
thinking that um
the rules of a business like what you
can and what will become a test like a
like I will have like a claw-p
can you delete this file and and then
and then we build up a test suite and
that will be our checklist to know that
our security guard rails are working for
us or not you know like it's def it's
it's completely shifted left I
H that
>> what what you're talking about is
>> making sure the agent follows the rules.
So even if the human is unaware, the
agent gets like a guard rails around it
is what you're saying.
>> Exactly. So this is this is a big change
in the industry.
>> Yeah. I think I think
>> this is different sort of this is this
is different sort of engineering to what
most security people have been doing in
the past.
>> Yeah. But I also think that one one
important change is that we will have
less people involved in the agentic
loop, right? I mean, we've already
talked about this like originally it's
it's shifting left more and more. Uh
when you talk about prompting an agent
to do things, we we were originally
talking about specri development, coming
up with user stories, functional
requirements, then breaking that down
into tasks and so on. And then that over
time we're no longer really checking if
these tasks are like exactly what we
want. We just make sure that it matches
the the user stories and that we're able
to define what good looks like and how
you validate that good looks like that.
And now with the dynamic workflows,
humans are not even involved like nobody
there's no human prompting the agent.
It's just agents writing the prompts
which is also Steinberger's reach.
>> No no I think I think humans will be
writing the prompts. I mean
>> no
>> I think I think you went a little bit
too far there. I think you're not up to
speed there.
>> And it's it's easy to get like totally
far-fetched. And the other thing that
>> I think if you look into what's
happening, it's all in on agents now.
Like it's been all in on agents for a
while, but things have changed a lot.
>> I want to be the the voice of reason.
Like
>> like
I heard
a client wants to run agents in
production and like my jaw was like
>> I think it's very different between No,
no, no. Listen, listen. You're
conflating two things.
>> Giving an agent to put it in
>> agent writing prompts. That's I I I feel
you because you can uh use agents to
like refine
>> Let me talk one second here because
you're conflating putting agents in
production as in I'm going to let random
people consumers like you know you see
all these meme tweets about you don't
need to pay for cloud anymore. You can
just uh go into a chat session with
Chipotle's uh help menu and it's an LLM
and you can ask it to write Python. You
don't need to pay for cloud. That's if
you talk about putting agent in
production, that's what you talk about
putting agent in front of consumers.
>> Yeah. Or Yeah. Uh
>> I think it's very different between your
internal consumers like your developers,
product managers, product owners.
>> Yeah, you're right. You're right. You
need to define it like perhaps using
that lethal trifecta is probably the
best approach like so so let's say let's
say it's not it does it doesn't have
access to untrusted data but but running
an agent in production
just internally I feel is just just kind
of
kind of risky. I mean,
>> so basically,
>> yeah,
>> Peter Steinberger has put out a tweet, I
think last week, and it's been
repeatedly posted on Reddit. And it's
basically like you do not write the
prompt. You write the loop that write
the prompt. And and um and I think this
this is also what's happening with um I
can't access uh sitter here. Um but but
it's basically the same happening right
now with these dynamic workflows right
as long as I give it a a clear goal like
even with codex goal it spins up an
enormous amount of agents that all the
work and they are able to validate you
know they have their own loops feedback
validation loops to validate that what
they built works cuz if you look at like
6 months ago you would do the spec you
would do the plan you would go through
sessions with the agent with the agentic
shell right it's not an agent you would
using agent and shells. It would
complete the work. You would see the
stream. You would realize what was
working.
>> You would write deterministic code,
right?
>> And and the agent would write code and
you would have your verification
mechanism.
>> You don't believe in this stuff, do you?
Come on, Vincent.
>> I've seen this stuff. It works. You are
the one that's not not paying attention.
Read what's happening on Twitter in the
last week. I I I know I know people, but
like there's so much craziness on
Twitter and X and like I I guess my
philosophy is is my philosophy really
that out of date? My philosophy is that
you use the agent to create
deterministic workflows, deterministic
code to do the job.
>> That's what dynamic workflows ultimately
boggles down to. Yes. You you
>> ship in production deterministic stuff.
You don't want to ship
>> that's not into production. We're not
talking about like when you say
production, what do you mean? Because
this is not in front of consumers.
>> Stuff that's going to hit like consumers
and
>> No, no, we're not talking about consu.
We're talking about software factories
and internal like software delivery like
AI delivery. Um AI AI DLC.
>> Okay. I think big shift.
>> Okay. AI delivery. Okay. I can I can get
behind that whole agents. You write the
loop to write the agents. I can get
Yeah.
>> So why every time I mention it, you go
you fall back to like, yo, but that's
not that's not what you do. But then
then you say, "Oh, no. I can get behind
it like it's very simple.
>> Okay.
>> Claude Code shipped
>> this this dynamic workflows agents have
been doing the work completely
independently and coming back with the
solution because it has validated.
>> I'm getting I'm getting a little bit
fixated about running this sort of stuff
in in in production with um with noobs
or you know like non-engineers I
suppose.
I mean it's very closely linked to your
your your worry because you say you need
an agent you need an agent shell and you
need guardrails. So if an engineer does
not take into account security
boundaries uh the agent shell guard
rails will stop that. What I'm trying to
say is there's no engineer sitting
there. There's no no human there
anymore. like what's happening. The
humans are involved with the planning
phase that creates a dynamic, you know,
workflow which is a deterministic
procedure of steps that includes up to
10 12 agents minimum that are doing the
implementation phases that are running
the feedback like verification loops.
But that only works if you can verify
what good look like and what
verification means
>> and then you get back.
>> I mean, we're getting a little bit stuck
with terminology. I mean, so it's
probably me that's the problem here, but
like I guess what you're saying is like
there's no more human in the loop.
There's human human on the loop. I heard
that expression. Would you get behind
that?
>> Whatever. What I'm saying is that
there's no more human um you know
directly reviewing the stream of
commands happening and approving
commands. That's kind of like agents are
executing on their own.
>> We're like all in on auto mode type.
>> Yeah. Yeah, we are. Since I can say auto
mode was one thing that was released
maybe a few weeks ago. Then they
released dynamic workflows and they made
an additional follow-up post about all
of examples of dynamic workflows and how
they work. And they give you examples
like hey you can call on dynamic
workflows to go through all of your
previous session and filter out like
common mistakes the model make to
exfiltrate better cloud rules global and
locally within the project. There's many
other examples in there. Um and and so
dynamic workflows plus auto mode have
significantly improved the accuracy of
agents working in the like on their own.
Like I can tell you back in November I
tried to run like hey I have a beautiful
plan fivephase implementation several
modules that are completely decoupled
and they need to do individual
development work. Now spin up five
agents and do them independently and it
would come back with all these agents
creating mock functions for contracts
between the modules and it would be a
complete mess. Um, now I have complete
trust. Like it like it just works. Um,
which is a massive shift. I mean, it's
kind of in line with what like Stripe
was saying that they have like minions.
They just have tickets in in a in a in a
Jira board or in a linear board and then
the agents go off and do all the work.
And and I think that
>> well so it's like guest town is becoming
uh
>> yeah guest town is built into Claude Code
directly now with with dynamic workflows
>> you don't need like uh you know five max
accounts and and and additional because
that's kind of built into Claude Code
with this dynamic workflows
>> I mean my god things move so quickly
don't they mean like oh my god oh
there's something I want there's
something I wanted to point out to you
if I can just find the link.
>> But I think like it changed so fast in
just the last two months, but
it just means that this aent like aic
reality is is coming way faster
to everyone. Like it's become a
commodity. It's not just like, oh,
there's this one stripe company that
software.
>> We're definitely like we're definitely
uh
the pioneers. I have to pinch myself
sometimes. I I really thought I don't
know if you saw this one how how we
contain uh clawed across products.
There's this footnote here that made me
laugh.
Claude code auto mode delegates command
approvals to modelbased classifier. I'm
not too sure why they call it a
classifier as opposed to an agent, but
whatever. It minimizes friction.
So, and it interestingly says 0.4% 4% of
ben of benign arguments or commands were
blocked. So basically things that were
pretty inconsequential, right? And then
it said that um
17% of ogre overeager actions which I
which you can interpret it at you know
like uh how how would you interpret that
that you would you could say yeah risky
17% of risky actions got through.
Don't you think that's an interesting
footnote to make on on on the topic of
auto mode? Are you still there Vincent?
Yeah, I mean in terms of like approvals
to a modelbased classifier when when I I
read classifier I thinks it has a very
strict category like it can be A B or C
and I mean it's not like an large
language model that puts out pros a
classifier would you know puts out a
verdict approved reject
um and then roughly 0.4% 4% of benign
comments blocked which means like this
comment is benign like I'm not English
native but the way I understand it is
that would be a comment that is that is
fine but uh it was blocked anyway which
is a little like you know it's a bit sad
but it's better to be uh you know overly
blocking things and then that's a cost
to that's okay to to pay for missing the
the what's the rest of the sense I don't
know but then in terms of like 70%
of uh riskier commands coming true. I
think what the the main point is what
they're making is you still need to run
this in a sandbox like um you cannot
trust this thing not to nuke your
machine. Um you know
>> that's basically what they're trying to
say. And then on the on the topic of
running Claude in a sandbox, I've been
using that Docker sandbox tool and I
think it's great actually except that
it's proprietary and uh
and that's a bit annoying and and it
seems to require a login and that's also
bloody annoying. Uh but otherwise this
this sandbox tool to to run um agents
inside I think is pretty nice cuz you
can you can isolate the configuration.
You can isolate the guard rails. Of
course you have a nice little UI to show
what network calls it's making. Not that
it's actually useful to be honest but
but I mean how else do you run uh agents
in in sandboxes? Uh, previously I was
doing it with like systemd and spawn but
like no one has systemd on a Mac OS
machine. Like how are you doing it?
I mean
when you use cloud the app on Mac OS and
you use co-work
I think I'm not I I haven't looked at
all into any technical deep dives in how
that works but the way it behaves it
really seems that it's using some type
of containers or some type of of uh you
know OSX native kernel like
virtualization
uh because it cannot just read files on
disk normally it needs to like mount
files into the co-work see. So when you
say sandboxes, you're talking about
co-work cuz I think
>> I mean I'm talking first about co-work
which is one one uh you know consumer I
guess less engineering focused solution
and you give it access to to Gmail and
crazy things like that. Um so that one
does like very um very strong isolation.
Again I don't care how it just works.
>> You're actually using it.
>> Yeah I am like I've got like this PDF
file in Gmail. I hate, you know, filling
things out. I just straight up, you
know, asked the cowork to do it for me.
Fetch the details. I got this contract
that I needed to do some stuff on. I was
like, "Hey, what's my action points
here?" And you know, my details just
fill out whatever you need to fill out
in this thing and revert it, please. I
don't even care. And um I mean that's
where co-work is perfect for me like end
of month type of stuff because I'm doing
some freelance stuff. So then you need
to do all this additional administrative
things. I just go like, "Yo, here's an
extra he I just literally go into the
time sheet web page. I sometimes capture
the network calls and drop drop in the
HAR file."
>> Well, I see. So, this Yeah, co-work has
more UI functionality does, doesn't it?
Um yeah, but I'm still approaching it
very engineering like like I just drop
dropping a network recording like
network calls recording uh from Chrome
into the into like the HAR archive and
then I also I just write I do inspect
and then I cl copy out the HTML elements
and paste it in. I was like hey can you
grap out the cuz these are stupid. I
mean I I think why is is HR system still
so horrible? Like I'm I have to work
with several HR systems.
>> Salesforce.
>> Sorry, it's not Salesforce,
>> isn't it? Oh, okay.
>> No, it's HR. It's not CRM. Does
Salesforce do HR now, too?
>> No. Well, I I Well, I'm not going to
give away how the HR time sheets are
working in my company in my company, but
it's something to do with a third party.
I actually asked Claude Codes to look at
the HR website and figure out how to
enter a time sheet and it couldn't. It
was such a nice software that even the
HR couldn't figure it out. It tried to
click on a couple of buttons and it
didn't work.
>> Yeah. I I feel like in every
organization there needs to be like a
test like how can you um can you tell
buttons to on board? Yeah. C can the can
the agents on board? Can the agents fill
in a time entry? Can the agents uh
>> that was actually very funny because my
wife asked me she has to do some she
works for product um what is it it's
like it's textile labels and stuff for
uh it's very very like manufacturing in
Vietnam and they have to build some
product development portal and she did
some outsource work and she's like I
need to do a usability study and I'm
like
>> let me install coowork point coowork at
it and do a usability study with co-work
seriously I'm not going to bother like
>> that's actually a good idea.
>> Oh yeah. Um,
>> so back to sandboxes like the sandboxes
in cowwork I I I I trust them. And then
>> does the sandbox I was just looking
through these notes here like for
example the docker sandbox uh it doesn't
do um it it basically by default it it
has skip dangerous it's in yolo mode by
default. Is co-work in co-work is not
yolo mode by default is it? Yeah,
>> I don't think so. But um but another
agents that get spun up is is basically
in Claude Code in Claude Code, right? It
spins up a whole bunch of sub agents and
they do whole bunch of things and
they're all, you know, guarded by this
auto mode and I don't think it does much
sandboxing there, right? I think it's
purely running agents directly in cloud
code shell. Uh I think they really
boosted Claude Code uh runtime
environment to may make it possible to
run so many agents. I think they're all
running locally. It's it's crazy. Yeah,
we we discussed this and I'm I'm being
cynical like I guess they're they're
going on they're leveraging agents just
to basically get people to spend more
money, right? I mean, I don't I I I
don't know. But apparently I figured out
that I'm on a max 5x account, which is
why I could never which is why I never
hit any limits. And I was like, "Oh, why
is everybody hitting limits?" Um, yeah,
I'm overpaying. But now I started
actually hitting my limits and I
actually start to use it properly. Um
but but back to sandboxes, right? Um I
actually was inv was looking into like
using firecracker or there was this
other um there was this other framework
that that I saw on LinkedIn and I asked
Claude Code like hey could I use this? Um
and it was like nah man it's like two
months old not worth it. Yeah cloud is
pretty fun. Don't use it and you should
go for firecracker. It's way more
>> how are you doing your Wait a minute.
How are you doing your re your research
with core code? Are you just using web
fetch to get the
>> um
I'm I'm so I'm on the cloud.ai like I'm
on the app and on the browser for the
chats. Uh and I always tell it to ground
itself like go and web search and and
find uh proof.
>> You you shared I mean you've been
showing me a lot and to be honest I
haven't read through them all but like I
think you like some time back you you
mentioned firecrawl. Is that something
to do with research?
>> That's that's a funny thing, right? I
was working on some curriculum for a
course and it's a course on AWS. So
there's a lot of AWS docs there and I
had an initial like road map and
curriculum with modules. Uh and then I
wanted to break that down in like real
content u you know that matches the
actual product offering but I was like
too lazy to go through every page. and I
asked cowwork and it scraped maybe 80
pages of like subsections of some some
areas. So that code Claude Code does it
automatically. Okay. But when you set up
an agent so the another reason why I'm
way way more in on agents um like I
never liked open claw. I didn't see the
purpose of it. You need to take need to
go.
>> No no no I'm just I just noticed my
light wasn't on.
>> Okay. I I I never liked OpenClaw because
I didn't see the purpose of it. But I
decided that because of my son is on
holiday, he should, you know, set up an
agent and then figure out if him and his
friends can use it on a Telegram chat or
whatever on Discord where they are to do
something with it. Uh, I said I'll pay
for the credits and actually we use
codeex um budget and he set up Hermas
and he's not really playing a lot a lot
with it but I have it on discord so I
just keep like asking it can you do this
can you do that and it can do this and
it can do that and it like it can do
things uh it really started to sell me
like when you earlier you said like you
cannot like an agent's not going to be
doing things on its own device but what
I I really like with an agent like
Hermes which by the way needed this for
fire crawl because it says I need the
fire crawl endpoint to do web scraping.
I mean, it supports many other
integrations, but firecrawl is one of
them.
>> Yeah, bright is a good one. But carry
on.
>> Yeah, I Yeah, I didn't pay for anything.
And I asked like, can you run this on
the on on your two CPU cores and 8 GB of
memory? And it was like, yeah, I can run
this on on on on two cores. And and what
does it need? Well, Chromium is good
enough, and you just need Docker and
Okay, let's go. I don't going to pay for
for it. So I'm running fire crawl stack
local and um and it's fun uh and and so
I started running a heras and I
connected it with a I gave it a GitHub
account and I connected it but it
doesn't have any permissions on any
repository. So it only it's like an
external contributor. It has a GitHub
account. It doesn't belong like it's not
a collaborator. It doesn't have any
permissions. It can only look at pull
requests. It can fork the repo, create a
pull request. I can do a GitHub search
because you need to be authenticated for
that, don't you?
>> Yeah. Yeah. It has its own account. It's
authenticated and they block agent mail,
but I aliased it on another CNAME and
and it worked. I was able to sign up
with my one of my domains and it has its
own account on agent mail. So, it has
the ability and agent mail is pretty
cool actually. You know, you you set up
the agent and it has an inbox. It looks
a bit like Gmail and you get quite a lot
for free. Uh I don't remember how
many,000 emails.
>> Something else. I don't know.
Why did you poo poo open claw? Because
Hermes agent sounds the same as
OpenClaw.
>> Yeah. Yeah. Absolutely. Except they say
I did a comparison. Apparently Hermes is
a little bit older, uh, a little bit
more security focused. I haven't tried
OpenClaw, so I haven't really I don't
really have a good comparison to it.
>> But you enjoy.
>> Wait a minute. We
>> And and would you say Fire Call is
better than Claude's native web fetch?
That that's the thing that I was trying
to get out of you.
>> Yeah. So, so it came to the stage where
Claude Code um which is like sorry cloud
co-work which scrapes a bunch of pages
but obviously it's using the entropic um
infrastructure and it's limited to avoid
getting blacklisted I guess. Um, and
then I was like I was playing so much
with this Hermes agent and it has this
whole firecross tag and I moved
everything to a local desktop. So it has
like uh it now has 12 uh threads or six
cores uh on an Intel um CPU. It's no
longer
>> ARM can now can thrash the internet
>> and I was like go crawl. It's like how
many pages are there? And it uses
browser as well. So, so Hermes goes
browser use, looks at the page, extract
extracts, takes a snapshot, extracts the
navbar, figures out all the I mean
honestly crawling a web page, you could
probably do that like 10 years ago with
a basic Python script, right? You don't
need an LLM for that either. Um,
>> well, you do you do need it to run a
browser because a lot of websites are
JavaScript front ends and things like
that. Yeah, but but yeah, I'm I'm
crawling AWS docs and that's where I
also asked you then why because I did
the whole crawl. It was it crawled down
400 pages into markdown with fire crawl
in just like 5 minutes
>> co-work didn't it it just was really
limited it was blocked
>> and so with with her
>> with AWS docs MCP you definitely don't
need to
>> yeah so so then I I came to the
realization because I was um I was
having a lot of fun because this this
desktop has a GPU so I was like it has 4
GB of RAM in the CPU and apparently
Hyperl is using 2.5 GB of it so I have
like a little over a gigabyte of RAM on
my GPU available and I was like yeah I
can't do anything with that right but
cloud was like no no you can run you can
run embedding models locally and you can
you can run a reranker locally and I was
like oh that's cool and and I'm running
like a full semantic memory layer uh on
on a local stack like it's it's using
another really cool find out of that is
PG0 um from vectorz.io IO which runs PG
vector embedded. Uh so it just spins up
a Postgress. It's almost like SQL light.
You know how annoying SQL light is
because it doesn't have like prop I mean
people love SQLite. SQLite is great
because you just have a single file and
you get started and you can run some
integration tests but it's not the
proper postgress diag
schemas. Now you can PG0 you can run a
embedded Postgress of a file just you
know without having to stand up like a
whole Postgress instance.
Didn't know that. So PG0 is like a is
like an SQLite version of Postgress is
it?
>> That's how I see it. So So this whole
hindsight hindsight stack uses PG0
embedded Postgress from the same company
that built it vectors. Is that related
to Hermes agent? What is hindsight?
>> Yeah, hindsight and mem zero and others.
There are memory systems. Um they
basically create semantic links. So some
of them are very basic. Uh there's
another one that I came across. This is
all in a use case of doing like
researching and scraping the internet
and stuff.
>> I mean it's very very very closely
related to like last year when you're
doing um retrieval augmented generation
basically when you have an incoming
query you first go and fetch uh using
those queries semantic related words and
then you fetch the articles and then you
inject it in the prompt so that that the
the model the LLM is grounded with
actual real um you know data. So this
whole idea of doc of like chunking docs
and putting them into a vector database
to be able to do a vector search um you
know semantically similar terminology.
So how so how did you get started with
all this stuff that you discovered it
with Hermes agent firecrawl and then it
just
>> I mean I don't know about how open claw
works because it probably has the same
things but like Hermes agents has this
basic memory file system like you have
the memory MD the soulm the user MD that
describes the user there's a couple of
markdown files on disk but if you start
with Hermes it has like quite strict
limits on how many data there can be in
the file and it very quickly like wants
you to use some type of memory
system and it advises you like if you
use mem zero you just sign up you can $5
credit out of the box.
>> On the topic of of memory systems
uh during that conference I went to last
week there was this misilla project
called CQ and the idea is that um
>> yeah uh let me just
launch the GitHub. I don't know if your
memory program does the same thing. I'm
just mentioning it uh because what they
demoed was that you run CQ on like an
organizational or team level, right? So,
so you store memories as a team so that
when you're when you're working through
some issue, it it does it uses the CQ
skill to grab what they call a knowledge
unit, which is basically the same as a
memory, uh to to basically do the job.
And I thought that was really
interesting. So have you explored using
memories as on a team level? I guess you
don't need to do that but I I do think
it's very interesting idea
shared agent because this is okay. So my
exploration was like originally I was
like should I just set up obsidian
because this it lays out on markdown on
disk right and then a human can browse
the obsidian and see the relationship
between the docs um and and it's very
famous now people are like you use
obsidian for a memory layer for your
Claude Code sessions and things like that
so I thought that's one option that's
one of the first options I proposed to
cloud say like hey maybe obsidian would
be great to for me to talk to manage the
memory of my Hermas agent. Um but or
maybe I I have a Postgress stack because
I have firecrawl. Maybe I can use PG
vector or maybe and then it came down to
what actual integrations does Hermes
support and it supports me zero. It
supports hindsight, it supports some
other solutions. Um I don't see CQ in
there. Um I then asked it to evaluate
like local versus hosted because on one
way I don't want to be locked behind
like I don't want all my memor my my
knowledge to be locked behind an API
that is done very costly in retrieving
information because a lot of these they
make it very easy for you to put data
but they actually try a lot to query to
get data because then you get it's hard
for you to get the data out and it's
also one
>> argument for actually running an agent
is that like you use chbt sometimes
Sometimes it's very annoying like Claude
is very annoying with that when you say
something he says like oh because of
your other project this is very in line
with what you want to do it's like I
didn't ask you in line of my other
projects right I mean I just asked you a
different question um so I don't know
how to tell cloud maybe it's possible to
tell cloud like ignore all my me all the
memories about me and my other project
just answer this question
>> yeah that I I I know what you mean by
that I really know what you mean
>> I think but on the other side the
concept of tagging and domains. So like
you might have a different a different
organizational hierarch you know you
could split your knowledge by the way
your organization looks
>> it has I think I think it's important I
think markdowns on disks is definitely
>> like getting limited at some point and
also like a a confluence database and so
on it's it's proprietary more and more
of these memory systems have these type
of features that are semantically for
agents I I think this really interesting
this confidence I don't know exactly how
it works but I'm assuming
>> the same
>> assuming it has like a way of like if
you use the knowledge and it actually
was useful is sort of feedback that it
was useful you know what I mean or or
wasn't you
there's some sort of like full life
cycle management going on I think it's
pretty interesting
>> I should I I can ask okay I can we
>> and you can still post this by the way
you don't have to use their thing Okay,
that that was one of the the important
factors like there were some very basic
markdown, there was some very basic
lexical search solutions. Um I thought
like do I need semantic like do I need
uh embedding? Do I need this vector
database? I don't know. Uh I tried just
using markdown on disk for a week and
then uh I thought maybe it would be
better. Actually, I asked the the agent
I migrated it from EC2 to my local
desktop and then it shut down the
gateway on the EC2 instance and it spun
up the gateway in my local desktop and I
asked the agent because it migrated
cloud migrated everything cloud you know
that was very nice to see it. It did an
SSH tunnel to the um EC2 box and a tar
stream down to the
>> agent. This is the Hermes agent that
did.
>> Yeah. So I had I had been using it one
week in the EC2 box. It has a whole
bunch of like environment configuration
memories it created. It had its own
state
>> shut it down.
>> Yeah, it had its own state like um you
know apparently there's a state database
with all the sessions data in there and
and then you know cloud helped me
orchestrate the migration. It said and
it actually did like a compression of
all the files on the EC2 box streamed it
down SSH and then decompressed it onto
the local box which is like like
>> Okay, you you're running code via
Hermes. Okay. No, no, no, no, no, no.
I'm not I'm not I don't want my account
to be banned. And I was also very afraid
that that Antropic would detect the
Hermas keywords and and ban my account.
What I was doing, I was running a
control laptop that has SSH to both. And
Cloud was orchestrating everything.
Cloud was running SSH. Yeah, Cloud was
writing shell scripts to migrate and to
like get all of the data from the EC2
box into my VM box. Yeah, I've seen I've
seen Claude work over SSH before and
usually it just goes S sh.
>> It doesn't obviously it doesn't have the
power to maintain.
>> Yeah. Yeah, it does. But it works. It's
insane. Like every because it's the same
like running a command and bash on your
local, right? It it writes this whole
command script or whatever and it sends
it over SSH and then waits for the box
to settle and then gets back and says,
"Okay, cool. This is all all
configured." So it did the whole Q emu
virtualization like VM setup, memory
allocation, disk allocation, natural it
created two bridges. Uh it gave the VM a
local LAN IP bridge. It gave it a um a
neted bridge with so there's host only
communication. So the whole I want to I
want to show you like the the hindsight
dashboard because it looks really cool.
You get like this semantic like
knowledge graph of all of the memories
of the agent.
>> Sure. Sure. Although
>> but it's all isolated. It's all locked
away behind a virtual inter like bridge
network that's only accessible between
the whole
>> do a tunnel cracky.
>> No, no, no. I have to like I have to
open the the UF the firewall to to open
the port.
>> Well, that's the it's the same way for
openclaw I think. Hey. Yeah. Actually,
in in the interest of time, I uh because
I need to get to work actually.
Actually, part of my work today is to do
a risk assessment with the Slack MCP and
the Gmail MCP.
I mean,
>> what's the SLE risk assessment like for
people to use it and and risk
>> for for for everyone to start using SL?
I mean, I I'm a little like
I wish I could say I'm not sure this is
a good idea. Stop.
Like what? Why use Slack MCP? Now
everyone's going to launch agents into
the Slack channel to do what?
>> That's the thing, Kai. That's that's
what organizations want now. like this
Hermes agent that I'm doing is
preparation for my job that I'm going to
start because agents are going to be
part of the of the of Slack of Jira of
Confluence and we're going to assign the
tickets directly to them which is I'm
already doing with with the CDK terrain
open source project. Yeah.
>> Um I'm giving it access I mean it
doesn't have any access but it only has
read access. So I'm telling it there's
actually there's actually a poll on the
chrome that pulls request that pulls
down a list of pull requests keeps it a
tracker of when was the last time it saw
it. So it's like a state on disk a
simple JSON file that says I've seen
this PR. It keeps a list of trusted
authors and it it reads the PRs of
trusted authors as long as they're not
in draft and then it reads the common
thread. It's all deterministic. No LLM
involved, right? The script is purely
terministic and it masks out any comment
that is from not from a trusted person.
So then it sends the full prompt to the
LLM to Hermes and then Hermes does an
evaluation which doesn't just read the
code. It looks at the file diff, it runs
the tests and it looks at the function
of that PR and then runs its own things
like for example I introduced release
please and we wanted to validate that
after it was merged to main um that when
we merge a feature it will not bump from
0.22 to 0.1.0 zero because we are in uh
pre- major like alpha and so so I asked
it after the release please PR was there
like hey do a simulation and Hermes just
creates a complete separate work tree or
or directory simulates the merged calls
this branch main adds a bunch of commits
runs the release please command and
validates the PR that would be created
if it was not a simulation I can run
release please
>> I'm a little bit confused I mean surely
this could have been done with Claude
why do you No, you can't. What do you
mean? I was I was eating breakfast and I
sent a message on Discord.
>> I see. So, so Hermes basically
coordinated that work for you. Okay. All
right. All right. But, but like
>> the point is that this is the work the
way we were going to work. Agents are
going to be
>> going to work. Yeah. But like I do feel
that we're crossing a boundary very very
quickly. Like for example, okay, Slack
aside, never mind Slack. I can
understand maybe that you would want
your your agent to be aware of some
context that's inside
>> because that's where you define the
conver like the the conversation defines
the the product. But I think you need to
dedicate those channels. You need to say
only these channels.
>> But then the next thing that the the
that people I'm going to ask for now I
want MCP for Gmail and then but now that
Gmail is a completely different beast
because because you can get external
mail into into your Gmail inbox unlike
>> [ __ ] So, so my my my Hermes agent has
uh an email account, but it doesn't read
the email. Like the email gets posted
with a chrome drop into discord and I
choose to act on it or not. Again, my
agent cannot be directly talked to by
anyone. It doesn't have any permissions.
It's always an external contributor. It
doesn't have any external like except
for the one deterministic that I say I
have trusted authors, but it doesn't
like it reads the PR thread, but it
won't respond.
Well, if somebody asks and tags it on
the PR thread and says, "Hey, can you
tell me this?"
>> I like I like to think that I set up the
same rules for an open claw instance.
So, basically, I got open claw running
on my dad's uh WhatsApp. And the funny
story was is that so um my sister had a
WhatsApp group with my parents including
my father and my father was basically
talking and like everyone couldn't
understand that cuz he's not on
WhatsApp. Then later um I was invited to
the same group and I was deadly
surprised that my that my uh openclaw
instance was chatting to my my sister
and my mother u without without me in
the mix.
There's me screwed up that ACL or
whatever. I mean all I'm doing is like
uh open claw you only talk to me but it
didn't work in this instance. Maybe
that's where Hermas is really good
because when you set up an a gateway
integration, you have to identify the
whitelisted like user ID on discord
thread like channel ID. Uh we give we
give the discord bot token permission to
create a difference with open claw but
through the upgrades maybe something
something broke. Oh god. Okay. I I need
to write some risk assessment and fix
some stuff at work. But anyway, it's
great talking with you, Vincent. Uh, and
>> agents,
>> hopefully anyone who's listening can
weigh in and say that we're doing it all
wrong. And
>> like I I'm an absolute noob with agents.
But I I was I was I was an absolute, how
do you say, pessimistic? Uh, skeptic. I
was very skeptic. Um, didn't trust it.
Didn't think it would work. Didn't think
it would give me any value. And in one
week it just um I can't I can't find
enough jobs to give to the agent. It's
like can the agent do this, can the
agent do that?
>> You you need to record you need to share
some of your workflow with me to maybe
just
>> I shared I shared the comment thread. I
showed you the whole pull request back
and forth between Claude Code web cloud
code and the web basically while I was
eating breakfast was implementing the
release please. Then in this course I
told the a hermes agent can you like run
the simulation and it ran the simulation
and then it commented back like I ran
the simulation and these were the
scenarios that worked that didn't work
and but the worst part is after all this
back and forth um after it was merged
because of the markdown wasn't properly
formatted in the change lock it injected
the but release please is not LLM right
it is derministically so the regular
expression injected the next version
release uh notes in the wrong section on
the change lock which was after all this
back and forth with the agent and all
these simulations nobody looked if the
change lock was actually you know
properly generated like yes it's
injecting it yeah but it's injecting it
in the wrong bloody section
so there's still still not like I think
if a human would have looked at it he
would say like yeah but it's not in the
right place so yeah still still there's
always some disappointment isn't it
>> yeah I mean yeah let's don't trust talk
about my sessions.
>> The risk assessment. Don't Don't trust
the agent on the risk assessment.
>> I'm going to ride as much. I'm just
going to
>> It's so hard.
>> It's so hard for me not like to just go
like, you know what, let the agent do a
first draft and then the draft gets so
bloated that you go like, you know what,
let another agent review the first
draft.
>> Are you not maxing out your max plan?
Must be getting close.
>> Not yet. Let me see.
>> I have 100% left right now. Weekly I
have 72 left weekly. Resets in two days.
42 in reserve.
>> Tropics surely are going to do a bait
and switch. And this is
>> they're they they're in the summer.
They're doing promotions. school work is
like increased limits and and and I
think also
>> once the IPO is done and dusted uh well
fast forward a year you're probably
going to be living under
>> but that's why I want all of my memories
out of Jet GPT and inside my own memory
bank if I have a sufficient hardware to
run it
>> I believe that's CPU and um and let me
know what you and yeah or whatever
memory thing you use. Okay. Anyway, I
really need to get back to to work.
Great speaking.
>> Read the blog. Read the Hermes blog. It
tells you how to migrate out of CHGPT
using the entropic posts. Tells you all
of the lessons learned in setting up
hindsight on a local desktop with a very
old GPU with
>> I will give it a try. I will give it a
try at some point.
>> A lot of
>> I do have a spare machine. I got a Oh, I
didn't did I mention to you I have a new
M5 48 GB beast in front of me now.
>> That's That's plenty.
>> Yeah, it's nice. And did you did you see
that I was running uh Gwen LLM locally
on Twitter?
>> So, one thing I learned because I ne I
never really cared about it, but the
Deep Seek is actually really cheap. Um
>> I think I sort of coach Flying by.
>> So, you you're using that locally or you
just use it? You got the little
subscription?
>> No, not local. I like I don't have the
the laptop like you, but even then like
you were saying that the the the token
per second is extremely Oh, no. You were
on Twitter and somebody replied the
token per second is just uh abhorent. So
So for me,
>> the tokens per second is not terrible. I
thought
>> they said for for coding work you need a
really high token per second because you
will not you will not
>> yeah actually to be honest you're right
cuz I noticed my token tick claude goes
to a,000 in one prompt
so 50 into a th00and is uh yeah
>> yeah you need really so so for me when I
was looking at local LLMs I mean my
hardware isn't good enough and then I
was like you know worst case we fall
back on like I put in a wallet on Deep
Seat and let's see how he goes. And it's
been
>> three days and I barely used a dollar
and I was like, "Huh, this is so good."
>> Yeah, I actually configured Open Claw to
use XAI and I was surprised how little I
spent there. And XAI is some
>> rightwing not drop off an LL
>> and XAI is really good because it has
all the Twitter info which is good for
research I feel. Okay, that's ended. See
you everybody. Please like the video.
Comment below.
>> Good point. I need X. I need I need some
Twitter info because everyone post
>> Exactly. So, get ready to for that
SpaceX IPO. See you. Bye. Okay. Bye.