Episode 31: Checking for AI guard rails

How does it go? How did it go? Did you
have a play? Did you do what you said
you would?
>> To be honest, I I wanted to I did, but I
kept on running into issues because the
project that I had the std spec driven
development in mind was a mapping
project which has a ton of dependencies
in order for it to work with the open
street map. And I kept on having
problems with Docker and
>> yeah because I needed Docker to build
the sort of dev container with all the
with all the open street map.
C++ boost balls. But let's not let's not
talk about that because
>> but I think it highlights a very
important thing, right? Um you you
speced it out. You had the user stories
but you got stuck on technical problems
like the
>> No, I didn't even I to be honest I
didn't even get that far.
>> Okay, I need to revisit that.
>> But today I was going to conveniently go
around that way. I hear this echo.
>> Let's Hold on. Let me just see if I can
>> probably my my head too loud or
something.
>> Oh, echo cancellation.
>> You mean you hear yourself? Maybe from
my can back into the mic.
>> Maybe it's me.
>> Yeah, the I mean changing the topic cuz
the spectrum I mean need to I I just
I've had a few just abortive attempts to
spectrum development. I just need to
give it another try. So have you been
doing it? What what is your status with
spectriven development before we move
on? So I've been working on a rewrite an
open source version of the back end of
Spec Ledger. So currently Spec Ledger
public repo is a Golang CLI and the
actual web app is a Versel hosted Nex.js
app with serverside rendering and I am
not a fan of that. Um I also prefer
monor repos and I don't mind. I think it
will be more interesting if the back end
is open source directly within the CLI
repository. So, I've been specking out a
new version of of the spec ledger with
organization support like multi-tenant
from the get- go. So, you can organize
projects per organization and then you
have your private workflows of if you
want to do openspec or if you want to do
specit or if you want to do Google
conductor
>> then you would should be able to choose
internally with like or you can build
your own version of your of your
templates. Basically what I want is I
don't want the prompts the um the
workflow to be embedded within the
Golang CLI. I want the Golang CLI to
like you know you authenticate and you
pull it from your organization or maybe
maybe there is the first version is is
is is available so when you
>> organization organization why do you say
organization is that just to set set the
context or something? Yeah, because I
believe the main purpose of or the main
problems that I'm trying to solve is
stakeholder buyin and sharing access to
the artifacts that need to be reviewed
and a review environment. And then the
second thing I want to solve is keeping
your specdriven development projects
aligned in terms of workflows so that
you can version and pull the workflows.
And then because specdriven development
workflows are heavy-handed, you should
be able to customize your workflows to
have maybe less steps for your for maybe
the data team doesn't want to do certain
steps of it. So you could customize
basically right now on Reddit I see
quite a lot of posts about we had the
problem of distributing skills within
our organization so we build an internal
skills registry which I think is already
hold on
>> does mpx skills.sh SH support private
ones I wonder cuz that's the way I I use
to distribute my own skills.
>> Yeah, so Versel is very simple the NPX
skills from Versel. I actually ported
that TypeScript source code because it's
open source into Golang specure CLI. So
um so basically because I I don't want
you to have to run multiple right beats
is inside the Golang CLI. the skills is
inside the Golang CLIs manage. So you
just download
>> building your own bundle package of AI
tooling or something.
>> Yeah, I mean you could argue that you
take the best tool for the job. But I I
think all of this is related like by
adding the skill search and skill
install capabilities directly to the
Golang CLI of the spec ledger. You can
also trigger it by the agent. So when
the agent goes down to the plan phase
and identifies a certain text tech like
hey you're building a react it then
actually goes and searches using the SL
skills any react skills that we can use
and then it proposes like I found these
skills and the versel uh registry has
metadata from snik about vulnerability.
So the agent does that
>> so it sounds like every organ
organization needs to heavily customize
this spec ledger workflow or how do you
describe it? Yeah.
>> Yeah. Because I think skills are like I
don't know my experience with skills is
that they have some skills I cannot
share because they have some hard like
some information that is private. So yes
you can use the versel skills dot you
know skills registry which is actually
it's interesting because after porting
it from Typescript over in Golang cloud
told me a couple of interesting things.
Apparently the way that skills are
indexed by that tool is if you use it to
install a skill.
>> Yeah. Then it comes up on the
leaderboard.
>> And then it comes up on the leaderboard.
It's through metadata to usage metadata.
So, so my Golang CLI is also sending
usage like every time you run the specs
ledger CLI to install a skill, it's
sending the same metadata like this
skill was installed. Yeah. And then the
second part is the skills live inside
GitHub, right? There nothing else.
There's no there's no no other registry.
It's just simple. A skill is referenced
by uh a git repository. There's a
shortcut for the Versel skills which are
served directly from the registry
endpoint. So for every or pretty much
every other skill, the npx skills CLI
actually does a get clone of the repo
and and then my Golang CLI is like
trying to use GitHub API, but then there
was a bug because it wasn't like getting
all of the references like we get get
the main
>> API is a nice way to get some private
stuff.
>> Yeah. And it would be that's my goal
ultimately. My goal is SL CLI, you know,
and as part of an organization, you have
your own skills that you can't make
public and that you don't want the users
data to be registered. So, so then you
you should be able within teams to run
the SL binary and then uh hide or or
just have private skills as well on top
of the Versel registry hosted skills.
>> Let me just
>> But the same with agents.
>> I'm trying to find um how to share
something. Where's the chat? There's
chat here somewhere. Oh, yeah, it's on
the right on the side. So this this this
rewrite of spec ledger is really a big
like a thing that I wanted because the
way I envisioned it, it always had like
the ability to create your own workflow
within your organization and share those
across repositories and that was never
possible because of the way it was
implemented. So I'm really excited about
this rewrite but um I haven't been able
to actually you know complete it.
>> You can see what I'm writing here.
>> Yeah, it's a big box with spec ledger.
Maybe you want to zoom in because when I
watch these um recordings on YouTube the
it's always super small. I usually watch
from mobile.
>> Okay. So, um I I love how like in the
last one I'm we we called this this
whole thing
um you know this whole STD thing a
planning tool uh you know driving
alignment and now this thing like you're
doing a package a bundle of of necessary
AI tooling or something is the way
thinking I'm processing what you just
said to me. Uh yeah it's more like um
boilerplate management. So if you want
to bootstrap a repository for a certain
type of development it will you know
with SL init you will get the full spec
like spec kit workflow which is the same
spec ledger workflow
>> and then every prompt is telling the
agent to hey if you run the SL binary
you can find open comments that people
has left on on these documents. you can
find um issues that have been created
and and task dependency graph and what
is open what is available what's the
definition of done has this been
completed
>> oh so do what are you using for like a
beads visualizer
>> no visual I yes sorry so when you push
git there's a web hook that parses the
JSNL
>> uh and then visualizes into a conbon
view and um a tree view in in the web
app so
>> Oh so so that's part of your your web
app
>> yeah that's the back end so the back can
is the web app is is really so I I did
um a little like brain
storm with um I think it was with CHPT
to identify if um Spec Ledger should
really be a plug-in for backstage or a
collection of plugins for backstage
because I felt like backstage has is an
internal developer platform that gives
you a single pane to discover you know.
>> Yeah, I'm not I'm not fan. Yes, I know a
lot of people hate backstage because of
past experience because it was hard to
customize because it required you to
write TypeScript plugins. I think a lot
of those arguments and I'm very excited
to challenge that opinion of me are gone
with AI because now asking and also
backstage has repositioned itself more
as like a context delivery or like AI
native tool and like remote MCP. So a
friend told me like I think we discussed
this last time. He said that having a
remote MCP server unlike having all
these skills inside repositories and
distributed across machines is much
harder to keep aligned than having a
remote MCP. And a remote MCP is always
up to date to the version that's
deployed for everyone. So that made me
think like actually um having this like
this IDP as a context provider.
Therefore specledger should really just
be like a small AI workflow plugin to
it. But at the same time, I don't want
Spec Ledger to pre-require um uh
backstage obviously because a lot of
people don't want to run it. So that's
why I was like, is there a way for me to
build specger as a bunch of plugins and
then mock the whole of backstage as just
um you know, if you don't want
backstage, you can just have like um a
virtual one that just runs spec ledger
only. Um but it looked like it's really
not a good idea to do that. But it also
surfaced very interesting stuff. So,
backstage does have some document
plug-in that basically uses MK docs to
visualize the um the data and and I I
was basically telling TGPD I feel like
specledger is just um a like markdown
render like a web app that renders
markdown out of a git repository, right?
But it's not really um because Specled
Ledger gives you metadata about the
ability to to leave comments. It also
visualizes specifically the workflow
because that's a common problem with
specd driven development is like I've
done this what's the next step I
remember our early discussions every
single time was like what's the next
step so so spec ledger visualizes that
process so it's it's a very customized
um for the spec spec development
>> and then how does GitHub fit in with
this
>> GitHub right
>> is git is the source of truth right but
you're assuming GitHub is there and
going to be running like your CI/CD and
the rest of it, right? I mean, just to
build a complete picture here,
>> um, I mean, Git is a source of truth.
Specled Ledger just generates documents
that are version controlled in git and
the spec spec web app is currently only
built for GitHub web hooks, but it could
support any git version control system.
Uh, and just visualizes the contents of
your git repository also across
repositories, right? um as an
organization then you should be able to
like one one other feature that specled
ledger I feel must have is the ability
to search like is there any like have we
if this uses story somewhere is there
like like you know to index and find and
source like what what you get cuz
basically it's kind of like a knowledge
base like what do you need if you if
you've replaced confluence with kit you
need still the ability to see the
documents in a nice uh printable view
mobile so it's possible to you know have
a mobile version to to easily review and
look at documents. Um that's really bad
on the GitHub interface because that's
not what it it's written for, right? And
uh the search functionality, right, with
Confluence, you can very easily go and
find related documents and things like
that as a KB. I mean, in the end, it's a
bit like Obsidian then, right?
Creating my gosh, I'm really messing
this up. Okay, so I think I I I
understand where you're going here. I I
think all this tooling
um has its place and I think it's kind
of interesting. But I just I wanted to
just contrast this with with my reality.
And my reality is like
um
I'm usually like in a in a in a team. Oh
my god, what happened to my you know I'm
usually in a team. I mean this is great
for for alignment, right? But I'm I'm in
a team with like a couple of other
people. We're all generally using um but
that's you know something like claw code
or codeex or whatever
>> or open code
>> or even open code. Yeah. Well, we all
Okay, let's let's see if I can draw a
picture that's a bit more
representative.
>> There's a massive CDK terrain release
coming by the way. Lots of fixes.
>> Nice. like we we all have like a our
individual uh uh
sessions whatever and um and we all I
don't know working on different sort of
things
I I guess I guess in reality this in a
proper software project this turns out
to be uh PRs I suppose
>> yeah which is also I mean I mean as
you're drawing arrows which is also a
feature that specledger built which is a
a session like um indexing engine,
right? Every every time you do a commit
spec ledger CLI is um triggered as a
hook. And to be honest, right now that's
not very configurable. It's a bit like
beats. It's it's like a virus and it
stalls themselves. But basically, when
it detects a get commit, it will capture
the human um conversation of what led to
that commit inside. Yeah, it does.
>> Okay, this is what I was looking for. A
a get commit.
>> That's how it has been implemented. I
don't know like I let somebody else
decide what's the best integration
pattern.
>> So it grabs the the transcript or
>> it grabs the delta. So every like every
time um it does it uses like a
sophisticated regular expression
uh and it it identifies um which context
that you're on like for example are we
working on this spec or that spec which
branch is he on um and so on. And then
it will capture all of the conversation
since the last capture point uh and then
put it inside superbase. And then the
idea is then to surface it, right? To be
able to see this comet was really good
or this like this whole feature went
wrong. So every single like ultimately
all of the messages that led for one
feature spec are all tagged by the
feature spec and we can break it down by
commit to see how did the user come up
to that commit. Um but that part of
actually mining the data displaying the
data is not implemented.
>> So we do have like data being in in
superbase. I have never looked at the
table. Somebody else implemented it. I'm
not sure if this is the best way like
one of the approaches we thought was
like why don't we just you know just
drop the whole session in inside S3 or
something like that like obviously with
the consent. Um yeah, but um yeah,
that's not really there, but that's
definitely one of the things that we
want
>> that we want to expand on.
>> All right, cool. Cool. Cuz like I
there's a few things that that that come
to mind.
>> I keep saying the same thing because I
mean I also keep saying the same thing,
but you shared Simon Willis like
visualization of how you capture your
sessions and put them into an HTML
format.
>> Yeah. I'm just revisiting it. I'm just
revisiting it because there's still some
questions in my mind like um
like the PRs like like like guard rails
here like like like things I think are
quite sensible to do like like how how
would you like limit the size of the PR
because I mean I I mean I'm guilty of
doing this myself when I'm I'm and I'm
in a bit of a session with Claude or
Codeex and my PR that I just created is
huge like wouldn't it make sense?
>> I mean this is a common problem with AI
development, right? That that it's very
hard for humans to review. I I honestly
think with
that we have to trust our CI/CD and our
tests more that we cannot review the
code. Um but I also I have two ways of
working with with uh with the code,
right?
Um
I have smaller PRs when I tend to use um
cloud plan mode and I do that when I and
so no spec driven development when I'm
working on like an issue that was
created. So I have two ways two work um
streams. One is specd driven development
creates the issue list next issue ready
you know based on the spec and the user
stories and everything that we agree the
humans involvement is the review of the
user stories and the validation that the
end to end thing work and the user story
is is implemented as is exe executable
specs should be the ideal proof that it
works we don't care about the code
>> okay I don't think I've this before but
that makes a lot of sense the human okay
let me just let me just pen that before
I lose it
the human does the um I'm I send you the
link in the chat by the way. Um a story
review. So that comes I I'm not too sure
what the the whole cycle looks like. And
then I'm looking for a I don't know an
AI. The AI does does the um or the agent
does the the the code review is what
you're saying, right?
>> Uh
code review. I mean awesome. I think
right now people are going back and
forth this a lot between because some
people like I mean back in January a lot
of people step up and says I'm an
engineer that have like senior engineer
lots of experience I don't look at my
code anymore right lots of people said
that back in January I think people now
in May are coming back from that a
little bit and saying yeah you know we
did that we we didn't look at our code
it wasn't really good we found like so
much duplicated functions so much uh
>> I don't know like running situations
>> so you were just telling me I mean sorry
I was maybe half listening because I was
trying to
>> cuz I was telling you that the human's
involvement is purely and I don't user
stories yeah on the user stories and
then validating that like I mean you
have you should have automated
validation like we talked and endlessly
about that um how do you really have
fully automated tests and the ability to
bootstrap a full ephemeral environment
and validate everything works exactly
the way you expect or how you have
executable specs uh and then on the
other ite you have the the code. Um,
right now people are going crazy, right?
You have agents reviewing agents. My
friend was like that's that will never
work. Agents will always say that they
are doing the right job. But to be
honest, you can do adversarial reviews
and agents can be very critical. You can
create dedicated skills that focus on
identifying security issues and so on.
There's a lot of startups right now with
huge amount of investment um for
for um you know for AI review systems,
right?
>> Yeah. Yep. So I mean I mean that's the
next thing I wanted to get on to with
Okay. So so I'm just I'm still trying to
imagine this PR. So So you still have a
huge PR and you just basically rely on
your your CICD and your your a your
adversarial agents to review it. But
like and and then as you said like for
security you also have a particular
agent to to
>> it doesn't all have to be agents right I
mean right now you have very decent like
Golang evaluators that say complexity of
function is too is is too
>> yeah like go sec and go
um go lint something whatever
>> yeah it will say you have three layer
like you you this is there's a nested if
branches um this is too complex uh fall
is not clear.
>> So there's some deterministic uh things
you can run over it.
>> Also test coverage even if you want to
consider it as a valid metric.
Um that will say like I don't know I
really don't like test coverage after
you know the problems I had with with AI
because of the amount of refactoring you
have to do and how I feel tests fail
when you have to do massive refactoring.
But maybe it's a it's a skill issue of
initial planning. I I I I I'm I'm I'm
not a fan of test coverage. I feel like
almost ripping that out just because I
don't want to encourage it because like
sometimes I think the the Go core team
they basically
uh they have very light uh testing in a
way. Their coverage is not for the full
Golang source code. They they they test
where they think it's meaningful to
test. And I noticed that in my
personally in my own projects when I get
AI to do um you know try to get 100%
coverage like it like completely
re-engineers my my source code and
completely uh you know creates these
interfaces just so that they can
properly be tested and it m it just
balloons the source code in my it makes
the source even simple projects kind of
difficult in my opinion but I know it's
probably better engineered but I just
for a hobby project I just hate it. I
had an interesting experience very early
on when I started learning Golang. Um I
was writing small little projects back
like 10 years ago and I was trying to
write go tests and it wasn't going very
well and I ended up rewriting a lot of
my like I implemented all the stuff and
then I was like okay now let me write
some tests like I haven't done that
before. Uh I start writing tests and I
can't test this thing. I can't test that
thing because you you can't like execute
it individually. you can't you you need
to like the function needs to uh you
know maybe the interface must be what
you call it inversion of control must be
injectable must be this type of patterns
and I I was like wow after I finished
writing the test I feel my code is so
much nicer I've like and I go to my
colleague and I say like well I just
learned how to write go test it was
pretty cool I rewrote all of my original
code and he was like if you have to
rewrite your original code to write go
test that's probably not a good thing
and I was like to be honest I believe
the original code was really bad and I
believe rewriting your code to implement
the tests made it a lot better like
you're just saying oh it creates I I I
think the fact that you had trouble
adding the tests or the AI ask you to
you know to re interfaces because it was
all tightly coupled and and not possible
to test again where do you want to do
the tests do you want to do unit test
>> I think on on a on a client project I
would have done the proper thing it's
just it's just I find I find it's uh I
was just trying to make a facicious I
was trying to make an argument that when
you go to 100% uh when when you want to
achieve 100% coverage, weird things
start to happen in my opinion.
>> Yeah. No, like every metric once it
starts to get gamed, you're not getting
what you want. And I also had a
colleague like who said my my coverage
is super high, but that's usually that's
like you have some organizations where
you can't have more than X percentage of
coverage. Can't be more than 70%. If if
it's more than 70%, you're you're you're
you're writing your test to technical
implementation details. So I think
there's there's some assumptions that I
think that we need to capture here
because I I heard a story where there's
an organization that's using Claude
everywhere and and in some of some of
the you know the different teams the
different skill levels and some people
are basically producing like
uh artifacts using claude or codecs
which maybe don't go through a CI/CD you
know like like like how do you prevent
claude from like publishing to a public
site or something like this these sort
of things. Um like I feel I feel like we
this is quite a rosy situation where
everyone's like
>> I disagree. I mean if you're telling me
that the organization doesn't have
proper uh you know development practices
and people are like I think the biggest
issue that people have with AI messing
up things is that they are not engineers
and they have never worked in in an
organization that actually follows
proper sane engineering practices.
>> But this this is the
>> having separate environments not getting
giving access to production going and
promoting artifacts and as you increase
the trust. I mean a lot of these stories
that I hear about AI is because you're
not following the most basic DevOps or
security practices that even the
smallest startup has.
>> Well, I mean I could come up I can come
up with some scenarios like for example
like for like you you in a in a even in
a a small or larger organization you
might have people who are like um you
know working in finance and things like
this and then they want to come up with
some sales report. I mean to them this
this whole engineering stuff is
completely foreign to them and then they
end up
>> but but it is a problem where where
you're not like an AI engineer or an AI
um what what would you say an AI agent
is basically need to be on boarded onto
the systems right and the boarding
practice must um must be solid if you do
the same with a human you could argue
that like you tell a human it's going to
do it's going to make a mistake but you
tell it not to do it again it's going to
do it again and AI agent doesn't really
learn unless you have like this like
memory system or or so on, but like it's
still going to keep making the same
mistakes. I I would accept that as an
argument, but still it's a problem with
access. If you're saying like, oh,
there's a finance guy now gets wants to
get cloud access because um it's all the
rage and he believes himself um an
excellent engineer because AI tells you
this is so easy and let's do it and then
it just goes and publishes um a
financial report which would be private.
Uh I can agree with you that that that
happens. Um and and we actually have a
DevOps day talk about that last year in
Singapore where if your organization
doesn't have a proper policy on AI
adoption already a year ago they were
saying like well how are you going to
govern um AI like if if you don't have a
policy people are going to use their own
subscription and their own tool.
>> So for example the finance or the
nontechnical people in an organization
let me get this straight. So you're
saying that you wouldn't give them
access to to claude on AI tools?
>> No, no. Like the talk that I refer of uh
DevOps day Singapore last year
explicitly said that as an organization
you have to have an AI policy. You have
to put in place um the contracts and
also you have to give them access you
have to give them a proper way to access
it. That includes through a company
subscription um with limitations that
like will
um you know first of like guarantee that
there will be no training on this
internal data. No internal data will be
used by the inference provider. So
that's a proper contract between the
company and the um and the inference
provider. and and then other suggestions
that was in that talk which I'm off the
top of my head was like to actually have
a router that looks at the AI usage. Um
basically everyone is using AI through
this router and then that gives you a a
layer of accountability and also ability
to block things.
>> So on on that topic how do you how do
you route all the the AI network
requests through a router? This is
something I I I I was meaning to look at
today. I I
>> so I can I I could find you that talk
because he he gives an example. It's a
year old so obviously a lot have
changed. Um and also funny I think we
mentioned this in the previous call as
well that a lot of these routers are
actually like stealing um your your
credentials and like trying like crypto
wallet stealers and so on. So you have
to be careful which one you use. But but
I think another important part of that
aspect is or another important aspect of
that is something that's really
happening a lot right now with agents
messing up is like agent like you just
mentioned guard rails right I talk
frequently with George who works at
stackpack and they kind of pivoted
towards more of aentic uh because they
are focused on devops tasks so they're
focused specifically on how do we put
like a harness that catches
damaging actions and then uh stops them,
right? I think there was recently an
announcement from AWS or uh they said
like hey here we give you a sandbox was
it I think Verscell announced agent
sandbox where they say now you give we
give you a full sandbox and um we
guarantee that the agent cannot do
anything anything silly. I'm not sure if
it was Versel or AWS. There's been a
whole there has been a few announcements
like I I thought I was of the opinion
that AWS Bedrock had had a lot of stuff
in this regard, but I'm I'm just talking
up my ass really.
>> I remember seeing an announcement
somewhere on on on on um LinkedIn being
like we we guarantee you that the agent
cannot do something damaging.
Like in my previous client, I was I
think I was whining and bitching about
it with you, but like it was heavily uh
limited in the sense that we could only
use it in co-pilot
and I guess it was a bit for for
fortuitous, but we were using uh dev
containers. So when we were using dev
containers, we were using co-pilot and
we're using AI and dev containers and
everything was very was sandboxed by
default like without us even thinking
about it. Um so
of course
of course most people who who using AI
are not really using it in a sandbox
right at this point in time. I I know
maybe clawed code work tried to get
there
uh but like most people who who who just
run clawed code are don't have a proper
sandbox really especially when they when
they go you know dangerously skip
permissions or something.
>> Yeah I think
but so Claude Code added a new
interesting thing which is auto mode.
Um, so originally when you did a plan,
you had the option to go auto accept
edits and then it would still constantly
ex ask you for permission because I I
don't use I don't run dangerally skip
permissions. I am the bottleneck. When
it asks me a comment, I haven't
whitelisted. I have to approve it or
allow listed. Then now it has this allow
mode which I've been trying to use a
couple of times and successfully
sometimes but lately I actually kind of
avoid it. So auto mode is a separate
eval agent that like a small little
model that does a quick eval like is
this a dangerous activity or not and it
rejects things because it says things
yeah but that's kind of like affecting a
third party system and we should not be
affecting third party system like it's
very uh generic type of evaluator. It
doesn't take into account the context of
what we're trying to do. It just looks
at the command and says could this
potentially have a negative impact? If
it does it just rejects it interesting
and then it gives the reason. I thought
we're going to maybe talk about hooks
and things like that. Have you have you
played with clawed hooks?
>> Yeah. So the the spec ledger CLI is a
hook that um that that acts whenever a
command get executed.
>> Yeah. But like are you talking about
like a cloud hook or github?
>> Yeah.
>> Cloud. So it happens whenever the agent
triggers a git commit because a lot of
the prompts and a lot of the work I do
um now is I I have no problem with the
agent. Actually, before I never allow
the agent to commit, but now I always
ask the agent to commit because it
writes a really good it stages the file
correctly. It writes a really good
commit message that I like to see. So,
um, so that's why the agent it requires
you to trust the agent to do the commit.
So, when that happens, um, it's a bash
call that that doesn't
>> your claw hook uh sort of make sure that
there's a good uh get.
>> No, no, no. the only thing it does
because it has to do something real fast
uh and get out of the way because if the
hook fails it it really breaks your
agent's shell. We have experience where
if CLI breaks for whatever reason and
then everything the agent tries to do
because it was a a bash tool hook. So
every time the bash tool was called to
do a git command or whatever to identify
if it was a big git command. It was a
nightmare. No. suddenly like I would
download the new spec ledger CLI uh
release and then I would try to use
cloud and it would just break like the
model would just constantly every
command and it wouldn't tell you why it
would just be failed failed failed and
after investigation we figure I figured
out like oh it's because of the SL
binary hook is is exiting exit one and
the hooks are failing and there was no
information why
>> that's the one thing I always get
frustrated with uh all the tooling
really is that like sometimes something
hit something went into my global thing
and then and then and then there's
something in my project thing and I
don't know what the precedence is like
or what what's being used
>> but that what I really like about that
is that whenever something like that is
failing I always ask cloud okay go and
look what is going on and it goes
through its local settings it goes
through the global settings it goes here
and there and says like these are all
the things that are currently connected
that could be impacting this and and
here it is this is the fix
>> so the next thing I want to talk about
is things.json like
um do you ever touch this file like I I
know it's probably more of a Claude
thing but like do you ever because in
the settings.json you can like uh I
think Claude also does it in skills in
other places. Hold on I'll get to skills
in a bit but like in in the
settings.json you have your like
permission sets you can say that like
>> and also arguments for the model and
like API endpoints for the model. Yeah,
there's a bunch of stuff that goes in
there. Like everything goes in there
basically. Like would you say it's um it
makes sense to ship a settings.json in a
project as to you know maybe disallow
users from I don't know RM or something
like that
RM minus RF. You know what I mean? So,
so what what specledger uh the Golang
CLI does is it does have the concept of
agent config because one of the people
that was working on the project was
either using open code or Claude Code
with quen local or a separate uh
endpoint. So when he I built a feature
that launch launches Claude Code and with
a prepared prompt and then goes straight
into some activity. So you would run SL
and then it would bootstrap the repo and
then would immediately launch Claude Code
and says um we're bootstrapping the repo
and it has the prompts and the skills
and it would just go off and he'd be
like that really breaks for me because I
have to set a custom endpoint. I have to
do all of these additional settings
configuration when I launch Claude Code
there's all this like arguments that I
pass in and so he built a feature on the
on the CLI that that gives you the
ability to configure different agents uh
profiles um like and and so your
settings.json JSON but but with two
sections like secrets versus non-secs
overridable what is collected what you
would commit into the repo because maybe
you want some of those settings be get
included and some of this like local
that you don't want to include so so
there is quite a extensive feature spec
on on how to set up the Claude Code
settings and so on but to be honest I
don't use it because I I'm using cloud
code as is I don't have any like custom
flags or something so there was only one
other person using completeness. Yeah, I
wanted to mention like get um what is it
called? Git hooks
>> attribut hooks.
>> What is have you heard of this new
project called PR? Hold on. What is pre?
Oh, pre-commit. Sorry.
>> Yeah, of course. What do you mean new?
>> Okay, whatever.
>> Um
>> I I've been using Precomit since 2019, I
think seven years for all of my
Terraform.
>> Autofixers are like
>> Yeah. Yeah. Yeah. But PRE is the new
Rust one. Whatever. uh
>> new wish. Okay,
>> you just given me an idea uh Vincent
like how about the vibe coding a tool
like this a tool where you can go like
um you know
uh AI show guard rails or something like
this and it would basically look through
your your uh your settings your
pre-commits
um your your your I don't know your
switches your flags and basically
basically you know you know, give a
score about about your your your
session, how safe it is, you know what I
mean? Cuz cuz a lot of people, including
myself, to be honest, in a personal
context, I go yolo in my little sandbox.
But like from an org point of view, you
you want people um well to to set up to
to use AI safely. But in the in the case
that they haven't for one reason or
another, maybe it would be good to sort
of gify it to say like, hey, you okay,
you had a great session with AI, but you
know what? um you know this could have
gone wrong that could have gone wrong
and you know your score is 58%
you know next time um AI could have done
this you could have improved and become
you know 88%
uh if you um had this guard rail in
place or something like that you know
what I mean
>> yeah I would definitely ask clout or
chat to do a explore if anything like
that exists um it Sounds interesting,
but it also sounds it also sounds like
something that I would go like that's
wrong. You don't trust me. I don't care
cuz yes. I mean I mean these tools are
popular and people use them and I think
they're good for um non-technical user
but like for me I find them offensive.
>> I guess for us we would find it
frustrating and dumb or something like
that. But I do
>> Yeah. It would it would say like you
only have a 50% secure setup and I would
be like f you. I know what I'm doing. Uh
but but but um while we were on the
topic actually earlier about
settings.json and when I was talking
about doing I mean I think the AI show
guard wheels is cool but what I wanted
to to highlight is something that I I
learned and I saw a funny meme. So I
think it was on Reddit and people and
people are like when you work with AI
and um you say like commit this then
cloud goes like here's the commit and
then it like writes the whole commit
message and then at the bottom it says
co-authored by um cloud opus 1M or
whatever like the model and the context
and so on. a lot of times because maybe
it's something that you don't want
people to see that it was generated with
AI. you may you know either I I I
usually I don't give it a permission to
do the git commit sometimes and I just
in those projects I don't give it the
permission and I would go like no I
would copy the command remove the
attribution and then run the command
myself and I would be like I already
committed it uh I had to fix the
pre-commit um go continue I would I
would usually because actually it's in
projects usually
>> get commit gas amend right
>> but the very funny thing is and it was a
funny meme of a picture of of a of a guy
uh I I forgot the picture. Okay, you got
to find my picture for for
>> I got to find it. I might I'm not sure
if I saved the comment because there was
a very interesting comment. Apparently,
you can in the settings of cloud disable
attribution. So, by default, yeah, by
default, cloud will add its like
co-authored by cloud in the commit
message and also if you create a pull
request, it will also say generated with
cloud in the in the pull request. Uh you
can disable both or one or the other.
So, so you have all of that uh
capability. So you can actually and then
right after I read read that comment on
on Reddit I went to cloud and I said can
you disable attribution and it was like
both PR or commit and I was like both
and it was like okay done. I was like,
"Holy shit." You know, so many times
I've just like told it not to do that
and then went in and and and and did it
myself. And the funny thing is when I
did that and it then shows the whole
pre-commit like pass pass and then or
maybe with an autofixer and then I go on
stage and then do the commit again and
then I have a reason, right? And I say,
"Hey, I had to do it myself." And then
cloud next time it's like here's the git
command for you to run and I was like,
"Okay, thank you." So it kind of
remembers like this guy doesn't want me
to commit.
>> But but that's that's kind of a bad
practice though. I mean we you want to
show that you used AI, right?
>> We depend on the project.
>> For this for this AI show guardrails
that
>> I'm going to give you an example that
somebody told me. Okay. Not me.
>> It's a friend of a friend. Absolutely
not me.
>> Totally not you.
>> Remember the whole thing I said about
like AI routers, API uh AI router
governance contracts in place. You
cannot use like you even said it. You
cannot use certain models. If I do a
commit with Opus 4.71M, but it's not
available on whatever the the plan that
that that I'm supposed to use, that's a
problem, right? I cannot have a commit
that says Opus 4.71 did this commit.
>> Yeah.
>> I mean, not me, right? Somebody told me
that.
>> Yeah.
>> Yeah. I mean, this I'm actually
>> using the right.
>> I want I want to write this program
because I
>> I know it's going to catch people like
me.
>> Catch people like me. catch people like
you Vincent for example but this this is
the problem I'm seeing all every day at
work people are basically
um maybe maybe token usage there's a lot
of things that could go in here do you
think this okay now do you think this
makes sense to be a spec driven develop
spec driven develop project yeah I mean
I think this type of project is
something I mean you could probably okay
you could probably vibe code this with
the going into plan mode and it will
make a lot of assumptions
Um, but if you want to have the real
control of exactly what are the metrics
that you want to capture, then doing it
with spec driven development, it might
actually help you more like spike on the
different type of guard rails that you
want to build.
>> Okay, I'm I I just realized the time I
need to take my kids to school in 10
minutes, but okay, I I'm I'm going to
record myself. I'm I'm gonna mark this
the the places where I might be a bit
confused. I'm going to record myself
developing this in spec with spec uh the
spectrum development on the specket and
then and then maybe we can use that as a
topic for the next pod, right?
>> Yeah. Um it might be like if you're used
to plan mode, it will be frustrating but
it might also be rewarding because you
might see all the assumptions that the
AI gives you uh and then decide that yes
this is a great assumption and actually
no I want to go more in this direction.
Um but you will it will definitely be
frustrating in the token usage. Um you
will definitely be like I have used 80
like 30% of my session budget and I have
not even written a line of code yet and
I've only got these stupid spec
documents. That will definitely be the
first impression.
>> Yeah, I I I like this. I mean this this
seems like a
I can't remember the name. Can can you
can you dig up that talk from from
Singapore Dev Days or whatever?
>> Yeah, DevOps Day Singapore. um which I
think this year is going to be by the
end of the year. Um last year it was
around May. So pretty much
>> yeah the whole networking uh log VPC oh
whatever capturing the the the the
networking of um cuz what the lethal
trifecta for for AI agents is private
data access to the internet and
untrusted content
uh untrusted content if I remember
correctly which is and untrusted content
and internet access is almost the same
thing. So I'm I'm thinking the network
angle is something I want to investigate
further. But in the meantime, I will
specdriven development some sort of AI
show guard rails. This is going to be my
my my next vibe project. How do I I
don't know how to use a cruel code
anymore. What are you what are you
doing? Thanks. Was there some other
basic DevOps practices to add here that
come to mind? I'm I'm drawing blanks
here.
or have having a a pipeline test gates
can't hear you. You you muted yourself,
Vincent.
>> Ah, so earlier with my amazing jokes,
you didn't hear any of them.
>> Yeah, they were that good. Okay, thanks.
I think that was a I think that was a
productive session. I'm I'm I'm going to
do some Spectre and development on a on
a guardrails
uh linting tool. Okay,
>> I want to ramble off one more fun fact.
Okay,
>> there's a branch on the bun repository
of a rewrite into Rust. Reddit and had
like five or six threats going crazy
about, oh my god, Antropic, they have so
much money to have AI rewrite bun from
Zigg into Rust. Haven't they ever heard
of Dino? What's wrong with them? Um, and
X I guess the same type of backlash. And
the author of the branch came uh in one
of those threads and said, "You guys are
blowing this way out of proportion. I
just want to know what a rust version of
bun looks like. What's the memory
profile? How does it act and so on. Uh
but it's kind of interesting to that we
are at a day and age that you can just
go and let me rewrite this massive
framework from zig into rust just to see
what's the you know.
>> Yeah.
>> Yeah. But that those sort of experiments
should be private. Why the hell is he
publishing it for Christ
>> sake? because um bun is an open source
project and it's a branch on the
opensource uh repository rewriting and
he's doing and and I don't know I think
I shared with you because he has this
context document in the branch that says
like this is how you rewrite um from
zing
>> super interesting because um that's one
of my original AI that was my DevOps day
talk last year by the way uh using AI to
rewrite uh something from AWS CDK cloud
form into CDK for Terraform. So to see
that how the rules that he he defines
are very very very cool. Um
>> Right. Right. Right. Oh on the topic to
have you just to rattle off a couple of
things the
for setting up um
like a like a fresh AWS
um
landing zone like a whole organization,
right? Would you say the Terraform
tooling or the CDK uh tooling is about
the same? It makes no difference like
using AWS control tower, setting up your
logging account and your cloud trail
blah blah blah. Does like which IA
solution would you go for? I think e
either or would you say Terraform's
better? Like especially if you had to
like you know do do general things like
uh you know link up with uh with octa
and things like that. Does it make
>> I think my my experience is dated when I
use AWS control tower um I think four
years ago it missed a lot of the
features that I wanted. I have a mixed
where I do like AWS cloud AWS
organization and cloud formation stack
set which allow you to target a child or
member account based on the uh unit like
organization unit ou. So you can apply a
stack set basically you can define the
cloud for stack at the organization
level and then on these organization
units automatically deploy them.
>> Sure you're using cloud for I guess
>> no bootstrapping. So I said I have a mix
setup like I use AWS or I don't use
control tower um because of the landing
zone uh product was pretty bad
>> as another project I can't remember.
Sorry.
>> What is AWS or?
>> So AWS or is where you are able to
manage multiple AWS account
>> under Yeah. Yeah. Yeah. Sorry. I thought
you were referring to some sort of
framework or something. Sorry.
>> No, no, no. But but that all of those
services have evolved so much um that I
think
>> it's the kind of thing you don't do very
often. um usually and also that's
another very like an argument that that
is very hard for me to argue against
which is like yeah but we only have five
accounts and we only on boarding them
one one every year or one every quarter
so why do we need to have this all
automated so my my personal preference
right now that I have that I use is I
use stack set to bootstrap the terraform
because I I don't use tacos uh hosted
service I use Atlantis and Terraform
with with my own setup you're using when
you say stack set you're using like the
AWS provided stuff I guess right
>> so stack sets are an AWS orc cloud for
feature you have to give the I think you
have to give a service linked account uh
linked role permission uh you have to
enable cloud for service to run AC
across your AWS org so that it can go in
into different accounts and do things so
I use that to bootstrap my accounts
automatically they get bootstrapped with
uh an IM role that has a trust
relationship with the AWS account where
my my Terraform automation lives. So I
have one which usually is called like
managed services or shared services
account or they call it in in AWS CDK
you have this as well. AWS CDK pipelines
uses the DevOps account as as a as an
idea. That's where your pipeline is
defined and it runs across all of your
other accounts. So I have the same
setup. I have one account where where
the um the pipeline lives and all of the
other accounts when they're on boarded
they trust they create a trust
relationship with that account.
Have you had any things that like like
easily bootstrap Atlantis and things
like that or is that just
>> Yeah, I do. And I I so part of my terra
constructs uh project was to basically
come to the point that I publish a a
terraform mod terf or a CDK terrain
construct that just you just run it and
it just bootstraps your whole um account
and terraform automation.
>> That would be cool. I mean like people
ask me about this and I'm like yeah use
control tower. How do you do Atlantis?
Uh do it manually?
>> I feel Atlantis is lagging behind in a
lot of features and it's designed in a
very rigid way. Like it's very hard to
to make changes like even just making a
drift detection with the API endpoint is
kind of really is broken. So I use
Atlantis but I don't think it's a very
good solution. I think there might be
other like I don't know maybe state
graph if there but there should be more
like a better automation solution for
that. But
>> yeah, agreed. But as you said, like you
you generally only do this like once.
How often do you need to do it? Okay,
cool. Um, I thought that was a
productive session. I will I will pledge
to do some some sedd for you to review
in a bit. Have a great day. See you.
>> You would have a lot better experience
if you use spec ledger, but maybe you
should use specket first to know.
>> Yeah, I'll use packet first and then
then we can exchange notes. Okay. See?