Episode 24: AI's broken mirror

Yeah, I'm just making a conscious
decision with this podcast to try speak
a bit slower because I've noticed when I
was trying to make a short of us
talking, we we talk over each other
almost every every moment. So, um I
think I think it's cuz we're I think
it's cuz we're vibing in all honesty,
but like when you're editing it, it's
like, "Oh no, he got cut off.
he didn't make his point clearly and
such and so forth. So, I'm just like, I
got to speak clearer and not cut Vincent
off so I can get that sweet
sound bite.
Yeah,
>> we complet
Oh, wow. It took a while.
Okay,
>> maybe maybe it's not worth the effort to
be honest. So, we just keep on doing
what we're doing. I mean, like it's
we're having fun really, I suppose.
Okay, let me let me share the screen so
I can demonstrate what I mean. Okay,
first little question for you is
I know you you're you're a proponent of
uh specit because you're behind it, but
like is is there a quicker way to share
your your prompt with your team other
than using something like claude code
transcripts? Is there some other like
lightweight way of doing it? Do you just
throw uh some history into uh a gist or
something? Cuz I'm because I'm I'm I'm
now between Claude and Codeex
and this tool doesn't seem to work for
Codeex. So great.
Oh. Um yeah, I was just about to sense
or share something about
And uh it's like sddddd align. No, no,
no, no. There was this other project
agent diff session capture. Oh, that's
an interesting one. But I don't know if
it supports others than cloud.
>> Okay.
>> It's a CLI tool that hooks into cloud.
>> Then there's co-pilot. Yeah. I I just
feel I I don't know if it's even worth
doing this on in sometimes cuz like
because sometimes like when people share
things to me the session is just too
>> Yeah. So I think we need to identify and
here we were talking over each other and
then I interrupted you but I think what
we need to do is we need to identify why
do we want to share sessions? what is it
that we want to get out of it, right?
And we could share
>> the full session log with all the
mishaps or we could find some type of
flow that allows us to capture
important key decisions and information
that we want to share to
>> accompany the code changes that we are
proposing. Right.
>> Yeah. Yeah. like these important
decisions need to be captured and marked
up as much and and I don't know of any
tool that does it like yeah like these
sessions are hard to follow and if you
could if you could sort of like
highlight like I don't know the the bit
where something happened which which
plays into a certain
uh
AI usage pattern that you want to get
across to your team or something then
that would be great but I just maybe I
need to build the damn tool that I want.
>> Sounds like it because what you're
saying is a very different use case from
what I think about um session sharing to
me. It sounds like you are and I think
this is something that I also do. you
are trying to show to your team members
if you're faced with this type of issue
or hey did you know that you can prompt
the model to think deeper in this way or
to identify other edge cases in that way
is that what you're trying to do right
you're trying to share
>> yeah I'm trying to share like it's not
just about me it's like I would like to
see what other people think
>> okay this this leads me nicely onto what
I really want to get at actually I came
across a brilliant um
like web page where they essentially try
to plot all the things that we've talked
about
from time to time
about about uh you know obstacles as
they call it challenges to using AI um
antiatterns things that you shouldn't
maybe do patterns which which which
might help
and this is a this is the best
collection I've seen
and for this the purpose of this podcast
today. I thought it would be cool if we
just quickly go through them if if we
agree on these things or not or maybe
what they missed or something like that.
>> I know there's a lot there.
>> I'm curious to hear from you which one
of like you're showing a diagram and I
don't know where to start and did you
>> read it and is there anything that
stands out that you say this really?
Well, there's there's a few I I dipped
in here and there and everywhere cuz as
you as I mentioned the last podcast, I'm
I am essentially taking
uh
some platform documentation and we're
working on how to make it AI friendly
and uh so I'm keen to to meet all the
patterns uh around
making the knowledge accessible to to
AI. So like like when I dipped into this
knowledge composition here, it was like
saying,
you know,
don't when you keep everything in a big
file, it might not work out or something
like that. So it it's recommending uh oh
my gosh, what's what is that? It's
called uh what is that skill? Uh it's in
the claw documentation like gradual.
>> Yeah. Yeah. Um what's the name?
progressive discovery or something.
>> Progressive disclosure. Yeah,
>> disclosure.
>> So essentially they're saying
progressive disclosure here. So instead
of having like one huge best practices
which is actually the case with my
current client like we just have huge on
boarding guides and uh of course they
recommended you know breaking it down
with architecture tech stack deployment
API docs
>> and this is simple stuff. This is
something that I've noticed as well in
the skills that so basically as you
alluded earlier um spec ledger is is a
is a framework around specri development
for team collaboration for sharing
information and we also have implemented
the idea of capturing the messages from
the user that and the model that led or
the agentic um shell that led to a
commit. So we're using um a hook
whenever a get commit happens to
basically capture the delta of messages.
Then we tag those messages by the commit
ID. So every commit ID should have and
then you should be able to identify
messages related to a specification and
a specific commit.
>> So you're trying you're trying to roll
up the the prompt that went into the
commit basically.
>> Yeah, but it's too detailed, right? So
then we have a separate GitHub issue
that was um referring to an MCP to
basically convert session data into
knowledge into a knowledge base. And
when I looked at it, it was like it was
a bit silly because we're not storing
sessions as blobs uh you know inside a
datab inside a an object storage. We're
actually already analyzing messages by
commit. So it did highlight a couple of
features that we could add onto spec
ledger which is the ability to prune and
and merge and actually derive
knowledgebased articles um global
knowledgebased articles to out of
session data
>> and and that's one of the um you know
GitHub issues and and and and area like
areas of work that we want to explore.
So we haven't implemented that yet. But
then when it comes down to what you just
highlighted, the problem where we we
have specled ledger CLI and we have a
bunch of commands. So the spec kit, it's
built out of bash scripts and Python CLI
and some other stuff and then the agent
is instructed to run a a bash script and
then the bash script needs to be a
PowerShell for Windows. So it's very
messy, right? So with Spec Ledger, we
have one CLI. We've got written of
beats. Everything is is rolled into one
CLI and then we have a bunch of skills
for that CLI. like this is how you
create tasks, this is how you um you
know review comments, this is how you
put comments.
>> Yeah.
>> Yeah. So we wrote a bunch of skills and
when you init your repo we we inject we
we write on down those skills embedded
into the Golang binary down to your disk
so that they can be progressively
discovered or disclosed by by the agent
fell to the model.
>> And I actually noticed that those scales
are too big like exactly what you
highlighted. So today I just created an
issue which is like um I used the
entropic skill creator and they have now
the whole evaluation framework in there
as well. Um and I I basically pulled it
into the repository and I did a full
analysis on all all of our skills and
basically what do we need to do to make
these skills more aligned with the way
that the agent shall or Claude Code will
progressively um disclose this
information to the agent.
>> How did you notice that? This is where I
feel we get into, you know, touchyfey
spider sense territory. How did you know
that your skills were too big? Just
because they they didn't seem to be
working or actually so we did a huge
streamlining effort of of like getting
rid of the bash scripts and and and
converting into subcomands into the
Golang binary. So you have only one
binary need
>> and in that effort we also realigned all
the scales. We defined a four layer
framework um of each layer's
responsibility of like this this belongs
to um shell hooks. This responsibility
belongs to skills. This responsibility
belongs to the CLI output also took an
efficient output. All this is is defined
in like one uh feature spec and we
actually created a bunch of rules around
that and then we realigned it. So, so
only today I finally had time to
actually download the binary with all
the latest streamlining and it actually
works really well. Like um I was working
I picked up a task and I was like let's
implement it with the latest CLI and it
was working well. So I didn't have an
issue with the skills at the moment but
I noticed that maybe there there can be
an improvement. So it wasn't really
driven by like data. It was just that um
>> No, I think I think I think Vibes and
Spider Senses does is something uh just
that it it's a bit weird.
>> It actually worked really well. But um
but the thing is I I also wrote a Jira
skill, right? A Jira CLI and
>> I noticed that it was really messy and
it didn't really work well and I did the
same thing there. So, so it's it's more
like I I I
>> was your Jurro CLI like actual code or
is it like a skill?
>> No, Mark. Well, well, so basically I use
I I forked the the Golang CLI. That's
the most popular one apparently. Uh it
has a lot of open PR. So I forked it and
merged some of those open PRs and to
make it act more like the GitHub CLI
because the GitHub CLI works really well
with uh with with Claude Code
>> and um for example, you have an escape
hatch. can you can put API and then it
can do REST API calls.
>> And um I'm not sure if I mentioned this
to you the last time, but what I noticed
was
I had I was using the the Jira CLI and
according to the Atlassian docs when you
want to move in my case I had to move a
card from the backlog to a board. So if
you go to backlog you can right click
and you say move to board, right? Uh
maybe this is going into an anecdote,
right? But um but basically I I was like
showing um claude I said like hey I need
to move this ticket to the to the board.
It's an when I create ticket it goes
into the backlog and when then I want to
assign it to the board and and and then
cloud was like yeah according to the
Atlassian docs it should be this
endpoint uh and it it does this endpoint
and then it will move the card to the
board and it wrote this down in the
scale and then I was like okay great uh
but I already moved it so let's do it
later and then it created another ticket
and uh and then I was like okay move it
to the port and it failed it got a two
or four response empty response
>> I was going to I was going to say as
much I was going to say that I feel I
feel like it's a trap when you think
that the skill is going to be your CLI.
It's uh it's never going to end well
>> because deterministic, right?
>> Yeah, absolutely. But the thing was um
the thing was when I asked
cloud to move to ticket, it didn't work.
And and and really cool what happened
was that I asked Cloud, can you see my
Chrome? And it it launched Chrome MCP
like DevTools controlled and it told me
that it enabled the hook. Um it it sorry
it enabled a spy in the in the Chrome
session uh managed by cloud and it asked
me to move the ticket
>> and I said okay I click on the button
and and then I said I've done it and it
says now let me look at the lock from
the hook and it says I can see that it's
using uh green light API and not I'm
just making up the name or something
greenhouse or green light API
>> and it's not using the rest API when
it's moved to move the ticket
>> and and and so it it updated the skill
saying um don't use the documented rest
API the actual conference Jira API like
the web application is using this API
and and and now it works like what the
hell
>> well I mean that that sounds like a
rabbit hole to me the the on the topic
of Alassian
>> what do you mean rabbit hole
>> I mean like because you're you're in
like skill land as opposed to something
deterministic with the CLI No, no, no.
Absolutely not. You missed the point.
>> The point was
>> that the Atlassian REST API told you to
to do a certain API post request.
>> Yeah.
>> And that it would move the cart, but
that it didn't work.
>> That the web app works
>> and that cloud was able to hook into the
Chrome session,
>> enable the spy when I click the button,
identify the actual endpoint, and then I
don't care that it goes into the skill,
right? The point is
>> it's insane. Like at that at that point
I I could go like okay my Golang U CLI
deterministically should always call
that agent
>> but to complete this flywheel to get
back to determinism.
you you we we you should have spun up an
agent, reported the bug, splabbed on the
issue to say that uh the the this uh
this post didn't work as intended to
move the card from this column to this
column and this code
>> the the the people the upstream of this
Jurro CLI that you're using using
>> it's
so there's no Oh, I thought you were
using some go code around
>> I am using a go code but the go code
doesn't have the command to move the
board the cart. So I use the pull
request
>> contribute back for instant.
>> Yes, there is already a pull request
that wasn't merged almost a year ago
which exposes a backd dooror to to
actually do an API. So you have the Jira
CLI you can do Jira uh new ticket and it
will create a new ticket and all that.
Those are all there but there's a couple
of things missing and in that case you
know the GitHub CLI you can do GitHub
space API and then you path put the API
path and then you can actually use your
GitHub CLI authentication to call the
API endpoint right it's like know about
that okay all right
>> yeah so it's like a it's like a escape
hatch so if if the CLI doesn't have the
content so that already existed there is
already an um a pull request to
contribute the API subcomand So that you
can directly hit end points.
>> I see. Okay. Oh, this is pretty nuanced
then. So you using that to keep the
creds and
>> yeah so my skill says until we have a
subcomand to move tickets you can use
the escape hatch which is through the
API and originally according to
Atlastian docs you should use this
endpoint but that didn't work so we
removed that. And in the scale it says
use this green light or whatever it is.
>> This yeah this escape
>> hatch. It's quite an interesting
concept
>> feel design. Yes. You don't need
>> I still think it's
>> Yeah. Like after working with Atlassian
products. Yeah. This this thing that I'm
sharing with you right now I'm I'm I'm
like
uh
so there's two ways
>> this is this for confluence?
>> Yeah, there's two ways to obviously work
with confluence. You can you can work
with the curl API. you can work with an
okay there's three way these there's a
bunch of ways there's the MCP way which
is basically doing curl commands and
there's um and then and then there's
this way using a tool called mark which
has got the highest rated I should star
it the highest rated uh tool to to push
to to confluence
and
it the the thing that I wanted to say is
that with Atlassian products They're
actually amazingly complicated to work
with. The whole active markdown to HTML,
all these crazy um macro templates they
have for like everything from like
little notes to I don't know table
layouts to database to to to Jira ticket
layouts. There's a lot going on
actually. So, so
this tool is actually really good and it
does have its problems and there's some
open pull requests, but like if you're
working with confluence from AI, you
should be using a tool like this.
Otherwise, I feel you'll be catching
some edge cases sooner or later and
you'll be scratching your head and this
is the deterministic tool to get you
somewhat there.
Okay, that said, let's just sp let's
just let's just spend like a lightning
one minute on each thing just to say
whether it's whether it's whether we
think it's correct or
or or it sucks. How about that, Vincent?
>> Yeah.
>> Well, I don't even know where to start.
Okay, let I'll start at the top left. Do
you think uh yeah I think context raw is
still a thing even though I noticed that
codeex is definitely better at it.
>> Uh I think context rock is still a
massive problem. Yeah. And it should be
>> I don't know about codecs but um even
with Opus 1 million context window um
it's very difficult to I noticed that
LLM agents do not keep track of um
details very well. I don't know if this
is a regression or we're just used to
them doing a lot more than before and
now we just expect the world.
>> So most people are feeling it's
regression but I don't think it's
regression. I think it's just we getting
used to
>> yeah the the expectations are always
growing. So context management um treat
context as a scarce degrading okay I
think this is
like do you have an approach to context
management like to be honest I you know
I do click compact conversation
regrettably from time to time when I
have a bit of downtime.
So for me, since I'm now using the my
plan with Opus 1 million token access, I
honestly
I have the status bar tell me how much
percent is used. I also like that after
the plan, it always tells me how much
percent was used by the plan and it
offers me to clear and and then do the
implementation. But I noticed with Opus
1M, the usage is usually like 3% or 17%
very low. So I'm like I'm not even
clearing it anymore. I'm like, "Oh,
looks good." But maybe that's also why
sometimes details get lost. Um like even
though it's a 1 million token context
window opus LLM model um and it tells me
that it's only using 10 or 15% of token
window details are not being tracked. So
sometimes I do go to like the basic opus
and then I do aggressively clear. I do
not uh I do not compact.
>> It's quite a lot of work, isn't it, to
to reset it in some ways. So,
>> we get
>> I mean I have my checkpoint uh so in
spec we added the idea of checkpoints
and and what they do is they basically
that's a bit related to the question
earlier about like how do we share
sessions and what do we want to get out
of sharing sessions. Um we actually
started to move more towards either
building knowledge base out of sessions
or with a checkpoint command we asked
the model specifically
give us a detailed like progress update
of the work that we've completed in this
>> related back to the actual thing.
>> Yeah. Summary.
>> Yeah. related related specifically to
like the work items that we're supposed
to work on because there's so many other
things about happening in between when I
when I encounter a bug and I say go and
create a bug I don't want really that to
be to be collected or whatever
>> that summary is pretty much what I like
to do and what what actually confuses me
is that the compact conversation like I
don't even know what it's doing which
makes me uncomfortable like is it
summarizing where is the summary I don't
know maybe that's me just not knowing
showing how compact works.
Okay, let's just move on. Knowledge
document keep AI. I think we talked
about this. It's just saying the same
thing again and again uh about keeping
things
uh ground rules. This is like just say
what your
I I think we talked about agents.
I mean, it's been a topic on on on
a couple of channels about how
agents.mmd or claw.md might be polluting
everything. I I still have a claw.md
where where I have some basic setup, but
I don't I I don't do like um prefer
simple solutions or
I I don't know if you do this sort of
thing like
For me I I really don't use the clouded
MD um because spec kit has this
constitution and it gets invoked as
opposed
certain after certain steps uh or phases
the constitution check gets invoked and
then we do a full analysis
>> but constitution would be the ground
rules I suppose. Yeah, I mean I
understand that that cloud MD is more
like an a little additional
customization on top of the system
prompt to make sure that your agent
works in a consistent manner. And
I'm I'm not sure that we use it
effectively because what we do is when
we use specit we it's a bit like mini
waterfall, right? we we ahead of time
plan out the work across the different
areas and then also the definition of
done and acceptance criteria and and
basically what we need for validation.
So that gets reinforced
>> in the agent loop, right? You pick up
the next piece of work. You're going to
work on this. These are the files that
you should be looking at. Maybe the
model goes in a little bit of an explore
to like confirm its understanding and to
make sure that the file system still
matches with the task definition. And
then it um when it's thinks it's done,
it needs to go through an type of an
iteration of like you know is this
really you know validated like did I do
it correctly? Are my acceptance criteria
met? Are my test passing?
I understand you
has a lot of has a lot of stuff like
that, but like like I mean when you're
vibing in a in a clawed session,
>> you you you don't have like any project
rules that that apply across all your
different chat sessions.
We just introduced a new rule amongst
ourselves which is if we like what what
happens what I frequently do is I see a
problem and I just tell cloud hey go and
create a github issue and it very
beautifully captures the problem maybe
there's a bit of an exploration to find
out what could be the reason for that
error and then you know it usually does
a sub agent and then creates a github
issue
>> you have a command or you just tell it
you just chat to it like
>> no I I barely do I don't say anything
anymore recently uh I think it's it's
gotten that good that I just say You're
not using beads anymore. You just tell
it to make a kid issue.
>> Well, I guess that's one way. I guess it
>> it uses beads.
>> No, we use beads for our spec driven
development, but if we encounter a bug,
I ask it to go and create a GitHub
issue.
>> Oh, because then the team gets wider
knowledge or wider alignment to to the
issue.
>> It's more like beats has become feature
specific task three. It's like a to-do
list, but it belongs to a specific
feature.
>> So, you know, originally beast is like
at the root of the repo and it's shared,
but then you get a lot of merge
conflicts when you have different uh
feature branches.
>> So, we had a lot of issues with that.
So, we so when we rolled beads into our
our CLI,
>> we made it that um create feature
specific task lists
>> and and they're basically little task
trees that belong that's part of our
mini waterfall, right? We we have a
feature that we need to work on. We
create um a plan, research uh the
technologies, do some spikes on certain
approaches and then generate a task tree
uh dependencies phases and MVP path.
Basically what you get with K as well
>> and and the GitHub issues is for team
alignment, right? It's it's
>> and the GitHub issue is like [ __ ]
there's something else came up and I
can't really work on that right now.
That makes a lot of sense.
>> It's more like a global like a bug
report or something like that.
>> And that leads to this thing called
extract knowledge. I don't know what
it's going to say. Not yet because you I
wanted to link this to the GitHub issue
where uh I wanted to link this to what
you just say about VIP coding. So the
new rule that we introduced is that we
don't do spectctors in development for
these type of bugs unless all right it's
a funny rule because I I introduced it
and then two minutes later I changed it.
So the rule is when you have an issue
like that I noticed some of the our our
team members they started this huge
specdriven development on like one line
fix and it generates a whole spec user
stories edge cases and and and I was
like ultimately the the code change was
literally like one line um I'll give you
a concrete example it was the go release
populating the version field there was a
bug it was populating the wrong constant
and so when the the CLI ran it didn't
show the version correctly.
>> So it was a bug report like hey when we
run we run SL version it doesn't show us
the right version and it created like a
massive user data a user story around it
and I was like guys for this type of
stuff can we just vibe it uh and then
the room change was I had another issue
and I was like I'm going to vibe it. I I
just picked up the GitHub issue and I
literally said nothing to cloud. I I
boots I I just started cloud. I said I
want to work on #72
and cloud went let me run GitHub issue
view 72. There it is. So you use the
CLI. I didn't tell it to do that. It
just automatically went I'm in a
repository. It's on GitHub. I'm going to
use the GitHub CLI to view the issue. I
didn't say any of that. It just pulled
out the the issue description from that
particular session that created it. And
then uh I I switched it to plan mode
obviously because it was a vibe and I I
wanted to um you know I wanted it to
explore and and identify and then tell
me what it's what's going to do before
we actually start doing things. So it's
not the full like user alignment with
spec driven development. It's very very
short little plan to do this and and
then it pulled in a whole bunch of
additional stuff. It's like okay um in
this case um there could be use scenario
where this and that happened. What do we
do? And I was like oh my god this is
getting too big. this is not a oneline
change. Let's do a spec driven
development. So then I told then I told
cloud okay that's a nice plan but I
think we need team alignment on that. So
then I told it to to feed that
information into spec ledger specify and
then it created user stories and edge
cases
>> because it blew up my my my uh bug was
um get attributes whenever you do a pull
request some of the files are purely
generated and you want to linguistically
ignore those so that the PR doesn't like
expand them. So I wanted those
autogenerated files to be autoc
collapsed. Uh, and then the reason it
blew up was that some pe some
repositories when you initialize it
already have git attributes. So how do
you merge things together? So that's
where it was like created a whole bunch
of additional user stories.
>> Um,
>> so I'm I'm summarizing this in my head
that like you you have different types
of tasks that you do at work.
your your bug fixes, your business as
usual type stuff where you don't need so
much team alignment and then there's the
things that kind of blow up or whatever
the features. Yeah,
>> where you need team alignment.
>> I think that's the criteria.
>> Yeah,
>> I think the the key word is team
alignment. the moment it's more than a
oneline change or it requires like a
user stories and um you I mean the
moment that it's more than a oneline
change and you want other people's um
>> views you should push it as a spec
I'm just equating this to the the sort
of sensible defaults at work like we
have these processes called um uh
kickoffs for for u for team alignment
before you start the work and and desk
checks for team alignment as you
complete the work. And um I feel like
these these uh engineering practices
that we've been doing for years can be
sort of mapped to to working with
agents. So
>> something I need to write about. I think
what we've learned is that that all of
these practices that we used to do for
years where we have a lot of overhead in
communication uh between people finding
time to discuss or getting a review
>> have been collapsed significantly with
agents accelerated. A lot of them become
um literally less of an issue because
you can get the turnaround much faster.
>> Um but you still need like those good
practices of
>> finding team alignment. I'm just
thinking about it and I might as well
mention it is that
traditionally um
the best desk checks, the best kickoffs,
they've always been in person and it's
always been uh like a synchronous thing
with your team.
Um, and I noticed that since I'm
becoming more remote that this this
relationship is kind of degrading
because, you know, there's people on the
call that don't even say anything
sometimes, which is never a good sign.
Um, but now we have agents which can't
talk, you know, they can't sit in around
a table or a whiteboard. So now we have
to adapt our workflow so that we can
bring in the voice of the agent or
whatever you want to call it the input
from the agent as well as the input from
team members. And this is where I think
things fall down because because like
yeah the
it's easy to have a whiteboarding
session and talk about things but then
how do you make sure the team is aware
of that sort of that plan from the
agent? You know what I mean? Like people
don't necessarily read all that stuff
before the for the before the meeting
sort of thing.
>> Yeah.
>> So this is where things get tricky. as
human team
team alignment. Yeah, it's damn I
haven't seen it working perfectly.
That's for sure.
And I've also noticed that um so one of
the major things that I really want to
focus on in the next few days or over
the weekends with specledger is to find
the deterministic validations because I
feel without it even if you do spec
development even if you do user stories
and you have um team alignment on the
user stories there is still a big
quality problem with the output
um if you do not have more deterministic
validation around
um
>> the the the output basically like I
think this relates back to original
discussion where and Adam's Jacob was
saying like you have two two teams one
team is still very much hands-on with
the code and another team is very much
um you know
>> focused on on just accepting the code
but that second team must have very
strong testing practices
going back to what we was discussing in
the previous part like there needs to be
like a this role almost or I mean we're
talking about roles just generally but
like yeah I think I I would definitely
agree that like this these engineering
practices of having a separate QA team
is actually probably more important than
ever
you need that sort of like uh that
quality control yeah okay let's
>> I think you need at least to have the
expertise to make sure that the quality
control is part of the feedback loop
Yeah,
>> because I don't think you need a
separate team. I think you need people
with the expertise.
>> Yeah, the people with expertise. That
That's a good point because I'm just
harking back to the usual QA experiences
that I have. They usually they're not
very technical people.
>> Yes. You know, doing very mundane
things. And things have changed now
because now because now with AI you can
the the name of the game is to make to
make processes deterministic
using AI. So
so things are different now because
because that that quality control team
should be able to to do the that oh you
you sent me a um screen grab. I don't
know where it was from, but essentially
I thought it was saying what I'm trying
to say very well. The the division is
simple. Deterministic computation and
prob probabilistic reasoning. They're
complimentary. The pipe between them is
where the work happens.
Getting getting um yeah,
what was the point I'm trying to make
here? that the quality control stuff
needs to somehow prove that the stuff is
correct in a in a deterministic way or
something like that. It's the same thing
like I shared a post on from Reddit and
it was removed maybe because it was AI
generated but it the the core point of
the post was that he's part of a team
and they have front end issues 40 layers
of react hook linked together I don't
even know like so basically they lost
control over the quality of the product
and um he's saying that on the back end
the it's kind of okay the the
performance of the AI generated code is
all right But um they're like he's
really concerned about like long-term
maintenance. And then the top comment
uploaded was um you know it's not
against the law to write to use AI first
to write tests. And it was funny because
it was just something we discussed right
which is like
>> I we think that the first thing you
should do if if you have an existing
code base is to booster your test
harnesses before you unleash it on your
code.
>> Yeah. and and test harnesses.
>> The thing that I find about test
harnesses is that people
>> I think the more experienced you are,
the more you realize that you could test
here, you can test this way, you can do
a regression test here, you can do a
unit test here, you can you can do the
Swiss cheese model and those people who
are experienced can do this can can
cover have some intuition to do that
very well.
But other times like if if you if you
pass the quality if you pass the QA to
someone who who doesn't have the
experience, it's not going to be a great
experience. It's not going to be a great
results. That's the biggest issue with
with um AI coding is that you really
need a strong engineering like an
experienced engineer to see patterns
that the AI is going down and um and
highlight them. For example, I was
asking um to do a bug report. Uh when I
ran a command, it didn't update the the
data the way I expected. It actually
gave me an error. And I was like, I
create a back bug report. It shouldn't
give an error because the the the file
system is okay. And it identified that
that the command is running using
current working directory. and it didn't
identify the git route to find the
relative path to the actual location of
the file where it's expected whatever
that is right so so it's it's it's it's
identified the problem it's using
current work directory it also noticed
that there is a a feature to identify
the git root so to find you can ask git
uh what is the root of the repo
>> yeah yeah like I thought and so
relatively go back to something yeah
>> yeah it's something uh get reverse parse
tree and then I I don't like to get the
get root. It's a weird command,
>> but it the the AI went and said like,
okay, you know, there's actually a
command. It's in this um in this um
subcommand
this this git root utility. And I'm
like, stop right there because you're
just going to start using this
functionality, this utility from this
other like you don't have a dependency
on this other component and you're going
to use an internal utility. And I'm
like, don't do that. Like I if you let
AI do it will just go ahead. Oh, it
already exists. There's a function
there. And it creates a dependency
between two components that should have
a dependency. And in reality, it should
move that utility function into a
utility library. Create a shared
dependency on the utility library unless
you know in Golang it's very common to
to just duplicate the code uh to to
avoid creating dependencies between
components or creating shared
dependencies. But those type of
questions if you don't even ask those
questions to begin with right is already
the problem. You already have to have
experience like okay but
>> it's so it's so nuanced like I'm
>> reuse and shared dependencies are are
are painful too. So like where do I go?
Which way do I go? I'm just thinking at
work we have like code sense code scene
sona cube running on every goddamn PR
and the uh you know the the little hints
it gives you like oh the complexity is
moving from scale 7 to scale 8 or what
have you so bad and
or or maybe I'm just uh flippant about
it like a lot of the determinist istic
ways of me of of measuring code quality
just judging by my jud judging by my
anecdotal experiences with these code
scene whatever tools
I'm probably just throwing shade but I
just don't feel that they're very useful
and and then and if I don't feel them
that they're useful then I don't feel
that they're going to be useful for AI
either you know like these I don't know
if you've ever played with these tools
that
>> it's it's nuanced to do what we this
this quality really
>> when you start out as an engineer you
may try to go for 100% unit test
coverage
>> right
>> and I see your by your face that you
already say
>> yeah and and those tools and those tools
installed by enterprises are like pretty
much suggesting that oh your coverage is
low no no and
>> but now most organizations go like
coverage should be between 70 and 80%
not not less and and so on, right? I
mean, the thing is it goes takes
experience to realize
>> everything so nuanced
>> about what we do.
>> There's no
>> it takes experience.
>> There's no like cleancut way which makes
it it makes it really painful to talk
about anything. But that means it it
ties perfectly back to the original
statement which is um you need expert
experience to guide AI still like you
need an experienced QAT person that
understand the test harnesses to to to
be able to make good decisions about
which responsibility how it should be
tested.
>> You need you need
>> some taste. Yeah. I just I'm just almost
frustrated that that's the case almost.
But it's I guess it's keeps us in the
loop somehow. It's just it's just
annoying.
>> For how long, right?
>> Yeah. It's just a big cringe really.
Like these things happen and you just
go, "Oh no,
>> for how long?" Because we just talked
about agent personas and about, you
know, skills and then people are trying
to capture every single operational
expertise detail into skills. So for
example, exactly what I just said. I am
not an expert at writing um Claude Code
skills. So when I look at the skill, I
can't say if it's good or bad. I can
only anecdotally experience if the agent
is nondeterministically
um using the agent correctly. But
Antropic released agent skill
evaluators. They created a skill creator
that includes it instructions to cloud
on how to evaluate skills. So we're more
and more of these expertise
um are trying to be moved into these um
>> progressively
there's definitely some sort of maturity
level right like the the the you know
somewhere in the middle there needs some
experience
but then at some point that experience
that need for that experience probably
drops away really and then then the the
loop becomes you know
As with a mature product, the
things can be automated and it will be
like a flywheel. It will be a machine
and you'll work on a new machine or
something. So, it's so it's everything
is so bloody nuanced and the way things
fit in is Yeah, I think I I need to
start writing some things down otherwise
I'm not going to capture all this stuff.
>> You'll get a summary from this podcast.
I don't think it'll be good enough
um until the next model comes along and
then we can do it all over again.
>> Okay, let's let's just get through this
top row. Let's just let's just chug
along. Limited context. I think this has
been said before. I'm not going to go on
to that. A distracted agent
using one agent for everything.
Ah, I I do feel sometimes because I
create all these issues along the way, I
do distract the agent. But for me, the
ultimate like my agent only needs to
focus on one thing. If I'm in in the
specdriven development, I need to focus
on getting the plan and user stories
out. I made it commit and then do the
web view. It's like, you know, I don't
know if you had this, but a lot of
people, at least early on with screens,
were like, I need to see the printed out
version. I need to print it out, hold
the paper in my hand, and then mark it
down. put a little unders underline
here.
>> Well, so I didn't understand. Oh, do you
mean like
>> So, so for me, right,
>> the really old fashioned way is when you
print things out and in the printer like
it's
>> Yeah. So, how that relates back to what
I just said is that the agent is writing
a bunch of markdown. I can go into the
code editor and read the markdown. But
ultimately, what I really want is I want
to commit and push it. So, I can see it
in my web dashboard and I can like
highlight these things, put a little
comment next to it, highlight that, put
a little comment next to it. And then I
go back to my terminal and I say let's
clarify. So the clarify command in
specure pulls all the comments from the
web dashboard and then pulls in all of
the edge questions that are sprinkled
around the
>> Okay. Okay. So you streamline that whole
prompting process with the
>> Yeah. So even though I have a distracted
agent, ultimately I have one artifact to
review and make sure that that artifact
is focused and then I can clear the pro
clear the session and and have a new
agent look at it. But it's fact it's a
fact.
>> There was a new there was a new command
in claude called by the way. So like
when it's chugging away you can type by
the way.
>> Yeah. So I haven't used been able to use
cloud because of some work commitments.
Uh and I used it today and I I used
voice which is pretty cool. So you can
slash voice and then you can hold
spacear and
>> it's just working on the terminal.
>> Yep. And it it it even changes the size
of your cursor based on on on the on the
amplification of your audio.
>> Oh god,
>> it's insane.
>> So it's funny like
>> what the
>> So this distracted agent is an anti
pattern, but then but then Claude
releases this by the way command. So it
makes
>> it's funny when I'm not sure how much of
the by the way is inside the context,
but the way that when I used it is like
this. You do slash by the way. So it's
doing things and I'm like slash by the
way because I'm used to queue up
messages. So I'm like let's queue up
messages using by the way. I thought it
was work the same thing but it doesn't.
So when you do slash by the way it seems
to suspend the other agent answer your
question and then wait for you to
acknowledge that. And then the moment
you press any key I don't know how but I
started typing. I thought it was going
to be like it was a in the in the in the
message log, right? I thought it was in
my response reply. So, I started typing
something and the by the way disappeared
and when I went to scroll, where is the
by the way answer? It wasn't there. So,
to me, it really seems like this by the
way thing allows you to to like take
some of that context, suspend the agent,
answer a question, and it doesn't go
into your distracted agent thing.
>> Okay. Okay. That's how I haven't
actually played with it. So though the
distracted agent
>> is is a symptom of uh
>> and I think the by the way actually
helps you avoid the distracted agent
because it suspends the agent, answers
your question and then goes back to the
agent and then if you want to queue up a
message based on that answer, you can
then put that into the context.
>> Limited focus. I guess I feel like a lot
of these things are just describing the
same thing in different ways. So this is
just like a
>> it's lop it's AI generated. It's a
hallmark of AI generated where it's
describing the same thing in a different
way and it's the hardest thing. It all
it all like makes sense and they all
have like a slightly different angle but
they're like 60% the same thing. It's so
annoying with AI.
>> Yeah, that's that's true. It's just like
it's like a broken mirror. There's like
a million ways of seeing the same thing
and it's fragmented. It should just be
>> that's actually really nice way of of of
looking at it. Um when I ask
>> Yeah. Because this is my biggest
problem. I have a a couple of core ideas
that I want to get out and then
I don't know about lately but at least a
while back when then I I have like um
you know give me the key points and then
it creates them and then but isn't that
the same one as the previous point? Yes,
there's a slight different angle but
it's almost the same. And then I say
okay you know what I'll solve it when
I'm in the slides. So then I generate
all the slides from it and then every
slide kind of repeating the same thing
but slightly different. I'm like, "Oh my
god, how do I how do I get ex like
concise
>> to purely identify the exact
differences?"
>> It's really like a blocker mirror.
>> Yeah. This is where summaries this is
where I guess you need to put your foot
down and make a commandment make a con
make a call. Um
ah this is another thing about Yeah,
we've and we started we started off
here. Okay. We did the top row. I think
that's enough for one session. It was
kind of painful because it kind kind of
repeated itself. But still, I think this
is a good exercise. And what I like what
what they've done here is that when
they've in, you know, they vibe coded
something so that like um uh Oh, I
thought they were like they were doing
some pretty good linking to each other.
Um, and now I can't find the link. Like,
yeah, it's like you can see that the
blackbox AI obstacle is related to this.
It's caused by this,
solved by that. Sorry,
you're going to have to cut that. Um,
but basically what I wanted to say was,
um, I mean the sides looks nice, right?
But I do feel like it it has a lot of
repetition. Um,
>> yeah.
is detrimental detrimental to trying to
get a point across
>> is if you cannot filter that out.
>> Yeah.
>> Um you know what I saw yesterday I think
>> and it's not easy it's not easy to
distill this though. So a point that I
saw yesterday and maybe it's not the
same thing but I saw a post on on our
terraform on Reddit and it was I've been
working for five or six years as a ter
as a infrastructure or devops consultant
or platform engineer or security was
like security engineer and I have
collected here and published pre
publicly
and it was like 150 modules for AWS 200
modules for GCP 120 modules for for A is
Azure
all on one like website and the website
name was horrible so it's clearly not
sales so maybe it actually is solid
solid telephone code but when I see that
>> how bad the name is
>> I then I see that I'm like yeah okay
there's no way in hell like maybe
90% of that is pure AI slop like how are
you going to publish you know over 400
different Terraform modules
>> and just say here you go like how how
can I I I was even
>> there's probably tools around this
already but there should be like a tool
to recognize AI slop you know you said
this you said this here it's very
similar to what you said just there you
know minus 10 points you know like
there should be a tool to recognize that
something can be summarized or
>> I a
frustrated with Reddit because I'm
reading posts And if they I guess if you
can keep it short enough for me not to
go like holy [ __ ] wall of text. If it's
like a wall of text, you start reading.
You start reading and you go like hold
on, how much more is there? And then you
keep scrolling for like five swipes and
you go like what the [ __ ] I'm not going
to read all of that. Then okay, that's
clear. I'm not I'm just going to exit
that thread. But if it's like short
enough and I then spend time to comment
on it and then
maybe I come back to my comment because
I'm like, "Hey, why nobody replied to my
comment or what's going on?" And then
removed. Hey, and then you look at the
comments, AI slop, AI slop, AI slop
every single time. Now, everywhere on
social media, people go like, "Oh, it's
all AI slop. Oh, I'm not even going to
read that." And then, and then the
moderators usually put the rule like,
"If you don't spend the time to write
it, then we're not going to spend the
time to read it.
So, but the way AI slop is recognized
right now is just by human intuition,
right? Or there's no
>> Yeah, I think they go like, "Oh, there's
an M dash AI slop."
And you know what? I had a very funny
situation. I used AI to to um I use
cloud, right? But I didn't want cloud
co-author on my commit um for reasons.
And I let it I told it like give me the
commit message and then I do the commit
and then uh okay open the GitHub pull
request and when it does a pull request
it doesn't put itself as an author it oh
maybe it does sometimes but it didn't in
this case and I was like happy happy and
then the next like two hours later I go
back to the PR and I noticed the title
has an M dash. I was like god damn it I
didn't see the m dash
>> and now I have to rewrite my history.
Ah, yeah. No, no, it was an M dash in
the title of the PR. And then if you go
and edit the title, of course, everyone
can see Vincent changed the title from
two. But can you see the actual
difference? Because it was changing a
damn dash to a dash. And I was like,
yeah, nobody's going to nobody's going
to know what I did.
Well, let let's change um the topic a
little bit to to AI um to infrastructure
type stuff because this is the AI
infrastructure podcast. I wanted to
mention to you that I I tried swamp
swampy donkey.
>> Do you understand now? Because
>> I do pretty much understand it and and
as I was evaluating it, I was asking uh
codeex at the time because I've run out
of bloody claude credits. Um
like how how did swamp help? So like
swamp essentially
as I understand it, it grabs the model
definitions from AWS. Yeah, like model
definitions is like I don't know this
what I think we talked about this
>> uh the what
>> the like the open API or JSON schema or
something around
>> yeah like the JSON schema it it grabs
down
>> yeah it grabs down the the schema as
such from from
>> I mean I I guess they have their own
repositories of the models
um which are probably derived like the
same way you do well the the way CDK
works but Okay, not the same way CDK TF
works, but anyway, it grabs it down from
AWS and then and then you basically talk
to the the agent and then and then Swamp
sort of creates this environment so that
it's geared up for doing
uh infrastructure work. So you you
create like um you know you say to it
like create an EC2 instance blah blah
blah and then it basically sets up a
workflow and then you then once you've
got that workflow then you deploy it
in in a sort of structured reproducible
way. That's the way I understand how it
works at the moment. Um so instead of
like infrastructure as code it creates
workflows that do repetit like to um
repeatedly
>> yeah that's the way I I to be honest I
haven't repeated this thing I I'm a
little bit confused like the on boarding
is a bit strange because like you do an
init and then expects you to go into
your into your uh into your chat session
with Claude or Codeex to do the thing.
This is where I thought was a little
bit, you know, confusing here. Like it
says in it second step, you know, do do
tell your agent, but it did it didn't
explicitly say, you know, MPX claude or
codeex or something. So that that was um
a thing. But the the last bit I was just
chatting with um Mr. Stack, Mr. Paul
Stack was uh that that I didn't
understand how it links up with the AWS
CLI. And then I think he's trying to say
that you need to just set that up.
>> It just uses the environment.
>> Yeah,
>> I guess it uses the a AWS CLI credential
chain which is you either have a
>> So my my my first taste of Swamp I think
I think I get what they're doing
actually. Um and it seems to be quite
sane.
>> Um so that my initial first impressions
is is good. I'm going to give it a try a
bit more time. So yeah.
>> Yeah,
>> give it a try.
>> And
>> I found they get they they they got more
and more aggressive when people keep
asking questions.
>> Yeah, I think I think it's like I can
understand the deal with this one
because like I guess
>> it it is a big bit of a mindset shift to
sort of like
>> um
>> Can you explain something?
>> Try it out. Yeah. like why does it need
to be a specific
workflow
and not cannot be just a generic like I
don't need a specific front- end
workflow or a backend workflow right I
just use cloud it can write front-end
code it can write backend code it can
write infrastructure code it can write
github actions why do I need a separate
tool or a CLI or a shell to do something
specific
Well, I I I think it's consolidating,
but anyway, I don't want to say any too
much more because I haven't even
deployed the resources yet. So, I still
have
>> that's the question that I have because
the same question I have to every single
company launching an agent shell is
like, why do I need your agent shell?
And then the most common response I've
received is that
>> if you're an expert in AI and you know
how to use an agent shell, then you can
do everything with one generic shell.
But if you're not, then you don't know
how to set up your agent shell or how to
pull in the skills or how to set up the
things. Then you need a specific agent
shell. That was the con the the response
I've received from um Stackpack when he
says, "Yes, if you're a power user, you
know, you don't need this, but most
people don't know how to use it." And I
don't think that's a good enough answer
to me. Like,
>> yeah, I don't like things that are not
aimed at a power user. That's for sure.
>> I don't think he says it's not aimed at
a power user. It's more like um because
a power user can still use a tool to
good effect. It's just that I don't
think the tools should exist.
The the one the the one thing that I
that worried me with Swamp was that the
every time I ran Swamp on my Raspberry
Pi, it took a few seconds to work. So So
for a power user, that's one thing I
that's a bit of a smell test is that
things need to be fast. And if that's
slow, you need to tell me that you're
doing a network request or something.
And we're waiting for this network
request. But anyway, the the the other
final thing because I I need it for
work. Actually, I need it for work. So
CDK question for you, Vincent. I might
have asked you this before, but I'm just
going to ask you again.
>> Big big CDK landscape at work. We've
created constructs. The constructs, the
idea of the constructs is that we have
repeatable patterns within a governance
ADR signed off being copied and they
diverge. Diverged.
>> They've been copied and they've
diverged.
>> What do you mean the patterns diverged?
What do you mean?
>> No, you you mentioned last time you have
like reusable construct but they've been
copied around and they have slightly
diverged and you're trying to realign
them.
>> Yeah, we're we're trying to realign
them. This the problem is is that when
you deploy a stack stack is that the
right term or an app I'm
>> cloudformation type an app has multiple
stacks
>> okay yeah exactly when you when you
deploy an app multiple stacks we need to
prove that that that those constructs
were used in in that stack
>> oh
>> so it it can't it can't be a simple like
>> bill of materials coming out of cloud
form because that doesn't make sense
>> we we need to say that this um you know
acme construct for for buckets was used
acme construct for Dynamob was used acme
construct for I don't know um certain uh
support role was used
when I was chatting about it to to AWS
docs uh MCP
essentially it suggested just tagging
tagging the constructs and and then
deriving the tags from from the
from the deployment which seemed a bit
lame but but seemed but seemed fine at
the same time. I just I was just
wondering
>> yeah I've heard that before like
basically you track which module version
of in terapform has been used by looking
at the resources tag like they they
identify deploy I've been deployed by
this version.
>> I guess that's it's probably just good
enough to rely on tags. I just thought
there might be I I mean initially I was
I was going down this rabbit hole where
you know when you deploy on CDK you get
the CDK metadata field that's put in by
>> normally you would do this through
through aspects
um and you can have like a whole bunch
of um compliance aspect aspect so what
they do aspects is
>> aspects is that an AWS thing
>> it's yeah they they ported it over to
CDTF as well so what it does is it they
are invoked they are attached to nodes
within the tree and they're invoked
um during synthesis uh when when when
constructs have been generated
um and they are then visiting the nodes
and then some functional code gets
executed. So aspects can be used to
validate things. Um if you were to
like for example there's one aspect in
the AWS CDK library that validates
you can you can activate it to ensure
that IMDSV2 is used the uh information
metadata service
>> um where you have v1 did not require a
token but v2 requires a token and it's a
security compliance thing usually that
you want your AWS instance first time
I'm hearing thank thanks for the tip I'm
going to have to explore that
>> yeah aspects are very powerful. Um, they
can be used.
>> I kind of assume that that sort of thing
would be an SCP, but one problem at work
is that we don't even know who controls
the organization. Like this there's
there's an organizational breakdown
where the people who admin organization
don't seem we don't even know who they
are.
So rolling out MCPs SCP sorry is um okay
confusing.
Yeah. So aspects are a way to
programmatically and you Yeah. Aspects
are a way to programmatically validate
certain and ensure that that that
resources are created in the construct
tree or validate the configuration of a
resource in the construct tree. So you
can do a lot of things like is this
resource of this type and if that's the
case does the field this particular
configuration field match this
particular um value. So that's why when
when Anto Babenco started talking about
compliance.tf
uh introducing a whole bunch of
compliancy around Terraform modules,
>> I was like that would be so easy to do
on top of CDKTF because you just create
aspects. You just create an aspect and
you just run them uh across your
construct tree and you validate that the
resources are configured correctly.
>> That sounds like quite a cool thing. I
mean I'm just looking at the page right
here. That sounds like because in
terraform you would have to
do some weird I'm trying to think of the
name of the like OPA you need like some
sort of weird policy.
>> So there's several ways to do it in
terapform you can do OPA or Sentinel
from Sentinel is the Ashikorp way that's
only in Haship cloud. If you want to do
the open source one then you use open
policy agent which is basically uh like
a rule based matching engine like
prologue um and you basically define
your rules and it works across
Kubernetes for Kubernetes admission um
hooks but it also works across terraform
plan files or JSON you can apply it
across JSON so then it understands the
concept of like a plan and what's the
before and after of a resource and then
it can validate that the after still is
um is
>> I find like you have to be a super
mature shop to start using OPA. Um
>> we use we use it at so we use it for tag
compliance. So we have a couple of rules
that identify which are the allowed tech
like required tag keys. So if a resource
is going to be created or modified the
after snapshot has to have these tags uh
these particular keys and then per key
which are the valid values. So if you
use a if you make a typo in your if in
your value it will highlight it. It says
that's not a valid you know using value
list like this reference key. It's
called a reference list right you have
this key has this value list.
>> Yeah I can
>> um so we use opa for that with because
it helps with like
>> incident escalation cost.
>> I think I've seen as much before.
>> Okay. Sorry I need to take my kids to
school. We're out of time.
We're out of time.
>> Yeah. So you can do tags. Um, and you
can do opa.
>> Well, if you can think of anything else,
just let me know. I I need that's the
that's the thing I need to do today. Uh,
I mean, I'm a I'm a big fan of like the
whole bill of material thing. Like I
don't I don't feel that's done enough in
our industry to say like these these and
these and these things are deployed.
>> But yeah, the problem is there's so many
it doesn't make a lot of sense. You need
you need
>> I was going to say with AWS CDK that
lives in your Python requirements.txt or
in your package.json, right? which are
the constructs that you depend on, but
it doesn't really mean that they're
being the ones invoked into the code and
>> Exactly. Exactly.
>> Right.
>> Yeah. So like like uh I I'll I'll get
into it in another time. Okay. Thanks.
See you. See you, Vincent. Cheers.