Episode 25: How can we trust AI?

Hello and welcome to episode 25 of the
AI infrastructure podcast
with me Kai Hendry in the UK, Southwest
UK to be exact
and
my
co-host Vincent De Smet over in Vietnam,
Ho Chi Minh City
I believe.
We're both
infrastructure engineers
by day and uh
and and uh by night and in the morning
here I am a AI enthusiast. Um in this
podcast you can expect us talking about
AI and infrastructure.
Um what does it all mean?
And I I hope you enjoy it.
Do please comment.
Do please like. Do please subscribe.
And um we're trying something new today.
We're trying Riverside. So do tell us
what you think of the recording. Is it
better than the typical Zoom recordings
that we've been doing up until now?
You tell us.
Thank you and enjoy.
>> Hey, how are you? Having a good uh day?
>> Uh
I just woke up really.
I'm uh
Well, I just
There's a lot to cover. It is almost
overwhelming every time we meet cuz like
there's a few things I want to cover
>> Mhm.
>> and I'm like
where's Vincent on this? Where's Vincent
on that? I mean yeah, what about
yourself?
>> I had a lot of fun from like in the last
four days using the tool that I built
because I haven't had the opportunity to
play with it and and and I had some time
to play with it and it just feels so
good to to see how it has evolved, what
features has been added, what works,
what doesn't work. I also played with
Riverside like you. Actually basically
because you sent me the message I was
like, okay, let me try it. It was pretty
fun.
>> Yeah, well, I'm I'm desperate to improve
the production quality and try different
things actually. So well, here we are,
trying different things. So cool. So,
your tool is Spec Ledger or something
else?
>> Yeah, yeah, no, no, it was Spec Ledger.
So, basically, there were a couple of
annoyances that had that I had filed,
and then they had fixed it, and I wanted
to see if it the fixes were there. I
think one of the things I might have
mentioned this that we were suffering
some quality issues, right? Even though
that we were using the spec driven
development, and I had very good results
with it. Once we scaled it out over a
team with people of different
experiences, I felt that even though we
had team alignment on what to implement,
there were still, when I finally got the
implementation done by other people,
areas where I discovered, "Hey, that's
not done. This is not done." And so, I I
felt like
we talked a lot about how do you
validate, how do you ensure that the AI
implements, and I always talked about,
"Yeah, you know, you need to build those
validation, you know, testing around
it." And then, the thing that I have was
completely failing. So, I was like,
"What the hell?"
>> [laughter]
>> So,
how is
How are you doing it in a team? Cuz I
can't help but think all those the
quality assurance is is getting back to
basics, like, you know, my employer
preaches shift left, preaches having a
dedicated QA person on the team.
>> Yeah, so
>> it emphasizes quality. Um so, as a team,
you can't just expect AI to to do the
quality, right? You have You have to
have someone responsible or playing the
role of a tester or something.
>> I felt the hardest thing is when you are
writing quite detailed and maybe too
large of a user stories, is how I only
find out maybe a week later, like, "Hey,
I remember I put some requirement there,
and when I'm using it, it turns out it's
not there." So, then I get frustrated,
like, how did that not get there? Like,
in terms of verifying it, it felt very
random, and gave me like a lot of doubt
about the quality. But, when I started
playing with it, I actually realized
that none of the quality guards that are
built into Spec It were properly
initiated. So, I had not initiated the
repository, and so we were missing the
constitution. We didn't have proper
checklists. We didn't have a proper
design like
>> Well,
it sounds like you're in that sort of
waterfall mindset where you're saying
like the reason we had this quality
issue is because our spec was wrong.
>> No, it's it's the the the validation
wasn't there.
One of the things that I put in the
constitution is that a quick start
document, in this case for a CLI, must
document the user stories in a It's a
command line interface, so it must show
user story one. You have You invoke the
command line with this parameter, then
this needs to happen. And that directly
must translate to a formal like test in
the test. So, that I had that in my own
project, but I noticed it was missing.
We didn't have like these tests there.
>> All right, so the the tests are not
dropping out of The AI isn't isn't
following instructions.
>> No, the problem is the instructions were
missing.
>> Really?
>> I had not initiated this, and the
constitution was still empty. I thought
it was it was filled out. So, it has no
principles. So, there was no rule for
the AI when it finishes the thing and
then validates the task list. Maybe you
can say that's because I I thought like
waterfall plan wasn't right, but it's
very important because the final phase,
which is the the the validation and
testing and polishing phase, needs to
have a task that validates it. And a lot
of the times the task was manual
verification instead of And that's
because it wasn't configured properly.
And and so basically what I did for the
weekend was exfiltrate what I had done
in my own project, where I have a very
strong test suite, and basically copied
over the principles into this one. Cuz I
showed you the grid, right? I showed you
that I have flu full play right
workflows. In this case, it's It's just
a CLI, it's actually a web interface
with Keycloak authentication.
I'm running everything there and and the
and I had no quality problems, but I was
also doing it alone, right? So, that
means I was defining the plan and I was
also driving the implementation. In a
team setting, you might only be involved
in defining the plan, but you are not
driving the implementation. And then you
find out the person that did the
implementation didn't actually validate
it.
>> it makes me think that you need to be
there for the whole process. You can't
be
>> want to be there for the whole process,
right? So, I am trying to solve that
problem because
>> Why don't you want to be there for the
whole process? Because it's tedious? Is
there some reason?
>> Because we individually must own each
feature that we're working on. Like,
there's basically a team exercise where
humans decide and collaborate on what is
it that we want to implement, and then
it's an individual and ideally not even
a human involved, purely AI agent
implementation, right? So, today we are
letting juniors drive that
implementation process. I don't know if
that's a good thing or if it's a bad
thing. Maybe they are distracting the
agent and therefore the agent did
deliver, but I don't think that's a
problem. I think they are letting the
agent implement and they're not stopping
it when it diverges. And And that's a
problem. I don't want a human to be
there in the end to to stop it when it
diverges. I want it to be so
straightforward that when the agent is
done, the output works. I don't want to
be there for that implementation.
>> Why doesn't So, essentially you're
saying that the junior is not catching
the diversions or something like this?
And
then you're going back to the drawing
board to make sure the AI doesn't
diverge. It sounds like that sort of
thing is happening.
>> Yeah, and I think the contributing
factor is that the junior is actually
not very much involved in the plan
planning process. They're just But, it's
not even on juniors only. We even have
like a very big user story, like we had
too many, like 10, and we split them off
into the individual ones. But, we had
aligned on the on the 10 big user
stories and we realized this is way too
big of a scope to try and go into
implementation. So, we split it off in
three smaller work streams of three each
and one of four, and and and I let that
I let
and then execution by a like a more
experienced person, and we still had
things where, "Hey, the original 10 user
stories had a detail here that was not
captured. It could have been lost when
it was split, or it could have been lost
during the implementation." So, even the
split I should have been involved, but
Yeah, it's just
>> It's tricky. I guess you're you're
trying to scale with the help of AI, and
>> Yeah. And that's why I really love
>> of problems.
>> Exactly. That's why I really love this
project because it
it's really a real true test of scaling
out spec driven development across a
team of different skilled people.
>> So,
on that note,
let me just share something that I
thought was really
um awesome.
>> And I say different different skilled
people, but I also found mistakes that I
made, by the way.
It's not like
>> Yeah, I mean the the last podcast, or
the podcast before last, I was I was
actually like in shock about a bug that
came through into
Well, it didn't go into production,
thank god, but
>> But you feel that the the workflow would
have easily been found out if you had
paid closer attention.
>> Well, if Yeah, if we if we just did it
the traditional way, it would it would
have never got through. But, of course,
I was like giving the PR to a colleague
to review, and the
and the problem was is that the
colleague trusted me, and I trusted the
AI, and then and then this bug bug got
through. So, the the whole the whole
trust thing and I felt it wasn't so much
I I put the bug through, it's also that
now now my colleague thinks I'm an
idiot, for want of a better word,
because I let a trivial bug through, and
now he's lost confidence in me because
I didn't review I didn't see this AI
introduce the issue to me.
I mean, I'm feeling like I'm blaming AI,
but it but it was really me that was
that was doing the wrong thing.
>> I don't know how much data it had to
like you had Like was it part of a huge
chunk and therefore it got it flew or
did you really just not give it
damn guy.
>> Anyway quality I feel like we can talk
about quality for days but this blog I
think is very succinct on the topics. He
starts off by saying that every time you
have a review it makes you 10 times
slower and of course PR reviews we all
know they suck but they they are a
quality gate right and and he was he was
pointing out the more reviews you have
like I'm in some heavily regulated
environments and it takes like two
approvers to get something in and and
what he's saying here doesn't sound
>> [snorts]
>> insane but it it definitely accumulates
about how
reviews take a long time and then he
makes a very interesting point to say
that AI can't fix this.
And like just like what we talked about
like a lot of people are are creating
these AI
flower wheels or AI harnesses or
orchestrations where like you know oh I
mean this I mean doesn't it sound like
you right now in a in a in some way of
instant like well I have this prototype.
>> And this a month ago.
>> Yeah exactly but the but the prototype
is getting busy buggy sorry we need to
tell AI to fix this problem.
>> I didn't see I need to
to let AI fix it.
I said I want a formal validation
framework around AI.
>> Sorry I'm I am making this a meal of the
straw materialization but like I I think
you get the point here is that the a lot
of people are getting into this like
trap where like AI caused the problem
but maybe I can get AI to fix the
problem and then now we have a different
sorts of problem.
>> Absolutely and this is what I also said
said to my friends I mean the fact that
we have this problem and we know that
because we're taking something and we're
scanning it out of a team and clearly
the quality is a problem so we need to
solve it and that's definitely what's
going to be the focus of the next few
months, which is the validation
mechanisms of the AI output. Cuz
everybody's on board with SDD now,
right? I mean, it's going all over the
place. My friends are constantly sharing
Singapore GovTech is doing a
a trial of it. Um other organizations
are trialing it.
>> Oh, okay.
>> Yeah, spec driven development. That's
That's kind of like been being adopted
everywhere, right?
>> I feel like the problems that you just
talked about like that you
wrote the spec, but then somewhere down
the line there was a quality lapse or
some junior didn't understand. Like
>> The whole purpose of SDD is to catch
this. And And I know for a fact that my
friend wrote a blog post talking about
how vibe coding creates all these
problems and how spec driven design
solves them.
And I tell him like I would
>> But surely surely if we we ran the clock
uh 20 years back, I'm sure some
waterfall
IBM proponent would have said the same
thing
about
his specs. He would have said like, "Oh,
yeah, if the spec would have caught that
problem because we would have done the
upfront design." What What What about
the whole agile thing where you
>> No, this is not my This is not the
problem that I'm saying that there is,
right? I'm saying the spec clearly
defined it this way and the
implementation does not align. I'm not
saying the spec
um
wasn't properly designed. I never said
that. I said the spec was very clearly
defined, but the implementation doesn't
match the spec. The agents took
liberties.
And then the funny thing is when I see
the agent take liberties when I'm I'm
driving it, I stop it and I asked it
why. And often times there's a very good
reason. Sometimes it was because the
task description said one thing and the
original plan, like the the very first
functional requirement in the spec, said
something else. And the agent says,
"Spec rules all. So, I went with the
spec." And the reason that the task
definition was different because halfway
through research, when it filtered down
into the task, I made a change and I
didn't back update it into the spec.
Something like that. That that's one
case where I'm
>> wrong with the with the process.
>> Yeah. And and I think sometimes it's
because I didn't do the full cross task
plan and spec
validation cuz I do skip that one
sometimes.
Um which also made me think like, "Hey,
we need to make sure we need to keep a
flag to see if somebody at least once
has run that validation uh or
verification thing." Which is a very
extensive read cross artifact um
validator that, you know, you can use
different models. I used to run those on
Codex when I would always run out of
tokens. Today, I don't really switch
models.
>> So you use different models.
>> I don't think different models is
required. It was just because I ran out
of tokens. But yeah, but that's just one
case where the implementation diverges.
I want to give you one more example,
which is
um and now I'm forgetting it.
So one was that that the task and the
plan did not align. The second one that
I've seen is that the plan says we can
use this library this way and then the
agent says it doesn't work. Well, let me
go back and simplify this or something
like that, right? You see it happen,
right? The
you make a plan with the agent, it it
reads the docs of the library, it
decides to do things certain way, and
then finally when it's implementing, it
doesn't work. And then sometimes Cloud
Code goes like, "Well, you know what?
This is too complicated. Let me simplify
it." Then you know that no, stop. So I
stopped it and I asked it very clearly,
"What is the problem?" And and it's it's
totally valid, right? It's totally
valid. But the the most the biggest
problem is the agent just keep going
instead of stopping and saying like,
"Hey, plan doesn't work." And it's very
hard to define when when are we okay
with the agent stopping and when are we
not okay with the agent stopping?
Because ideally, we want like a Ralph
Wriggum type of loop that keeps going
until the task is complete, right? So
>> Yeah.
>> Yeah. So that's that's the problem.
>> that there's different philosophies
here. Like I'm I'm I'm more in the like
just chat to it camp.
>> Just chat with it. Yes, but you don't
want to be in the loop the whole time.
Maybe maybe it's it's good, right? Maybe
it's like yeah, we still have a job.
>> Well, I just feel like the small
iterations get to the solution better
than going back to step one in a sense.
Hmm.
>> So, you could argue that that the spec
was too large and it should have broken
down more. Um
you could argue that, you know, don't do
spec driven development in Vibe. I don't
agree with that. But you could say if
you do spec driven development, really
scope down. And that to be honest is one
of the one of the principles I put in
the constitution now, which is shortest
path to MVP, short-lived branches, scope
down, scope down, scope down. Like do
not Let's all YAGNI, you ain't going to
need it. Nothing that has a clear use
case should ever be included in the in
in in the implementation plan.
>> Mhm. So, I'm just highlighted agent
framework and I'm just I can't help but
have this thought that like what is the
difference between spec driven
development that you're creating and an
agent framework? Cuz surely
the difference
there's not a lot of difference there or
it can easily be construed as a
framework, couldn't it?
>> I'm not sure, but to me, when you start
talking about agents and what the
majority of like articles I read are
doing, they're looking at like agent
teams and personas and all that. I I
don't want to go there. I don't want to
look at like agents completely
individually doing all the work. I I
think you need to have a way to define
the work. You need to have a way to to
to really
you know, master and manage what the
work is.
Um because I mean, a lot of these teams
that are saying, "Oh, we we have a
linear uh board with issues and we just
assign it to our uh swarm of agents and
they they work on it and then we get a
working solution." Either maybe that
that their SDD spec driven development
is is basically their classic product
owners
um and engineers defining
>> Mhm.
>> the actual ticket uh down to a very
small task that an agent can complete.
And yeah, I mean, if you do that, that's
perfect, right? But I'm focused on that
that life cycle part. Like
>> Yeah.
>> taking it So, the whole like agent
framework and individual working again,
like I said, I don't want to be in the
driver's seat, but I very much feel I
have to be right now. And to me, it
feels like you I I I want to control
the the you know, the task definitions
and how the work goes into the agent
framework. That's where I'm focused on.
>> So, the other thought, because this is
applicable to a problem I have with at
work right now,
is documentation. So, your spec, is that
>> It's not a documentation.
>> No, I had the same and it's a very good
question that I did a session with my
friend. I said, "Hey,
um basically what I'm doing is I'm
giving people the chance to
um give me some project they want to
work on, and I use my cloud Claude Code
agent tokens, and I spend 2 hours, and
we have to complete it within 2 hours,
and I'll uh give you a working solution.
So,
I initialize the repository with or you
give me a a repository, you give me a
feature to work on, and the condition is
that I can record and publish it, and
then I hopefully give you a working
feature implemented onto your code base.
So, I did that with my friend, and he
asked me exactly the same question. So,
the spec, is that a documentation? And
and I actually really didn't realize it,
but you need to mean I think you also
asked me this last time. No, it was my
friend.
>> This is the big problem I have at work.
I can maybe explain to you the problem.
But yeah, so what was
>> Sorry, I interrupted you before you
explained.
>> What was your resolution? How did you
What's your approach to documentation?
>> I believe it should be um
in the repository as um
This is what I did for Spec Ledger,
right? I I created doc/design that
documents every layer. Like, what is our
our design philosophy? How do we
organize the command line interface? how
do we organize the what's the
responsibility of each layer within this
agentic design framework.
>> Okay. So, you have a separate
documentation, but who edits
documentation? Agents or humans? Or or
both? What's your How do you keep the
documentation?
>> needs to update the documentation. That
is the humans need to confirm and and
review. I mean, ultimately the agent
writes the docs, but the humans need to
review and and and can modify the docs.
>> So, I feel partly responsible for this
misalignment with my my colleague. But
like for example, one thing that I like
to refer to a lot is this
um
this French site called I think I showed
you this before probably.
>> Yeah.
>> Whatever. Don't even know how to
pronounce that word.
We have these different sort of styles
of documentation, right? So, that we
have some documentation at work. It's
quite a lot of documentation, like at
least 500 files of markdown
um exported from Confluence. And what
he's done
is pretty good, but at the same time
it's kind of made made things a bit more
problematic because he's taken the 500
documentations and then he's generated a
how-to guide. He's generated an
information reference. He's generated um
some explanation documents. He's He's
generated some tutorials from that
original
um
corpus. So, now we have actually more
documentation than what we had
previously. And now it's like becomes a
like like where's the source of truth?
>> Yeah, you They They become
out of sync.
>> Now you have a broken mirror.
>> Well,
yeah, something like that.
>> Yeah, because one the how-to guide is
still talking about A, well the
reference has been updated and actually
it's A A A uh A alpha or A beta. And and
now they're like slightly different. And
now how to realign
>> Exactly. And that now we Yeah, we now we
have a serious alignment problem. Um
um
But like
my colleague is he's good.
It's I just I just that I feel like
that
I mean the things he's generated are
actually a lot better than what we have
currently, but the trouble is it's like
there's no way to
keep things
synced in my mind without
running AI over every like every time we
update this thing, then maybe I have to
have an AI job to create the tutorial or
update the tutorial and update the
explanation, update the reference.
>> You're just going to
>> you run the AI, it's going to find a
difference and tell you there's another
difference.
>> Exactly. I'm just
I'm already in this like
Kafkaesque loop and I can't help but
think when we went back
The original documentation that we have
in Confluence, even though sometimes it
was wrong, at least you had the source
of truth, right? Like all the
documentation for this particular
feature is here and and that's that's
where it is. It is not like in four
places now.
>> And it's also something that that that
Opus or Sonnet or all of them like to do
is when I when I have like, "Hey, we're
creating this index document." And then
it goes, "Do you want a short summary of
like the other document or just a direct
external link?"
Um and every time it creates a short
summary, that's that's something that
potentially gets outdated and and
doesn't show the original like doesn't
actually show the correct contents of
what it's linking to, right? I mean, I I
tend to
I always ask it to generate options and
then ask for my alignment and then it
always gives me this option to like
duplicate some of the information here
and then create a link.
>> Yeah.
>> Which sounds interesting because it's
kind of like progressive disclosure,
right? You you read a little bit and
then you can go in into deeper details,
but it just creates an another thing
like earlier when I said the task
definition is a one thing, the plan kind
of hinted at it and then the the spec
was completely different. You need to go
all the way back to update all of them
or you get all these misalignments.
>> Yeah, this this
Yeah, when you when you talk about uh
spec-driven development, I do really
like the source of truth element to it.
I mean, I'm I maybe made a a brash
assumption there, but with your
spec-driven development, like there's
one document that is the source of truth
for that
or or or there's a part of the document
that's the source of truth for that
feature, say, right?
>> Yeah, but if you if you So, the thing it
it I guess it depends on which framework
that you use because I know that Open
Spec apparently has the idea of um like
archiving and um but but with the one
that we currently have with Spec Kit and
we haven't changed too much about it,
you end up with 20 20 spec 20 spec
folders. And each one of them have a
slightly iteration on top of the
previous features, right? So, your
question earlier, like where is the
source of truth? It's not in the spec
folders because you have the version one
which was we do this initial spec with
these initial initial features. And then
we build on top of that in in the second
iteration, we we add a bunch of new
features, we modify the original user
story slightly. And then the third one
but and we don't go back to the first
one because that's the original user
stories, right? So, I do feel you we
need to have a canonical root of repo
once this user so once the second spec
is merged all the user stories in in
like the the root need to be aligned.
So, because now we have the original
user story here, we have the new user
story there, and we have the slightly
modified original plus use changes that
were introduced by the next one. So,
that is currently not in the Spec Kit
and that is something that I put in
That's not in the Spec Kit. That's not
That's not in
>> And wait, wait, are you in Spec Kit or
or or
I mean, is it not in your spec ledger or
I don't I don't quite follow what what
you mean by that?
>> It's It's No, it's not It's not in Spec
Kit. I don't know if Open Spec is doing
it. Maybe I should I should investigate
it further, but it's something that I
put in my constitution now, like I just
explained, like I just added a docs
design markdown index, and I said when
we are building, we go and update this
the documentation that defines the the
system like design.
>> Yeah, that seems like a fundamental
thing that needs needs to be updated and
it needs to be canonical. Like I
I was thinking naively at at my
workplace that we could just do what AWS
docs
uh do, like I don't know if you've ever
used this uh this I I I know you're
going to probably revolt when I say MCP,
but this particular
>> I am I am controversial, right? Now,
everybody says MCP is that? No, I say
actually I really enjoy MCP.
>> Yeah, I
Well, this this MCP uh AWS docs
uh feature, I don't know how you want
this this service where where uh you can
jump into Claude, set up MCP, and then
ask questions about AWS docs. It's it's
wonderful. And this is what I wanted at
work. I wanted something like this where
>> For your docs.
>> where your docs the the canonical docs
of AWS or canonical docs are queried,
and and the cool thing about using an AI
agent like Claude is that like if you
wanted a tutorial, you can you can get
you can say to
uh your prompt to say like give me the
steps to set up an S3 bucket with with
the I don't know, an access point or
something. And it can read the
documentation, and even though it's not
a tutorial, even though it's like I
don't know what the AWS docs are.
>> It can generate a tutorial for you.
>> It can it can generate a tutorial. So,
this is what I wanted at work, but the
trouble is we don't have an MCP.
>> I don't think the problem is the MCP.
First off, you can what with the CDK TF
my fork, we we are we got OSS support of
Mentlify, and they provide a doc MCP out
of the box out of their platform. So,
you put your
um handman hand crafted
um documentation. Actually, we have two
two parts of the CDK TF docs, and that's
nothing we built. It's what the original
CDK TF project had. Which is on one
side, there's the, you know, concepts
um type of information about, you know,
what's a stack, what's what's an aspect,
and all that information. And on the
other side, there's an API reference,
which is completely generated from the
TypeScript uh type definitions, you
know, from through JSII. Um you get the
core schema, like almost like JSON
schema type script schemas, and you
generate it cross-translated. It says,
for Python, it means it's like this, it
uses JSII. So, it generates the API
reference. It's like JS docs, Java docs,
if you remember, right? It generates
this huge um
API reference uh libraries, right?
>> Which are pretty cool. I mean
>> Yeah, and I think that's
>> a bit a bit dry, but they have a lot of
Yeah, it's it's interesting.
>> Yeah, uh when when we when we learn
programming in school or in university,
you had to go through the Java SDK API
docs, right?
>> Yeah, the PHP
>> to
>> I always think of the PHP docs for some
reason. The PHP API docs were amazing.
>> We had to learn the standard uh library
of Java, and we had to like understand
all of these collection classes, string
buffer, and when to use what. Um
And and you had to understand the SDK.
Even when I was learning .NET, and we
were learning about like threads and
asynchronous,
um the whole threading in .NET.
>> Yeah, that's that's a that's a good
exercise to read through the base like
With PHP, it was I I I I guess I was a
bit distracted half the time because the
comments section of the PHP docs was was
always hilarious to me or very
interesting.
>> Okay. So, yeah. So, but but we I think
we can agree that we have long solved
this problem, right? You put some Java
doc string on your code, on your
interface, on the property, and you get
an auto-generated, very deterministic,
maybe dry, but
>> you any
of that.
>> to these decisions, of course.
>> It depends how good you document it. But
like because if you look at AWS CDK,
they actually have like very detailed in
the Java doc, they have code snippets.
Like sorry, in the JS docstring,
JavaScript documentation string. They
have a code snippet. Like there's a
constructor of the VPC L2 construct,
there will be a code snippet. This is
how you build a basic VPC right above
the constructor. So inside the API ref,
um you go to that their docs. They have
a right at the top introduction. This is
a VPC construct, then little example.
This is how you you invoke it with basic
>> to send me a link to that cuz it's
probably something I completely missed
actually.
>> AWS CDK L2
>> Because we
cuz at work we're trying to create level
two constructs and our level two
constructs are basically living the
documentation's living in confluence as
an ADR that got signed off by security
and whatnot.
And
I think that's what we're missing at
work actually cuz the constructs, the
way that they're implemented, there's
very little documentation in the source
code. Everything's in confluence and we
need
>> So this is the API reference. So on one
side you have the developer guide that
explains concepts that
that explains that explains concepts.
And then you have the completely
programmatically generated API reference
that you can view in all of the
supported languages. So these language
translations
>> Three.
>> are fully automated, right? And then the
example I've gave for example was for
the VPC, right? Let's go to the VPC. So
if you go to the Where is he? Seriously.
>> Well, I'm more interested in like a
level two construct or something like
that.
>> Yeah, the the the VPC level two
constructs or maybe they
they they got rid of it. But I want to
show you the level two construct, not
the level one. Is this
Where are the level twos? Let's just I
want I want to take one that I know.
Like let's take um
Dynamo. So DynamoDB, oh I know why I
couldn't see it because it's under the
EC2.
Eh, where did it go again? It's under
EC2. But, they they also have the V2.
AWS EC2. Okay, you're going to have to
cut a little. So, here you have AWS VP
EC2. They have the overview, which is
the root readme of the the readme.md
from the library, right? In the library
folder. And then you have every single
construct, and this is an L2. And right
in the construct here at the top, you
have a little example, right?
Um for example here, and then you have
another example here.
>> Mhm. Mhm.
>> And how is this created? If you go to
AWS Labs, is it AWS CDK?
>> Yeah, but examples if you go to the
GitHub
>> That's one form of documentation. I I'm
thinking like
>> No, no, hold on. Let me finish.
Um
>> Okay.
>> AWS EC2. Where is EC2? Oh, no, it's AWS
EC2. Where is the source for this
because what we just said is that AI can
generate all of this stuff, but it's
it's it's like slop. So, the root readme
is this thing, right? Which
completely matches the root readme here,
okay? This is import. This is examples.
So, that's that thing, right? Exactly
the same thing. Then the second thing is
you have the library. And then if you
look at the actual VPC here, right in
the JS doc string, you have these
examples, right? Um in the constructor.
So,
VPC creates a VPC. I guess there's one
on the cloud There's two. One One is on
the class. There's a JS doc on the
class.
Um This is the static import. Where is
the public Oh, there it was right there
at the top. Here. So, export class VPC
VPC base. And here we have the original
like for example, there's a VPC new.
There's So, you see it's even like
annotated TypeScript. So, that's use
what you see at the top here. When you
look at this, you see it right here.
That's is a
JS doc, right? And then when you look I
don't know where the other example is,
but I I it's here. Well, where is the
actual constructor?
>> here's the constructor.
>> I think those are good for like
development
uh
It's not because of the library things,
but like what what
like
often enough we we have a little to
construct at work where it's for a
particular business unit for a
particular feature and it's not the sort
of style. It's like this this security
consideration, security reviews and
reasons why it's done this way for
certain reasons and
>> So, we we have
>> I don't think [clears throat] the source
code is the right place to put it, is
it? I'm not too sure.
>> I absolutely think that as much as
possible your documentation should live
life together with your code.
>> Mhm. But like
in a lot of enterprises this there's so
many non-technical people involved. Like
I dare say the security people are not
terribly ever going to look at the code
and the product owners are definitely
not going to look at the code and the
business analysts are not going to look
at the code.
But this is the Mint the Five. Is this
Mint the Five driven MCP or AI feature?
>> Yeah. So, in this case, right? I don't
think I have any example in my
documentation that shows you exactly how
to how to create a Terraform stack with
the S3 backend. But because of the way
that these API references and everything
is generated, you should get exactly
this. Like this is how you do it. And it
shows us there's an API reference here
and there's a backend config reference
here. So, this is kind of what you
expect that you're getting out of the
AWS MCP, right? You can ask it the
question even though that the the
reference data, which is auto-generated
and very deterministic out of the, you
know, API contracts and open API spec,
which AWS what they they have their own
like Smith
Smitty language to define API contracts.
They they they document it very
auto-generate all the documentation and
then you layer layer of AI on top of it
and you get this type of response where
you ask for an example and it gives you
this. And in they actually give you in
the pro plan the ability to expose this
as an MCP so that people can install, I
don't know where is the option here.
>> Okay, so this is this is Mintlify
uh
feature, right?
>> Yes, these are all Mintlify features.
And and um and and it kind of combats
the problem of we have a whole bunch of
markdown AI slop generated in five
different ways. No, we have the core
truth ideally or deterministically from
the code source code, and then we layer
we layer the AI on top and it has a
vector database and it can in it has
indexed your docs on whatever they get
published or whenever you do a library
release, and then uh the vector database
quickly sources the data so that it can
>> I like this. This is great.
Though I mean, you hear me in the sense
that like we have a lot of non-technical
people involved in in these enterprises,
and they just I don't think this is a
good fit for them because they they live
in
in Jira Confluence land. Like how do you
how do I incorporate
>> I mean, it depends on what is the
documentation that you need to generate,
right? If the documentation What do you
mean? Like if Who owns Who Who has the
knowledge? Who needs to write it?
>> Well, the the trouble is in a in a big
enterprise some people I think the
knowledge is kind of split because like
some people have technical knowledge
like the docs and things like that and
they the implementation knowledge. Some
people have the the business knowledge
like why it is this way because of this
decision. Unfortunately, in a lot of
enterprises a lot of this information is
kind of gatekept to be honest and it's
really hard to to shake out like why
this decision is like it is
and things like this, but
this is this is the problem I'm dealing
with and I'm just starting to think it's
it's a really challenging problem.
>> If it's a challenging problem, it's
probably worth money.
Because I think you know how much money
Mintlify has been raising? I mean, I
just showed you what they can do with
like they are
>> Well, this is this is the great thing
about AI is like we're looking at
problems again and we're coming up and
there's innovation happening everywhere
and we can hardly keep track of
everything. It's just incredible and
it's really exciting space. I mean, I'm
a little my mood this morning is
probably not great, but like I'm
actually excited because there's so many
but there's so many like things to solve
now. There's so many things. But let me
let me get back to that blog that we
started off with because I think there's
some there's some really good points
that I made. Like for example,
like why we review.
We we we want to review so that
there's alignment that we have a quality
well, I don't want to read out this
text, but this is what I'm
this is what I'm I'm getting at.
Uh
this is how I interpreted
interpreted this blog.
I think we talked about quality. But the
the really interesting thing I think he
makes it is is trust. Like he he made
the point that like uh
what one of the things that made
Japanese
uh
stuff so good is that there was an
implied quality, right? Like if you
bought this Japanese thing, it's going
to be quality. So when you when when you
have that sort of uh expectation and
trust,
uh it makes your quality process a lot
easier because you don't need to have
all these checks and rigors if you know
you're going to going to get a quality
item um
you know, cuz it's got this Japanese
labels on it. But I think it's I think
it's true like back in the day as I
don't think I don't think it's the case
nowadays because like Japanese products
are probably not even manufactured in
Japan, but who knows?
>> I mean the core the core point is that
with a label on it, it uh
it's trustworthy. But the funny thing is
also related
>> Well, no, the label's the wrong thing.
It's like it's like the trust is there.
>> You know, yeah, but like it's the same
like you trust a certification program
or you trust a security scanner or you
trust uh it's a it's a base layer of
um guarantees, right? That you can build
upon.
>> Exactly. Exactly.
>> Uh but but if you want to talk about
Japan quality, when I talk about Japan
quality to my father
uh who is 80 years old, he says it's
funny because back when he was young,
um Japan did not have the the the
reputation of quality and and it was
like where China was maybe a few years
back.
Um where, you know, made in China was
not was like oh it's it's it's not good
quality. It was like that with Japan. He
says also like they used to come to
Belgium and take pictures of everything
and everyone hated it like because they
take pictures and then they go and make
it back in Japan for cheap. Um
but it's funny now because now
uh made in China is actually like, you
know, it's good quality. I don't know if
you you still have that connotation, but
for me, made in China means it's
actually futuristic, it's good quality,
it's
>> Yeah, there's definitely something. Like
I mean
my my parents used to go to Japan in in
the in the '80s to buy fabric because
there was sanctions in in South Africa
and you couldn't import fabric very
easily from other countries, but in
Japan
um they were nice to South Africa.
>> Contraband.
>> I've I've got a few I've got a few
Japanese toys and the quality is
incredible um for the '80s. So, there
was there was a golden age for sure.
Uh
but like
yeah, going back to this
going back to AI and
>> Yeah, we went to
>> teams
>> We got off topic totally.
>> and teams and things like this. The
trust Yeah, like
I mean, I know from working with a as
being I know from being a consultant for
the last 15 years five years at least in
my current consultancy, like trust is is
the number one thing. Trust is the key
to for a client relationship. Trust
trust trust. And like I'm a little bit
shocked in some ways that people are
using AI and uh basically destroying
that trust because like that the client
doesn't know that much about AI and then
someone comes along and another a
consultant says like, "But if you use
AI, this could be done so much quicker."
And then all of a sudden quality issues
occur and then the trust is basically
destroyed. And this could be a colleague
of mine.
Um
So that anyway, I guess I I don't really
have a solution here but like
but like if I don't know how you even
measure trust really.
But but this is the thing that needs to
be
I anyway, this this blog made me just
think aloud that like trust needs to be
it needs to be protected. We need
integrity here for what whatever we do,
trust and alignment or whatever you want
to call it is so key with all this
innovation that's happening. It's just
like
>> You know what I also You know how how
>> how um we used to look at CI/CD
pipelines. One of the concepts of
immutable infrastructure is that you
build an immutable asset and that asset
or artifact moves down the pipeline and
the further along it goes to the right,
the higher our trust is into this
artifact until we are sufficiently
convinced to deploy it into production.
>> Yeah, passing all the promotion gates
and such and so forth.
>> And I think that's still very much like
the requirements is we need verification
mechanisms around these
>> But then but then but then but that is
in contrast like the speed of AI and the
slowness of reviews and quality gates.
>> Yeah, which is why everyone always says
like Agile and I think we mentioned this
like a long time ago, Agile best
practices are really important and I
think even like Adam Jacobs and all when
they talk about AI they always say like,
you know, we chef, we do Agile, we do we
do all of this, you know, devops best
practices. We do validation mechanisms
and the focus was always on automation,
right? It was always on linting to
reduce the overhead and like, you know
why linters exist, right? We we want to
remove all of the white space
differences and formatting issues that
people would argue over. Like, no, but
you know, curly braces would be on the
set next line, not on the same line.
Linting rule.
>> Yeah, we Yeah.
>> We don't want to have We don't want to
waste our time with with those
discussions, right? So, the more that we
can put in uh formal and automated
deterministic validation mechanisms
around produced code, then the more we
can
>> Every makes the same point with Go
format. Think of the people who created
Go format. I don't know if you've ever
worked with Go. You have, right? Go
format is fantastic.
>> And and Rust expanded on that because
then aside from Go format, Go test is
also an amazing framework.
>> Yes, it
>> originally those test frameworks, they
were not in built-in. Like, Node didn't
have a test, right? Then you have
Jasmine, and you have so many different
iterations of it.
>> Yeah, and they're slow and they're God.
>> So, a language that is built from the
ground up with formatting, linting,
testing, like Rust, Go, what else? Those
languages, I mean, you have
well-established frameworks for Python
and for for JavaScript, Jest or Vitest.
So, you have a lot of them, yeah.
>> I mean, like, we know we know what to do
in some ways that we need to add
automation, add tests, add
>> to build trust.
>> automation and sense add trust into the
whole into these new technological
advancements, AI. But, I just feel like
it is very challenging. Like, this is
what we need to do, but I feel like we
need to
go back to the drawing board, like make
all these mistakes, and then automate
these things again. And that And that's
quite challenging in an enterprise
environment which is used to
being slow and used to
having all those quality
in place. You know, you can't just like
flip the table and say, "Hey, we're
using AI and we're making And we're
going to be a startup again." And and
and we're going to learn a lot and
things like this. It's really
challenging.
>> And I think the most important thing is
that AI has been trained on the test
frameworks and we're trusting AI to
write a test and we need now to need to
trust the AI to write the test. Like we
don't trust the test because AI wrote
it, right? And that's the thing. That's
That's where I'm like That's where I'm
excited because AWS with Kiro, right?
They said the ears format or the easy
way of defining
>> Yeah, like
That's That's an excellent point that
you made there. It's like
We talk about trust, but like if we
asked AI today to write us a test suite,
do we trust that this test suite is is
actually a value? I I I can't.
>> But the problem is that this the the
amount of I think the problem is the
volume, right? Because the volume code
output is much higher. So human
reviewers are overloaded. The volume of
generated tests are much higher, so you
can't validate that every test is really
testing what you expect it to test. Are
the assertions right? Is it not putting
in a mock somewhere?
>> Mhm.
>> So then
you're shifting You're shifting the
abstraction layer, right? Because when
you talk like low-level assembly, it's a
massive volume, right? We We used it by
going up in the abstraction layer. And I
think that's where then now with AI
we're able to live at a much higher
abstraction layer, the English language,
and we are defining our user stories and
we're defining our tests in the English
language, right? So things that were not
possible in the past, automatically
parsing English language into validation
framework,
become a little bit more realistic now
because of AI has the ability to
interpret a paragraph, narrate
a text that is easily for a human to to
understand and validate, formally then
convert that into an actual
um rule, like property-based testing or
Gherkin BDD, these frameworks. Which
Which exist for for years and everybody
Maybe you will write them off say,
"Yeah, we tried it, it didn't work." It
We tried it, it didn't work, but it
there was no AI.
>> I mean, Vince, when I when I'm listening
to you, it sounds like
the solution to the problem is AI. Like,
if if only we had AI
>> No, the solution to the problem is to
apply AI to higher layer and then have a
deterministic parsing. Like, we're
parsing a formal language down to
>> But that that higher layer requires
requires agents to to do the uh
>> No, the parsing down will not. Okay, so
so imagine this, right? The way that
Kiro is designed is to to define a user
story. From those user story, like, as a
developer, I want this because of this.
This is a primary reason why I want to
do this, blah blah blah. There gets
functional requirements, requirements.
When I do this, then this happen, like
happy path. When I do this, then that
should not happen, like sad path. These
are very very parsable strings of text,
right? They're in between, they're from
the user story, which is like a
paragraph of text to actual bullet
points of rules.
>> Yeah, but like
>> requirements. They are parsable. The
parsable part means deterministic, means
we don't let an AI generate the text.
>> need you still need some
>> Yeah, but as a human
>> test harness to to to run that and and
and the AI could
>> No, the AI should not be involved there
at all.
>> But how do you How do you How do you How
do you execute these requirements
against the implementation of the
>> the concept of cucumber and gherkin,
BDD, behavior-driven test uh
behavior-driven development. They have
defined a markdown format that can be
parsed down into a validation framework.
So, the AI agent writes English in those
specific strings and then you fail the
parser if it doesn't match that format.
Or you pass the parser and it generates
a deterministic validation framework.
Your role as a human is to validate
those requirements.
>> Okay.
>> And if you fail that
then it then you cannot trust it the the
completely deterministic generated
tests. There's no AI from that point.
>> Okay, I think I need to maybe you should
try conjure an example for me cuz I
guess I'm I'm lacking the trust because
I just feel that the AI is smart enough
to circumvent that in a way. Like
Like for example hear me out here. Like
the way I usually test some work when
I'm working with the AI stuff is that
I I introduce a bug to make sure that
that my test harness sort of caught it
and that and that for me gives me the
reassurance that these tests that were
generated are doing a cap are capturing
the bug right. You know what I mean?
Like there's probably a name for this.
Like I I basically randomly enter a bug
and make sure that
>> Kiro does that and they call it
property-based testing. They This
basically like fuzzing, right? You
generate a whole bunch of variables and
you test the rules, the invariants
against those variables and you
basically generate
like programmatically, randomly all the
possible inputs until you find a
counterexample.
>> I understand fuzzing is. It's just It's
just that the like my sort of review
process is to is to sort of spot check,
introduce not so much like a boundary
problem, more like something more
fundamental. Well, I say that
and I can't even think of a good example
right now. But
that's that's where my my mind is going
when it comes to testing. It's like you
you are given something to review.
There's so much volume as we were saying
with the code and and the tests and then
you you almost have to go in there,
break something just to validate what
this what this
PR is is trying to do or something like
that.
>> Yeah, so I had a discussion with Ion
Murdock about this where I keep saying
like we need to have Gherkin or BDD
because this is the my my one of the
approaches. Like I said over the last
few days I I realized that the
constitution wasn't there, the checklist
wasn't there, the cross-reference verify
wasn't there. That's all AI agent
driven, right? But my original idea was
assuming that they were all there. Even
if they were there, they all failed
because they are
unreliable. So, my
>> unreliable, untrustworthy.
>> Yeah. And so, I always want to explore
this basically what I heard or
understood Kiro was doing. And when I
talked to Ion, he said, "Putting more
rules and formal methodology around
these agents is constraining them too
much. You're just going to get worse
output because the agents are working of
their ability to to I guess have freedom
to execute. And the more you start
throwing errors at them when they run,
the more they get distracted and the the
less they can, you know, perform and
give you the output that you want.
Because my assumption is like I'm going
to just immediately throw an error if it
doesn't pass, right? So, he said, "Too
many rules,
even like too many injecting error
messages when things are not according
to certain expectations is not going to
improve the performance of the agents in
a way. That's what I understood from
him. But I feel like I need to try it. I
also wonder why I haven't really seen
that in Kiro. I saw a lot of
presentations from AWS talking about how
they do this like deterministically
validating.
But I haven't really seen a lot of
actual examples of that.
>> Well, I mean there must be a reason why
it doesn't work, right?
>> It's not a solved problem. It's not a
solved problem by any means. Anyway, I I
got to take my kids to school now.
Anyway, I it's a thought-provoking
discussion. Thank you again, Vincent.
>> I want to try I mean, in terms of giving
you examples, it's definitely my
intention to build like a Gherkin parser
into Spec Ledger. And basically,
whenever the agent writes on the
requirements, run the parser. If the
requirements don't pass, feed it back to
the agent say like, "Okay, great. You
wrote all the specs, but there's a
parsing error on the requirements. I
can't I can't
>> validate this.
>> Instinctively, I'm thinking just like
everything in the world, I feel like
that human that the like a human needs
to be at the beginning and a human needs
to be at the end. There's got to be a
relationship there. There's got to be
trust. And I'm thinking that like when
you take the when you take the artifact
off the production line, there needs to
be that human review before it goes out
there.
>> approval.
>> Exactly. That that QA mark.
>> The human that that that signed the
contract and that's held responsible if
the thing blows up.
>> Yeah, exactly. Insurance. The insurance
policy and things like that.
>> The guy that gets paid $20,000 a month
or more at least.
>> maybe not the insurance thing. I do find
that horribly bureaucratic. I I hate the
whole insurance industry. But yeah, I
mean I
I see where you're coming from. I I
I hope you're right in a way. I hope
you're right. Though at the same time
I'm I'm just thinking to myself that
>> I hope I'm wrong because where did the
human What's human role if if it can be
The human role seem to be the only thing
that generates the ideas and then the
whole process
um is completely AI. There's no Like if
you can completely trust the framework
that it builds exactly according to what
>> Well, let's say there there could be
another test. Like for example, there
might not be like a tedious QA stamp,
but there could be a test a market test
like if this product is good, then then
then humans will will buy it. And that
that is the ultimate test, of course.
Uh
>> Well, I wanted to show something, but
you have to go. I also next time you
have to show me I'm going to watch the
your your experimentation with swamp and
see if I am convinced.
>> Yeah, please do. And uh please ask
Ashani. Okay, so let's just wind this
up. Oh my gosh, I think I need to do an
intro in the beginning.
>> Yeah, you can edit I I have to say River
sounds really nice because the the video
was all recorded directly from my
machine and it's uploading. Quality is
much higher. Audio is really good.
Second thing is it's going to give you
audio channels. So you can mute me if if
you want to talk. I can I can mute I
mean you can I do that with my my
friend. I was saying something while he
was talking or he was talking when I was
saying something. You you mute it. So
individual audio channels. perfect,
right?
>> Okay, I'll do an intro. So, that thanks
for listening everybody and hopefully
this Riverside works really well. See
you. Bye. I'm clicking stop.