Episode 30: Managing context is the challenge

Yeah, so this is episode 30. I think
this is quite a milestone. We've chatted
30 times about AI infrastructure.
I hope not to repeat it too early.
I don't think repetition is a bad thing
necessarily. I I I like I had the
thought this morning our brains are
probably changing and I was just
thinking like wouldn't you wouldn't you
say AI requires some spinning of plates
like I'm hopping in between sessions all
the time which is that the opposite you
know sage advice of focusing on a
particular problem and doing it well.
Anyway. Yeah, I think that's one of the
discussions that we had like what are
the skills that you need as an engineer
in an in the AI future. You need to be
able to like you said switch contexts
more actively, be able to critically
review an architectural design and kind
of predict or know. Like they say on one
side it kills the junior engineers
because you um as a junior maybe you
don't know yet what's going to work and
what fails. I did I did notice that I'm
I'm handing off my brain to the AI a lot
more even in area I mean I guess more in
areas my brain. It's more in areas that
you're not familiar with, right? If
you're very familiar it's a funny quote
where they say like it's amazing like if
I work with with ChatGPT if it goes into
a domain I know very well it makes so
many mistakes but if I'm learning
something new it's amazing it knows
everything and I just can't trust it
fully. What is it What what you said
about switching context I almost thought
you were talking about AI because
because when I when I think of um I know
what you mean you you're you're saying
that us humans need to switch context
but at the same time each one of my AI
sessions feel like if if anything I'm
curating the context you know I'm making
sure that it's got the right it's in the
right directory it's in the it's got the
right claw.md it's got the right MCP if
I was going to go down that way. You
know what I mean like each one of my
folders is like curated with the right
context for it to do the job. Yeah. I
mean that's I had an interesting
discussion a few days ago for an
organization that are looking for like
platform and platform engineering and
and my opinion is that for platform
engineering as we were the enablers for
CICD and DevOps and and Agile that our
role now extends towards AI and so I I'm
having conversations with organizations
where I'm saying if you're looking at me
as just doing you know business as usual
I see and get ups get up. I mean that's
not what I'm looking for, right? I can
do that. I'm going to drive cloud to do
it because I've done it so many times
and it's like short-term I can output
text but long-term it's not a viable
strategy because the model's going to
get so good to get it all of it context
it needs. And I'm not just going to be a
basic context provider finding all of
the like you know connections.
>> provider. Long-term I want to be you
know you know what what what what the
whole idea of DevOps and Agile is we are
a feedback loop. We provide feedback
loops you know continuous improvements.
The whole idea of of DevOps is to close
that loop so that we are able to source
the relevant information and measure
what matters so that we know the
effectiveness and and that mindset of
DevOps you know comes with the Dora
metrics comes with measuring
effectiveness of of teams like in a lot
of organizations we've been relegated to
do pure ops
a little more than a system
administrator and maybe a little bit of
a expert log reader.
>> business as usual is the way I
understand it. Yeah, we are like log
readers. Like people come to us with
errors and we go like you didn't read
the logs. We read the logs and we go
like
>> I can use cloudwatch obviously you
can't.
So so our role has been to surface
information. If we're not relegated to
to a title change that's still the old
system admin Quick [clears throat] side
note when I was at the AWS summit last
Wednesday in London I couldn't help but
think observability was one of the most
popular sort of third-party providers.
You know there there was about like you
know honeycomb data dog new relic um
I think there's a new one called uh
Bronte. There was about there was about
10 of them. Yeah. I think they they need
to reposition themselves as like context
providers for AI, right? Like a lot of
the honeycomb talks from Charity Majors
on LinkedIn it's like well it's what
we've been doing all the time, haven't
we? We've been providing you all of the
data for your services in in operations
and now it's no longer needs to be
serviced to towards humans it needs to
go towards
>> towards the agents, yeah. agents. This
whole context provider it feels like a
meme but it's bloody true. But
I I I think that's an important
evolution, right? I don't think many
DevOps engineers had the opportunity to
take on more of like a platform type of
role of like I create the measurable
like I create the data dog monitors. I
help you define your runbooks.
Ultimately we're not the one that's
supposed to write the runbooks. We're
the ones that's supposed to put the
systems in place and dashboards in place
so that others when they build a
solution can build those runbooks. And
if they didn't then we were embedded and
then we would be the ones like site
reliability team engineers responsible
for highlighting if you want to go live
with this feature this is probably what
you need to be aware of. This is where
we need to put some you know monitors
around and how we we we measure the
effectiveness of the solution. So I mean
the idea has been shared responsibility
and enabling others to take on that
responsibility not taking over the
responsibility from them, right? Our
role was to enable them to to put in
place the systems like observability
platforms like CICD checks that give us
the trust and gives us the ability to to
to
to pull out the information.
>> On that role I mean you you define that
role pretty well I think. Though perhaps
I've done the wrong thing in in past
projects or maybe I just just got roped
into it. Like I can't help but think in
a lot of my I'm just thinking of all the
clients that I've worked with
in my in my consultancy
job. Many times I've not even though my
title is infrastructure
infrastructure lead I think technically.
I I've I've gone into I've wore
different hats in the sense that I've
I've got my I've I've went into full
development mode on some features of the
product, right? And I never got and yeah
I've played like maybe the security
champion you know information security
dude. But the same but like in the job
that I've done I would say that I've
I've [snorts] not had the
opportunity to to wear one hat and wear
it well. Just I'm actually
having to wear many hats
in my typical job.
And hence I've I feel like sometimes the
focus has been lost and I've never
proudly you know
on you know
100% on getting the Dora metrics done in
a way because I was like you know
juggling some other balls. I mean to be
honest for an organization usually they
don't want you to spend time on that
anyway, right? They're like what's your
how are you contributing business value?
Like I mean
If you're being questioned as a platform
team how are you contributing business
value?
>> Yeah I mean that's the age-old question.
Right. Then then the platform team
will will When things go well like oh
what did you guys do? Oh my lord.
Yeah. And then the worst is when you're
most valued if you're inside a startup
and it's failing all over the place and
everyone is depending on you to keep the
system running. Yeah. And
>> I I hate it when you like
you know John was putting out fires over
the weekend like he's such a hero. Like
John doesn't know what he's doing.
But but I think so for for AI future
like when I'm looking at roles now when
I'm talking because I am talking to
companies I I am asking them like I got
an interesting question which they ask
what is it that you want to do and what
is something that you absolutely don't
want to do if we were to like hire you.
>> That's a good question to ask anyone.
Yeah and I said well I'm okay with being
I can drive cloud to to do your Argo CD
migration from helm add-on to a EKS
managed add-on but it will obviously be
cloud and it will obviously be me
generating the plan getting alignment
with you guys and then just having cloud
deliver it and driving cloud and that's
not a long-term strategy. I can do that.
I don't mind doing that but for my role
it must be I would rather be the person
that helps cuz I I talked to a lot of
organizations that said they have these
phases of AI adoption. Initially a
little bit reluctant we don't want to
give it access to our critical systems
then everybody goes all in maybe they
got cursor licenses or they start
rolling out Claude Code or teams
independently adopt these tools anyway
like as an organization you can't stop
them because they are so productive and
they save so much time that teams go in
a wild sprawl and every team does it
differently and then after now 1 year
that was 2025 now 2026 a lot of the
organizations are like we have a wild
sprawl of tools. Some teams are
efficient some teams are giving their
product managers AI access and they are
releasing to production and we have a
huge slope problem. We have reliability
issues. These systems haven't been
properly designed and we need to put in
place guardrails right now. We need to
get that under control. A lot of
organizations are at that at that stage
where like we've gone into a wild west
and we have no control and I'm like well
It's so easy to happen. It's so easy to
happen. I've seen it myself already a
couple of times almost. I've seen it
with my own work in a way.
Yeah I
we have to fail in order to succeed. I'm
worried because you told me that you had
scenario where you trusted it and then
it it ended up giving biting you in the
back. I've had a scenario where I'm I'm
like I said handing over the the brain.
It's doing work that I have like a grasp
of what we're trying to do and the AI is
going off on these massive quests of
we're going to do this thing and that
thing and I'm like, well, kind of makes
sense, but I'm not 100% sure, but
honestly can't be bothered. And I'm
like, this is going to bite me. So, I I
have the one agent It's going to bite
the whole industry. Like, during the AWS
keynote, I mentioned to you over
WhatsApp, I I've got to put the slide in
somewhere in the description or
somewhere just so I'm not making this
up. The AWS partner with a
uh with a local company called Motorway.
And they were like boasting that they
can ship a million lines of code. Now,
with Kiro, they're shipping a million
lines of code per month now. And I mean
The industry is I'm just I'm just so
flabbergasted when people say that.
I mean, a lot of people go like saying
"Swans is key lock a measurement of
productivity." Lines of code has a never
been a measurement of productivity.
>> And like, how AWS stand up and and like
and like slap this guy on the back
saying this is where you want to be.
Yeah, I feel like it feels as a horrible
thing. It's like a broken trust. Uh
it's not cool and it's really really not
cool.
Quite a lot of people in the like the in
the in the classic Agile DevOps uh space
where we're quite
you know, heavy on culture, heavy on
measure measurable outcomes, um on
defining what you want to measure and
validating that they are the the proper
metrics. So, in my circle on LinkedIn,
there's quite a lot of people that are
going like heavily
against what you just mentioned, like
sales standing up and saying "Ah,
they're shipping so much code." Which is
like, what do you mean you're shipping
code, but isn't the purpose to ship ship
features and measure that we're actually
like getting the results that we're
looking for?
>> the guy the guy that was on stage was
like the director of engineering, which
is just just absolutely scary. I mean, I
don't want to throw shade to the guy. I
mean, maybe maybe it is working out for
him, but like I I didn't
get the
I didn't get the the upside of of
shipping a million lines of code. Maybe
maybe they were
>> director of engineering using key lock
as a metric and standing up upstage and
being proud, then I guess you're going
to get a lot of backlash. Yeah, I
believe I I believe I
No, no one like if it was if the
auditorium was full with people like me,
there would have been an audible groan
or sigh. But, everyone was like, "Yeah,
this is amazing." Like, no, this not
amazing. And and I did have a chat with
an AWS guy that I know for some time. I
won't name I won't name names. And he
we were we were talking about in the
context of serverless stuff. And like,
one thing that was amazing about and
still is amazing about serverless about
lambda functions is that it sort of like
encouraged engineering to be small,
right? And I'm missing the word
containable. Not containable, but like a
unit of work that could be easily
understood. Lambdas are brilliant for
that. Of course, you can do what I do is
shove in a monolithic
binary into a lambda context, but
but you know what I mean. Like, lambda
is actually like such a breath of fresh
air in the industry to keep things
manageable. But, I mean
>> it's it's almost like Heroku, right?
Like, the whole problem of of people
building stuff was that how do we get
this in front of a lot of people? And
then if you're scaling, then you have
problems of like, what is my monthly
cost of running this solution? How do I
make it highly available? And then you
have solutions like Heroku and lambda
where it's like you don't need to worry
about that. You just define your
functionality and you get it deployed.
And then you have larger engineering
teams which are like, we build this
monstrosity of hundreds lambdas and API
gateways and we have no idea like we
would have been better off with building
a you know, a monolithic API and
deploying it and that's way more
easier to understand and manage and run
locally and deploy. I mean, there's
there's pros and cons. There's This is
one of the questions they asked me as
well, like from you what have you
learned? What are some of the mistakes
that you would avoid if you would start
now?
>> interview question. Yeah, but like I
felt it's like on one way I was saying,
well, you know, don't need sometimes
these like serverless solutions and you
can get away like you know, the DHH type
of a spiel of like you can get away with
an instance and you can scale out quite
quite strongly early stage with a simple
virtual machine
>> machine under the desk. Yeah.
Um you can ship fast with that as well.
You can avoid, you know, mono poly repo
setups and go mono repo and or you don't
even need a mono repo.
>> because you want the agents to to
develop quickly and avoid all this
costly
>> even talking about like in the AI world.
And then I said, well, and then on the
other side what I've learned because
I've joined a lot of Usually you they
hire DevOps or site reliability when
they're having scaling issues, right? If
they are starting to have more teams,
they're starting to slow down in
shipping features or they have
compliance requirements. And now
suddenly they need to be SOC 2 compliant
or whatever and then they realize that
every team is doing their own thing. One
team's running on Google Cloud with
their
analytical database and then another
team is using SageMaker on AWS and
nobody has any control and now suddenly
they need to run all like control access
to production data and they need to roll
out uh privileged access management
solutions. And how do you do that like
from from um like a platform team
perspective or DevOps? Like, if you're
embedded within the small teams. So, I
said like
>> have infrastructure as code and like,
well, what what is this Terraform stuff?
You got to teach them.
>> So, this was my answer. I started saying
like you need to find a balance between
depending because I was talking to a
sensitive industry. So, I think your
compliance was important for them and I
said this is actually quite a big of a
trouble if you're trying to do this on a
startup scale. But then I I really
regret because over the course of the
conversation, what I really think I
learned after, you know, the call, I
stopped and then I thought like, you
know, what I really think I learned is
that now you have so like, what is the
thing that you usually miss when you go
into an organization? They hire you,
they say "We want to be able to stand up
an ephemeral environment. We want to be
able to test if new feature development
completely individually and stand up
everything." And then you're like,
"Yeah, but do you have database seats?
Do you have the ability to stand up an
environment on its own?" Like, all of
this is usually missing. I don't know. I
don't know in your client engagements,
but most of the organizations I joined
have absolutely no way of creating an
ephemeral environment or doing an
integration testing environment.
And and and I it did kind of come out of
the whole conversation because I did say
like if today with AI adoption and AI
agents having the ability to create an
ephemeral environment or to do an
end-to-end integration test or to set up
your Playwright scripts is absolutely
crucial. And it's also the first thing
and we talked about this. It's the first
thing that you can employ AI agents on
to brownfield environment. So, this is
what I should have answered and I regret
after the call.
>> I hate it when someone asks you a
question and then you know you've asked
answered it wrong 10 minutes or in the
middle of the night is for me rather.
Okay, I need to take my kids to school.
Maybe we can resume the conversation,
but yeah, I need to take my kids to
school now. So, it's a 20-20 minute
call. I'll catch up with you later. Bye.
So, are you having off? Are you having
off, sorry? For the weekend? I mean, for
like Labor Day? Labor Day? First of May?
Next week? I'm not [clears throat] sure,
actually.
I should
UK holidays?
Um
No, I think it's 25th of May is the next
holiday in in the UK. Yeah. In In
Vietnam, 30th of April is King Hung
holiday and 1st of May is Labor Day. I
think in a lot of the ex communist
countries, Labor Day is very important.
It's a very important Russian kind of
celebration.
>> Russian? A USSR type of celebration.
>> Yeah, I sometimes think to myself like
with the with AI and what does labor
mean?
There doesn't seem to be any of that
highbrow
>> question. There There doesn't seem to
highbrow political philosophy being
applied to AI, but like in our industry,
the churn or the uh
the fear is real. Like, people losing
their jobs and everything like that.
It's it's pretty real. The workers uh
rights seem to be not on the forefront
of discussions.
Yeah. They're being
>> Yeah, it's tough it's tough these times.
There was a post on Reddit the other day
of an engineering manager saying he's
part of these organizations that are of
course claiming productivity gains with
the layoffs and adoption of AI, but that
there is a resurgence in hiring senior
engineers because AI turns out to be
harder to control than they imagine. And
then a second thing that he mentioned
was that a lot of the employees are not
happy, are you know, using AI to do the
work because they have to and they're
not resigning because they're afraid
that they can't get another job. So,
they're all just begrudgingly like
burning out and
ending AI. Yeah.
I
There's a part of me that that loves AI
because, you know, I think it's new and
cool and it's really fun. But, as a
47-year-old experienced dev and just I
guess talking with my peers, I guess
that the fear does trickle down. The FUD
does affect me. For me, like I said,
I've had very positive and very scary
experiences. Like, I have half of my
brain afraid sometimes where it I feel
like AI is going off on its own and I'm
like, yeah, I'm not 100% sure what
you're doing, but makes sense to me.
But, I've also had scenarios where like
I told you, there was this report of a
problem and I saw that the screenshot
was shared and I I had a cursory glance
at the screenshot and it was in an area
that I had worked on for redacting
sensitive information from Datadog agent
logs. And I had set up like log
redaction like regular patterns on the
agents to to prevent this issue and
apparently some of the logs were not
redacted. So, I had assumed that it was
in the area that I had worked on. But
the screenshot showed So I worked on
like an EC2 instance deployed agent and
I I told the AI give it all the context
that I have like hey there's these log
forwarding
um regular expressions to detect the
type of sensitive information to redact
it and so on. They're in this area,
they're in that area and Opus went and
said well the screenshot shows that the
agent on the EC2 instance is redacting
the data correctly. The other screenshot
is from a lambda. So it's coming from
the data dog lambda forwarder
integration in your AWS account. Which I
had totally not given any context about.
And it had gone ahead and said like I
can see that you have a Terraform module
that manages your data dog integrations
and it's deployed across all these AWS
accounts. It's straight passing through
the cloud formation stack configuration.
So there's a Terraform variable for
every cloud formation stack input and
there's a parameter that allows you to
set the data dog lambda forwarder
redaction like regular expressions as
well. So not only did it like identify
that I was giving it the completely
wrong direction. Cuz I told it it's
agents deployed on a instances and it
says no it's the data dog lambda
forwarder that's deployed inside your
AWS account.
>> because of of its efficacy sort of
thing. No in this case I wasn't scared.
It's like I said I have scary
experiences where I have it handed it
off and I have no idea and I have
wonderful experiences where I give it
the wrong direction and it comes back to
me pushes back against me and the only
and also goes a step further
highlighting exactly what I need to do.
So we I know and I know this very well
because I've done all of this Terraform
configuration. So everything it said I
knew 100% was correct because I had
written that and carefully cross it like
over the last 3 years. So I knew very
well what it was talking about. I didn't
know that there was a variable for the
cloud formation stack to configure your
data dog lambda forwarders. Obviously
the problem is in the lambda's logging
sensitive details to cloud watch cuz
forwarding them and then removing them
from data dog doesn't remove them from
cloud watch. They're still there, right?
So that's the second And then again the
agent and this is what I was thinking it
helps to know the terminology because
the agent was like as a defense in death
we're going to make sure that the
forwarder doesn't send the sensitive
data. But the real fix is obviously to
go back to the product teams and these
particular lambda's like it even
highlighted these are the lambda's
writing sensitive data. These are the
ones that you need to go back to the
product teams to tell them not to log
this data. Because they're still in
cloud watch. But we're also going to
deploy a fix so that they're not being
indexed. So it's like
>> But but it's getting to a stage where
where it's going above and beyond and
it's becoming the master of us in a way.
It's like you didn't think about this
you know like like you become the junior
getting schooled by the AI. And and and
that's where you you're easily then hand
off. Like in this particular case it
proven above and beyond that it knows
exactly what it's doing. Because I was
very well aware of what is the setup and
I know that what it says is correct. And
I was telling it it asked me a couple of
clarifying questions. Do you want me to
deploy this first to dev and then
propagate it? And I was like yes we need
to propagate it. And then we have like
what I I call trunk based development
with Terraform modules. So we have these
environments using the live module like
from head of branch. Trunk based
basically. So if I'm bumping because I
have two problems. One is the cloud
formation stack that deploys the lambda
needs to be updated to include the
regular expression to mask out sensitive
data so that the lambda doesn't send the
sensitive data down to the indexing in
data dog's third party API. And then the
second thing is I have a runtime. So
this has been left to a little bit on
its own and it needs to be updated and
there's a lambda runtime update that
needs to happen. So I told it we need to
do this as well and it was like yeah but
that's a separate issue like you
shouldn't put this in the same pull
request, right? We need to keep this
dedicated on the on the issue at hand.
And I was like yes but if we're going to
make a change to the lambda to the
cloud formation stack the runtime is
going to be blocked, right? If we make a
reconfiguration it's going to redeploy
the lambda the lambda's going to be
blocked because the Python runtime ran
out end of life. After a certain support
period lambda will block any roll out.
It will keep the lambda's running and it
will not provide any security patches
for them. But if you try to make a
change it will fail. So I mean I didn't
even say it in that many details. I said
we need to update the runtime and then
it says based on the research it did
says like yes you're right because the
runtime is too old and if we try to make
a change to cloud formation it's going
to fail because the lambda runtime will
be blocked. Like we won't be able to
reconfigure the lambda and and we need
to update the runtime as well. So we
will include it into this pull request
as well. So it's always very careful
like are we going to scope our work?
Like are we going to keep it within the
same scope? Because actually Opus now is
very aggressive in like this is not my
problem. This is not related to my
change and I'm not going to make a
change to that. And then I have to push
it. I say actually in this case I want
you to also do that on top of that. And
I like it. I like it. I've noticed that
in my last vibing session where it was
like
like I'm going to set up a different
issue for this and like no and then we
had this little fight. It's like what
the what's going on here? Yeah but I
mean this is a perfect scenario where
the models have become and the harness
is so good that it's giving it system
instructions the model is defensive and
is able to
you know do this in a vibe
vibing manner, right? I on one side I I
have a Jira ticket that records the
issue and I told it like I pasted the
screenshot in this chat and I was and I
know that it basic C4 encodes your
screenshots and stores them inside JSON
as well. And I was like can you take the
screenshot that I sent to you and put
them inside there? And it says no
because there is a leaked password in it
and I'm not going to post the leaked
password on the Jira ticket. And I was
like god damn. So I then opened the
screenshot marked redacted the leaked
password and I said I'll paste it it
myself.
That is crazy.
It is crazy, right? I mean they made it
a lot more defensive.
Okay let's maybe talk about one thing
that came up on my radar was this whole
new lever with with saying
the thinking level. Is it the right
term? Yeah they added extra high high
now in Or they have max now which is
>> Or the ultra think it's been there for a
while, right? If you type it then it
becomes a regular
>> ultra think.
>> Ultra plan is like a second planning
session. I wish I kind of wish that they
didn't introduce these variables because
like because in that blog post where
they talk about how they messed up. I
don't know if you saw that one. What one
of the issues was that they basically
changed the
the thinking level from high to medium
or something. And yeah like isn't there
like an adage when it comes to software
engineering? Like the more freaking
variables you give the more the more
surface area your API has the more
likely things are going to go wrong. I
mean
if they can't get it right like what is
it what is it going to be it's like how
many developer teams are going to be
perhaps just confused and befuddled by
someone who specified extra high versus
high versus medium. I mean it's going to
going to have probably different results
and everyone's just going to be in
disarray. It's going to be a mess.
That's what a lot of people are saying,
right? I haven't read but I've seen it
posted like 20 times on LinkedIn and
Reddit every single one. And Anthropic
has released a postmortem on their
degradation on why performance was bad
and and they talked about the caching
issues that we talked to you community
kind of highlighted on its own. I
honestly I think I've been I've been
vibe coding a lot with it because of all
of this like concerns that people have
been raising. So I've been a little bit
more hands-on with like small little
vibe plans instead of like long long
lift um I'm not giving it a 20 task. One
of the issues was that it was it was
clearing up its context from from
previous memories. But that wouldn't I
don't think that would have affect
affect me with beads. Yeah but I feel
like there's
there's a whole bunch of schadenfreude.
Like people are going overboard. I'm
really tired of the community just
constantly complaining. Like back in
November you were barely able you felt
like you couldn't use it and then you
said December was amazing. Like I was on
a honeymoon and it was crazy and then
everybody has these these what do you
call nostalgic posts about I'm
reminiscing about how amazing my January
February life was. I I don't care. I
think I think it's I think it's very
human in a way to complain. I think it's
not necessarily the worst thing ever as
long as it's obviously constructive. And
I think it has been constructive for the
most part. Isn't it? I don't know. Like
I mean there's a few constructive posts
like where the director of AMD has some
factual data.
Some of the GitHub issues that have been
linked with the Anthropic team responds
to. They have been very technical and
detailed. This is also the feedback of
Hashimoto, right? He says like he's
getting technical reports by people that
never wrote Zig's programming language
but because of the AI is able to surface
all of these details. They're now
getting these super detailed bug reports
and I think Anthropic is getting the
same. But I think at the same time
Anthropic is also pushing back. They're
like well you know a lot of these
extended thinking or chain of thoughts
reasoning are not part of your
issues.json. So you're saying that the
model is like being overcharged in token
usage but all of those thinking tokens
are spent on the service side and you're
not seeing the full picture. They're
having these AIs run loose on client
side information that they have and then
of course the AI is great in making
assumptions and comes up with all of
these stories of why it's going wrong
and then they have to go and debunk and
every single one of them. I mean yes
there's there's signal and there's noise
and I think that that's the hard bit for
Anthropic to go and filter out and
correlate what the community is saying.
And then of course there's a whole slew
of people that are saying well it's all
slop it's all AI code that Anthropic
are doing.
>> to I to
like think the best of people's
intentions in a way. Because as we as we
started, you know, we were talking about
Workers Day. I mean, we're we're humans,
we're workers, we have feelings, and AI
is moving so quickly. It it can't I
think we're in a very privileged
position because we can experiment a bit
in our
>> Yeah. uh AI AI infrastructure field. A
lot of people are perhaps relying on the
Claude code for their business, and then
of course they might just go, "Well,
actually this this makes me think like
some people do complain
at
um
a little bit too ferociously. And I
sometimes think to myself like, "I wish
they gave the context of what they're
trying to do." Like, you know, "I want
to I want a refund. This is balls."
Like, what what what were you doing? I
was What were you doing that was so
important and time-sensitive? I had
three accounts and I cancelled them all.
I'm using ChatGPT now. Yeah, okay.
>> Like, what were you doing that was so
important and time-sensitive? Like, what
were you what Like, are you are you
working on assignment with Like, what
what Like, I see at the bottom of their
post they have that that /
feedback command. I I do I do think as
humans it's like we we need to like have
some norms when it comes to
>> Well, I think what people don't realize
and also if you look at how Anthropic is
controlling access to these
subscriptions is that they don't realize
how heavily subsidized this is, right? I
mean, this is something that we've
always been saying. These subscriptions
we're in an in the heyday of AI,
everything is subsidized. Right now you
get for a certain amount of money you
get a certain service that you know, a
long time was the the best of the best
AI model available. Open AI obviously is
trying to undercut them and is trying to
offer more access, more tokens, maybe
throws more raw power to it to to to
lure you away. You're you're getting all
these Chinese and then Google sometimes
coming in with with with a some some
solution to try and lure away some of
that accounts so that they can talk to
the board of how many subscriptions that
they have and increased the subscription
count. But people are just extremely
spoiled right now and acting like,
"Yeah, I mean, I've you know, I've been
able to build my business with this and
it's completely failed me in the last
month. It wasn't able to do the simplest
of things. And of course it's all AI
slop and I can't do it myself. Not me,
right? That's what people post on on on
Reddit. But like
The thing the thing that bothers me is
that people make it sound like this
Claude thing has enabled them to build a
life support system that you know,
basically help I don't know some some
very critical need. But at the same
time, I don't actually see
or may maybe I'm probably just building
a very hypothetical example here, but
like if Claude code and all this AGI
stuff is so great, what do we have to
show for it other than people just
whining and going mental? Yeah, I mean,
I think this is one of the aspects of an
If you're going to rely on an AI, I
think you have to be realistic that you
are relying on a subsidized token
industry that you're at um a privileged
position. That if it works out for you,
that's great. I mean, obviously somebody
has to hold them accountable. They're
complaining about it's bait and switch.
You're luring us in. Now they removed
Claude code from the pro account level.
>> like can you can you think of a business
that has like, "Hey, thanks to Claude,
now we are you know, now we've 10x." I I
mean, I can't think of a public maybe I
don't know what to say.
>> that I wouldn't be able to build so many
things without all of the Like like I
said, a lot of the stuff that I I kick
off with, you know, spec ledgers and
some specs.
>> like are you you're not like the
greatest is all I mean, I don't want to
put you down, but like like
I'm not making any business or money out
of the process.
>> Yeah, like I I I can't green field fun
projects for me. Some people say they
have like 7K recurring rent monthly
recurring revenue. Yeah, exactly.
building something small with it. I
mean, I think the biggest issue is and I
think what I can see is that those
people have built something that they
cannot maintain because they have
something and they rely heavily on the
performance of the model to be able to
maintain it. And and that gets that's
the point that I wanted you to make or
us to make. They built something they
can't maintain. This goes back to what I
was saying about the AWS summit. When
you're producing a million lines of
code, you've built something that you
can't maintain. On your own like I mean,
at the same time people are saying like
Gemma 4 on an M4 Mac Mini or even some
of the models like Gemma you mean the
Yeah, Gemma from Google that is now a
local LLM that you can run.
>> one you mean? A lot of those self-hosted
ones like even if suddenly all the
prices go up, um you know, you cannot
use Opus 4.7 on an cloud Claude Code or
cloud pro plan anymore. You can you have
to pay max you have to be close to $200
to get access even to a decent model. I
mean, decent if you want to $200 isn't
much in the scheme of things, but carry
on. You're very privileged to say that.
>> [laughter]
>> $200 is a lot for a lot of people per
month just on the AI. That's a lot. But
come on, what is the average hourly rate
of a developer? Not that much anymore, I
would say. I think that's that has
collapsed. I think
if you if you go look at at at
engineering job posts and rates, they
have really really collapsed a lot. $200
a month is not nothing. Definitely not
like in Vietnam or is the
I'm just typing this into Gemini. So
anyway, about like building something
you can't maintain, I do think that's
absolutely a reality for me. I have a
lot of projects where I absolutely have
no idea what is the code look like and I
don't care because my unit tests pass,
my integration tests pass, the thing
works the way I want it to work, and I
don't I really don't care. But then if
the Claude code is this, you know, phase
where I can't really do any work with
it, then I will be in trouble because I
can't really change much about it
because it's so But at the same
time, I believe that if that's really
the case that I cannot get done what I
want to get done and I haven't faced
that issue to be honest. I have always
been able to get it to do what I wanted
to do, so I'm not in that boat at the
moment and knock wood I don't want to be
in that boat. I can't believe I'm not
saying that those testimonies are wrong.
They probably are in those situations,
but um I still feel that there will be a
future because I will be able to use a
local LLM.
Um I hear that their performance is
getting more better and better. I don't
understand why you're
like you're saying that
that you're not dependent
on Anthropic.
>> am dependent. Okay, say we are dependent
on a on a model whatever.
>> I'm dependent on it for a lot of
projects. Not not my day-to-day job. I'm
still confident I can do what I need to
But still, how do you it doesn't really
help you with that whole question you've
built Is it a question? The statement of
you've built something with AI that you
that now you're struggling to maintain.
Are you are you are you saying that
like, "Actually no, I can maintain it
because I can run a
I mean, I I hear a lot of people saying
that they built something and they can't
maintain it. I did not face that
problem. So far AI has been able to do
what I wanted to do. I haven't faced
those issues. Like I said, knock wood. I
don't know if it's because of the way
that I manage the context for it. I
don't know what allows me to maintain it
or if I'm able to engineering like break
down the problem. Maybe I'm a more
technical user. Yeah. But I have not
faced the issue that I cannot use AI to
maintain it. I do face the issue that if
I don't have AI, I'm pretty sure I can't
maintain
>> I see what you're saying. I I thought I
thought you're making a general
statement that like when you build with
AI, you're going to have an an
inevitable maintenance problem. I think
what you were saying really was that
when you build with AI, you become
dependent on AI, and this is why you
need to look at self-hosted models as a
backup or something. Okay, okay. That's
what I said also. I mean, I said that I
currently do not have a problem with AI
getting the things done that I wanted to
get done. A lot of people say that they
have are at the stage with Claude code
that they send one prompt, say hello,
and 70 and 20% of their their 5-hour
session budget is gone. This type of
statements I have not faced.
>> Oh yeah, like anyone who tries I well,
this was a couple of weeks back, but I I
think I I signed up a friend for the
free version, and then of course he said
hello and and something else, and then
that the free version was already gone.
Which is I think why it makes sense they
remove access on those
subscriptions because and I think they
responded to the complaints of the
community because community is saying,
"Oh, you're taking things away that used
to be there." And they said, "Well,
those things didn't exist when those
subscriptions were created. We gave you
a $20 per month subscription."
Originally with Claude code you get like
maybe one or two hours of usage out of
it and then you're out of your 5-hour
session budget with Sonnet 3.5
and maybe with Sonnet 4 and then some
Haiku that worked fine. Now we're on
Opus 4.7, the token cost is so high,
they make it the default model, and they
feel like people are not getting good
enough quality uh out of the product.
And they're saying, "Look, we're we're
just simply not going to advertise
Claude code as as an as a as an offering
of this pro plan anymore because with
$20 a month you're not going to get a
good experience out of Claude code." And
then
I think it's expectations yeah. probably
getting quite high. Like, I was actually
thinking to myself like I'm I'm just
looking at Hacker News here that GPT 5.5
is out.
Uh
I mean, surely we're going to get to a
point where maybe they're not going to
even
tell you what the new models are
or something because well, A, people are
like using it through a back door to do
their things. It gets to a point where
like expectations are so high, do they
do they want to like release a new
version to like make everyone's head
spin or they just want to hide it?
>> They have to, you know, because Claude
just launched 4.7, right? And then Open
AI announces their competitor 5.5, so
>> but I I do feel it's a bit of a charade
or a bit of a show. You don't think
those are like really newly trained
models that are
new capabilities. They are publishing
book benchmarks each time about the code
bench.
It's funny that every model now released
is always saying this model excels as
coding tasks. That's the same with 5.5
again.
>> Exactly. It's getting a bit samey. It's
getting a bit samey, but I'm sure they
are improving. I'm sure they are
improving. Though there's more there's
more to the model numbers now, isn't
there? There's the harness, there's I
think Open AI is significantly
subsidizing their GPT models. I don't
know if they've
I mean a lot more than Anthropic. I
think Anthropic has lost a lot of their
margin with the Department of Force in a
situation and they're having a huge
influx of customers and they're having
an inference problem. So they are not
being able to subsidize as much as Open
AI, which is still having the Department
of War contract, unless they have really
built such a good model that is doing
inference much cheaper and more
efficient, but I don't believe it. I
think Anthropic is like cutting back and
that's why they're making the plans
changes and they're having to compete
and they have to uphold a certain level
of quality and people have been super
critical.
Yeah, that makes sense. It absolutely
makes sense. I guess we're living
through living through a phase right now
and it's cool it's cool. It's cool
nonetheless.
It's just It's just that it's full of uh
I feel like I'm
I'm very out of touch. I'm also like by
default on my on my Claude Code Opus 4.7
is running on extra high and I read the
effort levels and like extra high is
when you have to do some serious
thinking in the midst of an
implementation. Whereas if I'm using
Spec Ledger, I have it run on extra high
to do the research and come up with the
implementation plan and generate a task.
And but then I switch it to lower effort
because why would you need it on extra
high and then it really redefines this
task because when it's on extra high, it
actually goes and says, "Well, this task
is kind of stupid. I'm not going to do
that." So you have to have find those
balances.
I think that the
when it comes to my my own workflow with
AI, the thing that I still haven't
figured out really is that I I'm still
using beads like a caveman. I'm still
using caveman like a caveman.
Are you using caveman? No, no, no. No,
I'm just joking. That whole caveman
thing is a total meme. Now, the thing
that I'm struggling with Claude code
especially is that we you know, that we
do the BD ready and we implement a task
and then even even if it has no
questions for me, it then stops and
says, "Do you want me to continue with
the next thing?" Yes, I want you to
continue. Do I have
>> I just showed you to put this in a BD
ready loop with a while? I mean I just
told you that that
that having So, okay, so beats for me
and I have the equivalent of beats but
in a simplified version without all of
the extra whistles all the bells that
that Stevie got added. It's a little bit
crazy. So I've got a very simplified
version of beats, which is more of like
a persistent task graph across sessions,
which Claude Code by the way also is
trying to build as well. They have the
ability to create task across sessions
as well and Claude Code has task
dependencies as well built into the
harnesses too. But I also had the same
problem where like you said it it goes
off and does a couple of beats ready
tasks and then after a while it stops
and says, "Okay, I've completed this and
I want to continue." But I just told you
I showed you at the start of the
recording it was running for 35 minutes
at that time and when I came back now
after having a bagel, it's done. It's
implemented 24 of 24 MVP task. Five
phases have been completed. It says I
haven't validated it yet.
Like I just told you like at the start
of the call I updated all my prompts to
have it read and so basically the
prompts are like get the task graph, get
the phases and the MVP task that you
need to do.
And then I tell it
No, stop. Then I tell it to actually
create with its to-do tool the actual
task. So it says in its particular
session also because these are 1 million
token windows now. In this particular
session it say it it actually created a
task graph on top of the beats graph. It
knows that after it has it has done
this, the next task is going to be that,
which is just a pointer to the full task
description and definition of done that
it has created. And then it goes off the
task list. It just keeps going. It
doesn't stop because it has a a task
graph and it Do you Do you
Do you Do you spin the task off in an
agent or something?
>> No, no, no, still not doing that because
I had bad experience with that and and I
like your styles your graphs, which are
sorry, your stages slide that you share,
which is like um you start with live
coding plan plan modes and you go
towards like building hooks and like you
know, giving it in like inline like with
system prompts adjustments while it's
working with hooks and then you go
towards agent like agent background
and then you go to agents teams as like
the full master of orchestration. I'm
not at that at all. I'm at hooks. I
should write a hook to do a BD ready.
You've given me an idea.
>> that. Beats has like a whole bunch of
stuff it puts inside your context. It
has like landing the plane. It has like
when you when you do a compact, it's
it's injects a whole message inside the
compact and says, "We're we're doing a
compact, but after that you have to do
the beats." Anyway, but like I said, I
use beats similar thing for like the
full five phase with like three phases
for MVP uh implementation plan, which
includes like dotting the eyes and
polish phase and all that, which is all
extra stuff. But in this case, I
launched the prompt the spec ledger or
spec it implementation prompt, which
tells it to read the spec, read the
plan, the task, everything and it picked
up and it asked me when it started. It
says, "What do you want me to do? Do you
want me to just focus on the MVP tasks?"
And I said, "Just do the MVP task." And
then it created 24
I'm going to do this. I'm going to do
this. I'm going to do this. I'm going to
do this. I'm going to do this and it
didn't stop until all of them were done.
It ran more than 30 minutes. I think
close to an hour and it says, "I have
completed all the MVP. I have
implemented all the phase features,"
which is in this case it's building a
a web dashboard
for like a node graph visualization.
Does your to-do you know
use the full description? No, it it it
it had like I'm going to do task SL1556
or whatever. And then I guess when it I
haven't really looked at what exactly
the content is of the to-do list, but it
says, "I'm going to work on this item."
And when it reads that item, it sees a
full definition of done of each item.
But I tell you right now, it's asking me
to do an adversarial review of the
implementation. I'm telling you this is
actually going to read and this is where
you have Spec Ledger as as a good thing
because I have a very well-defined
definition of done and what I'm trying
to do. And if I run this adversarial
review, 90% it will tell me it says it's
completed, but it hasn't done ABCD of
the definition of done. 90% every single
time.
I'm just trying to summarize this last
conversation. It just sounds like you're
pretty excited that you have got a
better workflow going with Spec Ledger,
which is great, which is great.
>> Yeah, I can show you like
the where it's at right now. Yeah. Well
screen here.
>> Well well it's
I mean maybe
I was thinking of of changing the
conversation to the the other thing you
were mentioning about the or the So it
says here implementation I'll zoom in.
It says here
it has completed all the MVP phases
phases five to nine. Um it has deferred
something. So it's already telling me
that has deferred something, which it
always does.
Interesting.
I don't know what the slices part is. I
think it's because this is a basically a
Terraform state graph visualization
thing
for from fog and it's the serve command
is going to serve a JavaScript web page
that uses Sigma JS to visualize it
because apparently it can render a lot
of nodes like tons of nodes in with
WebGL in the browser. So it allows me to
drag all of the Terraform states around
with all the edges attached to it.
So it has completed all of them.
And one of the things that I really like
is I think because of the task graph and
like handoff type of setup of the
prompts right now is it is very
forceful now. It's forcing you into this
form into this workflow where it says,
"I have you know, the agent says
everything is done. You need to run a
checkpoint." And the checkpoint
is going to do
a full coherence check, but it's going
to validate its own work, which is
always going to be like, "Yeah, yeah,
yeah, it's all good. I haven't done
this, but it's fine." It's always going
to do that. And then if I say yes, do
that. So it's it's forcing me into this
flow. When it has finished doing a full
review of what it has completed against
the code and everything it's written,
it's then going to propose to launch an
agent in an empty context with no
information other than here is the task
list and the definition of done. Here is
the code that has been completed.
Now validate it. And that agent in a
completely fresh context is absolutely
every single time going to be brutal.
It's always going to go like, "You know,
the agent says that everything's been
completed, but this definition of done
is clearly not A few minutes ago I asked
you about using agents, but you you you
said that you don't use agents cuz you
had a bad experience, but of course this
this particular agent the adversary
agent you're now saying to me Yes. is
actually pretty amazing. If I do agents,
I'm not running them that much in
parallel, but I am using more of like or
like a planning agent that helps build
the plan and then an implementation
agent that implements against the plan
and then a adversarial review agent that
validates the work of the implementation
agent against the plan. And you need to
run that one in an in a in a separate
context because again, if you're asking
self review, it's always going to say,
"I haven't done it, but it's okay
because blah blah blah." And the
adversarial review is going to be
brutal. It's going to be it's not done
whatever excuse is not valid. Well, I I
I like to pretend that I have a fresh
context when I when I do a BD ready, but
the fact of the matter is is that I do
keep my context open because it's just
easier to work that way and and probably
I get better results. And I think it's
the same way that you're working. You're
basically
uh leveraging a a large context to to to
uh strike off your your to-do's, which
which isn't um the optimal way, right?
The optimal way would be each to-do it
in a separate agent. But I'm sure we'll
get there and like, you know, one step
at a time, I suppose. Yeah, so the idea
idea originally with with Speck it as
well as it always identifies
parallelizable tasks where it says these
tasks can be done in parallel. They're
in different areas of the code base and
very early on like back in October when
sub-agents were early in Claude Code, I
told it very ambitiously, go and spin
off like five parallel agents to do all
of these parallel tasks.
And every single time it would come back
and it would have implemented uh
a utility shared function over and over.
It would have fought It would There
would have been assumptions where like
agent A is going to work on task A and
agent B is on B, but then agent B would
have made assumptions and like
implemented mocks against task A saying
like, "Oh, the code isn't there, so I'm
going to build these additional mocks to
to simulate this work." That's what I
used to do. I think they've fixed that a
lot. I haven't tried it since. But it
used to do that all the time and it was
like at the time I was not on Max Plan,
so a $20 per month gives me like 2 hours
of effective usage and I don't want to
have 2 hours of crap.
Cuz if after 2 hours it gives me all of
these duplicate implementations with
mocks like
>> to button and say that I'm not I'm not a
huge fan of parallelization. Like
what like the Is there a real need for
that? I I thought the
I think the the rough wagon loop to
doing things sequentially is is fine.
>> See, so it has completed. So what it has
done in the prompt, which is a
checkpoint prompt, which is not the
Claude Code checkpoint, but my Speck
Ledger checkpoint, it has found some
divergences. It says it's testing the
wrong scenario. As I said, 90% chance
there's a whole bunch of issues. Uh and
if you don't run this adversarial
review, you will get crap code and you
will find problems all the time. And it
has this done. It's a self-evaluation.
And it says, "Oh, none of these are
important." But then it also goes ahead
and um says that right now, uh do you
want to launch an independent
adversarial review? And it has actually
I have embedded the adversarial agent's
prompt I I can look at your
your agent.
>> Yeah, it's actually a an issue that a
feature request that I have opened for
myself, uh I mean, for Cloud in the
future, is to actually ship in agents
because right now the the agent's the
adversarial review prompt is baked in
the in other prompt. So this this agent
is going to launch another agent with
the prompt. In reality
>> walk me through this, Vincent. You you
you spin off the adversarial review. It
basically is brutal with the work that's
been done. Now what happens? How do you
rectify the
the the issues that raised by the
Is it Is it just another bunch of to-do
lists? Yeah, the adversarial will come
back with um some probably find the same
issues and then also some some other
ones. Usually it will highlight that
some definition of done or some some
task has been closed where the
definition of done has not been checked
off. I don't know. This was back in 4.6.
I don't know if 4.7 does the same thing,
but 4.6 would just close the just say
it's done, but actually didn't check it
off.
>> the the task. So the adversarial agent
will just come back with a report. So
they they generate
um
>> Oh. They they generate here, but the way
that it calls these reports are I'm not
very happy with, but like it did a
divergence review on this time and uh it
basically wrote this down. I don't think
the adversarial agent will read this.
Could it could be It could that it reads
this and just goes off this.
Yeah, I hope not.
>> like like many teams, you know, I've
I've worked on some teams where
where the the card would go back into
the um the Kanban, reopened. And then
I've worked in some teams where like,
"Nah, okay. I mean, we've done this.
Let's just start a new issue cuz we
found this issue in this after we
deployed it or something like that." So
I guess I guess all this the way that
you're defining your own um you're
defining your own your own workflow here
and and that's perfectly fine, in my
opinion. You're you're And that's what
you That's what every business needs to
do in a way, to find their own find what
works for them
in their problem space. And this is what
you're doing. Um That's what every
business did for the last year and now
have like one team that's able to
deliver decent code and another team
that's delivering slop and has a massive
amount of
quality issues. So I think we're past
that. I think we're past the point where
every team is on their own coming up
with their own workflows and so on. I
think we're at the stage that we need to
start measuring what works and go back
and and and actually do things with
metric like driven and not just like how
much KLOC we ship, but uh how many
efficiencies do we have and how many
times did we have to roll back a
deployment, you know, that sort of
stuff.
Yeah, but but but but there's It's got
to be uh the
Yeah, I'm with you. I mean, there's
there's some maybe some top-down things
you can do.
Here.
I mean, wouldn't you say wouldn't you
say, Vincent, that that a lot of teams
need to to find their own way? They need
to go They need to adopt AI from a
grassroots level as well, you know.
Yeah, absolutely. And I think that's the
most um important thing. Like if you're
in a platform team, the biggest
resistance against a centralized
platform team by most product teams
within an organization is like, "You're
going to guys You guys going to be the
bottleneck. You're going to tell me what
to do. I want to be able to do my own
thing. I want to be able to explore this
framework and this tool on my own." I
mean, every single organization to have
that tension between um
we have a certain way of doing things
that we want to standardize so that
people can move between teams and and
things are aligned so that we can be
better in compliance. And on the other
side, everyone is free to explore the
tools and find the best way of working
and
>> Yeah, there's there's a real conflict
there, isn't there? Like I've I've seen
it
uh on a on a couple of clients where
like where
like so someone with the best intentions
like, "Hey, I'm coming up with a
framework." And then and then he
>> pushback. or she inevitably says,
"Everyone needs to use our framework so
that we we pick the Jira card up and we
implement it to our coding standards."
And then like
and then people going like, "No, I don't
want to use your your framework. I want
to do this my way."
>> I mean, this tension's been in DevOps
for ages, right? And they always say
DevOps the culture of DevOps is a
bottoms-up and up-top-down. Whereas you
build the best tool and people adopt
them because they're good, not because
they're forced to use them. So that's
part of the idea of DevOps culture is to
um you know, find buy-in and bring those
tools that you want to build with for
for the other people to benefit them.
That you work with them to to to see
where it fits in.
>> to
to to do that approach, right? Because
there's so much tooling, there's so many
ideas there that that top-down like,
"Hey, use Speck Ledger in our company."
It's It's not going to work, is it? I
mean,
Yeah, absolutely. Um
I It's a tough one, isn't it?
Yeah, I mean, the way that I see it
We're going to go through some sort of
trough of disillusionment or some chaos
before before uh real practical
standards really emerge in a way. What I
hear the most in organizations that are
adopting AI heavily is that they they
have a whole framework, right? To to
basically share brown bags, shared
repositories, skills registries, agents
registries. We are sharing an agent that
is able to pull all the context needed
and that is able to help you with your
work. We have agents that can live where
you live, inside Slack, inside
Confluence. And we are making these
available across the organization and
it's up to you to use them or not, but
we're advocating for them and we're
we're promoting them across the
organizations and we're doing brown bags
and we have
um an actual effort. Uh but I guess if
you do that again, you're going to have
the same
That's always a problem with the
platform or an internal workflow team.
It's like, "How are you improving our
workflow?" No matter where you work,
they're always going to ask the question
and that's where then you can grope
grasp at DORA metrics or similar and
say, "Well, we are gathering all the
information." And this is actually
something interesting because I have
some friends uh that I used to work with
and they're saying, "We've we've got
backstage now. We didn't have it when I
was there. Um we've got backstage and we
have set up Spotify
>> Spotify backstage, right? Yeah, Spotify
backstage, which is inherently
considered very hard to to deploy
because it's a bunch of TypeScript
plugins and they're saying like, "It's
you you There's a lot of like this
Roadie and other hosted backstage
solutions that are saying backstage
sounds great, right? It's like
self-hosted.
But once you deploy it, then you need to
build all your plugins, then you need to
integrate it within all your systems. I
think today with AI, that's not an an
issue, right? Integrating it with an
existing system and go and find the
information where it lives is so much
easier with AI and to have it built a
TypeScript extension into backstage is
so much easier with AI. You can
literally can give the task to an AI
agent, it goes and builds the thing. And
and so I think today you don't face the
same problems that you would if you were
to roll out backstage 3 years ago where
you need to be a TypeScript expert, you
need to learn how to write backstage
plugins, you need to know how to connect
them to everything.
>> was was to be an internal developer
platform, a dashboard that you can
connect all the information to.
standards, right? Yeah, the idea was to
give it a remove friction. To deploy a
dashboard that that you can connect your
DataDog dashboards into, that you can
connect your Confluence space into, that
you you connect your GitHub resource
into, that. Every team, whatever tool
they're using, can integrate into it.
So, no matter if you're part of the
team, you can go and get like a little
dashboard overview of like, "Okay, so
this is the This is the commercial
platform team, and they have their
source code live across these
repositories. They are in the adoption
of Python 3.14. Uh They have mostly
upgraded their code base. They're about
70% of adoption because we can see that
as the metrics are being pulled out of
repositories." So, this If you have a
internal developer platform like that
Backstage, which a long time people have
said this is the future of platform
engineering, you need to have an
internal developer platform. I think
today with AI, you can then connect MCPs
for it. And I was like, "I hate MCPs."
But then my friend who then he says
like, "Well, MCPs can be remotely
hosted. Therefore, everyone always gets
the up-to-date information. You need to
You don't need to distribute your CLI
and your skills to every every person.
An MCP can be a remote server that you
remotely update, and then you connect
that to your Backstage, and then
everybody can just immediately via their
agents, they get the context for this
particular team. They know where the
code lives. They know where the DataDog
alerts are. They know which workflows
matter for deployment." I think you're
painting a picture of
of an enterprise having a harmonious
coding standards driven by some
dashboard
>> path page.
I think that's the hardest bit bit about
golden paths is like it is no Nobody
wants to adopt a golden path exists.
Like there's no purpose. It's very hard
to build a golden path. And once you
have it, and once you start deploying
and managing your your your product
efforts with golden paths, you just have
to update the golden path, and then you
can keep rolling it out. So, the picture
that I've been painting is a picture
that be painting to me by people that in
an organization that I've left. And
they're saying that this has been a very
successful adoption of AI and and MCPs
within their organization. And I do not
know the details, but this is what they
say, and I can see that that that's
something I can play devil's advocate.
I'm I'm
almost of the opinion
Okay, I'm just maybe opinions aside,
couldn't we expect a world of chaos
because a lot of people have strong
People are going to probably get
stronger opinions in order to deliver
with AI.
You know
and with these strong opinions, there's
going to be a conflict of
of settling on to
a golden path that's going to apply to
to a team of teams, you know, like
there's going to be conflicts because
maybe the team doesn't want to work with
the other team because like they know
better or they want to do things
differently.
I do feel that agents that AI empowers
at least myself to do what I want to do
and and take up and take up more scope
and be more ambitious. And now that's
going to be a conflict in a in a typical
enterprise because
because
there's going to be
you know, like we we want to do this.
And I think the conflict, I'm not sure
if you touched on that, but the conflict
on that I also heard from other people
is that they went all in on AI, and then
they are generating so much code that
there's a bottleneck. Um and I think we
mentioned that before as well.
>> going to be
a bottleneck in terms of like review.
Like you're putting out too much code.
We don't know if this is
>> Exactly. The There's the PR like like
PRs are are well known to be
problematic, and you can't expect
uh
stack sets or whatever to save you,
right? A lot of PRs are going to
If If you expect a PR to be the way to
distribute knowledge in inside an
enterprise, I don't see it really
especially in the volume that agents
produce. Yeah. I can't help but think
we're we're going to head for quite
tough times, really. But But this is
also where I don't I think Codium just
raised like 10 million
I don't know how many on maybe a I don't
know how many dollars, but a lot of
these AI agentic code review systems
now, they are raising tons of money
because And I'm using one for SpecLedger
as well.
Um whereas like I just now I did the
full close to an hour implementation of
the MVP path, and it highlighted that I
need to do a manual verification because
there's no Playwright script. I think I
explicitly put because I'm building a
browser and a web app, and I put
Playwright as a deferred task for
another feature to to do full end-to-end
testing. So, the AI is blocking. It
says, "You need to validate it." Um but
then I've done the adversarial review,
which came back and said, "There's a
couple of like I don't think the
adversarial review actually came up with
a lot of like high findings. I'm
surprised." It says like, "There's a bad
equality test missing according to the
data model. You should have a specific
test that you had that hasn't been
built." And so on.
Uh but then what I usually get from
Codium
And it's really good. Codium Yeah,
there's quite a few. There's like Grapp
Tile, Codium.
Um
One of them is doing a a ton of
advertisement, so that always makes me
super suspicious of their actual
solution. If you If you need that much
advertising to sell me your solution,
it's probably I'm sure there's
going to be a lot of invention with with
new tools
Where is it? Oh, here it is. No, but to
be honest, I think this is the only
solution. I mean,
we're going to Humans are clearly the
bottleneck. We're going to have a way
more output of code generated
PRs. Like you said, there's going to be
a huge battle. People are are going to
complain about the quality. I definitely
have a lot of like complaints when I
have somebody else use AI and send in
code that I do not trust their code.
Whereas I generate code and I trust my
code because I believe that I've put in
place the mechanisms to catch the
problems. But then
as I use the product, I find a lot of
errors down the line as I because I
built it for myself. And then I said,
"Oh, I thought I I I thought I I said it
specifically to do it correctly, and it
did it wrong."
>> [laughter and gasps]
>> Be extra careful mean on mistakes.
But But
Yeah, we definitely cannot trust the
output of our peers and their AI.
Okay, say you have an amazing amazing AI
uh
way of reviewing your code. But still
the problem remains that that your team
is going to potentially be unaligned.
And I feel like no one's No one I I keep
mentioning this like a broken record. If
I'm sure I've mentioned this hundreds of
times before now. It's like I I still
don't understand how you're supposed to
collaborate with your team. The teams of
anything need to be smaller
to improve communication.
And If If you If you're working in a big
team like five, seven people banging out
code with AI, it's I don't see it
working. Maybe I just lack the
imagination.
So, where people are That's definitely
the million-dollar question, right?
That's the problem that everyone's
facing. We've heavily adopted AI. Some
people stand up proudly that they've
been
shipping tons of code, which we disagree
with the metrics that they use.
Um but
a lot of them are coming back from that
saying that some teams deliver quality,
other teams do not. So, we need to find
a way to align the teams so that they
can deliver the same amount of quality.
How do we validate the quality? We've
talked about it several times. Do we
find a way to have executable specs like
behavior-driven development?
Um
you know, using Cucumber to define the
what what we want Are specs formally
specific enough to capture all of the
demands so that we can execute them and
that we can have enough trust in into
the code being built? Are we becoming
too reliant on AI? Are we unable to
maintain it because of the
stupid has generated?
>> Surely you would consider some teams
might want to do spec-it driven
development. A spec-driven development
uh driven development. Some teams might
want to do like test-driven
uh driven development. Some Some teams
might want to do
some other workflow, you know? And there
will definitely be an a battle on some
teams arguing that their way is better,
and they can trust it better, and they
don't want to adopt the other teams'
way, and you're going to have a a
divergence across the organizations on
how the code needs to be validated. It
will also that have very strong depend
on the industry and the level of risk
that the industry that the company is in
that it But we haven't even talked about
the recent leaks. Vercel being
compromised because of What is it?
Context 7?
GitHub being compromised and then
attackers being able to exfiltrate all
of the Vercel source code now being sold
on the black market. Lovable having of
complete
leak of their platform.
Basically, if you share or if you
if some tenants make made some code
public, then there was a way to
get access to other tenants' private
code or something. There was a leak. And
I have to say, I had the exact same
problem with SpecLedger
after I introduced a new feature where
it assumed a certain level of
permissions or a certain level
authorization against the user. But
because of the new feature, those
assumptions didn't weren't right
anymore. So, even though the feature
spec and the development was all good,
and it worked, and it passed the the
validation like our manual verification,
it turned out that it changed a
fundamental assumption in the security
model. And as a result, um you were able
to
to bypass some authorization um guards
on on the back end on the Next.js app.
So, this type of problem can sneak in so
easily, and it happens in Vercel, and it
happens in Lovable. Lovable being
billions of dollars and used by uh I
don't know, huge companies um to to
build their internal platform uh
solutions quickly. And there's a clear
security problem. There's a clear
quality problem.
Um and there's a clear Yeah, um
problem. Well, the If you can solve the
quality problem of AI,
the thing that concerns me is that
there's going to be a lots of problems
that require some
But most of these problems have Most of
these problems are valid with users as
well with humans as well, right? I mean
agents are just slightly better humans.
>> you know, I'm not a huge fan of using
Jira and sprints. Don't get me wrong,
but like there's there's an ebb there's
a flow there's a familiarity with that
sort of way of developing software and
now we're going to go into a stage where
where there's going to be a lot of
conflict no doubt and and essentially
people are going to lose people probably
potentially going to lose their jobs
because people are going to say now I
don't want to do it that way basically.
>> I mean
I think I think it's true that
because of the speed at which AI works a
lot of the mechanisms that we have
because I was about to say we have all
the mechanisms in place with human
review and with human quality control is
the same thing. You must have your CICD
security checks shift left whatever
practices that you had you have to still
apply them.
But the volume output of AI agents make
a lot of these original frameworks too
much of a bottleneck that a lot of
organizations are bypassing them or are
shortcutting them
or are just simply ignoring them because
of the bottleneck that they are posing
and they want to move fast and AI is
doing has passed has been doing a good
enough job long enough
>> Yeah, like if if if you trust it.
If you said to me
that
you we're we're enforcing some
organizational standards and the
standards would be something like you
know, we we're going to do it like a QA
phase and things like that. I can get
behind that. I can't get behind when
people say okay now we're going to have
a
everyone use you know
you you know one two three QA or some
some QA tool our QA tool like to do this
thing. Like I I'm not too sure I can get
behind that.
You know what I mean? It's like the
difference between
sort of like a high level in principle
what you want to do as opposed to like
hey, here's some tool that now we think
should be using now. Here's a model that
everyone should be using. It's great.
It's called GPT 3.0.
>> [laughter]
>> And that's and that's the standard that
everyone needs to use. It's like oh no.
Did you see also from Uber that they are
like they blew through their AI budget
and it's not even half half the year
gone. They're like fire all the staff.
They put in place a bunch of AI
workflows and halfway down the year
they're like we threw our budget. We
cannot afford the AI tools anymore. Like
we have to rehire humans.
>> [laughter]
>> Well, there's
there's lots of talk about yeah token
maxing is the is the meme, right?
Where you have leaderboards and
companies.
Yeah, this quote to quote from Jensen as
well saying if you have an engineer and
you're paying him 200,000 dollars a year
and he's not spending 50 like I don't
know what was the number.
>> Oh yeah, that's the other one.
20% of this annual budget on In the
enterprise in my my experience people
are struggling just to get access still.
People are just struggling to get
access. We we have to like pinch
ourselves Vincent but and and and who
anyone who's listening we are pioneers
here. We are pioneers. I'm I'm
I'm sorry to say put the put your FOMO
aside. We are basically pioneers at this
point.
We're
So just chill. Just chill a little bit.
Chill. Take stock.
I thought you were going to like when
you said we are and then I would like
the huge Invincible title screen would
pop up.
I don't know I've been watching
Invincible and so one of the things they
do is at the start of the episode they
will always frame a sentence and where
it Invincible is then one of the like
word plays that completes the sentence.
Somebody would say like you think I am
and then
Invincible pops up and say we are
pioneers.
>> Is it a Disney thing or Oh yes, it's a
Disney thing. It's a very gory it's a
bit like The Boys on Amazon Prime.
Invincible.
>> catch a series called
the Mr. In-Between on Disney.
It's really good. Mr. In-Between on
Disney? I stopped paying for Disney. It
was it wasn't worth it. It's
it's quite cheap here with the with the
ads. It's like five bucks. Okay, let
let's let's end it there. I I think
we've talked for a solid hour if not
more and I need to session alone and you
have two more of 20 minutes each so
that's like 1 hour and 40 minutes. I
need to
Have you ever done AWS certifications? I
need to do mine next week. Oh god.
>> Can't you let AI do it for you? No, you
can't.
>> You can you can ask Claude to build a
battle cards for you. Like in just all
those material. I've been actually
vibing
um
I've I've shown you command and control
on my mobile, right?
>> Flash cards not battle cards. Command
control yes.
I've prompted it just to just to just to
test me and I've been doing tests like
that on my mobile which is nice cuz the
tutorial Dojo online is is on the mobile
web is a mess. Okay, let's end the
conversation there for now. I thought I
thought it was good. I mean
I feel like we're just rehashing old
topics sometimes but I think that's the
part
>> in the way that the ecosystem is
constantly changing and some of the
topics we've raised just prove to be
true over and over like all of these
Yeah. We didn't even talk about the
supply chain attack now against check
marks and it even impacting Bitwarden
CLI which is like the secret management
solution now being impacted by a supply
chain attack. Today my my GitHub token
just expired and somebody was like I
know reminding me and I was like I'm not
giving a long live token. I don't want
the project to die because of a supply
chain attack. I want multiple small I
still have to get rid of the token
though. Yeah, it's funny. We had that
conversation a a few episodes ago when
you were talking about
the NPM what do you call it max age and
all that sort of stuff and I've just
been seeing in in my internal stuff like
how how people are just cottoning on to
this practice. I was like okay or you
know, some people are talking about it
but it's great because
uh you
Renovate bot apparently also has this
now as a flag that you can you have to
or you can set up your renovate to not
accept any updates that are younger than
a certain amount or a certain amount of
days on the on the registry even for
Terraform modules apparently. So that's
interesting. I mean I still hate
Terraform modules as a as a mechanisms
for your infrastructure as code but as
long as they're around it only makes
sense that people keep duplicating any
sane package manager features and
rolling it out out into this stupid
niche infrastructure as code packaging
mechanism. I I hate Terraform modules.
Okay. Why why do you need a like a
separate package manager for Terraform
modules and then you have to duplicate
all of this stuff that you just
mentioned against supply chain attacks?
Yeah, yeah. Okay, this is kind of like
premature defense because somebody was
saying like when it hits when the supply
chain attack hits on a Terraform module
at least you should have this already in
place before it hits, right? That's the
point. Anyway, there's lots to do lots
to talk about lots to talk about again I
feel though. Anyway, thanks thanks
Vincent for your time. I'm going to wrap
it up here. If you want this far please
like please subscribe please review
please email us about something you
would prefer us to talk about and yeah.
Toodle toodle pip. Bye.
>> Bye.