Episode 32: Where does Spec Driven Development fit?

And all this stuff it happens that you
you remember it just enough to to pass
the certificates. That's why I'm
personally I had this experience of
getting or being required to get like
five certificates in 3 months and then
each of those certificates expected like
3 to 6 years of experience and I was
just an intern. Not an intern, I was
just on probation and condition to pass
probation was to pass. And so some
people joined before me, they had
prepared exam cram guides and downloaded
a bunch and then on one of those
certificates I got like a 100% like a
complete passing score and I was like
they're going to be revoked. They're
going to
>> [laughter]
>> they're going to you know claim that I
That was actually the most interesting
one. It was Microsoft SQL Server 2005 uh
certification.
I remember now you're describing your
past when you were like uh
Yeah, and it's the same thing because to
be a Microsoft gold partner you have to
have a certain amount of people within
the organization that have those
certificates. Yeah. I mean it's 2026.
It's the same with AWS. I'm pressurized
to do the certificates anyway. I mean
the problem I learned from that is that
I never trust someone who has a
certificate. Like if somebody on a CV
puts certificates, I will usually if
it's in something I'm knowledgeable
about, I will make their life hard on
those topics. Well, fair enough. I mean
and it's
On certificates I do think
they're more useful than they're not
useful. And
and to be honest I I do encourage my
colleagues to get it because you do
learn things that you might not
ordinarily know. Um it's a bit lame but
I didn't even know about uh
I didn't know about
CloudTrail Lake until it was in the exam
question. Yeah, I don't know about it
also. But then what do you use it for?
Is it like like a data lake for your
audit events or is it like a Yeah,
exactly. So you don't have to Previously
I think you just used to dump everything
in S3 and and try to make sense of it
with Athena.
Yeah, but night
At one of my previous organizations,
they set up a huge Elasticsearch cluster
to ingest all of the security events
data. And to be honest, they just took a
off-the-shelf GitHub Python repo. It's
funny because we were doing Terraform
everywhere for 3 years, and then we
hired two people to do security with a
new CTO, and those people go off and
they do absolutely everything on their
own, completely in isolation. And they
like under direct supervision of the CTO
to build all these dashboards. And like
well, that's okay, but then they do like
integrate with our Terraform. So, they
set up all this account, we need to set
up a Kinesis stream endpoint to to, you
know, route all the events. But, I mean,
they they impressed me when I there was
a an US East 1 login outage, and I went
around the back to get like access via a
I don't know, I I think I use an access
key that is stored At the time, that was
like more than 4 years ago, we still had
some AWS access keys in some services,
and I just went to the secret store,
used the access key directly, and
started doing my stuff there because I
couldn't do IAM logins. And I
immediately got a notification on Slack.
Yo, Vincent, there's some unusual
activity on this access key. Is this
you? And I was like, oh, wow, nice. So,
it worked. Yeah, yeah. I didn't really
know about all the security tooling like
Security GuardDuty, Security Hub. I do
feel it would probably They have all
these like like out of the box stuff
that will help you catch that sort of
stuff.
>> And with
Another thing I didn't quite know about
was that, you know, when you have an
organizational unit in your master root
account, you can there's a command in
AWS Organizations to delegate the
account, so you can delegate
the security account and the IAM
account.
>> Yeah. I didn't actually know about
>> I know that people set up accounts to do
that, but I didn't know that you can you
can delegate it straight from the the
root account and and and in that way.
So, Yeah, we we have a few things like
Like um we have a shared services
account where we run all of our like uh
workloads like Atlantis and and like
even if you use CDK and you CDK
pipelines, it will recommend you to have
a DevOps like pipeline account and it
will provision the CDK pipeline into
that DevOps account and then it will
create trust relationships uh into your
across your other accounts and your
regions. So, we have this shared service
account and we have like some delegated
for resource manager, I believe, and
also to to aggregate some of data with
control tower. Control tower does a lot
of that out of the box. But yeah, I mean
what you said, sometimes there's some
services and if you keep them on the
root like the management account, it's
not a very good practice and it's much
better like they actually recommend say,
"Hey, you shouldn't do this in here. You
probably need to delegate this to
another account." Yeah. There's also a
feature I didn't know about where you
can basically
stop the root root account access.
And uh it forces you to do like a
temporary root access. I I can't
remember the name of the feature, but
there's a there's a there's quite a few
things. But that's accounts root, right?
It's not management account from the AWS
org. Yeah, but still like you don't want
root account Yeah, but it's different,
right?
What do you mean when you say root
account? Because in in AWS org, you have
a management account which is the
account that you use to manage your org
and root, what is root account in an AWS
org? Root is the like maybe um So, you
mean the root user? When you set up a
new account, you have like this master
and Yeah, you should absolutely disable
that. That's like like we have that
disabled.
I think it's probably AWS config rules
that I I'm brushing up on all the
security stuff because my next role is
more security sec sec eng. So, my new
role uh which I'll be starting next
week. We'll be administering
um
security preventions, monitoring, and
all that sort of stuff on a on a large
organization, making sure that nothing
untoward happens. With with a focus on
AI tooling because
they want That's why we
That's why that I'm was kind of keen on
the AI guardrail stuff. So, I probably
will continue working on that. Although,
and and that project is like the first
project I've used Spekit on or Spectrum
development. And yeah, I do as I
mentioned over WhatsApp, I I have I have
some questions or some concerns or like
I'm a bit I shared I mean, I shared them
with you, and I've also shared my
Spectrum development questions with my
colleagues. And I feel like I get the
same answer from you than with my
colleagues. Like, "Yeah, that's Spekit.
Have you tried this other STD
accelerator? Have you tried Be Mad? Have
you tried Get Things Done?" It's like,
"What?" I think generally it's the right
advice. You should try many. And also,
Spekit is has not been maintained. And
then, some of the like information you
share with me, it's clear that they are
changing things in Spekit without
updating the docs. So, the original
person building it is no longer with
Microsoft since like, I don't know,
since November when he joined Anthropic
to work on MCP specifically. And since
then, it has been in limbo for like
several months. And I guess in the
lately, like maybe in the last month or
so, they started adding new stuff to it.
Like, they rewrote all of the prompts
into skills. I think that's a pretty
interesting idea. I mean, all honestly,
Anthropic says that the commands /
commands and skills are kind of the same
thing. And you see Spek ledger also
Sorry, you see Opus or the agent shell
loading prompts as if they are skills.
>> commands are way more implicit. Like,
you like this is what's happening.
>> They've been completely If you go to the
official Anthropic docs, commands and
skills are the same thing. If you run a
/command, it loads the the skill
of that command. If you mention, we're
going to run the Spek ledger inspect.
>> Okay, so it's going to go and load it.
>> You should it's the same. You just go
slash okay, okay, fine. For the agent,
the idea of slash command has been
deprecated. It's no longer a thing.
They're still available but
>> I thought you were saying that that that
the
the steps in spec driven
development specify the plans, the
tasks, the implement is now skill in the
sense that you don't actually have to
explicitly type it, but it sounds like
the command is just hooked up to the
skill. Yeah, the agent sees the commands
and can load the command on demand. And
I I see this is very nice in in cloud
code with spec ledger. I think the front
matter of the command because the
implementation of Claude Code and the
implementation of GitHub co-pilot, they
at the very tail end of spec it, they
added this idea of like handoffs between
agents. And they even have this concept
of like in the because I watched the
videos of how he uses it with within VS
code with co-pilot and he has the chat
interface and it's very easy for him to
say, "Now I want to hand off to this
next command." And at the front matter
you have like if you run this command
then the handoff it next is that that
next step that you need to do. And I
just copy that over and when I ask cloud
code, you know, with the onboarding
actually with spec ledger, which is a
skill that we added when you in it a
repo, if it's a new repo, it
automatically triggers the onboarding
skill. If it's a existing repo, the
onboarding skill does an exploration and
then suggests the principle for you. It
will look at your code in the way that
you organize things and then it will say
based on like you know, don't repeat
yourself or write everything twice and
you're you're like it will it will take
a couple of those patterns that it sees
and it will say, "The principles that
you might be aligned with are these."
And that's the onboarding flow of spec
ledger that that spec it doesn't have.
But the same with in it, the in it
onboarding flow because of these
handovers between the prompts and
because if you use with Opus 1 million
token context with spec ledger, you get
this very smooth like, "Okay, we've set
up the constitution. Do you want to
build a new What do you want to work on
first?
The point of a handoff is to refresh the
context, right? To clear, right? So that
you have But not necessarily. It's It's
more like what's the next step, right?
We We We've done the constitution. We're
going to work on our first feature,
right?
>> the handoff is is is the is the sort of
line in the sand between a specify and a
plan and a Do you Do you have the Do you
have the same steps in Spec Ledger that
the specify a plan? Yeah, it's pretty
much the same except one person that
like to use Open Spec renamed one or two
things. Like instead of
instead of analyze, it's
or clarify.
>> It's clarify. Yeah. Or clarify is the
same thing.
>> Spec It is clarify. Then there was
something that was verify for a
cross-check. I don't know. He renamed
some of them, which I wasn't a huge fan
of. I was like, but people come from
Spec It. But I think more people use
Open Spec. And it's funny because I I
explained everything to a friend of mine
and then he announced this to everyone,
"Oh, I'm doing a workshop on how to do
spec-driven development." And like he I
was like Yeah, that was it.
>> I was learning it.
Exactly.
Anyway.
>> Yeah, I've seen things like that before.
It's not the first time.
>> Question Question Question time for you
is like if you have like a if you're
working on a spec-driven project and
there's a small change that you need to
make, do you go through all the steps?
Because there was like a bug in my AI
guardrail uh guardrail thing. It was
quite simple to fix. I was like, I could
even fix it by hand, like a cave man.
>> Yeah, absolutely. I've said that many
many times that I
that I have two ways of working with the
agent, right? Often because I'm working
on Spec Ledger using Spec Ledgers, then
while I'm working, I see, "Hey, this
doesn't work well." And then I ask the
agent, "Can you figure out why
it didn't do this? I expect you that
this would happen." And then the agent
says, "Well, the prompt said this and my
context has that. So that's why I didn't
do it. And then I say, "Okay, go file a
bug, right? Go file a bug. We're going
to address that. Uh let's continue with
the work that we're doing." And then I
will often open a second cloud uh for
what you just described, like there's a
small little prompt template issue or
the command isn't outputting something
that confuses the agent, causing it to
to do something unexpectedly. So, I just
open a second cloud session. I say, "Go
create a Git work tree and and then
using the Git work tree tool, and then
address this bug." That's the bug
number. Okay.
>> Just in plan mode. [clears throat] Just
in plan mode. No specifier, no spectrum
development. You do it in parallel.
Yeah, cuz it's cuz that's the next thing
I wanted to moan about was that going
through spec uh
spec it's but you know, specify, plan,
God, clarify. It takes so long. It takes
so
long.
>> I used to I used to tell it like even in
plan mode, I would use to also indicate
like, "Let's look at the impact." If the
bug sometimes it looks superficial, but
I ask it, "Go and investigate how much
of the code base needs to be modified
and what is the huge like how how big is
the impact of this bug?" And then it um
and then I might decide that this is
quite big, and I actually might want
this to be more of an STD flow. So, you
can easily evolve like going from a bug
planning phase and say, "No, I'm going
to make a a branch for it. I'm going to
create user stories and and get a larger
buy-in." And and the And the reason for
going for this bug workflow is for
speed, right? Just to get it done.
>> And then the one of the things I did was
I used to specify work on the bugs 1 2 3
4 5. And then it would create this
massive fe- uh massive feature, like
user stories, this bug needs to be fixed
because as a user this and that, and it
was this massive. And then I would it
would never get done. Like I would ask
other people, like, "Do you agree with
the approach for these bugs?" But a lot
of them were very straightforward. Like
it's a very easy bug. Uh so, there's no
need to get like a full alignment on
that. So, so then I was like these
branches affected too many areas, became
too big. And I think that's the biggest
danger with spectrum development is that
you go in these long-lift massive
feature branches. Which is great if
you're starting from scratch.
Which is a little bit less great if you
like if you're bootstrapping something,
the first commit you have to set up the
whole repo and you have to build out an
MVP and you have to have a you basic
testing. So, you can do this huge flow
and the branch can be quite big. You can
try and cut it down like, "Hey, I really
want just to have the web app with the
minimal interface and and and and all
that." Sometimes just even I want the
CLI with this input schema and then I'll
work on the web app later. So, it's very
dangerous to to go into these massive
feature branches that never land.
>> That leads me to my next issue is that
yeah, it's easy for the branch to get
quite big. And then, of course, the PR's
going to become massive if you ever had
to approve PR's in your in your in your
project. So, how do you Is there like a
clever way to say like
what we're working on here can't be more
than a thousand lines of code or
something like that? Cuz at the moment
AI agents This is your judgment and you
I don't think you can You need to cut
it. Like what I do, I tell the agent
looking at the number of user stories if
it's above five, I don't know. And then
I say, "Let's scope these out." Like
flag them as like um
a wish list or later. And it will just
put it like Again, I think we we have to
not be stuck up on the process cuz if it
feels like I mean, I understand on one
way you want to guarantee quality and
you want other people to follow a
process and and then people get confused
if the process isn't clear. But I think
the mistake that you're making is you're
getting so hung up on the process. Like,
"But hey, but how do I do this?" Well,
it's still an AI agent. You can still
talk to it. And I think one thing we
learned is that with these AI agents,
you just ask them. You say like, "Hey,
there's something wrong with your
settings." And it goes and finds the
setting and say, "Oh, you're right.
There's a setting here that I can change
and it will solve this problem."
>> I I mean, we're debating here. I can see
what you're saying, but though I do feel
the process should have a lot of a lot
of all like it should get you there
without without it should get you to
where you need to go to without too many
problems. And I do feel as you mentioned
the feature branch being too big is is
very very easy to happen. I think like
One thing that works for teams
very disappointing
>> you do it, you will get you will not
have that problem. I don't think I mean,
you could argue that yeah, but if
somebody news comes, he's going to face
these issues. But I I think you need to
let people do pilots and and experience
the problems and then walk back from it.
>> I mean, I can only tell you 100 times
Kai the same things that I've said on
this show about these feature branches
and scoping down.
>> I know my it's a bit irritating to hear.
My But the Okay, another thing that I
was a bit disappointed by is that Spec
Kit has this auto Git commit hook. And I
was like, "Oh, cool. So, when I do this,
it just Git commits." And then when I do
that, it it just Git commits. So, I So,
I can roll back. To be honest, I never
rolled back. And then when I looked at
the Git history, I was expecting to see
like an amazing like step-by-step about
all my genius prompts that I I was
giving. The
If you look at the Git history of the AI
God Rolls project, link below, whatever.
There's no
the Git commit uh message is just like,
uh you know, Spec Kit blah, Spec Kit
that. There's no no information in the
in the
uh Git Maybe that's just a a problem
with the
>> like I said, the people that are working
on Spec Kit right now, I think they are
not very high quality because they made
changes, but they didn't update the
docs. Like they don't show that this
There's a lot of documentation lagging
behind. And to be honest, this commit
hooks are are are something that didn't
exist. And right now my commits are
really nice.
I really love the the commits that I'm
getting, but I'm not using any hooks.
Okay, well, okay, it's I've got to I've
got to say something.
>> times stop using spec drift, right? I
mean, I've said I've said it like it's a
good first starter.
>> didn't use that one first, but okay, um
No, yeah, you use it first. You do a
small little demo and then you find all
the pains and and when you ask me all
these questions, I was like, I'm not
going to answer those questions anymore
until you try something else because
And I did And for the record, I did try
Kyra and that was a terrible experience.
It was terrible. It was so slow and it
it kept on like
uh you know, timing out and you know,
you kept on having to baby sit it. Not
cool. Not cool. And then yeah, the the
last note I have and I've talked to you
about but it for and I'm sure you're
going to tell me to do try another one
is is the spec drift problem where the
specs just become unaligned. But yeah, I
will Okay, I'll I'm going to try another
one.
>> again, I already I I said this several
times on this on these calls as well,
that it is really up to you to say to
the like let's say that the constitution
is kind of and the same with Google
Conductor, right? They all have these
project principles a concept.
>> forgot about Google's. Yeah, you tried
Google Conductor and one of the things
is it like it uses this JSON file at
least back like maybe between four
months ago and and it has this idea of
like how do you want to define like how
do you work with your team? And there's
a constitution, right? And and this spec
drift problem that you mentioned, like I
can't explain it
several times. I said like, you can
really create a constitution rule and
say that every spec plan includes
maintaining
um
a centralized documentation library.
Let's say that because again, the spec
folders, they will contain a history of
all of the specs that you've been
adding, but they're changing. Like even
if I scope something out in my five user
story spec and I say, let's not do user
story six and write it down." And then,
when I come back to it once it's merged,
I will often have changed my mind. I
will look back at user story six and I
see I realize that's not really what I
want because now I have something in
hand and I realize that actually it
should be different. And so, thinking
that those spec folders have any
meaning, I think it is wrong. And what I
do now is I tell in my constitution,
when you build your plan and you layout
your tasks, one of the final tasks is to
go back and update the documentation.
That's one way of doing it. The second
way is like we discussed as well is I'm
actually having a library of end-to-end
tests. And the user stories, they impact
it. I I ask it to write a quick start.
And if you look at the spec later
constitution, the CLI, it has a rule
that says the quick start MD that gets
generated has to match the E2E test
suite. So, because it's a CLI, it's very
easy to for it to write down when the
user runs this command, then this is
what needs to happen, right? And that's
my quick quick start MD, which is like
it shows you how the sub command looks.
It shows you what the arguments are and
the CLI, what's the UX a little bit. And
it basically translate very cleanly to
an end-to-end test on on on Go lang,
where it's like it builds the binary and
then it writes a test things
like do the assertions. And so, my quick
start.md per spec in those folders, each
time results in another changes into the
E2E. And if I don't have the central
docs of these are all the features and
all the sub commands, I have a fully to
E of all the features and sub commands
that are officially supported, right?
And if I run make E2E, it's going to run
is the user still able to run SL, um,
you know, in it. Is the user still able
to run this and is this the in it
behaving the way it is? And if I haven't
if I have changed the behavior of my
CLI, you know, because a new a new spec
has landed that has changed the
behavior, then any of those previous E2E
tests will have failed, right? So,
basically I have an executable Okay, so
a specification.
>> you essentially have a constitution to
say like don't break the tests. The the
tests are the source of truth. I say
there's two ways. You can say like I
have two in my in my constitution. I
always say like when you write the when
you do the research phase and and you
basically convert the very
product-oriented or user stories from
the specification into an actual
technical stack and down to the line
what is the actual, you know, command
that gets run, whether it be like I have
a mono repo that is because I love spec
kits in in mono repos or spec driven
development because you go like we have
the web, we have the CLI, we have the
back end, and they're all interacting
with each other. And so, the quick start
is usually, you know, we we we run a a
seeding script that set up the whole
environment, and then we do this and
that, and everything can be executed.
So, that's one way. I have an executable
kind of like centralized version away
from the spec folders. Or the other side
is I have a centralized docs because
ultimately when you build a
a solution, if it's a dev dev tool
focused, you will usually have a docs
website, right? Of like these are all
the features and this is how this work.
And so, part of the work that needs to
be done is to go back and update all the
docs accordingly, and then, you know,
when that lands, it pushes that. That's
why I think mono repos are great because
you can have everything
>> I'm totally team mono repo. I can't I
was speaking to a colleague and he says
that the on this enterprise project that
he's on, they they practice hexagonal
architecture. I had to look it up, but I
know what it means like
It's clean.
>> things are split into domains. So, he
was saying that like they're using spec
kit driven development on on a on a
project, and and for and for managers
and BAs and stakeholders, they can
generate user stories in a flash now
thanks to spec kit. But the trouble is
is that when you come to implement it in
a hexagonal architecture, you
essentially need to to cut like five or
six PRs for for a feature that's
crossing
um, you know five or six domains.
And as he says this is an absolute
nightmare because spec
STD doesn't help you there.
Yeah.
It's I've You should have an orc
his hot take like what why isn't STD
just called alignment driven development
or something because that's all it
really gives you right?
>> Alignment between AI and you and then
more multiplayer with other people. Yeah
it's like a more multi like multiplayer
driven development. To go back to your
previous point like when I have a a poly
repo I do actually in my work folder
where I launch cloud I clone multiple
repos. So I have my work directory. Yeah
and then cloud has access to all of them
and that's where some people like you
you create a like a spec driven repo and
then I'm not doing get submodules but
it's close to it right? It's like this
massive integration repo with modules
sub
In VS code you can use this workspace
feature which was a lifesaver in the in
a previous life. Because I used
workspaces and it's a different thing
though. Like I used it for well I used
it for a different purpose I guess. You
can have different checkouts and they
both and they all and like in code
spaces they come out in the workspaces
folder and they and then you can see it
nicely in your VS code. I mean it's not
that much different to checking into it
like one folder in a way. The one
problem that I had was that a lot of the
language servers of like typescript and
node JS and the integration within VS
code had issues with mono repos or I had
some problems with my configuration. I
remember one of the problems was the
debugger like if I need to run just
tests with a node JS debugger you could
only configure the debugger at the root
of the repo at the root of the project
not root of the repo. So I had a mono
repo with like five projects and then
the debugger would like use the get root
to find like to launch the the the and
he couldn't. Actually, it wasn't get
root, it was project root. So, that's
where I had to use workspaces. So, every
package had a different had a workspace,
and then I could open in this project in
workspaces, and then I could go to any
of them and run debug this package, and
then it would run the debugger attached
to this Yeah, I'm glad you brought that
up because like that's the typical thing
if you don't use a mono repo, you have
the different folders. And then does
your bloody intellisense work? Does your
code sense work between the different
projects? Because often enough that
someone's like saying, "I've got this
work around." But like the you know, the
typescript
type checking or the or the python
whatever it's called is broken.
Interesting. Yeah, and
Okay.
Another thing before I forget cuz I I
just made a note about it. Going back to
speckid, on the topic of all the specs
that you should look through, there
there was a command. I didn't try it. I
should have bloody tried it. And it was
I think it called speckid archive. Have
you Do you know what that command does?
I
I didn't have this yet. It's definitely
one thing on spec ledger it's in the UI.
But basically, the way it went, right? I
use speckid. I added task management
with with like um
a task graph instead of a markdown
because speckid was writing everything
down in markdown and it was like trying
to track dependencies and
parallelization. That makes sense.
>> I don't know that like like you have to
do that.
>> Does. Yeah, well well, I got rid of that
by by telling all of my structure
prompt. I noticed that well, they kept
it.
It's just an index. So, if you use spec
ledger, please do instead of complaining
and then never trying. Okay, I'm going
to try spec ledger next with I'm going
to I'm going to redo my projects in spec
ledger and then I
Yeah, so
>> Do you really want me to do this? Okay,
so if you use spec ledger, you download
a single go lang CLI and it has all of
the sub commands to do the spec info,
create a branch, do all the things. It
it up all of all your templates, and it
can download and update the template.
So, it's like a single thing. I don't
know. Originally, with Speck Kit, it was
like it created bash scripts. There was
one Python command to do bootstrap. It
was a bit bit messy. So, with with Speck
Ledger, it will then if you init, it
will ask you what agents that you use.
And if you want, it will then launch
Claude Code with a prompt, and it will it
will trigger the um the onboarding
process, which will launch an explore
agent, or if you if you do it in a
brownfield, it should launch an explore
agent and then come up with a bunch of
principles.
>> I can just bring my code over. Okay. It
should be. I'll try that. I forgot about
the brownfield. I'm living in such a
rosy life. But, one thing that I would
say, like the idea of Speck Ledger was
that you use the use the GitHub app in
the repo, and the moment you push, the
GitHub app will publish the markdown and
parse out the tasks and generate a
Kanban and generate a task graph to
visualize what the tasks are, and then
show the status of them, and then you
could click on them and see the details.
You using Beats under the hood, right?
Well, originally, yes. But, then we we
got rid of it because it's really a
pain. Like
>> One of the things So, you're back with
Beats. Yeah, so I'll explain it again.
So, basically, Beats uses a a task list
at the root of your repo, right? And
originally, I thought that's pretty cool
because that means that when I'm running
the command, like the the task get the
the list of tasks, even if I haven't
closed all of them in the next feature,
so the feature has been merged and I'm
working on a new one, it can see all of
the tasks because it's a shared central
one in the repo, not a per spec task
list. Well, that turned out to be really
painful because then they would create
merge conflicts. And I mean, Beats
always had this issue.
>> the way the branches work. Okay, yeah.
Yeah, and but Beats always had this
issue. It it it tried to work around
this problem by creating a sync branch.
So, every time you would commit, then
Beats would pushed the CLI would be
triggered via a
to go and update a single branch for
everyone every other branch to be
synchronizing into a single branch. So
they created this thing within beats
and it happens behind the scenes if you
don't pay attention and it keeps
propagating and recreating those
branches if you if you are not able to
remove all of the traces of beats. So it
was creating a lot of conflicts, it
wasn't able to handle the conflicts well
and so one of the people that was
working on spec ledger was like I'm so
tired of beats and I will just take the
the the idea but without running a demon
just JSON L keep a task task graph, you
know, take the structs from beats like
look at the beats source code and do it
per spec. That means if you run spec
ledger from the root, it will find
across all of the spec folders all of
the tasks. It can do that. You can see
all of the tasks across all of the
specs. But if you run it within this
feature, then you will only get the
feature specs like this the task for
that feature spec.
>> sounds bloody sensible. And then the
JSON L task list is in the folder so it
will not conflict between each other.
Because they're all like contained
within the features so they will not
fight constantly. So it's way way
better. So that's why it was written.
I understand I I I I understand the
drawbacks of beats. I also I find that
like some people are comfortable using
beats or something like beats and then
some people are definitely more
comfortable marked out but that's
another that's another separate topic. I
wanted to to to to check in with you
with
with the
the multi player what what what is your
vision when it comes to multi player
development
with spec ledger cuz that's the ultimate
that's the ultimate
underscore with spec
spec driven development is how the multi
player works. Yeah.
I'm I'm struggling to see it myself but
I've just obviously just did it in one
my mono repo with mono
Yeah, exactly. If you don't have other
people to work with you cannot see it. I
mean
I I saw the vision and when we all three
during the week like I was in Hanoi, one
person was in Saigon, another person was
in Singa in Singapore, we would do we
would be on a shared group chat because
we're not a company. And we would just
constantly like, "Hey, I specked out
this thing." And then during a meeting
we would make some notes and then we
would go off and you know, create the
different features that we want to work
on. And then we could share very
quickly. Yo, I I just had with my agents
and with the contacts I provided I
generated these user stories. Is there
any concern? Does this look like a good
approach to you guys? And then we would
just quickly go through, put some
comments, and say, "Hey, I think you
misunderstood." And and the great thing
is like I mentioned this before, I would
do a user stories and then somebody left
a comment and I was like, "This comments
makes no sense to me whatsoever." And I
would I was I was about to reply like,
"That makes no sense." And then instead
I use the Spec Ledger CLI and it
basically, you know that the Specket has
this clarify where you you take all the
user stories and there's always edge
cases and then it will go like, "There's
still unresolved questions. How do you
like these are the different ways you
could approach them, right?" So this
clarify command, yeah, in Spec Ledger
this clarify command, we actually push
the commit with all of the edge cases
and then we let people comment and the
clarify command tells the agent, "Great,
you've looked at the edge cases in the
documents. Go and look if anyone left a
comment on any of these artifacts." And
then the SL CLI will pull all the
comments that is in the web web app and
then pull them down and then look at
what's
>> down from from GitHub? Where are the
comments by the way? Metadata. That's
that's the one thing that is not stored.
It's like, okay, the way that I see Spec
Ledger is like the pull request view of
GitHub where you leave comments on the
PR except the PR doesn't organize the
files in a nice way. Spec Ledger
organizes the files in a way that makes
sense for Spec Ledger development.
>> this sort of view of the branch and then
people comment on it inside a SpecLedger
web application.
>> like I I basically looked at how Jira
organized documents. If you look at Jira
at the top, you have like these
artifacts have been created within the
spec phase or the these artifacts have
been created during the planning phase.
These artifacts have been created during
the tasks phase. And if And Jira
visualizes the process, it says you're
in the spec phase, you have artifacts
there. You did not do the plan and
research yet. So So the SpecLedger took
that idea and visualizes it on a web app
share it with other people and other
people can see, oh this feature branch
is already at the task phase. It has
already done the user stories. It has
already generated the research and it
has already generated the task graph.
That feature branch from Kai is still in
the specify phase. That means that there
are only artifacts related to that phase
inside this branch. There has been no no
research. There has been no task
generation yet. So that's the idea.
Compared to a PR where you just see a
list alphabetically organized where S
become comes after What is it? Research
or P Q R S? Yes. So you would see like
research.md and then you would see
spec.md and you'll be confused because
in the PR flow, if you look at the
GitHub pull request view, it doesn't
tell you you should read the spec first
you should go look at the research
because the research follows from the
spec, right? So that's the idea of
SpecLedger. It tells you that we have
done the specification and in here's the
spec and it has a little star because
that's the core artifact of this stage.
And and and that's the one you should be
looking at. You read the user stories.
And then if you want to like
I was looking for screenshots, but I
guess the the app is evolving a lot.
Yeah, I mean honestly this could be one
of the If it is it was a monorepo, but
the problem is that people I work with
didn't make it a monorepo. Um I I I want
to put everything under monorepo. Docs,
landing page, backend, CLI, everything
public. But um yeah, I
I haven't had a lot of time. I've
re-architected the thing because it's
super big.
>> architecture aside, I would love to um
capture the um um
What I'm looking to see is like an an
essence of what the workflow looks like.
You know, Vincent comes along, comes up
with the user story, Bob comes along,
has a comment here, Jane comes along,
have we thought about this? And then
these are this is great. I had like a
mock-up on speclager.io/
is it demo? Yeah. Yeah, that's very old.
The very first time I actually you can
click on that and that's the cool thing
about like sharing a little mock-up at
HTML. So basically we asked, okay
We had Cloud build a little mock-up UI.
This is all fake data, all JSON static,
all served from S3 behind CloudFront.
And this was the this was the design
that I wanted. So you can see we have to
specify phase, you can collapse that.
You can click on the little issues tab
at the top where you will see your
combine and you can see the the the
issues tab. You have artifacts and
issues and changes.
>> I'm sorry, sorry. I I was going
>> Yeah, and then you can switch that over
to tree view on the top on the top right
tree view. And that's that's what you
would see in beats or in if you would
use pal. So this is the web view of
that. And then the changes to be honest,
I hate that. It needs to be part of the
artifacts. You I want the Google Doc So
this is just the get diff view which to
me makes no sense. If you use Google
Docs and you work collaborative on
documents, you can click on you you can
see the versions of what this document
and you can compare two versions. And
and this whole changes being on a
completely different area is is really
bad and and that's one of the things I
will change when I finally get around to
do redoing the UI.
Um
Yeah, and then if you click you see you
have the two phases. As you see in this
case the in the workflow you see specify
and plan has been completed. Although
that the visualization is pretty bad.
Uh oh, you're okay, you're on a
different branch as me.
Um
Yeah, so you can see at which stage each
of them are. When you go to spec, you
can see the comments on If you click on
one of them, spec.md.
Click on the spec.md. Then you can see
you can select text.
>> Ah.
Select some text. Not there, not in the
comments. Yeah, there. Select some text.
And then you can click comment, and can
leave a little comment, blah blah blah.
>> And then all of those comments and they
are also threaded.
I'm sorry. I'm glad you're working on
this I'm really happy that you're
working on this.
>> on it anymore. We working on it like at
the start of the year. Now I want to
rewrite it because
>> what what what what why are you getting
distracted like is it cuz you're trying
to just take home some money? Mhm.
Like I mean I want to open source this
whole thing. It's a Next.js app backed
by Supabase hosted on Vercel, but I am
doing a full review of the Supabase. And
I honestly spent like almost 2 days
asking Claude to help me understand how
to properly architecture a Supabase app
because honestly
>> it's like it's just Postgres, isn't it?
Yeah, it's a tier It's like a two-tier.
It's like go back to stored procedures
type of thing. You know, it's back to
the '90s. What? Why are you architecting
it like that? It seems a bit bizarre.
>> Because that's how Supabase works. I was
asking like how do I
>> not depend on Firebase?
It's I mean if it's an open source
project, you can't depend on that.
>> What? I make it easily I want to I want
it to be super easily hostable. And
Supabase gives you like a CRUD back end
very quickly. But then when you look
into it, it actually sucks to be honest,
but Okay. Well, anyway.
Make it make it AWS hostable. I mean AWS
has a lot of stuff out of the out of the
box. The database stuff I think is
cheaper than it used to be. I know it's
not free. Yeah. But With Supabase you
get a free setup easily and also you can
self-host completely. You can you can
run it on localhost compose stack. So I
don't like Supabase and that's why I
spent 2 days trying to understand why I
should be using Supabase and why I
should like it. It's It's been
frustrating. Like the very first I did a
hackathon and they introduced Superbase
and I hated it. I was like, what is
this? This is from from client to
database directly. Where is the
the backend?
The application layer, where is the
logic? And then they built this thing in
Superbase because I I was like I had the
idea, I had the mock-ups of the UI and
then they spent like two people junior
to build it and um
and they use Superbase and I was like,
okay, I'll I'll guess I can I can try to
come up with that.
But now we all know that you're that was
not the best decision. But what is your
opinion of SQLite? That's another one
that I I like to use locally, but on the
cloud it's a different story. So SQLite
is super interesting, uh but it doesn't
do any like proper types in the in the
backend. When I build a Terraform state
backend, I support SQLite for like
having a single node backend, right? So
you can just run the Go language web
server backed by an SQLite, set up
Lightstream to have the write-ahead logs
streamed to S3 and have point-in-time
recovery very easily without doing like
daily backups. So you can do a lot of
really cool stuff with SQLite.
But the moment I started saying I'm also
supporting RDS and Postgres, then I
having so many issues because you can't
do it in SQLite. Like I I built like a
repository pattern and and then
integration and there's so much like the
database schemas I was using uh Boon
ORM, which is a Go language ORM that
supports SQLite and Postgres quite well.
And there was so many problems with
supporting it. The minute you you have
that requirement of or you have that
like support other databases problem and
you should earlier like if you want to
make it easily hostable, I mean
the hosting companies lock you in for
one reason or another. There's also
Turso that that they because SQLite
license is very strange. Like they give
you the software for free, but they they
charge you for the testing framework.
And then Turso I think rewrote SQLite I
think in Rust. I'm not sure.
And then they have three rights to worry
about.
Re-implement
or reverse engineer the test suite,
which is not available. And they made a
company and then they make like they
call it like Turso is a database at the
edge. So you can have like a distributed
SQL Lite database at the edge.
>> Mhm.
And they have some really cool ideas
there. It makes me think why didn't
Steve Yegge consider SQL Lite for Beats?
Because it is very
>> migrated.
They migrated to
>> to Dolt or something.
But which is Because because it solves
all of their like distributed Git
branches problem that we just talked
about. He had his demon, he had this
whole Git sync branch approach. And then
when they migrated to Dolt, I think they
solved a lot of those problems. But I
never like tried using it. Well, there's
so many things to try. I guess I have a
good a good problem. AI guardrails and I
can I have the time to try different
things. So I'm just going to try
different things. I'm going to try more
and more. Hey, did you want to talk
about anything else? Did you want to
talk about cuz got 10 minutes left. Did
you want to talk about another security
topic about the TanStack or because you
mentioned it and I didn't really
understand what the the problem was cuz
I cuz in my mind you educated me on the
PNPM supply security stuff.
Oh, I got educated by someone yesterday.
Oh no, two days
>> And I thought to myself, if you use PNPM
you're fine. No, but like
Actually very interesting feedback I
got.
Okay, what was the feedback?
Okay, so first off, of course it's been
a a roller coaster ride, you know,
because I don't know if we covered this,
but we had the copy fail local privilege
escalation Linux kernel bug.
>> That was allowing people to assume root
permissions if they
access. So local then somebody some
somebody reported a but a
bug called copy Sorry,
dirty frag, which researchers started
patching up on an open source repo, so
on an open branch. And an independent
researcher saw the branch, and the
branch name was like in the class of
copy fail. So, that drew his attention,
like what is this copy fail category
branch? Like is that another local
privilege escalation bug? And so, he
independently wrote a Python exploit
script based on the changes that he saw,
and then he published it. And then then
everyone suddenly was like, "Holy, this
is breaking the embargo. There has been
no fixes released. Like the kernel
hasn't been released. The distributions
haven't been able to patch anything. And
now there's a live exploit, which is a
zero day, that people can abuse." So,
initially the reaction was like,
"Embargo has been broken by somebody."
Uh but then he clarified on the mailing
list, "Sorry, guys. I just saw this
branch. It drew my attention. It didn't
know it was like It wasn't my intention
to create a zero day exploit." And for
me there was like, "I just finished
rolling out new AMIs, cuz I do immutable
infrastructure with the new kernel
build, and now I have to do it all over
again. And worse, there were no kernel
patches yet." So, that's where I think
we had this discussion. People started
releasing remediation instructions,
which is like you have to go and and
inside the Linux kernel disable these
modules if you don't use it, because
that's your right now your protection
against the LP. And the funny thing is
that What What is Sorry, what is LP?
>> Local privileges escalation.
>> Okay.
>> So, so the The funny thing is that the
these remediation instructions are being
distributed like agent prompts, right?
>> Yeah. Because like you don't need to
tell you don't need to like step one do
this, step two do this. No, you you can
just say give this prompt to your agent
and ask it to go and fix things. So,
originally when I was I was looking at
fixing the issue, I have AWS SSM, so all
of my Linux instances are available in
an inventory, and I can give the agent
access to it, and I say, "Find out all
the Linux instance or find out all the
instances that are running the AMI that
we need to go patch." And then it goes
and SSH into and and patches it up. Are
you using AWS Inspector to sort of
highlight the vulnerability or
We have Wiz.io to to
you know, inspect the runtime. It's kind
of like intrusion detection and
prevention.
Okay.
>> agents
So, they Wiz gives us like a graph.
Yeah. Okay. And does it know about
these copy fails and these It does. It
does. It It does highlight like Yeah, if
I don't uh patch the the instance, then
then we get an email, and we see a nice
graph of why and the the the risk
because if this is a Linux instance with
no public endpoints, and it's in a
private subnet, and it's usually fine. I
mean, like it's a lower risk factor.
I mean, do you have local users with
shell access to
>> No.
So, that's why also we are a lot So, the
risk is bloody low for you, isn't it?
Yeah. Yeah, the risk is much lower, but
but you want to patch these things
because
they do show up in
in those tools. And for example, my
Atlantis instance has a public IP that
is reachable by GitHub webhooks. So, it
will usually find and and highlight that
one because it says, "Oh, this Atlantis
instance is public IP like the EI
because it shows the AWS resources, and
it shows that there's an elastic IP
publicly attached to it.
>> if it's like a public IP running an
endpoint, you can't run that sort of
LPE, right? There's There's no way you
can do it. I mean, like I do I do
sometimes wonder like you were just
saying that like, "Oh, I just patched
AMIs, and then I have to do it all over
again." I mean, you're doing a lot of
work. No doubt. Actually, no, I'm not
actually doing
>> you I sometimes think it was like Do you
Do you need to do it? You don't really
need to do it, really, cuz there there's
no risk, in a way. If the If you don't
have This is defense in depth. This is
in case there's an RCE, somebody gets
like let's say that I had running an
Engine-X and I'm using OpenResty with
Lua scripts and there's an exploit for
someone to get remote code execution on
the host. But Engine-X no problem,
right? Because I'm running it under the
Engine-X web user in Linux and that user
doesn't have any permissions. And then
you have an LPE because you never put
patch to copy fail. So then that that
Engine-X user that has no permissions
suddenly elevates the permission to
become a root on that host and suddenly
gets access to write to an EBS volume
and and persist itself. Because I have
like some nodes that like they're in
ASGs and ephemeral, they get like
rotated, but they do have like for
example TLS certificates that are maybe
with Let's Encrypt stored on the on the
on a volume. And so the Engine-X web
server doesn't have access to that
volume. Well, if I'm using
>> So I still I still think that I mean who
uses Engine-X nowadays, but I still I
still find that hard to imagine
sometimes. Anyway, I'm not trying to say
that there's no
I'm not I'm not I'm not trying to say
there's no risk, but I'm just
>> you should look at the at the at the web
report on how much of the web is served
by Engine-X. I think it's still at like
50%.
Well,
I still find it hard to imagine that you
can get user access on an Engine-X
server in a in a modern hosted system. I
find it extremely hard to imagine, but I
feel free I feel free to prove me wrong,
listeners. I just I just hate how like
every security chat group I'm on like
hey, look at this copy fail thing.
Screaming, you know,
exasperated emojis. I like this is super
interesting.
>> dude, it's not going to be we don't
really need to do that we don't need to
drop everything and patch this because
the risk is incredibly low. We don't
have local users. You're forgetting
You know the amount of products that I
built that do not properly validate
inputs. When you say you don't have
local users, there's so many other
things. Sorry, you're using? We use go.
Okay, let's talk about the 10 stack 10
stack a hack because it was very
interesting that it it actually
showed that um I know I forgot. Cuz I I
said like one of the one of the things
that came out of this is basically 10
stack got published through the trusted
OIDC and was published as a verified
package. Meaning the package was built
on hardware that was running in GitHub
managed a software and it was signed and
all of the verifications everything
passed.
It was published from official CICD
pipelines from the project. Yeah, I mean
package that's signed and has a
vulnerability doesn't really surprise
me, but carry on.
No, but that's one of the more, you
know, the whole
security stance of a lot of these
projects is like you don't use
long-lived credentials because a lot of
these attacks lately have been
secret exfiltration, right? The Tree V
hack, Aqua security, it was all about
a package got executed, got access to
secrets, and we got like tokens to
publish. And I was like, why do you have
tokens? Why do you have long-lived
tokens? You should be using short-lived
tokens that are only valid for the
amount of time that you need to publish.
So, if your whole approach to security
was to adopt OIDC, that has now been
subverted. I mean like not subverted,
you always need to do defense in depth,
right? So, one of the main things was
that the 10 stack package was
through OIDC, short-lived tokens,
verified builders, signed package,
published. So, So, that So, they had a
good posture, but they still still got
hit or something. Is that what you're
saying?
>> Yeah, they got hit because there was a
way for somebody to do a PR, and so I
think it's Team PCP, right? The what's
it called? Hulut? Oh, yeah. Yeah. Mini
Hulut from Team PCP, they were able to
shy Hulut and mini shy who loot. So they
were able to create a PR and the caching
action of GitHub actions did not
differentiate the branch that the cash
came from. So they created another sort
of get
hack in a way through the cash thing or
something. Yeah, and it's funny because
lately somebody shared an article with
we was like we are bad citizens if we
are not using proper caching techniques
because everyone just assumes that NPM
GS and Python registries and Maven
public registry all of this
infrastructure is available for free,
but it costs money. And so you are
consuming a lot of compute whereas you
should probably use a proper cash and
don't redownload every package every
time you need to do a run. And now
caches are can be dangerous or it can be
an attack vector.
>> Well, if you do it again, all of these
attacks are exploiting some very minor
oversight in the way that things are set
up, right? The option is there and you
can use it, but if you do, there's a
whole bunch of conditions that you need
to do to protect yourself. In this case,
if you're doing this type of caching,
you need to make sure that the main
release branch doesn't use the caches
that are created and and and accessible
to untrusted code. So they did they did
a pull request from their code that they
controlled and they were in the cash
process part of the PR, it injected a
poison package inside the cash. And so
when the main release happened, the cash
got pulled in and during the node
install from the cash, node node modules
installed from the cash, a hook was
executed I think or something similar.
I'm not sure exactly how they were able
to trigger the poisoned package and that
package then was able to do whatever it
needed to do
with full credentials.
>> that to me, Vincent. Yeah, because
that's in these cases people are like oh
somebody of the maintainers had like a
token or his account was not secure
because this used to happen, right?
People got hacked, they took their
account got taken over or they were like
how did somebody merge this? Like when
when they talked about three V action,
then somebody had approved a pull
request that was pointing to a a commit
that it was accessible from a fork
repository. So so they were like why did
they not like why did they approve this
PR? And it's like usually because it's
hidden between a bunch of other diffs
and and things like that, right? So all
of these are very interesting and and
what they are trying to say is also that
with these AIs studying these
repositories and the way they are set
up, they're finding these
misconfigurations and these loopholes
much faster than we have. Like some of
these problems have been there for 20
years but nobody saw. And now these AIs
go through everything and they're like,
"Hey, the cache is reused across
branches. Exploit this." So a lot of
this is now automated being found and
then it's causing issues over and over
again.
Yeah, yeah, this is this is fascinating.
Okay, but I need to take my kids to
school. Thanks for your patience.
>> There was one more thing about using
package age. The package age like you
said like if you use PNPM and you make
sure that the package is over a certain
amount of age should be sufficient to
protect yourself. I got schooled on that
two days ago but now I forgot. He says
like I need to Wait, I I think I made a
note. I tried to be really fast here. I
hope I wrote it down. Well, or maybe
we'll put it in the show notes in in the
in the description how you got schooled
with PNPM. You can't just rely on PNPM.
Man, yeah. Well, I'm I'm I'm going to
I'm actually looking forward to doing
more security-related stuff. So I'm let
I remember.
>> I'm going to learn more. I saw the note.
He said that if you set up your project
to automatically update your
dependencies even if you make those auto
update flows make sure that they don't
update until the package has more than a
few days on the registry because then
you hope it will have been detected, you
will still be hacked. And the right way
to do it is to use a GitHub secret a
GitHub Dependabot feature, which through
secret through security. So, Dependabot
does a security scan on the packages.
And then it will only say, "Hey, this
package this dependency has a
vulnerability." And only update those
packages, not everything. So, a lot of
these upgrade flows, they will just look
is there a new package? Okay, upgrade.
And then is an upgrade and all the tests
pass? Okay, auto approve and merge. So,
always our project is upgrading, always
up to date. Your previous approach was
missing that security scan which
Dependabot gives you. Yeah, he's saying
is is like never do that. Only upgrade
if there's a CVE in a dependency and
only or if you have really there's a a
minor version patch that you need
because of a new feature, right? So, use
Dependabot, it will identify any
dependencies that have a
security vulnerability and it will
update those. I've never heard someone
advocating for Dependabot. Like usually
everyone says use Renovate, use more
advanced, you know, PNPM Because we're
on a public repo and it's for free. So,
he says, you know, you can get you get
it for free. So, and it's like easy to
set up.
>> that that comes out of the box with uh
with Dependabot or do you have to is
that a feature that you need to enable?
I think it's a feature. So, I did it's
something he told me like we have a lot
of those auto upgrade flows. Link to
I need to run although my kids will be
late. Thanks again, Vincent. We
appreciate it. Boy,
>> Okay. uh Bye-bye.