WEBVTT 00:00:02.520 --> 00:00:05.760 Hello and welcome to episode 25 of the 00:00:05.760 --> 00:00:08.320 AI infrastructure podcast 00:00:08.320 --> 00:00:12.000 with me Kai Hendry in the UK, Southwest 00:00:12.000 --> 00:00:13.520 UK to be exact 00:00:13.520 --> 00:00:14.560 and 00:00:14.560 --> 00:00:15.800 my 00:00:15.800 --> 00:00:18.560 co-host Vincent De Smet over in Vietnam, 00:00:18.560 --> 00:00:20.440 Ho Chi Minh City 00:00:20.440 --> 00:00:23.080 I believe. 00:00:23.080 --> 00:00:24.440 We're both 00:00:24.440 --> 00:00:26.960 infrastructure engineers 00:00:26.960 --> 00:00:28.960 by day and uh 00:00:28.960 --> 00:00:31.200 and and uh by night and in the morning 00:00:31.200 --> 00:00:34.960 here I am a AI enthusiast. Um in this 00:00:34.960 --> 00:00:37.000 podcast you can expect us talking about 00:00:37.000 --> 00:00:39.240 AI and infrastructure. 00:00:39.240 --> 00:00:41.240 Um what does it all mean? 00:00:41.240 --> 00:00:43.560 And I I hope you enjoy it. 00:00:43.560 --> 00:00:45.680 Do please comment. 00:00:45.680 --> 00:00:48.560 Do please like. Do please subscribe. 00:00:48.560 --> 00:00:50.200 And um we're trying something new today. 00:00:50.200 --> 00:00:53.040 We're trying Riverside. So do tell us 00:00:53.040 --> 00:00:54.560 what you think of the recording. Is it 00:00:54.560 --> 00:00:57.200 better than the typical Zoom recordings 00:00:57.200 --> 00:01:00.240 that we've been doing up until now? 00:01:00.240 --> 00:01:02.360 You tell us. 00:01:02.360 --> 00:01:06.520 Thank you and enjoy. 00:01:06.520 --> 00:01:08.760 >> Hey, how are you? Having a good uh day? 00:01:08.760 --> 00:01:09.280 >> Uh 00:01:09.280 --> 00:01:10.920 I just woke up really. 00:01:10.920 --> 00:01:13.400 I'm uh 00:01:13.400 --> 00:01:15.160 Well, I just 00:01:15.160 --> 00:01:17.080 There's a lot to cover. It is almost 00:01:17.080 --> 00:01:19.160 overwhelming every time we meet cuz like 00:01:19.160 --> 00:01:21.080 there's a few things I want to cover 00:01:21.080 --> 00:01:21.640 >> Mhm. 00:01:21.640 --> 00:01:23.280 >> and I'm like 00:01:23.280 --> 00:01:25.040 where's Vincent on this? Where's Vincent 00:01:25.040 --> 00:01:26.200 on that? I mean yeah, what about 00:01:26.200 --> 00:01:26.800 yourself? 00:01:26.800 --> 00:01:30.000 >> I had a lot of fun from like in the last 00:01:30.000 --> 00:01:32.320 four days using the tool that I built 00:01:32.320 --> 00:01:33.720 because I haven't had the opportunity to 00:01:33.720 --> 00:01:36.000 play with it and and and I had some time 00:01:36.000 --> 00:01:37.720 to play with it and it just feels so 00:01:37.720 --> 00:01:40.240 good to to see how it has evolved, what 00:01:40.240 --> 00:01:41.840 features has been added, what works, 00:01:41.840 --> 00:01:43.760 what doesn't work. I also played with 00:01:43.760 --> 00:01:46.240 Riverside like you. Actually basically 00:01:46.240 --> 00:01:47.720 because you sent me the message I was 00:01:47.720 --> 00:01:49.440 like, okay, let me try it. It was pretty 00:01:49.440 --> 00:01:49.840 fun. 00:01:49.840 --> 00:01:52.680 >> Yeah, well, I'm I'm desperate to improve 00:01:52.680 --> 00:01:55.120 the production quality and try different 00:01:55.120 --> 00:01:57.560 things actually. So well, here we are, 00:01:57.560 --> 00:01:59.840 trying different things. So cool. So, 00:01:59.840 --> 00:02:02.160 your tool is Spec Ledger or something 00:02:02.160 --> 00:02:02.280 else? 00:02:02.280 --> 00:02:03.680 >> Yeah, yeah, no, no, it was Spec Ledger. 00:02:03.680 --> 00:02:05.600 So, basically, there were a couple of 00:02:05.600 --> 00:02:07.600 annoyances that had that I had filed, 00:02:07.600 --> 00:02:09.000 and then they had fixed it, and I wanted 00:02:09.000 --> 00:02:10.720 to see if it the fixes were there. I 00:02:10.720 --> 00:02:12.000 think one of the things I might have 00:02:12.000 --> 00:02:13.640 mentioned this that we were suffering 00:02:13.640 --> 00:02:16.240 some quality issues, right? Even though 00:02:16.240 --> 00:02:18.040 that we were using the spec driven 00:02:18.040 --> 00:02:20.160 development, and I had very good results 00:02:20.160 --> 00:02:22.120 with it. Once we scaled it out over a 00:02:22.120 --> 00:02:24.320 team with people of different 00:02:24.320 --> 00:02:26.840 experiences, I felt that even though we 00:02:26.840 --> 00:02:29.160 had team alignment on what to implement, 00:02:29.160 --> 00:02:31.080 there were still, when I finally got the 00:02:31.080 --> 00:02:33.040 implementation done by other people, 00:02:33.040 --> 00:02:35.040 areas where I discovered, "Hey, that's 00:02:35.040 --> 00:02:37.640 not done. This is not done." And so, I I 00:02:37.640 --> 00:02:39.080 felt like 00:02:39.080 --> 00:02:40.600 we talked a lot about how do you 00:02:40.600 --> 00:02:43.480 validate, how do you ensure that the AI 00:02:43.480 --> 00:02:45.160 implements, and I always talked about, 00:02:45.160 --> 00:02:46.400 "Yeah, you know, you need to build those 00:02:46.400 --> 00:02:49.200 validation, you know, testing around 00:02:49.200 --> 00:02:51.760 it." And then, the thing that I have was 00:02:51.760 --> 00:02:53.280 completely failing. So, I was like, 00:02:53.280 --> 00:02:54.231 "What the hell?" 00:02:54.231 --> 00:02:54.800 >> [laughter] 00:02:54.800 --> 00:02:55.640 >> So, 00:02:55.640 --> 00:02:57.040 how is 00:02:57.040 --> 00:02:58.600 How are you doing it in a team? Cuz I 00:02:58.600 --> 00:03:00.480 can't help but think all those the 00:03:00.480 --> 00:03:02.080 quality assurance is is getting back to 00:03:02.080 --> 00:03:04.400 basics, like, you know, my employer 00:03:04.400 --> 00:03:06.959 preaches shift left, preaches having a 00:03:06.959 --> 00:03:10.560 dedicated QA person on the team. 00:03:10.560 --> 00:03:13.800 >> Yeah, so 00:03:13.800 --> 00:03:17.360 >> it emphasizes quality. Um so, as a team, 00:03:17.360 --> 00:03:19.840 you can't just expect AI to to do the 00:03:19.840 --> 00:03:21.680 quality, right? You have You have to 00:03:21.680 --> 00:03:24.320 have someone responsible or playing the 00:03:24.320 --> 00:03:25.800 role of a tester or something. 00:03:25.800 --> 00:03:28.600 >> I felt the hardest thing is when you are 00:03:28.600 --> 00:03:31.000 writing quite detailed and maybe too 00:03:31.000 --> 00:03:33.880 large of a user stories, is how I only 00:03:33.880 --> 00:03:36.720 find out maybe a week later, like, "Hey, 00:03:36.720 --> 00:03:38.840 I remember I put some requirement there, 00:03:38.840 --> 00:03:40.920 and when I'm using it, it turns out it's 00:03:40.920 --> 00:03:42.959 not there." So, then I get frustrated, 00:03:42.959 --> 00:03:44.840 like, how did that not get there? Like, 00:03:44.840 --> 00:03:47.560 in terms of verifying it, it felt very 00:03:47.560 --> 00:03:50.400 random, and gave me like a lot of doubt 00:03:50.400 --> 00:03:52.760 about the quality. But, when I started 00:03:52.760 --> 00:03:54.600 playing with it, I actually realized 00:03:54.600 --> 00:03:57.320 that none of the quality guards that are 00:03:57.320 --> 00:03:59.800 built into Spec It were properly 00:03:59.800 --> 00:04:02.080 initiated. So, I had not initiated the 00:04:02.080 --> 00:04:04.280 repository, and so we were missing the 00:04:04.280 --> 00:04:06.480 constitution. We didn't have proper 00:04:06.480 --> 00:04:09.800 checklists. We didn't have a proper 00:04:09.800 --> 00:04:11.480 design like 00:04:11.480 --> 00:04:12.400 >> Well, 00:04:12.400 --> 00:04:14.400 it sounds like you're in that sort of 00:04:14.400 --> 00:04:15.920 waterfall mindset where you're saying 00:04:15.920 --> 00:04:17.799 like the reason we had this quality 00:04:17.799 --> 00:04:20.160 issue is because our spec was wrong. 00:04:20.160 --> 00:04:22.360 >> No, it's it's the the the validation 00:04:22.360 --> 00:04:23.520 wasn't there. 00:04:23.520 --> 00:04:24.800 One of the things that I put in the 00:04:24.800 --> 00:04:27.120 constitution is that a quick start 00:04:27.120 --> 00:04:29.960 document, in this case for a CLI, must 00:04:29.960 --> 00:04:33.240 document the user stories in a It's a 00:04:33.240 --> 00:04:35.040 command line interface, so it must show 00:04:35.040 --> 00:04:37.480 user story one. You have You invoke the 00:04:37.480 --> 00:04:39.520 command line with this parameter, then 00:04:39.520 --> 00:04:41.720 this needs to happen. And that directly 00:04:41.720 --> 00:04:45.120 must translate to a formal like test in 00:04:45.120 --> 00:04:47.440 the test. So, that I had that in my own 00:04:47.440 --> 00:04:50.000 project, but I noticed it was missing. 00:04:50.000 --> 00:04:51.520 We didn't have like these tests there. 00:04:51.520 --> 00:04:53.640 >> All right, so the the tests are not 00:04:53.640 --> 00:04:56.400 dropping out of The AI isn't isn't 00:04:56.400 --> 00:04:58.120 following instructions. 00:04:58.120 --> 00:04:59.880 >> No, the problem is the instructions were 00:04:59.880 --> 00:05:00.320 missing. 00:05:00.320 --> 00:05:00.919 >> Really? 00:05:00.919 --> 00:05:02.400 >> I had not initiated this, and the 00:05:02.400 --> 00:05:04.560 constitution was still empty. I thought 00:05:04.560 --> 00:05:06.840 it was it was filled out. So, it has no 00:05:06.840 --> 00:05:09.360 principles. So, there was no rule for 00:05:09.360 --> 00:05:11.800 the AI when it finishes the thing and 00:05:11.800 --> 00:05:13.840 then validates the task list. Maybe you 00:05:13.840 --> 00:05:15.360 can say that's because I I thought like 00:05:15.360 --> 00:05:17.360 waterfall plan wasn't right, but it's 00:05:17.360 --> 00:05:19.080 very important because the final phase, 00:05:19.080 --> 00:05:20.880 which is the the the validation and 00:05:20.880 --> 00:05:22.880 testing and polishing phase, needs to 00:05:22.880 --> 00:05:25.120 have a task that validates it. And a lot 00:05:25.120 --> 00:05:26.680 of the times the task was manual 00:05:26.680 --> 00:05:28.640 verification instead of And that's 00:05:28.640 --> 00:05:30.440 because it wasn't configured properly. 00:05:30.440 --> 00:05:32.120 And and so basically what I did for the 00:05:32.120 --> 00:05:34.800 weekend was exfiltrate what I had done 00:05:34.800 --> 00:05:36.320 in my own project, where I have a very 00:05:36.320 --> 00:05:38.240 strong test suite, and basically copied 00:05:38.240 --> 00:05:40.160 over the principles into this one. Cuz I 00:05:40.160 --> 00:05:41.760 showed you the grid, right? I showed you 00:05:41.760 --> 00:05:43.960 that I have flu full play right 00:05:43.960 --> 00:05:45.320 workflows. In this case, it's It's just 00:05:45.320 --> 00:05:47.520 a CLI, it's actually a web interface 00:05:47.520 --> 00:05:50.000 with Keycloak authentication. 00:05:50.000 --> 00:05:51.640 I'm running everything there and and the 00:05:51.640 --> 00:05:53.520 and I had no quality problems, but I was 00:05:53.520 --> 00:05:55.160 also doing it alone, right? So, that 00:05:55.160 --> 00:05:56.720 means I was defining the plan and I was 00:05:56.720 --> 00:05:59.160 also driving the implementation. In a 00:05:59.160 --> 00:06:01.360 team setting, you might only be involved 00:06:01.360 --> 00:06:03.720 in defining the plan, but you are not 00:06:03.720 --> 00:06:05.760 driving the implementation. And then you 00:06:05.760 --> 00:06:07.400 find out the person that did the 00:06:07.400 --> 00:06:09.960 implementation didn't actually validate 00:06:09.960 --> 00:06:10.360 it. 00:06:10.360 --> 00:06:12.720 >> it makes me think that you need to be 00:06:12.720 --> 00:06:14.720 there for the whole process. You can't 00:06:14.720 --> 00:06:15.040 be 00:06:15.040 --> 00:06:16.600 >> want to be there for the whole process, 00:06:16.600 --> 00:06:18.560 right? So, I am trying to solve that 00:06:18.560 --> 00:06:20.000 problem because 00:06:20.000 --> 00:06:21.120 >> Why don't you want to be there for the 00:06:21.120 --> 00:06:23.080 whole process? Because it's tedious? Is 00:06:23.080 --> 00:06:23.840 there some reason? 00:06:23.840 --> 00:06:26.280 >> Because we individually must own each 00:06:26.280 --> 00:06:28.120 feature that we're working on. Like, 00:06:28.120 --> 00:06:30.600 there's basically a team exercise where 00:06:30.600 --> 00:06:32.680 humans decide and collaborate on what is 00:06:32.680 --> 00:06:34.320 it that we want to implement, and then 00:06:34.320 --> 00:06:36.320 it's an individual and ideally not even 00:06:36.320 --> 00:06:38.400 a human involved, purely AI agent 00:06:38.400 --> 00:06:41.280 implementation, right? So, today we are 00:06:41.280 --> 00:06:42.520 letting juniors drive that 00:06:42.520 --> 00:06:44.360 implementation process. I don't know if 00:06:44.360 --> 00:06:45.560 that's a good thing or if it's a bad 00:06:45.560 --> 00:06:46.760 thing. Maybe they are distracting the 00:06:46.760 --> 00:06:47.880 agent and therefore the agent did 00:06:47.880 --> 00:06:48.919 deliver, but I don't think that's a 00:06:48.919 --> 00:06:50.880 problem. I think they are letting the 00:06:50.880 --> 00:06:53.080 agent implement and they're not stopping 00:06:53.080 --> 00:06:55.520 it when it diverges. And And that's a 00:06:55.520 --> 00:06:57.160 problem. I don't want a human to be 00:06:57.160 --> 00:06:58.760 there in the end to to stop it when it 00:06:58.760 --> 00:07:01.080 diverges. I want it to be so 00:07:01.080 --> 00:07:03.000 straightforward that when the agent is 00:07:03.000 --> 00:07:05.240 done, the output works. I don't want to 00:07:05.240 --> 00:07:06.560 be there for that implementation. 00:07:06.560 --> 00:07:08.440 >> Why doesn't So, essentially you're 00:07:08.440 --> 00:07:11.000 saying that the junior is not catching 00:07:11.000 --> 00:07:12.960 the diversions or something like this? 00:07:12.960 --> 00:07:13.720 And 00:07:13.720 --> 00:07:14.919 then you're going back to the drawing 00:07:14.919 --> 00:07:16.840 board to make sure the AI doesn't 00:07:16.840 --> 00:07:18.480 diverge. It sounds like that sort of 00:07:18.480 --> 00:07:19.000 thing is happening. 00:07:19.000 --> 00:07:20.440 >> Yeah, and I think the contributing 00:07:20.440 --> 00:07:21.760 factor is that the junior is actually 00:07:21.760 --> 00:07:23.040 not very much involved in the plan 00:07:23.040 --> 00:07:25.800 planning process. They're just But, it's 00:07:25.800 --> 00:07:27.880 not even on juniors only. We even have 00:07:27.880 --> 00:07:30.080 like a very big user story, like we had 00:07:30.080 --> 00:07:31.640 too many, like 10, and we split them off 00:07:31.640 --> 00:07:33.320 into the individual ones. But, we had 00:07:33.320 --> 00:07:35.360 aligned on the on the 10 big user 00:07:35.360 --> 00:07:37.000 stories and we realized this is way too 00:07:37.000 --> 00:07:38.440 big of a scope to try and go into 00:07:38.440 --> 00:07:40.320 implementation. So, we split it off in 00:07:40.320 --> 00:07:42.600 three smaller work streams of three each 00:07:42.600 --> 00:07:44.800 and one of four, and and and I let that 00:07:44.800 --> 00:07:46.040 I let 00:07:46.040 --> 00:07:48.760 and then execution by a like a more 00:07:48.760 --> 00:07:50.640 experienced person, and we still had 00:07:50.640 --> 00:07:52.880 things where, "Hey, the original 10 user 00:07:52.880 --> 00:07:55.160 stories had a detail here that was not 00:07:55.160 --> 00:07:57.320 captured. It could have been lost when 00:07:57.320 --> 00:07:59.360 it was split, or it could have been lost 00:07:59.360 --> 00:08:01.240 during the implementation." So, even the 00:08:01.240 --> 00:08:02.840 split I should have been involved, but 00:08:02.840 --> 00:08:03.840 Yeah, it's just 00:08:03.840 --> 00:08:05.120 >> It's tricky. I guess you're you're 00:08:05.120 --> 00:08:08.280 trying to scale with the help of AI, and 00:08:08.280 --> 00:08:09.920 >> Yeah. And that's why I really love 00:08:09.920 --> 00:08:10.600 >> of problems. 00:08:10.600 --> 00:08:11.840 >> Exactly. That's why I really love this 00:08:11.840 --> 00:08:13.120 project because it 00:08:13.120 --> 00:08:15.440 it's really a real true test of scaling 00:08:15.440 --> 00:08:16.840 out spec driven development across a 00:08:16.840 --> 00:08:18.600 team of different skilled people. 00:08:18.600 --> 00:08:19.400 >> So, 00:08:19.400 --> 00:08:21.160 on that note, 00:08:21.160 --> 00:08:22.400 let me just share something that I 00:08:22.400 --> 00:08:24.160 thought was really 00:08:24.160 --> 00:08:25.080 um awesome. 00:08:25.080 --> 00:08:26.720 >> And I say different different skilled 00:08:26.720 --> 00:08:28.400 people, but I also found mistakes that I 00:08:28.400 --> 00:08:29.840 made, by the way. 00:08:29.840 --> 00:08:30.400 It's not like 00:08:30.400 --> 00:08:32.640 >> Yeah, I mean the the last podcast, or 00:08:32.640 --> 00:08:34.599 the podcast before last, I was I was 00:08:34.599 --> 00:08:37.280 actually like in shock about a bug that 00:08:37.280 --> 00:08:39.360 came through into 00:08:39.360 --> 00:08:40.760 Well, it didn't go into production, 00:08:40.760 --> 00:08:41.640 thank god, but 00:08:41.640 --> 00:08:43.280 >> But you feel that the the workflow would 00:08:43.280 --> 00:08:45.160 have easily been found out if you had 00:08:45.160 --> 00:08:46.440 paid closer attention. 00:08:46.440 --> 00:08:48.640 >> Well, if Yeah, if we if we just did it 00:08:48.640 --> 00:08:50.600 the traditional way, it would it would 00:08:50.600 --> 00:08:54.839 have never got through. But, of course, 00:08:54.839 --> 00:08:57.600 I was like giving the PR to a colleague 00:08:57.600 --> 00:08:59.160 to review, and the 00:08:59.160 --> 00:09:00.280 and the problem was is that the 00:09:00.280 --> 00:09:03.000 colleague trusted me, and I trusted the 00:09:03.000 --> 00:09:05.400 AI, and then and then this bug bug got 00:09:05.400 --> 00:09:07.400 through. So, the the whole the whole 00:09:07.400 --> 00:09:09.920 trust thing and I felt it wasn't so much 00:09:09.920 --> 00:09:11.680 I I put the bug through, it's also that 00:09:11.680 --> 00:09:13.920 now now my colleague thinks I'm an 00:09:13.920 --> 00:09:15.520 idiot, for want of a better word, 00:09:15.520 --> 00:09:18.480 because I let a trivial bug through, and 00:09:18.480 --> 00:09:21.480 now he's lost confidence in me because 00:09:21.480 --> 00:09:24.680 I didn't review I didn't see this AI 00:09:24.680 --> 00:09:27.120 introduce the issue to me. 00:09:27.120 --> 00:09:28.880 I mean, I'm feeling like I'm blaming AI, 00:09:28.880 --> 00:09:32.360 but it but it was really me that was 00:09:32.360 --> 00:09:33.760 that was doing the wrong thing. 00:09:33.760 --> 00:09:36.440 >> I don't know how much data it had to 00:09:36.440 --> 00:09:38.280 like you had Like was it part of a huge 00:09:38.280 --> 00:09:40.800 chunk and therefore it got it flew or 00:09:40.800 --> 00:09:42.960 did you really just not give it 00:09:42.960 --> 00:09:45.800 damn guy. 00:09:45.800 --> 00:09:47.480 >> Anyway quality I feel like we can talk 00:09:47.480 --> 00:09:50.520 about quality for days but this blog I 00:09:50.520 --> 00:09:53.960 think is very succinct on the topics. He 00:09:53.960 --> 00:09:55.400 starts off by saying that every time you 00:09:55.400 --> 00:09:57.000 have a review it makes you 10 times 00:09:57.000 --> 00:09:59.520 slower and of course PR reviews we all 00:09:59.520 --> 00:10:02.280 know they suck but they they are a 00:10:02.280 --> 00:10:04.880 quality gate right and and he was he was 00:10:04.880 --> 00:10:06.400 pointing out the more reviews you have 00:10:06.400 --> 00:10:08.880 like I'm in some heavily regulated 00:10:08.880 --> 00:10:10.560 environments and it takes like two 00:10:10.560 --> 00:10:13.480 approvers to get something in and and 00:10:13.480 --> 00:10:16.362 what he's saying here doesn't sound 00:10:16.362 --> 00:10:16.440 >> [snorts] 00:10:16.440 --> 00:10:19.240 >> insane but it it definitely accumulates 00:10:19.240 --> 00:10:20.560 about how 00:10:20.560 --> 00:10:23.120 reviews take a long time and then he 00:10:23.120 --> 00:10:24.520 makes a very interesting point to say 00:10:24.520 --> 00:10:27.280 that AI can't fix this. 00:10:27.280 --> 00:10:30.520 And like just like what we talked about 00:10:30.520 --> 00:10:32.000 like a lot of people are are creating 00:10:32.000 --> 00:10:33.400 these AI 00:10:33.400 --> 00:10:35.960 flower wheels or AI harnesses or 00:10:35.960 --> 00:10:38.960 orchestrations where like you know oh I 00:10:38.960 --> 00:10:41.000 mean this I mean doesn't it sound like 00:10:41.000 --> 00:10:43.800 you right now in a in a in some way of 00:10:43.800 --> 00:10:45.440 instant like well I have this prototype. 00:10:45.440 --> 00:10:46.760 >> And this a month ago. 00:10:46.760 --> 00:10:48.960 >> Yeah exactly but the but the prototype 00:10:48.960 --> 00:10:51.560 is getting busy buggy sorry we need to 00:10:51.560 --> 00:10:53.440 tell AI to fix this problem. 00:10:53.440 --> 00:10:55.040 >> I didn't see I need to 00:10:55.040 --> 00:10:56.760 to let AI fix it. 00:10:56.760 --> 00:10:58.839 I said I want a formal validation 00:10:58.839 --> 00:11:00.880 framework around AI. 00:11:00.880 --> 00:11:03.520 >> Sorry I'm I am making this a meal of the 00:11:03.520 --> 00:11:05.440 straw materialization but like I I think 00:11:05.440 --> 00:11:06.920 you get the point here is that the a lot 00:11:06.920 --> 00:11:09.680 of people are getting into this like 00:11:09.680 --> 00:11:12.680 trap where like AI caused the problem 00:11:12.680 --> 00:11:14.160 but maybe I can get AI to fix the 00:11:14.160 --> 00:11:16.240 problem and then now we have a different 00:11:16.240 --> 00:11:16.920 sorts of problem. 00:11:16.920 --> 00:11:18.720 >> Absolutely and this is what I also said 00:11:18.720 --> 00:11:20.120 said to my friends I mean the fact that 00:11:20.120 --> 00:11:22.920 we have this problem and we know that 00:11:22.920 --> 00:11:24.720 because we're taking something and we're 00:11:24.720 --> 00:11:26.600 scanning it out of a team and clearly 00:11:26.600 --> 00:11:28.800 the quality is a problem so we need to 00:11:28.800 --> 00:11:30.480 solve it and that's definitely what's 00:11:30.480 --> 00:11:31.960 going to be the focus of the next few 00:11:31.960 --> 00:11:34.080 months, which is the validation 00:11:34.080 --> 00:11:36.760 mechanisms of the AI output. Cuz 00:11:36.760 --> 00:11:38.440 everybody's on board with SDD now, 00:11:38.440 --> 00:11:40.040 right? I mean, it's going all over the 00:11:40.040 --> 00:11:42.000 place. My friends are constantly sharing 00:11:42.000 --> 00:11:45.120 Singapore GovTech is doing a 00:11:45.120 --> 00:11:47.960 a trial of it. Um other organizations 00:11:47.960 --> 00:11:50.080 are trialing it. 00:11:50.080 --> 00:11:50.440 >> Oh, okay. 00:11:50.440 --> 00:11:51.720 >> Yeah, spec driven development. That's 00:11:51.720 --> 00:11:54.480 That's kind of like been being adopted 00:11:54.480 --> 00:11:55.600 everywhere, right? 00:11:55.600 --> 00:11:56.920 >> I feel like the problems that you just 00:11:56.920 --> 00:12:00.040 talked about like that you 00:12:00.040 --> 00:12:02.160 wrote the spec, but then somewhere down 00:12:02.160 --> 00:12:04.160 the line there was a quality lapse or 00:12:04.160 --> 00:12:06.440 some junior didn't understand. Like 00:12:06.440 --> 00:12:08.240 >> The whole purpose of SDD is to catch 00:12:08.240 --> 00:12:10.600 this. And And I know for a fact that my 00:12:10.600 --> 00:12:13.320 friend wrote a blog post talking about 00:12:13.320 --> 00:12:14.960 how vibe coding creates all these 00:12:14.960 --> 00:12:16.720 problems and how spec driven design 00:12:16.720 --> 00:12:17.960 solves them. 00:12:17.960 --> 00:12:19.640 And I tell him like I would 00:12:19.640 --> 00:12:22.320 >> But surely surely if we we ran the clock 00:12:22.320 --> 00:12:24.440 uh 20 years back, I'm sure some 00:12:24.440 --> 00:12:25.920 waterfall 00:12:25.920 --> 00:12:27.560 IBM proponent would have said the same 00:12:27.560 --> 00:12:28.480 thing 00:12:28.480 --> 00:12:29.720 about 00:12:29.720 --> 00:12:31.280 his specs. He would have said like, "Oh, 00:12:31.280 --> 00:12:33.040 yeah, if the spec would have caught that 00:12:33.040 --> 00:12:34.680 problem because we would have done the 00:12:34.680 --> 00:12:36.640 upfront design." What What What about 00:12:36.640 --> 00:12:38.600 the whole agile thing where you 00:12:38.600 --> 00:12:39.960 >> No, this is not my This is not the 00:12:39.960 --> 00:12:41.839 problem that I'm saying that there is, 00:12:41.839 --> 00:12:44.000 right? I'm saying the spec clearly 00:12:44.000 --> 00:12:45.839 defined it this way and the 00:12:45.839 --> 00:12:48.040 implementation does not align. I'm not 00:12:48.040 --> 00:12:49.880 saying the spec 00:12:49.880 --> 00:12:51.000 um 00:12:51.000 --> 00:12:53.080 wasn't properly designed. I never said 00:12:53.080 --> 00:12:54.760 that. I said the spec was very clearly 00:12:54.760 --> 00:12:56.720 defined, but the implementation doesn't 00:12:56.720 --> 00:12:58.600 match the spec. The agents took 00:12:58.600 --> 00:13:00.040 liberties. 00:13:00.040 --> 00:13:01.400 And then the funny thing is when I see 00:13:01.400 --> 00:13:02.920 the agent take liberties when I'm I'm 00:13:02.920 --> 00:13:05.600 driving it, I stop it and I asked it 00:13:05.600 --> 00:13:07.400 why. And often times there's a very good 00:13:07.400 --> 00:13:10.320 reason. Sometimes it was because the 00:13:10.320 --> 00:13:12.920 task description said one thing and the 00:13:12.920 --> 00:13:15.480 original plan, like the the very first 00:13:15.480 --> 00:13:17.440 functional requirement in the spec, said 00:13:17.440 --> 00:13:19.320 something else. And the agent says, 00:13:19.320 --> 00:13:21.080 "Spec rules all. So, I went with the 00:13:21.080 --> 00:13:22.720 spec." And the reason that the task 00:13:22.720 --> 00:13:24.720 definition was different because halfway 00:13:24.720 --> 00:13:26.640 through research, when it filtered down 00:13:26.640 --> 00:13:29.040 into the task, I made a change and I 00:13:29.040 --> 00:13:31.000 didn't back update it into the spec. 00:13:31.000 --> 00:13:32.480 Something like that. That that's one 00:13:32.480 --> 00:13:33.440 case where I'm 00:13:33.440 --> 00:13:35.240 >> wrong with the with the process. 00:13:35.240 --> 00:13:37.040 >> Yeah. And and I think sometimes it's 00:13:37.040 --> 00:13:39.840 because I didn't do the full cross task 00:13:39.840 --> 00:13:41.600 plan and spec 00:13:41.600 --> 00:13:43.880 validation cuz I do skip that one 00:13:43.880 --> 00:13:45.160 sometimes. 00:13:45.160 --> 00:13:46.520 Um which also made me think like, "Hey, 00:13:46.520 --> 00:13:48.040 we need to make sure we need to keep a 00:13:48.040 --> 00:13:51.240 flag to see if somebody at least once 00:13:51.240 --> 00:13:53.200 has run that validation uh or 00:13:53.200 --> 00:13:55.040 verification thing." Which is a very 00:13:55.040 --> 00:13:58.400 extensive read cross artifact um 00:13:58.400 --> 00:14:00.760 validator that, you know, you can use 00:14:00.760 --> 00:14:02.520 different models. I used to run those on 00:14:02.520 --> 00:14:04.040 Codex when I would always run out of 00:14:04.040 --> 00:14:05.320 tokens. Today, I don't really switch 00:14:05.320 --> 00:14:05.560 models. 00:14:05.560 --> 00:14:07.000 >> So you use different models. 00:14:07.000 --> 00:14:08.200 >> I don't think different models is 00:14:08.200 --> 00:14:09.960 required. It was just because I ran out 00:14:09.960 --> 00:14:12.320 of tokens. But yeah, but that's just one 00:14:12.320 --> 00:14:14.120 case where the implementation diverges. 00:14:14.120 --> 00:14:16.120 I want to give you one more example, 00:14:16.120 --> 00:14:17.480 which is 00:14:17.480 --> 00:14:19.680 um and now I'm forgetting it. 00:14:19.680 --> 00:14:22.240 So one was that that the task and the 00:14:22.240 --> 00:14:24.640 plan did not align. The second one that 00:14:24.640 --> 00:14:28.040 I've seen is that the plan says we can 00:14:28.040 --> 00:14:30.040 use this library this way and then the 00:14:30.040 --> 00:14:32.400 agent says it doesn't work. Well, let me 00:14:32.400 --> 00:14:34.120 go back and simplify this or something 00:14:34.120 --> 00:14:35.760 like that, right? You see it happen, 00:14:35.760 --> 00:14:36.800 right? The 00:14:36.800 --> 00:14:38.360 you make a plan with the agent, it it 00:14:38.360 --> 00:14:40.240 reads the docs of the library, it 00:14:40.240 --> 00:14:42.400 decides to do things certain way, and 00:14:42.400 --> 00:14:44.080 then finally when it's implementing, it 00:14:44.080 --> 00:14:45.760 doesn't work. And then sometimes Cloud 00:14:45.760 --> 00:14:47.320 Code goes like, "Well, you know what? 00:14:47.320 --> 00:14:48.960 This is too complicated. Let me simplify 00:14:48.960 --> 00:14:52.080 it." Then you know that no, stop. So I 00:14:52.080 --> 00:14:53.880 stopped it and I asked it very clearly, 00:14:53.880 --> 00:14:55.800 "What is the problem?" And and it's it's 00:14:55.800 --> 00:14:57.800 totally valid, right? It's totally 00:14:57.800 --> 00:14:59.880 valid. But the the most the biggest 00:14:59.880 --> 00:15:01.720 problem is the agent just keep going 00:15:01.720 --> 00:15:03.280 instead of stopping and saying like, 00:15:03.280 --> 00:15:05.520 "Hey, plan doesn't work." And it's very 00:15:05.520 --> 00:15:07.360 hard to define when when are we okay 00:15:07.360 --> 00:15:09.040 with the agent stopping and when are we 00:15:09.040 --> 00:15:10.560 not okay with the agent stopping? 00:15:10.560 --> 00:15:12.240 Because ideally, we want like a Ralph 00:15:12.240 --> 00:15:13.600 Wriggum type of loop that keeps going 00:15:13.600 --> 00:15:15.880 until the task is complete, right? So 00:15:15.880 --> 00:15:16.240 >> Yeah. 00:15:16.240 --> 00:15:17.600 >> Yeah. So that's that's the problem. 00:15:17.600 --> 00:15:18.960 >> that there's different philosophies 00:15:18.960 --> 00:15:21.320 here. Like I'm I'm I'm more in the like 00:15:21.320 --> 00:15:22.680 just chat to it camp. 00:15:22.680 --> 00:15:23.960 >> Just chat with it. Yes, but you don't 00:15:23.960 --> 00:15:25.280 want to be in the loop the whole time. 00:15:25.280 --> 00:15:26.880 Maybe maybe it's it's good, right? Maybe 00:15:26.880 --> 00:15:30.120 it's like yeah, we still have a job. 00:15:30.120 --> 00:15:32.400 >> Well, I just feel like the small 00:15:32.400 --> 00:15:35.680 iterations get to the solution better 00:15:35.680 --> 00:15:38.520 than going back to step one in a sense. 00:15:38.520 --> 00:15:39.280 Hmm. 00:15:39.280 --> 00:15:41.160 >> So, you could argue that that the spec 00:15:41.160 --> 00:15:43.200 was too large and it should have broken 00:15:43.200 --> 00:15:45.720 down more. Um 00:15:45.720 --> 00:15:47.280 you could argue that, you know, don't do 00:15:47.280 --> 00:15:49.280 spec driven development in Vibe. I don't 00:15:49.280 --> 00:15:51.080 agree with that. But you could say if 00:15:51.080 --> 00:15:52.920 you do spec driven development, really 00:15:52.920 --> 00:15:54.680 scope down. And that to be honest is one 00:15:54.680 --> 00:15:56.560 of the one of the principles I put in 00:15:56.560 --> 00:15:58.840 the constitution now, which is shortest 00:15:58.840 --> 00:16:01.480 path to MVP, short-lived branches, scope 00:16:01.480 --> 00:16:03.520 down, scope down, scope down. Like do 00:16:03.520 --> 00:16:05.680 not Let's all YAGNI, you ain't going to 00:16:05.680 --> 00:16:07.760 need it. Nothing that has a clear use 00:16:07.760 --> 00:16:10.280 case should ever be included in the in 00:16:10.280 --> 00:16:12.040 in in the implementation plan. 00:16:12.040 --> 00:16:14.000 >> Mhm. So, I'm just highlighted agent 00:16:14.000 --> 00:16:15.800 framework and I'm just I can't help but 00:16:15.800 --> 00:16:18.160 have this thought that like what is the 00:16:18.160 --> 00:16:20.640 difference between spec driven 00:16:20.640 --> 00:16:23.320 development that you're creating and an 00:16:23.320 --> 00:16:25.920 agent framework? Cuz surely 00:16:25.920 --> 00:16:27.760 the difference 00:16:27.760 --> 00:16:29.160 there's not a lot of difference there or 00:16:29.160 --> 00:16:30.920 it can easily be construed as a 00:16:30.920 --> 00:16:31.960 framework, couldn't it? 00:16:31.960 --> 00:16:34.360 >> I'm not sure, but to me, when you start 00:16:34.360 --> 00:16:35.720 talking about agents and what the 00:16:35.720 --> 00:16:38.240 majority of like articles I read are 00:16:38.240 --> 00:16:39.760 doing, they're looking at like agent 00:16:39.760 --> 00:16:42.080 teams and personas and all that. I I 00:16:42.080 --> 00:16:43.520 don't want to go there. I don't want to 00:16:43.520 --> 00:16:45.160 look at like agents completely 00:16:45.160 --> 00:16:47.880 individually doing all the work. I I 00:16:47.880 --> 00:16:49.880 think you need to have a way to define 00:16:49.880 --> 00:16:52.520 the work. You need to have a way to to 00:16:52.520 --> 00:16:53.839 to really 00:16:53.839 --> 00:16:55.839 you know, master and manage what the 00:16:55.839 --> 00:16:57.320 work is. 00:16:57.320 --> 00:16:59.600 Um because I mean, a lot of these teams 00:16:59.600 --> 00:17:01.000 that are saying, "Oh, we we have a 00:17:01.000 --> 00:17:03.280 linear uh board with issues and we just 00:17:03.280 --> 00:17:05.920 assign it to our uh swarm of agents and 00:17:05.920 --> 00:17:08.040 they they work on it and then we get a 00:17:08.040 --> 00:17:10.520 working solution." Either maybe that 00:17:10.520 --> 00:17:13.400 that their SDD spec driven development 00:17:13.400 --> 00:17:15.880 is is basically their classic product 00:17:15.880 --> 00:17:17.079 owners 00:17:17.079 --> 00:17:19.040 um and engineers defining 00:17:19.040 --> 00:17:19.720 >> Mhm. 00:17:19.720 --> 00:17:21.680 >> the actual ticket uh down to a very 00:17:21.680 --> 00:17:23.360 small task that an agent can complete. 00:17:23.360 --> 00:17:25.079 And yeah, I mean, if you do that, that's 00:17:25.079 --> 00:17:27.240 perfect, right? But I'm focused on that 00:17:27.240 --> 00:17:29.440 that life cycle part. Like 00:17:29.440 --> 00:17:29.760 >> Yeah. 00:17:29.760 --> 00:17:32.120 >> taking it So, the whole like agent 00:17:32.120 --> 00:17:33.520 framework and individual working again, 00:17:33.520 --> 00:17:35.120 like I said, I don't want to be in the 00:17:35.120 --> 00:17:37.280 driver's seat, but I very much feel I 00:17:37.280 --> 00:17:39.040 have to be right now. And to me, it 00:17:39.040 --> 00:17:41.840 feels like you I I I want to control 00:17:41.840 --> 00:17:44.000 the the you know, the task definitions 00:17:44.000 --> 00:17:46.080 and how the work goes into the agent 00:17:46.080 --> 00:17:48.240 framework. That's where I'm focused on. 00:17:48.240 --> 00:17:50.160 >> So, the other thought, because this is 00:17:50.160 --> 00:17:52.200 applicable to a problem I have with at 00:17:52.200 --> 00:17:54.200 work right now, 00:17:54.200 --> 00:17:58.760 is documentation. So, your spec, is that 00:17:58.760 --> 00:18:01.560 >> It's not a documentation. 00:18:01.560 --> 00:18:03.080 >> No, I had the same and it's a very good 00:18:03.080 --> 00:18:04.560 question that I did a session with my 00:18:04.560 --> 00:18:06.160 friend. I said, "Hey, 00:18:06.160 --> 00:18:07.760 um basically what I'm doing is I'm 00:18:07.760 --> 00:18:09.920 giving people the chance to 00:18:09.920 --> 00:18:12.040 um give me some project they want to 00:18:12.040 --> 00:18:14.400 work on, and I use my cloud Claude Code 00:18:14.400 --> 00:18:16.880 agent tokens, and I spend 2 hours, and 00:18:16.880 --> 00:18:18.680 we have to complete it within 2 hours, 00:18:18.680 --> 00:18:21.160 and I'll uh give you a working solution. 00:18:21.160 --> 00:18:22.160 So, 00:18:22.160 --> 00:18:24.160 I initialize the repository with or you 00:18:24.160 --> 00:18:26.640 give me a a repository, you give me a 00:18:26.640 --> 00:18:28.880 feature to work on, and the condition is 00:18:28.880 --> 00:18:30.680 that I can record and publish it, and 00:18:30.680 --> 00:18:32.280 then I hopefully give you a working 00:18:32.280 --> 00:18:34.440 feature implemented onto your code base. 00:18:34.440 --> 00:18:35.760 So, I did that with my friend, and he 00:18:35.760 --> 00:18:37.680 asked me exactly the same question. So, 00:18:37.680 --> 00:18:40.120 the spec, is that a documentation? And 00:18:40.120 --> 00:18:41.680 and I actually really didn't realize it, 00:18:41.680 --> 00:18:44.600 but you need to mean I think you also 00:18:44.600 --> 00:18:46.760 asked me this last time. No, it was my 00:18:46.760 --> 00:18:47.160 friend. 00:18:47.160 --> 00:18:48.640 >> This is the big problem I have at work. 00:18:48.640 --> 00:18:50.200 I can maybe explain to you the problem. 00:18:50.200 --> 00:18:51.240 But yeah, so what was 00:18:51.240 --> 00:18:52.960 >> Sorry, I interrupted you before you 00:18:52.960 --> 00:18:53.280 explained. 00:18:53.280 --> 00:18:54.960 >> What was your resolution? How did you 00:18:54.960 --> 00:18:56.520 What's your approach to documentation? 00:18:56.520 --> 00:18:59.480 >> I believe it should be um 00:18:59.480 --> 00:19:01.960 in the repository as um 00:19:01.960 --> 00:19:03.200 This is what I did for Spec Ledger, 00:19:03.200 --> 00:19:06.440 right? I I created doc/design that 00:19:06.440 --> 00:19:08.960 documents every layer. Like, what is our 00:19:08.960 --> 00:19:10.920 our design philosophy? How do we 00:19:10.920 --> 00:19:13.280 organize the command line interface? how 00:19:13.280 --> 00:19:15.120 do we organize the what's the 00:19:15.120 --> 00:19:17.040 responsibility of each layer within this 00:19:17.040 --> 00:19:18.200 agentic design framework. 00:19:18.200 --> 00:19:19.400 >> Okay. So, you have a separate 00:19:19.400 --> 00:19:21.040 documentation, but who edits 00:19:21.040 --> 00:19:23.600 documentation? Agents or humans? Or or 00:19:23.600 --> 00:19:25.760 both? What's your How do you keep the 00:19:25.760 --> 00:19:26.320 documentation? 00:19:26.320 --> 00:19:28.320 >> needs to update the documentation. That 00:19:28.320 --> 00:19:30.880 is the humans need to confirm and and 00:19:30.880 --> 00:19:32.480 review. I mean, ultimately the agent 00:19:32.480 --> 00:19:33.840 writes the docs, but the humans need to 00:19:33.840 --> 00:19:36.560 review and and and can modify the docs. 00:19:36.560 --> 00:19:39.360 >> So, I feel partly responsible for this 00:19:39.360 --> 00:19:41.880 misalignment with my my colleague. But 00:19:41.880 --> 00:19:43.400 like for example, one thing that I like 00:19:43.400 --> 00:19:46.160 to refer to a lot is this 00:19:46.160 --> 00:19:47.040 um 00:19:47.040 --> 00:19:49.240 this French site called I think I showed 00:19:49.240 --> 00:19:50.360 you this before probably. 00:19:50.360 --> 00:19:52.120 >> Yeah. 00:19:52.120 --> 00:19:53.440 >> Whatever. Don't even know how to 00:19:53.440 --> 00:19:55.240 pronounce that word. 00:19:55.240 --> 00:19:57.240 We have these different sort of styles 00:19:57.240 --> 00:19:59.440 of documentation, right? So, that we 00:19:59.440 --> 00:20:00.840 have some documentation at work. It's 00:20:00.840 --> 00:20:02.760 quite a lot of documentation, like at 00:20:02.760 --> 00:20:06.480 least 500 files of markdown 00:20:06.480 --> 00:20:09.320 um exported from Confluence. And what 00:20:09.320 --> 00:20:11.440 he's done 00:20:11.440 --> 00:20:13.440 is pretty good, but at the same time 00:20:13.440 --> 00:20:17.240 it's kind of made made things a bit more 00:20:17.240 --> 00:20:19.800 problematic because he's taken the 500 00:20:19.800 --> 00:20:22.640 documentations and then he's generated a 00:20:22.640 --> 00:20:24.600 how-to guide. He's generated an 00:20:24.600 --> 00:20:27.000 information reference. He's generated um 00:20:27.000 --> 00:20:29.040 some explanation documents. He's He's 00:20:29.040 --> 00:20:31.040 generated some tutorials from that 00:20:31.040 --> 00:20:32.680 original 00:20:32.680 --> 00:20:33.760 um 00:20:33.760 --> 00:20:35.960 corpus. So, now we have actually more 00:20:35.960 --> 00:20:37.880 documentation than what we had 00:20:37.880 --> 00:20:41.400 previously. And now it's like becomes a 00:20:41.400 --> 00:20:44.160 like like where's the source of truth? 00:20:44.160 --> 00:20:45.960 >> Yeah, you They They become 00:20:45.960 --> 00:20:46.440 out of sync. 00:20:46.440 --> 00:20:47.640 >> Now you have a broken mirror. 00:20:47.640 --> 00:20:50.080 >> Well, 00:20:50.080 --> 00:20:51.520 yeah, something like that. 00:20:51.520 --> 00:20:53.840 >> Yeah, because one the how-to guide is 00:20:53.840 --> 00:20:55.520 still talking about A, well the 00:20:55.520 --> 00:20:57.160 reference has been updated and actually 00:20:57.160 --> 00:21:00.920 it's A A A uh A alpha or A beta. And and 00:21:00.920 --> 00:21:03.240 now they're like slightly different. And 00:21:03.240 --> 00:21:04.400 now how to realign 00:21:04.400 --> 00:21:06.600 >> Exactly. And that now we Yeah, we now we 00:21:06.600 --> 00:21:09.400 have a serious alignment problem. Um 00:21:09.400 --> 00:21:11.320 um 00:21:11.320 --> 00:21:12.200 But like 00:21:12.200 --> 00:21:14.960 my colleague is he's good. 00:21:14.960 --> 00:21:18.520 It's I just I just that I feel like 00:21:18.520 --> 00:21:19.480 that 00:21:19.480 --> 00:21:21.480 I mean the things he's generated are 00:21:21.480 --> 00:21:22.800 actually a lot better than what we have 00:21:22.800 --> 00:21:24.600 currently, but the trouble is it's like 00:21:24.600 --> 00:21:26.400 there's no way to 00:21:26.400 --> 00:21:28.120 keep things 00:21:28.120 --> 00:21:30.320 synced in my mind without 00:21:30.320 --> 00:21:33.640 running AI over every like every time we 00:21:33.640 --> 00:21:35.680 update this thing, then maybe I have to 00:21:35.680 --> 00:21:38.320 have an AI job to create the tutorial or 00:21:38.320 --> 00:21:39.760 update the tutorial and update the 00:21:39.760 --> 00:21:41.520 explanation, update the reference. 00:21:41.520 --> 00:21:42.160 >> You're just going to 00:21:42.160 --> 00:21:43.280 >> you run the AI, it's going to find a 00:21:43.280 --> 00:21:44.680 difference and tell you there's another 00:21:44.680 --> 00:21:45.240 difference. 00:21:45.240 --> 00:21:46.640 >> Exactly. I'm just 00:21:46.640 --> 00:21:48.640 I'm already in this like 00:21:48.640 --> 00:21:50.720 Kafkaesque loop and I can't help but 00:21:50.720 --> 00:21:52.600 think when we went back 00:21:52.600 --> 00:21:54.040 The original documentation that we have 00:21:54.040 --> 00:21:55.680 in Confluence, even though sometimes it 00:21:55.680 --> 00:21:57.480 was wrong, at least you had the source 00:21:57.480 --> 00:21:59.120 of truth, right? Like all the 00:21:59.120 --> 00:22:00.960 documentation for this particular 00:22:00.960 --> 00:22:03.440 feature is here and and that's that's 00:22:03.440 --> 00:22:05.040 where it is. It is not like in four 00:22:05.040 --> 00:22:06.200 places now. 00:22:06.200 --> 00:22:07.640 >> And it's also something that that that 00:22:07.640 --> 00:22:10.160 Opus or Sonnet or all of them like to do 00:22:10.160 --> 00:22:12.440 is when I when I have like, "Hey, we're 00:22:12.440 --> 00:22:14.840 creating this index document." And then 00:22:14.840 --> 00:22:17.160 it goes, "Do you want a short summary of 00:22:17.160 --> 00:22:19.680 like the other document or just a direct 00:22:19.680 --> 00:22:21.360 external link?" 00:22:21.360 --> 00:22:23.040 Um and every time it creates a short 00:22:23.040 --> 00:22:24.240 summary, that's that's something that 00:22:24.240 --> 00:22:26.920 potentially gets outdated and and 00:22:26.920 --> 00:22:28.360 doesn't show the original like doesn't 00:22:28.360 --> 00:22:29.840 actually show the correct contents of 00:22:29.840 --> 00:22:32.200 what it's linking to, right? I mean, I I 00:22:32.200 --> 00:22:33.280 tend to 00:22:33.280 --> 00:22:35.400 I always ask it to generate options and 00:22:35.400 --> 00:22:37.240 then ask for my alignment and then it 00:22:37.240 --> 00:22:39.440 always gives me this option to like 00:22:39.440 --> 00:22:41.120 duplicate some of the information here 00:22:41.120 --> 00:22:42.120 and then create a link. 00:22:42.120 --> 00:22:43.160 >> Yeah. 00:22:43.160 --> 00:22:45.360 >> Which sounds interesting because it's 00:22:45.360 --> 00:22:47.000 kind of like progressive disclosure, 00:22:47.000 --> 00:22:48.440 right? You you read a little bit and 00:22:48.440 --> 00:22:50.840 then you can go in into deeper details, 00:22:50.840 --> 00:22:53.520 but it just creates an another thing 00:22:53.520 --> 00:22:55.400 like earlier when I said the task 00:22:55.400 --> 00:22:57.320 definition is a one thing, the plan kind 00:22:57.320 --> 00:22:58.960 of hinted at it and then the the spec 00:22:58.960 --> 00:23:00.560 was completely different. You need to go 00:23:00.560 --> 00:23:02.600 all the way back to update all of them 00:23:02.600 --> 00:23:04.160 or you get all these misalignments. 00:23:04.160 --> 00:23:06.800 >> Yeah, this this 00:23:06.800 --> 00:23:08.440 Yeah, when you when you talk about uh 00:23:08.440 --> 00:23:11.200 spec-driven development, I do really 00:23:11.200 --> 00:23:14.040 like the source of truth element to it. 00:23:14.040 --> 00:23:16.400 I mean, I'm I maybe made a a brash 00:23:16.400 --> 00:23:18.480 assumption there, but with your 00:23:18.480 --> 00:23:20.600 spec-driven development, like there's 00:23:20.600 --> 00:23:22.800 one document that is the source of truth 00:23:22.800 --> 00:23:23.840 for that 00:23:23.840 --> 00:23:25.960 or or or there's a part of the document 00:23:25.960 --> 00:23:26.920 that's the source of truth for that 00:23:26.920 --> 00:23:28.160 feature, say, right? 00:23:28.160 --> 00:23:30.880 >> Yeah, but if you if you So, the thing it 00:23:30.880 --> 00:23:32.760 it I guess it depends on which framework 00:23:32.760 --> 00:23:34.680 that you use because I know that Open 00:23:34.680 --> 00:23:36.920 Spec apparently has the idea of um like 00:23:36.920 --> 00:23:39.800 archiving and um but but with the one 00:23:39.800 --> 00:23:41.840 that we currently have with Spec Kit and 00:23:41.840 --> 00:23:43.600 we haven't changed too much about it, 00:23:43.600 --> 00:23:47.080 you end up with 20 20 spec 20 spec 00:23:47.080 --> 00:23:48.840 folders. And each one of them have a 00:23:48.840 --> 00:23:50.800 slightly iteration on top of the 00:23:50.800 --> 00:23:53.040 previous features, right? So, your 00:23:53.040 --> 00:23:54.280 question earlier, like where is the 00:23:54.280 --> 00:23:56.000 source of truth? It's not in the spec 00:23:56.000 --> 00:23:58.680 folders because you have the version one 00:23:58.680 --> 00:24:00.680 which was we do this initial spec with 00:24:00.680 --> 00:24:02.880 these initial initial features. And then 00:24:02.880 --> 00:24:04.960 we build on top of that in in the second 00:24:04.960 --> 00:24:06.680 iteration, we we add a bunch of new 00:24:06.680 --> 00:24:08.600 features, we modify the original user 00:24:08.600 --> 00:24:10.440 story slightly. And then the third one 00:24:10.440 --> 00:24:11.920 but and we don't go back to the first 00:24:11.920 --> 00:24:13.720 one because that's the original user 00:24:13.720 --> 00:24:16.200 stories, right? So, I do feel you we 00:24:16.200 --> 00:24:19.440 need to have a canonical root of repo 00:24:19.440 --> 00:24:21.880 once this user so once the second spec 00:24:21.880 --> 00:24:24.520 is merged all the user stories in in 00:24:24.520 --> 00:24:26.880 like the the root need to be aligned. 00:24:26.880 --> 00:24:28.560 So, because now we have the original 00:24:28.560 --> 00:24:30.200 user story here, we have the new user 00:24:30.200 --> 00:24:31.920 story there, and we have the slightly 00:24:31.920 --> 00:24:34.760 modified original plus use changes that 00:24:34.760 --> 00:24:37.000 were introduced by the next one. So, 00:24:37.000 --> 00:24:38.640 that is currently not in the Spec Kit 00:24:38.640 --> 00:24:40.720 and that is something that I put in 00:24:40.720 --> 00:24:42.280 That's not in the Spec Kit. That's not 00:24:42.280 --> 00:24:43.040 That's not in 00:24:43.040 --> 00:24:44.440 >> And wait, wait, are you in Spec Kit or 00:24:44.440 --> 00:24:45.160 or or 00:24:45.160 --> 00:24:48.040 I mean, is it not in your spec ledger or 00:24:48.040 --> 00:24:49.800 I don't I don't quite follow what what 00:24:49.800 --> 00:24:50.920 you mean by that? 00:24:50.920 --> 00:24:53.080 >> It's It's No, it's not It's not in Spec 00:24:53.080 --> 00:24:55.040 Kit. I don't know if Open Spec is doing 00:24:55.040 --> 00:24:56.840 it. Maybe I should I should investigate 00:24:56.840 --> 00:24:58.280 it further, but it's something that I 00:24:58.280 --> 00:25:00.120 put in my constitution now, like I just 00:25:00.120 --> 00:25:02.120 explained, like I just added a docs 00:25:02.120 --> 00:25:04.600 design markdown index, and I said when 00:25:04.600 --> 00:25:07.480 we are building, we go and update this 00:25:07.480 --> 00:25:10.040 the documentation that defines the the 00:25:10.040 --> 00:25:11.080 system like design. 00:25:11.080 --> 00:25:12.360 >> Yeah, that seems like a fundamental 00:25:12.360 --> 00:25:14.960 thing that needs needs to be updated and 00:25:14.960 --> 00:25:18.480 it needs to be canonical. Like I 00:25:18.480 --> 00:25:20.880 I was thinking naively at at my 00:25:20.880 --> 00:25:23.760 workplace that we could just do what AWS 00:25:23.760 --> 00:25:25.200 docs 00:25:25.200 --> 00:25:26.920 uh do, like I don't know if you've ever 00:25:26.920 --> 00:25:29.800 used this uh this I I I know you're 00:25:29.800 --> 00:25:32.520 going to probably revolt when I say MCP, 00:25:32.520 --> 00:25:33.480 but this particular 00:25:33.480 --> 00:25:35.280 >> I am I am controversial, right? Now, 00:25:35.280 --> 00:25:37.000 everybody says MCP is that? No, I say 00:25:37.000 --> 00:25:39.040 actually I really enjoy MCP. 00:25:39.040 --> 00:25:41.000 >> Yeah, I 00:25:41.000 --> 00:25:44.880 Well, this this MCP uh AWS docs 00:25:44.880 --> 00:25:46.320 uh feature, I don't know how you want 00:25:46.320 --> 00:25:50.520 this this service where where uh you can 00:25:50.520 --> 00:25:52.800 jump into Claude, set up MCP, and then 00:25:52.800 --> 00:25:55.600 ask questions about AWS docs. It's it's 00:25:55.600 --> 00:25:57.600 wonderful. And this is what I wanted at 00:25:57.600 --> 00:26:00.240 work. I wanted something like this where 00:26:00.240 --> 00:26:00.760 >> For your docs. 00:26:00.760 --> 00:26:03.880 >> where your docs the the canonical docs 00:26:03.880 --> 00:26:07.400 of AWS or canonical docs are queried, 00:26:07.400 --> 00:26:09.800 and and the cool thing about using an AI 00:26:09.800 --> 00:26:11.400 agent like Claude is that like if you 00:26:11.400 --> 00:26:14.000 wanted a tutorial, you can you can get 00:26:14.000 --> 00:26:15.600 you can say to 00:26:15.600 --> 00:26:17.800 uh your prompt to say like give me the 00:26:17.800 --> 00:26:20.840 steps to set up an S3 bucket with with 00:26:20.840 --> 00:26:22.160 the I don't know, an access point or 00:26:22.160 --> 00:26:23.480 something. And it can read the 00:26:23.480 --> 00:26:25.440 documentation, and even though it's not 00:26:25.440 --> 00:26:27.440 a tutorial, even though it's like I 00:26:27.440 --> 00:26:29.080 don't know what the AWS docs are. 00:26:29.080 --> 00:26:30.520 >> It can generate a tutorial for you. 00:26:30.520 --> 00:26:31.840 >> It can it can generate a tutorial. So, 00:26:31.840 --> 00:26:33.480 this is what I wanted at work, but the 00:26:33.480 --> 00:26:35.440 trouble is we don't have an MCP. 00:26:35.440 --> 00:26:37.440 >> I don't think the problem is the MCP. 00:26:37.440 --> 00:26:40.480 First off, you can what with the CDK TF 00:26:40.480 --> 00:26:44.120 my fork, we we are we got OSS support of 00:26:44.120 --> 00:26:47.600 Mentlify, and they provide a doc MCP out 00:26:47.600 --> 00:26:49.440 of the box out of their platform. So, 00:26:49.440 --> 00:26:51.200 you put your 00:26:51.200 --> 00:26:55.080 um handman hand crafted 00:26:55.080 --> 00:26:57.840 um documentation. Actually, we have two 00:26:57.840 --> 00:26:59.800 two parts of the CDK TF docs, and that's 00:26:59.800 --> 00:27:01.440 nothing we built. It's what the original 00:27:01.440 --> 00:27:03.880 CDK TF project had. Which is on one 00:27:03.880 --> 00:27:06.920 side, there's the, you know, concepts 00:27:06.920 --> 00:27:09.120 um type of information about, you know, 00:27:09.120 --> 00:27:11.200 what's a stack, what's what's an aspect, 00:27:11.200 --> 00:27:12.960 and all that information. And on the 00:27:12.960 --> 00:27:14.800 other side, there's an API reference, 00:27:14.800 --> 00:27:16.200 which is completely generated from the 00:27:16.200 --> 00:27:18.160 TypeScript uh type definitions, you 00:27:18.160 --> 00:27:21.480 know, from through JSII. Um you get the 00:27:21.480 --> 00:27:23.640 core schema, like almost like JSON 00:27:23.640 --> 00:27:26.000 schema type script schemas, and you 00:27:26.000 --> 00:27:28.160 generate it cross-translated. It says, 00:27:28.160 --> 00:27:30.080 for Python, it means it's like this, it 00:27:30.080 --> 00:27:32.400 uses JSII. So, it generates the API 00:27:32.400 --> 00:27:34.320 reference. It's like JS docs, Java docs, 00:27:34.320 --> 00:27:35.920 if you remember, right? It generates 00:27:35.920 --> 00:27:37.400 this huge um 00:27:37.400 --> 00:27:40.440 API reference uh libraries, right? 00:27:40.440 --> 00:27:42.280 >> Which are pretty cool. I mean 00:27:42.280 --> 00:27:43.080 >> Yeah, and I think that's 00:27:43.080 --> 00:27:45.160 >> a bit a bit dry, but they have a lot of 00:27:45.160 --> 00:27:46.320 Yeah, it's it's interesting. 00:27:46.320 --> 00:27:47.560 >> Yeah, uh when when we when we learn 00:27:47.560 --> 00:27:50.040 programming in school or in university, 00:27:50.040 --> 00:27:53.320 you had to go through the Java SDK API 00:27:53.320 --> 00:27:54.280 docs, right? 00:27:54.280 --> 00:27:55.040 >> Yeah, the PHP 00:27:55.040 --> 00:27:55.520 >> to 00:27:55.520 --> 00:27:57.360 >> I always think of the PHP docs for some 00:27:57.360 --> 00:27:59.920 reason. The PHP API docs were amazing. 00:27:59.920 --> 00:28:01.840 >> We had to learn the standard uh library 00:28:01.840 --> 00:28:04.120 of Java, and we had to like understand 00:28:04.120 --> 00:28:06.200 all of these collection classes, string 00:28:06.200 --> 00:28:09.560 buffer, and when to use what. Um 00:28:09.560 --> 00:28:11.280 And and you had to understand the SDK. 00:28:11.280 --> 00:28:13.680 Even when I was learning .NET, and we 00:28:13.680 --> 00:28:15.120 were learning about like threads and 00:28:15.120 --> 00:28:16.520 asynchronous, 00:28:16.520 --> 00:28:18.560 um the whole threading in .NET. 00:28:18.560 --> 00:28:19.760 >> Yeah, that's that's a that's a good 00:28:19.760 --> 00:28:22.480 exercise to read through the base like 00:28:22.480 --> 00:28:25.120 With PHP, it was I I I I guess I was a 00:28:25.120 --> 00:28:27.120 bit distracted half the time because the 00:28:27.120 --> 00:28:29.880 comments section of the PHP docs was was 00:28:29.880 --> 00:28:31.720 always hilarious to me or very 00:28:31.720 --> 00:28:32.400 interesting. 00:28:32.400 --> 00:28:35.760 >> Okay. So, yeah. So, but but we I think 00:28:35.760 --> 00:28:37.520 we can agree that we have long solved 00:28:37.520 --> 00:28:39.280 this problem, right? You put some Java 00:28:39.280 --> 00:28:41.720 doc string on your code, on your 00:28:41.720 --> 00:28:43.960 interface, on the property, and you get 00:28:43.960 --> 00:28:46.600 an auto-generated, very deterministic, 00:28:46.600 --> 00:28:48.680 maybe dry, but 00:28:48.680 --> 00:28:49.720 >> you any 00:28:49.720 --> 00:28:50.440 of that. 00:28:50.440 --> 00:28:52.160 >> to these decisions, of course. 00:28:52.160 --> 00:28:53.840 >> It depends how good you document it. But 00:28:53.840 --> 00:28:57.120 like because if you look at AWS CDK, 00:28:57.120 --> 00:28:59.200 they actually have like very detailed in 00:28:59.200 --> 00:29:01.480 the Java doc, they have code snippets. 00:29:01.480 --> 00:29:03.440 Like sorry, in the JS docstring, 00:29:03.440 --> 00:29:05.360 JavaScript documentation string. They 00:29:05.360 --> 00:29:06.760 have a code snippet. Like there's a 00:29:06.760 --> 00:29:09.720 constructor of the VPC L2 construct, 00:29:09.720 --> 00:29:11.040 there will be a code snippet. This is 00:29:11.040 --> 00:29:14.120 how you build a basic VPC right above 00:29:14.120 --> 00:29:17.320 the constructor. So inside the API ref, 00:29:17.320 --> 00:29:19.560 um you go to that their docs. They have 00:29:19.560 --> 00:29:21.240 a right at the top introduction. This is 00:29:21.240 --> 00:29:23.640 a VPC construct, then little example. 00:29:23.640 --> 00:29:26.000 This is how you you invoke it with basic 00:29:26.000 --> 00:29:27.840 >> to send me a link to that cuz it's 00:29:27.840 --> 00:29:29.320 probably something I completely missed 00:29:29.320 --> 00:29:29.840 actually. 00:29:29.840 --> 00:29:33.440 >> AWS CDK L2 00:29:33.440 --> 00:29:34.480 >> Because we 00:29:34.480 --> 00:29:36.040 cuz at work we're trying to create level 00:29:36.040 --> 00:29:38.360 two constructs and our level two 00:29:38.360 --> 00:29:40.560 constructs are basically living the 00:29:40.560 --> 00:29:42.720 documentation's living in confluence as 00:29:42.720 --> 00:29:45.240 an ADR that got signed off by security 00:29:45.240 --> 00:29:46.600 and whatnot. 00:29:46.600 --> 00:29:47.760 And 00:29:47.760 --> 00:29:48.920 I think that's what we're missing at 00:29:48.920 --> 00:29:50.960 work actually cuz the constructs, the 00:29:50.960 --> 00:29:52.520 way that they're implemented, there's 00:29:52.520 --> 00:29:54.240 very little documentation in the source 00:29:54.240 --> 00:29:56.560 code. Everything's in confluence and we 00:29:56.560 --> 00:29:56.680 need 00:29:56.680 --> 00:29:58.960 >> So this is the API reference. So on one 00:29:58.960 --> 00:30:00.320 side you have the developer guide that 00:30:00.320 --> 00:30:02.360 explains concepts that 00:30:02.360 --> 00:30:04.920 that explains that explains concepts. 00:30:04.920 --> 00:30:06.160 And then you have the completely 00:30:06.160 --> 00:30:08.480 programmatically generated API reference 00:30:08.480 --> 00:30:09.880 that you can view in all of the 00:30:09.880 --> 00:30:11.920 supported languages. So these language 00:30:11.920 --> 00:30:12.720 translations 00:30:12.720 --> 00:30:13.120 >> Three. 00:30:13.120 --> 00:30:15.680 >> are fully automated, right? And then the 00:30:15.680 --> 00:30:17.360 example I've gave for example was for 00:30:17.360 --> 00:30:19.840 the VPC, right? Let's go to the VPC. So 00:30:19.840 --> 00:30:22.840 if you go to the Where is he? Seriously. 00:30:22.840 --> 00:30:24.280 >> Well, I'm more interested in like a 00:30:24.280 --> 00:30:26.040 level two construct or something like 00:30:26.040 --> 00:30:26.160 that. 00:30:26.160 --> 00:30:27.920 >> Yeah, the the the VPC level two 00:30:27.920 --> 00:30:29.720 constructs or maybe they 00:30:29.720 --> 00:30:31.240 they they got rid of it. But I want to 00:30:31.240 --> 00:30:32.920 show you the level two construct, not 00:30:32.920 --> 00:30:34.400 the level one. Is this 00:30:34.400 --> 00:30:36.440 Where are the level twos? Let's just I 00:30:36.440 --> 00:30:38.160 want I want to take one that I know. 00:30:38.160 --> 00:30:40.680 Like let's take um 00:30:40.680 --> 00:30:43.200 Dynamo. So DynamoDB, oh I know why I 00:30:43.200 --> 00:30:44.600 couldn't see it because it's under the 00:30:44.600 --> 00:30:46.240 EC2. 00:30:46.240 --> 00:30:47.760 Eh, where did it go again? It's under 00:30:47.760 --> 00:30:50.320 EC2. But, they they also have the V2. 00:30:50.320 --> 00:30:52.200 AWS EC2. Okay, you're going to have to 00:30:52.200 --> 00:30:54.160 cut a little. So, here you have AWS VP 00:30:54.160 --> 00:30:56.080 EC2. They have the overview, which is 00:30:56.080 --> 00:30:58.520 the root readme of the the readme.md 00:30:58.520 --> 00:31:00.200 from the library, right? In the library 00:31:00.200 --> 00:31:02.160 folder. And then you have every single 00:31:02.160 --> 00:31:04.240 construct, and this is an L2. And right 00:31:04.240 --> 00:31:06.120 in the construct here at the top, you 00:31:06.120 --> 00:31:08.520 have a little example, right? 00:31:08.520 --> 00:31:11.080 Um for example here, and then you have 00:31:11.080 --> 00:31:12.280 another example here. 00:31:12.280 --> 00:31:13.320 >> Mhm. Mhm. 00:31:13.320 --> 00:31:15.360 >> And how is this created? If you go to 00:31:15.360 --> 00:31:18.800 AWS Labs, is it AWS CDK? 00:31:18.800 --> 00:31:20.800 >> Yeah, but examples if you go to the 00:31:20.800 --> 00:31:21.200 GitHub 00:31:21.200 --> 00:31:23.200 >> That's one form of documentation. I I'm 00:31:23.200 --> 00:31:24.080 thinking like 00:31:24.080 --> 00:31:26.360 >> No, no, hold on. Let me finish. 00:31:26.360 --> 00:31:27.160 Um 00:31:27.160 --> 00:31:28.600 >> Okay. 00:31:28.600 --> 00:31:31.960 >> AWS EC2. Where is EC2? Oh, no, it's AWS 00:31:31.960 --> 00:31:33.640 EC2. Where is the source for this 00:31:33.640 --> 00:31:35.920 because what we just said is that AI can 00:31:35.920 --> 00:31:37.480 generate all of this stuff, but it's 00:31:37.480 --> 00:31:40.640 it's it's like slop. So, the root readme 00:31:40.640 --> 00:31:42.920 is this thing, right? Which 00:31:42.920 --> 00:31:45.640 completely matches the root readme here, 00:31:45.640 --> 00:31:48.360 okay? This is import. This is examples. 00:31:48.360 --> 00:31:49.920 So, that's that thing, right? Exactly 00:31:49.920 --> 00:31:51.520 the same thing. Then the second thing is 00:31:51.520 --> 00:31:53.120 you have the library. And then if you 00:31:53.120 --> 00:31:55.840 look at the actual VPC here, right in 00:31:55.840 --> 00:31:57.600 the JS doc string, you have these 00:31:57.600 --> 00:32:01.200 examples, right? Um in the constructor. 00:32:01.200 --> 00:32:02.440 So, 00:32:02.440 --> 00:32:04.800 VPC creates a VPC. I guess there's one 00:32:04.800 --> 00:32:06.640 on the cloud There's two. One One is on 00:32:06.640 --> 00:32:08.480 the class. There's a JS doc on the 00:32:08.480 --> 00:32:09.600 class. 00:32:09.600 --> 00:32:12.320 Um This is the static import. Where is 00:32:12.320 --> 00:32:14.520 the public Oh, there it was right there 00:32:14.520 --> 00:32:16.960 at the top. Here. So, export class VPC 00:32:16.960 --> 00:32:19.280 VPC base. And here we have the original 00:32:19.280 --> 00:32:22.360 like for example, there's a VPC new. 00:32:22.360 --> 00:32:24.040 There's So, you see it's even like 00:32:24.040 --> 00:32:26.120 annotated TypeScript. So, that's use 00:32:26.120 --> 00:32:27.520 what you see at the top here. When you 00:32:27.520 --> 00:32:29.320 look at this, you see it right here. 00:32:29.320 --> 00:32:30.360 That's is a 00:32:30.360 --> 00:32:33.600 JS doc, right? And then when you look I 00:32:33.600 --> 00:32:34.760 don't know where the other example is, 00:32:34.760 --> 00:32:37.120 but I I it's here. Well, where is the 00:32:37.120 --> 00:32:38.360 actual constructor? 00:32:38.360 --> 00:32:39.880 >> here's the constructor. 00:32:39.880 --> 00:32:42.240 >> I think those are good for like 00:32:42.240 --> 00:32:43.480 development 00:32:43.480 --> 00:32:44.720 uh 00:32:44.720 --> 00:32:46.080 It's not because of the library things, 00:32:46.080 --> 00:32:47.360 but like what what 00:32:47.360 --> 00:32:48.360 like 00:32:48.360 --> 00:32:49.840 often enough we we have a little to 00:32:49.840 --> 00:32:52.120 construct at work where it's for a 00:32:52.120 --> 00:32:54.480 particular business unit for a 00:32:54.480 --> 00:32:57.840 particular feature and it's not the sort 00:32:57.840 --> 00:33:00.240 of style. It's like this this security 00:33:00.240 --> 00:33:02.720 consideration, security reviews and 00:33:02.720 --> 00:33:04.680 reasons why it's done this way for 00:33:04.680 --> 00:33:07.520 certain reasons and 00:33:07.520 --> 00:33:09.040 >> So, we we have 00:33:09.040 --> 00:33:09.720 >> I don't think [clears throat] the source 00:33:09.720 --> 00:33:11.280 code is the right place to put it, is 00:33:11.280 --> 00:33:12.280 it? I'm not too sure. 00:33:12.280 --> 00:33:14.240 >> I absolutely think that as much as 00:33:14.240 --> 00:33:16.000 possible your documentation should live 00:33:16.000 --> 00:33:17.560 life together with your code. 00:33:17.560 --> 00:33:19.240 >> Mhm. But like 00:33:19.240 --> 00:33:21.560 in a lot of enterprises this there's so 00:33:21.560 --> 00:33:23.960 many non-technical people involved. Like 00:33:23.960 --> 00:33:26.360 I dare say the security people are not 00:33:26.360 --> 00:33:28.440 terribly ever going to look at the code 00:33:28.440 --> 00:33:30.160 and the product owners are definitely 00:33:30.160 --> 00:33:31.320 not going to look at the code and the 00:33:31.320 --> 00:33:32.560 business analysts are not going to look 00:33:32.560 --> 00:33:34.720 at the code. 00:33:34.720 --> 00:33:36.160 But this is the Mint the Five. Is this 00:33:36.160 --> 00:33:39.680 Mint the Five driven MCP or AI feature? 00:33:39.680 --> 00:33:41.640 >> Yeah. So, in this case, right? I don't 00:33:41.640 --> 00:33:43.640 think I have any example in my 00:33:43.640 --> 00:33:46.360 documentation that shows you exactly how 00:33:46.360 --> 00:33:48.120 to how to create a Terraform stack with 00:33:48.120 --> 00:33:50.440 the S3 backend. But because of the way 00:33:50.440 --> 00:33:52.200 that these API references and everything 00:33:52.200 --> 00:33:54.520 is generated, you should get exactly 00:33:54.520 --> 00:33:56.560 this. Like this is how you do it. And it 00:33:56.560 --> 00:33:58.600 shows us there's an API reference here 00:33:58.600 --> 00:34:00.080 and there's a backend config reference 00:34:00.080 --> 00:34:02.480 here. So, this is kind of what you 00:34:02.480 --> 00:34:04.240 expect that you're getting out of the 00:34:04.240 --> 00:34:06.800 AWS MCP, right? You can ask it the 00:34:06.800 --> 00:34:08.879 question even though that the the 00:34:08.879 --> 00:34:10.960 reference data, which is auto-generated 00:34:10.960 --> 00:34:13.840 and very deterministic out of the, you 00:34:13.840 --> 00:34:16.280 know, API contracts and open API spec, 00:34:16.280 --> 00:34:19.200 which AWS what they they have their own 00:34:19.200 --> 00:34:21.040 like Smith 00:34:21.040 --> 00:34:24.760 Smitty language to define API contracts. 00:34:24.760 --> 00:34:26.520 They they they document it very 00:34:26.520 --> 00:34:28.520 auto-generate all the documentation and 00:34:28.520 --> 00:34:31.320 then you layer layer of AI on top of it 00:34:31.320 --> 00:34:33.399 and you get this type of response where 00:34:33.399 --> 00:34:35.159 you ask for an example and it gives you 00:34:35.159 --> 00:34:37.200 this. And in they actually give you in 00:34:37.200 --> 00:34:39.800 the pro plan the ability to expose this 00:34:39.800 --> 00:34:42.840 as an MCP so that people can install, I 00:34:42.840 --> 00:34:44.280 don't know where is the option here. 00:34:44.280 --> 00:34:46.800 >> Okay, so this is this is Mintlify 00:34:46.800 --> 00:34:47.520 uh 00:34:47.520 --> 00:34:48.360 feature, right? 00:34:48.360 --> 00:34:50.879 >> Yes, these are all Mintlify features. 00:34:50.879 --> 00:34:53.120 And and um and and it kind of combats 00:34:53.120 --> 00:34:55.120 the problem of we have a whole bunch of 00:34:55.120 --> 00:34:57.320 markdown AI slop generated in five 00:34:57.320 --> 00:34:59.160 different ways. No, we have the core 00:34:59.160 --> 00:35:01.880 truth ideally or deterministically from 00:35:01.880 --> 00:35:04.640 the code source code, and then we layer 00:35:04.640 --> 00:35:06.560 we layer the AI on top and it has a 00:35:06.560 --> 00:35:08.400 vector database and it can in it has 00:35:08.400 --> 00:35:10.360 indexed your docs on whatever they get 00:35:10.360 --> 00:35:12.520 published or whenever you do a library 00:35:12.520 --> 00:35:15.080 release, and then uh the vector database 00:35:15.080 --> 00:35:17.040 quickly sources the data so that it can 00:35:17.040 --> 00:35:18.840 >> I like this. This is great. 00:35:18.840 --> 00:35:20.960 Though I mean, you hear me in the sense 00:35:20.960 --> 00:35:23.080 that like we have a lot of non-technical 00:35:23.080 --> 00:35:25.480 people involved in in these enterprises, 00:35:25.480 --> 00:35:27.600 and they just I don't think this is a 00:35:27.600 --> 00:35:29.800 good fit for them because they they live 00:35:29.800 --> 00:35:30.720 in 00:35:30.720 --> 00:35:33.080 in Jira Confluence land. Like how do you 00:35:33.080 --> 00:35:34.200 how do I incorporate 00:35:34.200 --> 00:35:35.720 >> I mean, it depends on what is the 00:35:35.720 --> 00:35:37.480 documentation that you need to generate, 00:35:37.480 --> 00:35:39.320 right? If the documentation What do you 00:35:39.320 --> 00:35:41.560 mean? Like if Who owns Who Who has the 00:35:41.560 --> 00:35:43.280 knowledge? Who needs to write it? 00:35:43.280 --> 00:35:45.040 >> Well, the the trouble is in a in a big 00:35:45.040 --> 00:35:47.400 enterprise some people I think the 00:35:47.400 --> 00:35:49.080 knowledge is kind of split because like 00:35:49.080 --> 00:35:50.960 some people have technical knowledge 00:35:50.960 --> 00:35:52.600 like the docs and things like that and 00:35:52.600 --> 00:35:54.200 they the implementation knowledge. Some 00:35:54.200 --> 00:35:56.520 people have the the business knowledge 00:35:56.520 --> 00:35:58.960 like why it is this way because of this 00:35:58.960 --> 00:36:00.520 decision. Unfortunately, in a lot of 00:36:00.520 --> 00:36:02.480 enterprises a lot of this information is 00:36:02.480 --> 00:36:05.000 kind of gatekept to be honest and it's 00:36:05.000 --> 00:36:07.520 really hard to to shake out like why 00:36:07.520 --> 00:36:10.360 this decision is like it is 00:36:10.360 --> 00:36:12.320 and things like this, but 00:36:12.320 --> 00:36:14.040 this is this is the problem I'm dealing 00:36:14.040 --> 00:36:15.880 with and I'm just starting to think it's 00:36:15.880 --> 00:36:17.440 it's a really challenging problem. 00:36:17.440 --> 00:36:18.760 >> If it's a challenging problem, it's 00:36:18.760 --> 00:36:21.120 probably worth money. 00:36:21.120 --> 00:36:22.640 Because I think you know how much money 00:36:22.640 --> 00:36:24.440 Mintlify has been raising? I mean, I 00:36:24.440 --> 00:36:26.480 just showed you what they can do with 00:36:26.480 --> 00:36:27.360 like they are 00:36:27.360 --> 00:36:28.480 >> Well, this is this is the great thing 00:36:28.480 --> 00:36:30.440 about AI is like we're looking at 00:36:30.440 --> 00:36:32.240 problems again and we're coming up and 00:36:32.240 --> 00:36:34.000 there's innovation happening everywhere 00:36:34.000 --> 00:36:35.360 and we can hardly keep track of 00:36:35.360 --> 00:36:37.000 everything. It's just incredible and 00:36:37.000 --> 00:36:38.680 it's really exciting space. I mean, I'm 00:36:38.680 --> 00:36:40.040 a little my mood this morning is 00:36:40.040 --> 00:36:41.520 probably not great, but like I'm 00:36:41.520 --> 00:36:44.560 actually excited because there's so many 00:36:44.560 --> 00:36:46.560 but there's so many like things to solve 00:36:46.560 --> 00:36:49.360 now. There's so many things. But let me 00:36:49.360 --> 00:36:50.600 let me get back to that blog that we 00:36:50.600 --> 00:36:52.320 started off with because I think there's 00:36:52.320 --> 00:36:53.800 some there's some really good points 00:36:53.800 --> 00:36:56.120 that I made. Like for example, 00:36:56.120 --> 00:36:58.000 like why we review. 00:36:58.000 --> 00:37:01.480 We we we want to review so that 00:37:01.480 --> 00:37:05.240 there's alignment that we have a quality 00:37:05.240 --> 00:37:06.680 well, I don't want to read out this 00:37:06.680 --> 00:37:09.440 text, but this is what I'm 00:37:09.440 --> 00:37:11.560 this is what I'm I'm getting at. 00:37:11.560 --> 00:37:12.320 Uh 00:37:12.320 --> 00:37:14.280 this is how I interpreted 00:37:14.280 --> 00:37:16.720 interpreted this blog. 00:37:16.720 --> 00:37:18.680 I think we talked about quality. But the 00:37:18.680 --> 00:37:20.280 the really interesting thing I think he 00:37:20.280 --> 00:37:22.760 makes it is is trust. Like he he made 00:37:22.760 --> 00:37:25.040 the point that like uh 00:37:25.040 --> 00:37:27.440 what one of the things that made 00:37:27.440 --> 00:37:29.040 Japanese 00:37:29.040 --> 00:37:29.680 uh 00:37:29.680 --> 00:37:31.840 stuff so good is that there was an 00:37:31.840 --> 00:37:33.480 implied quality, right? Like if you 00:37:33.480 --> 00:37:35.320 bought this Japanese thing, it's going 00:37:35.320 --> 00:37:37.400 to be quality. So when you when when you 00:37:37.400 --> 00:37:39.720 have that sort of uh expectation and 00:37:39.720 --> 00:37:41.040 trust, 00:37:41.040 --> 00:37:42.960 uh it makes your quality process a lot 00:37:42.960 --> 00:37:44.280 easier because you don't need to have 00:37:44.280 --> 00:37:46.160 all these checks and rigors if you know 00:37:46.160 --> 00:37:47.840 you're going to going to get a quality 00:37:47.840 --> 00:37:49.160 item um 00:37:49.160 --> 00:37:51.400 you know, cuz it's got this Japanese 00:37:51.400 --> 00:37:53.080 labels on it. But I think it's I think 00:37:53.080 --> 00:37:55.000 it's true like back in the day as I 00:37:55.000 --> 00:37:56.160 don't think I don't think it's the case 00:37:56.160 --> 00:37:58.000 nowadays because like Japanese products 00:37:58.000 --> 00:38:00.080 are probably not even manufactured in 00:38:00.080 --> 00:38:01.720 Japan, but who knows? 00:38:01.720 --> 00:38:04.080 >> I mean the core the core point is that 00:38:04.080 --> 00:38:07.240 with a label on it, it uh 00:38:07.240 --> 00:38:09.480 it's trustworthy. But the funny thing is 00:38:09.480 --> 00:38:10.080 also related 00:38:10.080 --> 00:38:11.360 >> Well, no, the label's the wrong thing. 00:38:11.360 --> 00:38:13.400 It's like it's like the trust is there. 00:38:13.400 --> 00:38:15.120 >> You know, yeah, but like it's the same 00:38:15.120 --> 00:38:16.960 like you trust a certification program 00:38:16.960 --> 00:38:18.800 or you trust a security scanner or you 00:38:18.800 --> 00:38:22.600 trust uh it's a it's a base layer of 00:38:22.600 --> 00:38:24.640 um guarantees, right? That you can build 00:38:24.640 --> 00:38:25.040 upon. 00:38:25.040 --> 00:38:26.360 >> Exactly. Exactly. 00:38:26.360 --> 00:38:28.000 >> Uh but but if you want to talk about 00:38:28.000 --> 00:38:31.560 Japan quality, when I talk about Japan 00:38:31.560 --> 00:38:33.400 quality to my father 00:38:33.400 --> 00:38:35.560 uh who is 80 years old, he says it's 00:38:35.560 --> 00:38:38.440 funny because back when he was young, 00:38:38.440 --> 00:38:41.200 um Japan did not have the the the 00:38:41.200 --> 00:38:43.160 reputation of quality and and it was 00:38:43.160 --> 00:38:45.480 like where China was maybe a few years 00:38:45.480 --> 00:38:46.440 back. 00:38:46.440 --> 00:38:48.880 Um where, you know, made in China was 00:38:48.880 --> 00:38:51.520 not was like oh it's it's it's not good 00:38:51.520 --> 00:38:53.440 quality. It was like that with Japan. He 00:38:53.440 --> 00:38:55.000 says also like they used to come to 00:38:55.000 --> 00:38:56.520 Belgium and take pictures of everything 00:38:56.520 --> 00:38:58.480 and everyone hated it like because they 00:38:58.480 --> 00:38:59.800 take pictures and then they go and make 00:38:59.800 --> 00:39:02.720 it back in Japan for cheap. Um 00:39:02.720 --> 00:39:04.920 but it's funny now because now 00:39:04.920 --> 00:39:06.960 uh made in China is actually like, you 00:39:06.960 --> 00:39:09.320 know, it's good quality. I don't know if 00:39:09.320 --> 00:39:11.480 you you still have that connotation, but 00:39:11.480 --> 00:39:13.280 for me, made in China means it's 00:39:13.280 --> 00:39:15.360 actually futuristic, it's good quality, 00:39:15.360 --> 00:39:15.960 it's 00:39:15.960 --> 00:39:17.600 >> Yeah, there's definitely something. Like 00:39:17.600 --> 00:39:18.760 I mean 00:39:18.760 --> 00:39:21.160 my my parents used to go to Japan in in 00:39:21.160 --> 00:39:23.760 the in the '80s to buy fabric because 00:39:23.760 --> 00:39:26.160 there was sanctions in in South Africa 00:39:26.160 --> 00:39:28.000 and you couldn't import fabric very 00:39:28.000 --> 00:39:29.480 easily from other countries, but in 00:39:29.480 --> 00:39:30.800 Japan 00:39:30.800 --> 00:39:33.880 um they were nice to South Africa. 00:39:33.880 --> 00:39:35.200 >> Contraband. 00:39:35.200 --> 00:39:37.120 >> I've I've got a few I've got a few 00:39:37.120 --> 00:39:39.120 Japanese toys and the quality is 00:39:39.120 --> 00:39:41.760 incredible um for the '80s. So, there 00:39:41.760 --> 00:39:44.440 was there was a golden age for sure. 00:39:44.440 --> 00:39:45.120 Uh 00:39:45.120 --> 00:39:46.240 but like 00:39:46.240 --> 00:39:47.680 yeah, going back to this 00:39:47.680 --> 00:39:49.400 going back to AI and 00:39:49.400 --> 00:39:50.040 >> Yeah, we went to 00:39:50.040 --> 00:39:50.440 >> teams 00:39:50.440 --> 00:39:51.560 >> We got off topic totally. 00:39:51.560 --> 00:39:53.040 >> and teams and things like this. The 00:39:53.040 --> 00:39:55.160 trust Yeah, like 00:39:55.160 --> 00:39:58.080 I mean, I know from working with a as 00:39:58.080 --> 00:39:59.840 being I know from being a consultant for 00:39:59.840 --> 00:40:03.120 the last 15 years five years at least in 00:40:03.120 --> 00:40:05.960 my current consultancy, like trust is is 00:40:05.960 --> 00:40:08.720 the number one thing. Trust is the key 00:40:08.720 --> 00:40:11.120 to for a client relationship. Trust 00:40:11.120 --> 00:40:13.280 trust trust. And like I'm a little bit 00:40:13.280 --> 00:40:15.560 shocked in some ways that people are 00:40:15.560 --> 00:40:19.200 using AI and uh basically destroying 00:40:19.200 --> 00:40:21.280 that trust because like that the client 00:40:21.280 --> 00:40:23.640 doesn't know that much about AI and then 00:40:23.640 --> 00:40:25.000 someone comes along and another a 00:40:25.000 --> 00:40:26.600 consultant says like, "But if you use 00:40:26.600 --> 00:40:28.880 AI, this could be done so much quicker." 00:40:28.880 --> 00:40:31.120 And then all of a sudden quality issues 00:40:31.120 --> 00:40:33.360 occur and then the trust is basically 00:40:33.360 --> 00:40:35.400 destroyed. And this could be a colleague 00:40:35.400 --> 00:40:37.160 of mine. 00:40:37.160 --> 00:40:39.480 Um 00:40:39.480 --> 00:40:41.400 So that anyway, I guess I I don't really 00:40:41.400 --> 00:40:43.880 have a solution here but like 00:40:43.880 --> 00:40:45.960 but like if I don't know how you even 00:40:45.960 --> 00:40:48.080 measure trust really. 00:40:48.080 --> 00:40:50.880 But but this is the thing that needs to 00:40:50.880 --> 00:40:52.480 be 00:40:52.480 --> 00:40:54.720 I anyway, this this blog made me just 00:40:54.720 --> 00:40:56.960 think aloud that like trust needs to be 00:40:56.960 --> 00:40:58.800 it needs to be protected. We need 00:40:58.800 --> 00:41:01.600 integrity here for what whatever we do, 00:41:01.600 --> 00:41:03.800 trust and alignment or whatever you want 00:41:03.800 --> 00:41:06.120 to call it is so key with all this 00:41:06.120 --> 00:41:07.600 innovation that's happening. It's just 00:41:07.600 --> 00:41:08.080 like 00:41:08.080 --> 00:41:09.280 >> You know what I also You know how how 00:41:09.280 --> 00:41:12.920 >> how um we used to look at CI/CD 00:41:12.920 --> 00:41:14.640 pipelines. One of the concepts of 00:41:14.640 --> 00:41:16.800 immutable infrastructure is that you 00:41:16.800 --> 00:41:20.160 build an immutable asset and that asset 00:41:20.160 --> 00:41:23.160 or artifact moves down the pipeline and 00:41:23.160 --> 00:41:25.120 the further along it goes to the right, 00:41:25.120 --> 00:41:27.400 the higher our trust is into this 00:41:27.400 --> 00:41:29.720 artifact until we are sufficiently 00:41:29.720 --> 00:41:31.440 convinced to deploy it into production. 00:41:31.440 --> 00:41:33.560 >> Yeah, passing all the promotion gates 00:41:33.560 --> 00:41:34.680 and such and so forth. 00:41:34.680 --> 00:41:37.080 >> And I think that's still very much like 00:41:37.080 --> 00:41:39.200 the requirements is we need verification 00:41:39.200 --> 00:41:41.480 mechanisms around these 00:41:41.480 --> 00:41:43.080 >> But then but then but then but that is 00:41:43.080 --> 00:41:46.160 in contrast like the speed of AI and the 00:41:46.160 --> 00:41:48.960 slowness of reviews and quality gates. 00:41:48.960 --> 00:41:51.080 >> Yeah, which is why everyone always says 00:41:51.080 --> 00:41:53.120 like Agile and I think we mentioned this 00:41:53.120 --> 00:41:54.600 like a long time ago, Agile best 00:41:54.600 --> 00:41:56.680 practices are really important and I 00:41:56.680 --> 00:41:58.920 think even like Adam Jacobs and all when 00:41:58.920 --> 00:42:00.760 they talk about AI they always say like, 00:42:00.760 --> 00:42:04.320 you know, we chef, we do Agile, we do we 00:42:04.320 --> 00:42:06.080 do all of this, you know, devops best 00:42:06.080 --> 00:42:09.360 practices. We do validation mechanisms 00:42:09.360 --> 00:42:11.280 and the focus was always on automation, 00:42:11.280 --> 00:42:13.800 right? It was always on linting to 00:42:13.800 --> 00:42:15.400 reduce the overhead and like, you know 00:42:15.400 --> 00:42:17.240 why linters exist, right? We we want to 00:42:17.240 --> 00:42:18.440 remove all of the white space 00:42:18.440 --> 00:42:20.320 differences and formatting issues that 00:42:20.320 --> 00:42:22.640 people would argue over. Like, no, but 00:42:22.640 --> 00:42:24.120 you know, curly braces would be on the 00:42:24.120 --> 00:42:26.280 set next line, not on the same line. 00:42:26.280 --> 00:42:27.400 Linting rule. 00:42:27.400 --> 00:42:28.760 >> Yeah, we Yeah. 00:42:28.760 --> 00:42:29.880 >> We don't want to have We don't want to 00:42:29.880 --> 00:42:31.120 waste our time with with those 00:42:31.120 --> 00:42:33.200 discussions, right? So, the more that we 00:42:33.200 --> 00:42:35.760 can put in uh formal and automated 00:42:35.760 --> 00:42:37.600 deterministic validation mechanisms 00:42:37.600 --> 00:42:40.120 around produced code, then the more we 00:42:40.120 --> 00:42:40.560 can 00:42:40.560 --> 00:42:42.880 >> Every makes the same point with Go 00:42:42.880 --> 00:42:44.720 format. Think of the people who created 00:42:44.720 --> 00:42:45.760 Go format. I don't know if you've ever 00:42:45.760 --> 00:42:47.960 worked with Go. You have, right? Go 00:42:47.960 --> 00:42:49.640 format is fantastic. 00:42:49.640 --> 00:42:51.320 >> And and Rust expanded on that because 00:42:51.320 --> 00:42:53.880 then aside from Go format, Go test is 00:42:53.880 --> 00:42:55.880 also an amazing framework. 00:42:55.880 --> 00:42:56.520 >> Yes, it 00:42:56.520 --> 00:42:58.040 >> originally those test frameworks, they 00:42:58.040 --> 00:43:00.240 were not in built-in. Like, Node didn't 00:43:00.240 --> 00:43:01.440 have a test, right? Then you have 00:43:01.440 --> 00:43:02.800 Jasmine, and you have so many different 00:43:02.800 --> 00:43:03.440 iterations of it. 00:43:03.440 --> 00:43:05.520 >> Yeah, and they're slow and they're God. 00:43:05.520 --> 00:43:07.200 >> So, a language that is built from the 00:43:07.200 --> 00:43:09.360 ground up with formatting, linting, 00:43:09.360 --> 00:43:12.680 testing, like Rust, Go, what else? Those 00:43:12.680 --> 00:43:14.000 languages, I mean, you have 00:43:14.000 --> 00:43:16.480 well-established frameworks for Python 00:43:16.480 --> 00:43:20.760 and for for JavaScript, Jest or Vitest. 00:43:20.760 --> 00:43:22.600 So, you have a lot of them, yeah. 00:43:22.600 --> 00:43:24.840 >> I mean, like, we know we know what to do 00:43:24.840 --> 00:43:27.280 in some ways that we need to add 00:43:27.280 --> 00:43:31.000 automation, add tests, add 00:43:31.000 --> 00:43:31.440 >> to build trust. 00:43:31.440 --> 00:43:33.400 >> automation and sense add trust into the 00:43:33.400 --> 00:43:36.400 whole into these new technological 00:43:36.400 --> 00:43:38.920 advancements, AI. But, I just feel like 00:43:38.920 --> 00:43:40.840 it is very challenging. Like, this is 00:43:40.840 --> 00:43:42.640 what we need to do, but I feel like we 00:43:42.640 --> 00:43:43.920 need to 00:43:43.920 --> 00:43:45.320 go back to the drawing board, like make 00:43:45.320 --> 00:43:46.840 all these mistakes, and then automate 00:43:46.840 --> 00:43:48.080 these things again. And that And that's 00:43:48.080 --> 00:43:49.680 quite challenging in an enterprise 00:43:49.680 --> 00:43:51.880 environment which is used to 00:43:51.880 --> 00:43:54.280 being slow and used to 00:43:54.280 --> 00:43:56.120 having all those quality 00:43:56.120 --> 00:43:58.160 in place. You know, you can't just like 00:43:58.160 --> 00:43:59.400 flip the table and say, "Hey, we're 00:43:59.400 --> 00:44:01.200 using AI and we're making And we're 00:44:01.200 --> 00:44:03.520 going to be a startup again." And and 00:44:03.520 --> 00:44:04.920 and we're going to learn a lot and 00:44:04.920 --> 00:44:06.080 things like this. It's really 00:44:06.080 --> 00:44:06.920 challenging. 00:44:06.920 --> 00:44:08.480 >> And I think the most important thing is 00:44:08.480 --> 00:44:10.360 that AI has been trained on the test 00:44:10.360 --> 00:44:12.000 frameworks and we're trusting AI to 00:44:12.000 --> 00:44:13.720 write a test and we need now to need to 00:44:13.720 --> 00:44:16.200 trust the AI to write the test. Like we 00:44:16.200 --> 00:44:18.920 don't trust the test because AI wrote 00:44:18.920 --> 00:44:21.560 it, right? And that's the thing. That's 00:44:21.560 --> 00:44:23.240 That's where I'm like That's where I'm 00:44:23.240 --> 00:44:25.520 excited because AWS with Kiro, right? 00:44:25.520 --> 00:44:27.680 They said the ears format or the easy 00:44:27.680 --> 00:44:28.560 way of defining 00:44:28.560 --> 00:44:29.720 >> Yeah, like 00:44:29.720 --> 00:44:31.160 That's That's an excellent point that 00:44:31.160 --> 00:44:33.280 you made there. It's like 00:44:33.280 --> 00:44:35.920 We talk about trust, but like if we 00:44:35.920 --> 00:44:38.680 asked AI today to write us a test suite, 00:44:38.680 --> 00:44:40.800 do we trust that this test suite is is 00:44:40.800 --> 00:44:43.400 actually a value? I I I can't. 00:44:43.400 --> 00:44:45.040 >> But the problem is that this the the 00:44:45.040 --> 00:44:46.920 amount of I think the problem is the 00:44:46.920 --> 00:44:50.320 volume, right? Because the volume code 00:44:50.320 --> 00:44:52.760 output is much higher. So human 00:44:52.760 --> 00:44:55.680 reviewers are overloaded. The volume of 00:44:55.680 --> 00:44:58.000 generated tests are much higher, so you 00:44:58.000 --> 00:45:00.120 can't validate that every test is really 00:45:00.120 --> 00:45:02.440 testing what you expect it to test. Are 00:45:02.440 --> 00:45:03.960 the assertions right? Is it not putting 00:45:03.960 --> 00:45:05.880 in a mock somewhere? 00:45:05.880 --> 00:45:07.240 >> Mhm. 00:45:07.240 --> 00:45:08.360 >> So then 00:45:08.360 --> 00:45:10.440 you're shifting You're shifting the 00:45:10.440 --> 00:45:12.400 abstraction layer, right? Because when 00:45:12.400 --> 00:45:14.600 you talk like low-level assembly, it's a 00:45:14.600 --> 00:45:17.000 massive volume, right? We We used it by 00:45:17.000 --> 00:45:18.920 going up in the abstraction layer. And I 00:45:18.920 --> 00:45:21.280 think that's where then now with AI 00:45:21.280 --> 00:45:23.440 we're able to live at a much higher 00:45:23.440 --> 00:45:25.280 abstraction layer, the English language, 00:45:25.280 --> 00:45:27.120 and we are defining our user stories and 00:45:27.120 --> 00:45:29.200 we're defining our tests in the English 00:45:29.200 --> 00:45:31.800 language, right? So things that were not 00:45:31.800 --> 00:45:33.720 possible in the past, automatically 00:45:33.720 --> 00:45:36.400 parsing English language into validation 00:45:36.400 --> 00:45:38.320 framework, 00:45:38.320 --> 00:45:40.400 become a little bit more realistic now 00:45:40.400 --> 00:45:42.160 because of AI has the ability to 00:45:42.160 --> 00:45:46.160 interpret a paragraph, narrate 00:45:46.160 --> 00:45:50.000 a text that is easily for a human to to 00:45:50.000 --> 00:45:51.840 understand and validate, formally then 00:45:51.840 --> 00:45:53.920 convert that into an actual 00:45:53.920 --> 00:45:56.640 um rule, like property-based testing or 00:45:56.640 --> 00:45:59.040 Gherkin BDD, these frameworks. Which 00:45:59.040 --> 00:46:01.560 Which exist for for years and everybody 00:46:01.560 --> 00:46:02.880 Maybe you will write them off say, 00:46:02.880 --> 00:46:04.360 "Yeah, we tried it, it didn't work." It 00:46:04.360 --> 00:46:05.640 We tried it, it didn't work, but it 00:46:05.640 --> 00:46:07.920 there was no AI. 00:46:07.920 --> 00:46:10.040 >> I mean, Vince, when I when I'm listening 00:46:10.040 --> 00:46:12.120 to you, it sounds like 00:46:12.120 --> 00:46:13.920 the solution to the problem is AI. Like, 00:46:13.920 --> 00:46:15.720 if if only we had AI 00:46:15.720 --> 00:46:17.200 >> No, the solution to the problem is to 00:46:17.200 --> 00:46:19.840 apply AI to higher layer and then have a 00:46:19.840 --> 00:46:22.120 deterministic parsing. Like, we're 00:46:22.120 --> 00:46:24.800 parsing a formal language down to 00:46:24.800 --> 00:46:27.120 >> But that that higher layer requires 00:46:27.120 --> 00:46:30.240 requires agents to to do the uh 00:46:30.240 --> 00:46:32.480 >> No, the parsing down will not. Okay, so 00:46:32.480 --> 00:46:34.120 so imagine this, right? The way that 00:46:34.120 --> 00:46:36.840 Kiro is designed is to to define a user 00:46:36.840 --> 00:46:39.280 story. From those user story, like, as a 00:46:39.280 --> 00:46:41.240 developer, I want this because of this. 00:46:41.240 --> 00:46:42.600 This is a primary reason why I want to 00:46:42.600 --> 00:46:44.160 do this, blah blah blah. There gets 00:46:44.160 --> 00:46:46.440 functional requirements, requirements. 00:46:46.440 --> 00:46:48.280 When I do this, then this happen, like 00:46:48.280 --> 00:46:50.360 happy path. When I do this, then that 00:46:50.360 --> 00:46:52.640 should not happen, like sad path. These 00:46:52.640 --> 00:46:55.880 are very very parsable strings of text, 00:46:55.880 --> 00:46:58.040 right? They're in between, they're from 00:46:58.040 --> 00:46:59.200 the user story, which is like a 00:46:59.200 --> 00:47:01.640 paragraph of text to actual bullet 00:47:01.640 --> 00:47:03.080 points of rules. 00:47:03.080 --> 00:47:03.640 >> Yeah, but like 00:47:03.640 --> 00:47:05.560 >> requirements. They are parsable. The 00:47:05.560 --> 00:47:07.520 parsable part means deterministic, means 00:47:07.520 --> 00:47:10.160 we don't let an AI generate the text. 00:47:10.160 --> 00:47:12.240 >> need you still need some 00:47:12.240 --> 00:47:13.480 >> Yeah, but as a human 00:47:13.480 --> 00:47:15.640 >> test harness to to to run that and and 00:47:15.640 --> 00:47:17.160 and the AI could 00:47:17.160 --> 00:47:18.720 >> No, the AI should not be involved there 00:47:18.720 --> 00:47:19.360 at all. 00:47:19.360 --> 00:47:20.760 >> But how do you How do you How do you How 00:47:20.760 --> 00:47:22.200 do you execute these requirements 00:47:22.200 --> 00:47:23.440 against the implementation of the 00:47:23.440 --> 00:47:26.000 >> the concept of cucumber and gherkin, 00:47:26.000 --> 00:47:28.360 BDD, behavior-driven test uh 00:47:28.360 --> 00:47:30.120 behavior-driven development. They have 00:47:30.120 --> 00:47:32.480 defined a markdown format that can be 00:47:32.480 --> 00:47:34.920 parsed down into a validation framework. 00:47:34.920 --> 00:47:37.320 So, the AI agent writes English in those 00:47:37.320 --> 00:47:39.320 specific strings and then you fail the 00:47:39.320 --> 00:47:41.840 parser if it doesn't match that format. 00:47:41.840 --> 00:47:43.720 Or you pass the parser and it generates 00:47:43.720 --> 00:47:45.720 a deterministic validation framework. 00:47:45.720 --> 00:47:47.920 Your role as a human is to validate 00:47:47.920 --> 00:47:49.000 those requirements. 00:47:49.000 --> 00:47:49.560 >> Okay. 00:47:49.560 --> 00:47:51.120 >> And if you fail that 00:47:51.120 --> 00:47:53.120 then it then you cannot trust it the the 00:47:53.120 --> 00:47:55.080 completely deterministic generated 00:47:55.080 --> 00:47:57.200 tests. There's no AI from that point. 00:47:57.200 --> 00:47:59.360 >> Okay, I think I need to maybe you should 00:47:59.360 --> 00:48:01.800 try conjure an example for me cuz I 00:48:01.800 --> 00:48:03.640 guess I'm I'm lacking the trust because 00:48:03.640 --> 00:48:06.760 I just feel that the AI is smart enough 00:48:06.760 --> 00:48:10.040 to circumvent that in a way. Like 00:48:10.040 --> 00:48:11.520 Like for example hear me out here. Like 00:48:11.520 --> 00:48:14.280 the way I usually test some work when 00:48:14.280 --> 00:48:17.520 I'm working with the AI stuff is that 00:48:17.520 --> 00:48:20.240 I I introduce a bug to make sure that 00:48:20.240 --> 00:48:22.560 that my test harness sort of caught it 00:48:22.560 --> 00:48:24.880 and that and that for me gives me the 00:48:24.880 --> 00:48:26.480 reassurance that these tests that were 00:48:26.480 --> 00:48:29.360 generated are doing a cap are capturing 00:48:29.360 --> 00:48:30.800 the bug right. You know what I mean? 00:48:30.800 --> 00:48:32.600 Like there's probably a name for this. 00:48:32.600 --> 00:48:34.640 Like I I basically randomly enter a bug 00:48:34.640 --> 00:48:36.080 and make sure that 00:48:36.080 --> 00:48:37.680 >> Kiro does that and they call it 00:48:37.680 --> 00:48:39.440 property-based testing. They This 00:48:39.440 --> 00:48:40.920 basically like fuzzing, right? You 00:48:40.920 --> 00:48:43.840 generate a whole bunch of variables and 00:48:43.840 --> 00:48:45.560 you test the rules, the invariants 00:48:45.560 --> 00:48:47.640 against those variables and you 00:48:47.640 --> 00:48:49.280 basically generate 00:48:49.280 --> 00:48:52.120 like programmatically, randomly all the 00:48:52.120 --> 00:48:54.320 possible inputs until you find a 00:48:54.320 --> 00:48:55.040 counterexample. 00:48:55.040 --> 00:48:57.120 >> I understand fuzzing is. It's just It's 00:48:57.120 --> 00:48:59.760 just that the like my sort of review 00:48:59.760 --> 00:49:02.840 process is to is to sort of spot check, 00:49:02.840 --> 00:49:04.720 introduce not so much like a boundary 00:49:04.720 --> 00:49:06.440 problem, more like something more 00:49:06.440 --> 00:49:09.240 fundamental. Well, I say that 00:49:09.240 --> 00:49:10.680 and I can't even think of a good example 00:49:10.680 --> 00:49:12.080 right now. But 00:49:12.080 --> 00:49:14.120 that's that's where my my mind is going 00:49:14.120 --> 00:49:16.120 when it comes to testing. It's like you 00:49:16.120 --> 00:49:18.160 you are given something to review. 00:49:18.160 --> 00:49:20.120 There's so much volume as we were saying 00:49:20.120 --> 00:49:22.520 with the code and and the tests and then 00:49:22.520 --> 00:49:25.000 you you almost have to go in there, 00:49:25.000 --> 00:49:27.400 break something just to validate what 00:49:27.400 --> 00:49:29.040 this what this 00:49:29.040 --> 00:49:30.840 PR is is trying to do or something like 00:49:30.840 --> 00:49:31.240 that. 00:49:31.240 --> 00:49:33.200 >> Yeah, so I had a discussion with Ion 00:49:33.200 --> 00:49:35.440 Murdock about this where I keep saying 00:49:35.440 --> 00:49:37.640 like we need to have Gherkin or BDD 00:49:37.640 --> 00:49:40.040 because this is the my my one of the 00:49:40.040 --> 00:49:41.760 approaches. Like I said over the last 00:49:41.760 --> 00:49:43.520 few days I I realized that the 00:49:43.520 --> 00:49:44.880 constitution wasn't there, the checklist 00:49:44.880 --> 00:49:47.040 wasn't there, the cross-reference verify 00:49:47.040 --> 00:49:49.600 wasn't there. That's all AI agent 00:49:49.600 --> 00:49:52.120 driven, right? But my original idea was 00:49:52.120 --> 00:49:53.920 assuming that they were all there. Even 00:49:53.920 --> 00:49:55.800 if they were there, they all failed 00:49:55.800 --> 00:49:56.800 because they are 00:49:56.800 --> 00:49:58.880 unreliable. So, my 00:49:58.880 --> 00:50:00.720 >> unreliable, untrustworthy. 00:50:00.720 --> 00:50:03.840 >> Yeah. And so, I always want to explore 00:50:03.840 --> 00:50:07.000 this basically what I heard or 00:50:07.000 --> 00:50:09.280 understood Kiro was doing. And when I 00:50:09.280 --> 00:50:12.040 talked to Ion, he said, "Putting more 00:50:12.040 --> 00:50:15.280 rules and formal methodology around 00:50:15.280 --> 00:50:17.320 these agents is constraining them too 00:50:17.320 --> 00:50:20.120 much. You're just going to get worse 00:50:20.120 --> 00:50:22.680 output because the agents are working of 00:50:22.680 --> 00:50:25.440 their ability to to I guess have freedom 00:50:25.440 --> 00:50:27.200 to execute. And the more you start 00:50:27.200 --> 00:50:29.080 throwing errors at them when they run, 00:50:29.080 --> 00:50:30.880 the more they get distracted and the the 00:50:30.880 --> 00:50:32.800 less they can, you know, perform and 00:50:32.800 --> 00:50:34.200 give you the output that you want. 00:50:34.200 --> 00:50:35.840 Because my assumption is like I'm going 00:50:35.840 --> 00:50:37.320 to just immediately throw an error if it 00:50:37.320 --> 00:50:39.960 doesn't pass, right? So, he said, "Too 00:50:39.960 --> 00:50:41.360 many rules, 00:50:41.360 --> 00:50:43.240 even like too many injecting error 00:50:43.240 --> 00:50:44.840 messages when things are not according 00:50:44.840 --> 00:50:47.440 to certain expectations is not going to 00:50:47.440 --> 00:50:49.720 improve the performance of the agents in 00:50:49.720 --> 00:50:51.480 a way. That's what I understood from 00:50:51.480 --> 00:50:53.440 him. But I feel like I need to try it. I 00:50:53.440 --> 00:50:55.240 also wonder why I haven't really seen 00:50:55.240 --> 00:50:56.480 that in Kiro. I saw a lot of 00:50:56.480 --> 00:50:58.640 presentations from AWS talking about how 00:50:58.640 --> 00:51:00.640 they do this like deterministically 00:51:00.640 --> 00:51:01.880 validating. 00:51:01.880 --> 00:51:03.240 But I haven't really seen a lot of 00:51:03.240 --> 00:51:04.880 actual examples of that. 00:51:04.880 --> 00:51:05.760 >> Well, I mean there must be a reason why 00:51:05.760 --> 00:51:06.560 it doesn't work, right? 00:51:06.560 --> 00:51:07.880 >> It's not a solved problem. It's not a 00:51:07.880 --> 00:51:09.960 solved problem by any means. Anyway, I I 00:51:09.960 --> 00:51:11.920 got to take my kids to school now. 00:51:11.920 --> 00:51:13.880 Anyway, I it's a thought-provoking 00:51:13.880 --> 00:51:15.880 discussion. Thank you again, Vincent. 00:51:15.880 --> 00:51:17.600 >> I want to try I mean, in terms of giving 00:51:17.600 --> 00:51:19.880 you examples, it's definitely my 00:51:19.880 --> 00:51:23.080 intention to build like a Gherkin parser 00:51:23.080 --> 00:51:25.080 into Spec Ledger. And basically, 00:51:25.080 --> 00:51:27.080 whenever the agent writes on the 00:51:27.080 --> 00:51:29.120 requirements, run the parser. If the 00:51:29.120 --> 00:51:31.120 requirements don't pass, feed it back to 00:51:31.120 --> 00:51:32.600 the agent say like, "Okay, great. You 00:51:32.600 --> 00:51:34.200 wrote all the specs, but there's a 00:51:34.200 --> 00:51:36.280 parsing error on the requirements. I 00:51:36.280 --> 00:51:37.800 can't I can't 00:51:37.800 --> 00:51:38.840 >> validate this. 00:51:38.840 --> 00:51:40.920 >> Instinctively, I'm thinking just like 00:51:40.920 --> 00:51:42.840 everything in the world, I feel like 00:51:42.840 --> 00:51:45.480 that human that the like a human needs 00:51:45.480 --> 00:51:48.160 to be at the beginning and a human needs 00:51:48.160 --> 00:51:50.080 to be at the end. There's got to be a 00:51:50.080 --> 00:51:51.240 relationship there. There's got to be 00:51:51.240 --> 00:51:53.240 trust. And I'm thinking that like when 00:51:53.240 --> 00:51:54.800 you take the when you take the artifact 00:51:54.800 --> 00:51:56.600 off the production line, there needs to 00:51:56.600 --> 00:51:59.120 be that human review before it goes out 00:51:59.120 --> 00:51:59.320 there. 00:51:59.320 --> 00:52:00.880 >> approval. 00:52:00.880 --> 00:52:03.080 >> Exactly. That that QA mark. 00:52:03.080 --> 00:52:04.600 >> The human that that that signed the 00:52:04.600 --> 00:52:07.080 contract and that's held responsible if 00:52:07.080 --> 00:52:07.640 the thing blows up. 00:52:07.640 --> 00:52:09.600 >> Yeah, exactly. Insurance. The insurance 00:52:09.600 --> 00:52:11.000 policy and things like that. 00:52:11.000 --> 00:52:13.640 >> The guy that gets paid $20,000 a month 00:52:13.640 --> 00:52:14.400 or more at least. 00:52:14.400 --> 00:52:16.040 >> maybe not the insurance thing. I do find 00:52:16.040 --> 00:52:18.320 that horribly bureaucratic. I I hate the 00:52:18.320 --> 00:52:20.160 whole insurance industry. But yeah, I 00:52:20.160 --> 00:52:22.520 mean I 00:52:22.520 --> 00:52:24.600 I see where you're coming from. I I 00:52:24.600 --> 00:52:26.560 I hope you're right in a way. I hope 00:52:26.560 --> 00:52:27.960 you're right. Though at the same time 00:52:27.960 --> 00:52:29.680 I'm I'm just thinking to myself that 00:52:29.680 --> 00:52:31.000 >> I hope I'm wrong because where did the 00:52:31.000 --> 00:52:33.800 human What's human role if if it can be 00:52:33.800 --> 00:52:35.280 The human role seem to be the only thing 00:52:35.280 --> 00:52:37.600 that generates the ideas and then the 00:52:37.600 --> 00:52:39.360 whole process 00:52:39.360 --> 00:52:42.680 um is completely AI. There's no Like if 00:52:42.680 --> 00:52:44.480 you can completely trust the framework 00:52:44.480 --> 00:52:46.640 that it builds exactly according to what 00:52:46.640 --> 00:52:47.800 >> Well, let's say there there could be 00:52:47.800 --> 00:52:49.000 another test. Like for example, there 00:52:49.000 --> 00:52:51.720 might not be like a tedious QA stamp, 00:52:51.720 --> 00:52:53.640 but there could be a test a market test 00:52:53.640 --> 00:52:56.440 like if this product is good, then then 00:52:56.440 --> 00:52:58.840 then humans will will buy it. And that 00:52:58.840 --> 00:53:01.800 that is the ultimate test, of course. 00:53:01.800 --> 00:53:02.360 Uh 00:53:02.360 --> 00:53:03.520 >> Well, I wanted to show something, but 00:53:03.520 --> 00:53:05.400 you have to go. I also next time you 00:53:05.400 --> 00:53:07.240 have to show me I'm going to watch the 00:53:07.240 --> 00:53:09.840 your your experimentation with swamp and 00:53:09.840 --> 00:53:12.600 see if I am convinced. 00:53:12.600 --> 00:53:14.720 >> Yeah, please do. And uh please ask 00:53:14.720 --> 00:53:16.240 Ashani. Okay, so let's just wind this 00:53:16.240 --> 00:53:17.800 up. Oh my gosh, I think I need to do an 00:53:17.800 --> 00:53:19.480 intro in the beginning. 00:53:19.480 --> 00:53:21.080 >> Yeah, you can edit I I have to say River 00:53:21.080 --> 00:53:23.240 sounds really nice because the the video 00:53:23.240 --> 00:53:24.720 was all recorded directly from my 00:53:24.720 --> 00:53:26.160 machine and it's uploading. Quality is 00:53:26.160 --> 00:53:27.680 much higher. Audio is really good. 00:53:27.680 --> 00:53:29.680 Second thing is it's going to give you 00:53:29.680 --> 00:53:32.480 audio channels. So you can mute me if if 00:53:32.480 --> 00:53:34.840 you want to talk. I can I can mute I 00:53:34.840 --> 00:53:36.480 mean you can I do that with my my 00:53:36.480 --> 00:53:37.920 friend. I was saying something while he 00:53:37.920 --> 00:53:39.480 was talking or he was talking when I was 00:53:39.480 --> 00:53:41.040 saying something. You you mute it. So 00:53:41.040 --> 00:53:42.800 individual audio channels. perfect, 00:53:42.800 --> 00:53:43.120 right? 00:53:43.120 --> 00:53:45.480 >> Okay, I'll do an intro. So, that thanks 00:53:45.480 --> 00:53:47.000 for listening everybody and hopefully 00:53:47.000 --> 00:53:48.840 this Riverside works really well. See 00:53:48.840 --> 00:53:53.040 you. Bye. I'm clicking stop.