WEBVTT 00:00:01.060 --> 00:00:04.840 So it sounds like judging by your manic messages throughout the, the night 00:00:04.880 --> 00:00:10.000 that you pulled another all-nighter riding high on Opus 5.5 or whatever 00:00:10.000 --> 00:00:11.080 [bleep] the you're doing. 00:00:12.860 --> 00:00:15.240 Unfortunately, it wasn't 5.5. 00:00:15.540 --> 00:00:19.980 Um, we're gearing up towards the release of, uh, of CDK Terrain, 00:00:20.700 --> 00:00:23.800 and I'm trying to get like as much as possible, 00:00:24.620 --> 00:00:26.120 like fixed and resolved. 00:00:26.840 --> 00:00:30.600 I want this release to go out and then focus on the AWS CDK bridge. 00:00:31.200 --> 00:00:36.680 I'm super excited. I just started looking into FOSDEM and, um, what it is. 00:00:37.460 --> 00:00:38.840 Um- You don't know FOSDEM? 00:00:39.740 --> 00:00:41.420 I've never been. I heard about it. 00:00:41.560 --> 00:00:44.180 I like when I went to Osia- It's a, it's a bigger, it's a bigger event than 00:00:44.240 --> 00:00:45.360 Config Management Camp. 00:00:46.200 --> 00:00:51.130 Yeah, obviously. They say thousands of people go there, uh- Yeah ... 00:00:51.160 --> 00:00:53.120 the number of recordings and talks. 00:00:53.160 --> 00:00:55.660 Yeah. I mean, I, I love it in the sense that, um, 00:00:56.600 --> 00:00:59.540 it's, it's like a... It's a, it's like 00:01:00.380 --> 00:01:03.870 no one... You don't have to register, and it- Yeah ... 00:01:03.920 --> 00:01:04.800 it's a bit manic. 00:01:06.200 --> 00:01:07.140 And, uh, 00:01:08.540 --> 00:01:10.140 it's just the best conference actually. 00:01:10.500 --> 00:01:11.320 It is the best conference. 00:01:11.860 --> 00:01:14.320 The, the trouble is that there is like... 00:01:17.760 --> 00:01:22.300 There is this like feeling that things can, can be, can go the other way. 00:01:22.400 --> 00:01:23.100 Like, for example, 00:01:23.980 --> 00:01:29.360 um, there's, there's people who are like protesting against Google, protesting 00:01:29.420 --> 00:01:30.340 against AI, 00:01:31.440 --> 00:01:35.980 and so on and so forth. Oh, yeah. And like for, for example, I think Mark Shuttleworth, the, the 00:01:36.040 --> 00:01:40.420 Canonical Ubuntu South African guy, he was gonna give a talk there, and then 00:01:40.440 --> 00:01:41.580 they had this massive... 00:01:41.620 --> 00:01:42.440 They had this mad 00:01:43.280 --> 00:01:46.720 demonstration saying that, you know, no kings or whatever. 00:01:47.400 --> 00:01:50.120 And, um, he basically withdrew his, um, 00:01:51.680 --> 00:01:52.640 his, uh, talk 00:01:53.620 --> 00:01:55.860 because of it. How, how is he related to the no kings? 00:01:56.800 --> 00:01:58.360 Well, I mean, I'm just... 00:01:59.060 --> 00:02:02.260 I, I don't know what the protest slogan was, but it was something like, you 00:02:02.300 --> 00:02:04.200 know, no billionaires, no, 00:02:05.020 --> 00:02:05.920 no, um, yeah, no kings. 00:02:05.940 --> 00:02:06.660 Wait, who is he? 00:02:07.500 --> 00:02:11.520 You know Mark Shuttleworth, the guy that owns Canonical, the, the Thawte SSL 00:02:11.580 --> 00:02:14.640 guy. Okay. And, and like what's... 00:02:14.820 --> 00:02:15.980 Like I know I don't know him. 00:02:16.960 --> 00:02:18.260 Uh, what, what is... Why is he the... 00:02:18.500 --> 00:02:20.040 Is he a king? Is he a billionaire? 00:02:20.640 --> 00:02:23.240 He's a billionaire. He owns Ubuntu, basically. 00:02:25.160 --> 00:02:26.740 Okay. But I mean, 00:02:27.800 --> 00:02:31.400 interesting. I don't know why people are- So anyway, FOSDEM has this like 00:02:32.040 --> 00:02:34.620 interesting... Like I've never seen it before in any other conference. 00:02:34.660 --> 00:02:38.920 It's got like a, like a... It's got like a counter aspect to it. 00:02:40.700 --> 00:02:43.040 Yeah. Like an unconference type of thing or what? 00:02:43.640 --> 00:02:44.700 No, not so much like unconference. 00:02:44.720 --> 00:02:48.780 It's just like there are people who like disagree with technology at the same 00:02:49.420 --> 00:02:50.680 meeting- Oh ... about technology. 00:02:51.740 --> 00:02:53.540 Oh, shit. So I'm, I'm gonna get beat up 00:02:54.420 --> 00:02:57.120 verbally. I disagree with a lot. 00:02:57.440 --> 00:02:59.480 Yeah. I mean, I'm, I'm open to debate. 00:02:59.560 --> 00:03:02.740 I think it's wonderful in a way, but like of course, if someone, if, if 00:03:02.800 --> 00:03:05.960 someone decides to firebomb you or something, that's a bit much. 00:03:07.340 --> 00:03:09.800 Bit much. Yeah. It would be interesting. 00:03:10.080 --> 00:03:16.160 Um, now I hope that people, that somebody that disagrees with this podcast doesn't 00:03:16.220 --> 00:03:17.000 decide to go. 00:03:18.700 --> 00:03:22.360 I'll tell him. Yeah. Well, yeah. Like- Yeah ... 00:03:22.420 --> 00:03:26.300 I think there's, there's a couple occasions people have recognized me and, uh, 00:03:26.520 --> 00:03:30.350 and of course I've sort of reinvented myself as a- You're polarizing ... 00:03:31.000 --> 00:03:34.180 I've, I've reinvented myself as an AI advocate somewhat, 00:03:35.040 --> 00:03:37.940 and they might just have beef with me and I'll be, "Okay, cool." 00:03:39.000 --> 00:03:43.500 Oh, so that's why you're concerned about like their stance towards AI. 00:03:44.020 --> 00:03:49.600 Well, well, well, because I'm, I'm, I'm talking to you here, and I do 00:03:50.240 --> 00:03:55.800 often... I was gonna say nothing in the projects that I work on, um, you know, has a 00:03:55.860 --> 00:03:57.280 very strong AI stance. 00:03:57.780 --> 00:03:58.780 But if you're saying... 00:03:59.280 --> 00:04:01.180 I don't think people would recognize me anyway. 00:04:01.620 --> 00:04:03.940 Oh, great. I'll, I'll, I'll make su- 00:04:05.080 --> 00:04:06.320 I'll make sure I look different. 00:04:08.320 --> 00:04:09.460 I'll fit, I'll fit in. 00:04:09.960 --> 00:04:14.000 I stop, I stop shaving from now until February. 00:04:14.320 --> 00:04:17.180 Yeah, that's a good way to already get like targeted. 00:04:17.779 --> 00:04:21.899 Well- Saying like I'll stop shaving from f- from September till February, and 00:04:21.940 --> 00:04:28.960 I'll fit in. I like to say that I'm not scared of anything really, but of course, if 00:04:28.980 --> 00:04:31.780 something ever happens to me, I'm probably gonna be traumatized by it. 00:04:32.660 --> 00:04:33.900 It is Brussels after all. 00:04:35.880 --> 00:04:37.500 Brussels is bad. Yeah. 00:04:37.540 --> 00:04:38.320 It is a bit... 00:04:38.600 --> 00:04:40.720 Brussels is a bit of a weird place, I must say. 00:04:42.500 --> 00:04:44.080 Interesting. Um, 00:04:44.900 --> 00:04:47.080 yeah, no, I, I, I haven't been... 00:04:47.280 --> 00:04:50.640 It pro- Brussels is probably the one place I would avoid the most. 00:04:51.080 --> 00:04:54.460 Um- As a Belgian. As a Belgian, yeah. 00:04:55.340 --> 00:04:58.700 Um, I never really liked to go there as a, as a, as a kid even. 00:04:59.660 --> 00:05:01.760 Uh, so, so, so the dev rooms are... 00:05:02.420 --> 00:05:07.760 So dev rooms are proposed until s- the, until October, then CFPs for those 00:05:07.820 --> 00:05:11.900 dev rooms open. And I guess if you, if you volunteer or propose a dev room, 00:05:11.960 --> 00:05:13.840 then you're the manager of that dev room. 00:05:13.880 --> 00:05:15.580 Like you must have at least another person. 00:05:15.860 --> 00:05:19.200 Yeah. And you must, you must manage the program based on the CFP. 00:05:19.200 --> 00:05:21.360 I mean, I, I can maybe volunteer to help you out a bit. 00:05:22.780 --> 00:05:26.500 I don't seem to have fi- I don't, I was- I don't seem to have 5.5 on my, on my 00:05:26.560 --> 00:05:30.120 system. Huh. Oh, you mean Anthropic? 00:05:31.840 --> 00:05:37.320 Shadow banned. I just relaunched one Claude instance and, um, because 00:05:37.380 --> 00:05:39.120 it was running. It, I don't know. 00:05:39.580 --> 00:05:43.620 There's a, there's this one, uh, uh, advice in the prompting guide from 00:05:43.680 --> 00:05:48.280 Anthropic around the Family 5 model that you have to ask it, um, to explain 00:05:48.300 --> 00:05:51.560 what it's gonna do before it, it does the thing, and also tell it that, you 00:05:51.600 --> 00:05:53.760 know, the user doesn't see the, the, the shell command. 00:05:53.840 --> 00:05:56.870 So if there's anything in the shell output you want the user to see- Mm ... 00:05:56.920 --> 00:06:00.420 then you must, um, you know, summarize and repeat it for the user. 00:06:00.720 --> 00:06:01.740 Mm. And then things like that. 00:06:02.240 --> 00:06:06.440 And I did those prompts in one session, and it resulted in- 00:06:07.288 --> 00:06:12.648 Um, Opus 5 for the last two days, every time I asked something, every 00:06:12.708 --> 00:06:18.168 turn it would say privately, "I need this and this to verify this," and 00:06:18.188 --> 00:06:22.628 then it would bullet point list everything it's gonna do, and then it would 00:06:22.848 --> 00:06:25.968 follow up with like a, a summary of what it did-- what it saw in the last 00:06:26.028 --> 00:06:30.008 turn. Um, like, um, so far the command output has shown me this and that. 00:06:30.348 --> 00:06:35.048 And it was infuriating to read because I always have to like skip past the 00:06:35.088 --> 00:06:38.668 privately and I would jump down to whatever summary there is and then realize, 00:06:38.848 --> 00:06:40.508 oh no, I have no idea what he's talking about. 00:06:40.828 --> 00:06:43.308 Go back to the privately part, which is so annoying. 00:06:43.348 --> 00:06:44.508 Like, I don't need anything. 00:06:44.688 --> 00:06:46.108 He's, he's saying... I don't know. 00:06:46.168 --> 00:06:47.008 I should-- maybe I should show the- Yeah. 00:06:47.028 --> 00:06:48.168 There- ... the transcript. There's a lot of news. 00:06:48.208 --> 00:06:52.268 Like you, you, you're probably referring to these prompt engineering best 00:06:52.328 --> 00:06:55.008 practices. I mean, like every model has their own one 00:06:55.888 --> 00:07:01.108 practically. No, like the family of 5, like Opus 5, Sonnet 5, F- Fable 5 and 00:07:01.128 --> 00:07:04.148 now 5.5, they have a very specific... 00:07:04.308 --> 00:07:05.988 Like they're way more goal-oriented. 00:07:06.028 --> 00:07:07.608 They're just going ahead and doing things. 00:07:08.028 --> 00:07:11.447 Uh, people are complaining on LinkedIn like, "I asked a question and Opus just 00:07:11.507 --> 00:07:15.608 went off a, you know, a, a series of commands to, to do the thing when I was 00:07:15.668 --> 00:07:18.628 just only asking a question." Well, that's the way they've been trained. 00:07:18.668 --> 00:07:23.148 They've been trained to, you know, act on, uh, information and, and, and, and 00:07:23.208 --> 00:07:24.408 just go ahead and do things. 00:07:24.708 --> 00:07:29.068 And then I, I bet you if, if they change that behavior, you will have a whole 00:07:29.108 --> 00:07:31.988 bunch of other people say like, "I'm asking something and then it never does 00:07:32.028 --> 00:07:34.238 anything and I have to prompt it 100 times to- Oh, yeah ... 00:07:34.328 --> 00:07:35.868 to make it do the thing, right? 00:07:36.768 --> 00:07:38.208 And then at least- You're talking about user-facing 00:07:40.668 --> 00:07:43.988 . Yeah. So, so I'll show you the, the thing w- I'm talking about, about the, 00:07:44.248 --> 00:07:46.108 the privately. I, I'll just read it out. 00:07:46.568 --> 00:07:50.968 It says like, "Privately colon nothing further is needed, both at is landed 00:07:51.048 --> 00:07:53.088 and the remaining work is yours. 00:07:53.448 --> 00:07:56.348 No tool calls." And then boom, it replies with the whole thing. 00:07:56.748 --> 00:07:58.108 So it's-- it was very annoying. 00:07:58.148 --> 00:08:02.808 But, but what I did was I compacted and then I relaunched with 5.5 and then 00:08:02.908 --> 00:08:04.568 it, uh, behaves beautifully now. 00:08:05.308 --> 00:08:06.668 Nice. Doesn't do the privately anymore. 00:08:06.728 --> 00:08:08.628 I mean, Opus 5 needed help, man. 00:08:08.688 --> 00:08:10.468 It needed some effing help. 00:08:10.628 --> 00:08:13.228 Yeah, a lot of people were very, very upset about it, yeah. 00:08:13.228 --> 00:08:14.668 I was-- I'm, I was sick of it, honestly. 00:08:14.728 --> 00:08:15.788 I'm, uh, I'm, I'm like 00:08:17.168 --> 00:08:20.248 cancel Anthropic type s-situation at work. 00:08:21.548 --> 00:08:23.428 Um- I, I was, I was stuck with it. 00:08:23.528 --> 00:08:28.208 So anyway, there's a lot more to discuss aside from FOSDEM and Opus 5 point... 00:08:29.508 --> 00:08:32.888 Okay. So, um, there's this one thing from 00:08:33.988 --> 00:08:39.068 the... I think Joe Beda or was it, uh, Craig McLuckie, one of the Kubernetes 00:08:39.148 --> 00:08:43.588 guys and they, uh, you know, the original creators of Kubernetes at Google, 00:08:44.268 --> 00:08:49.208 um, they created an open source cloud native harness to run 00:08:49.308 --> 00:08:50.528 agents, um, 00:08:51.368 --> 00:08:54.048 on, on Kubernetes as far as I think-- I, I can tell. 00:08:54.088 --> 00:08:57.808 Like get your harness off the desktop, so it's called Mecatl.dev. 00:08:58.608 --> 00:09:01.508 Um- Mecatl dot... What a odd name. 00:09:02.328 --> 00:09:05.908 It's funny that they actually put the pronunciation in the name now. 00:09:05.948 --> 00:09:11.148 Like everyone's complaining about kubectl versus kube ctl, uh, versus kube... 00:09:11.168 --> 00:09:16.268 What's the other one? Kube control. So now they put Mecatl and they actually put- Oh, how do you 00:09:16.348 --> 00:09:17.488 spell "meh"? Sorry, I didn't... 00:09:17.588 --> 00:09:18.268 I'm gonna get this. 00:09:19.548 --> 00:09:20.788 Um- M-E- I'll just... 00:09:21.908 --> 00:09:26.058 It's M-E-C-A-T-L. I think it's a reference to cattle- Oh ... 00:09:26.068 --> 00:09:28.168 because your agents are like cattle. 00:09:31.108 --> 00:09:32.048 Oh, it's from Stacklok. 00:09:32.508 --> 00:09:33.288 I forgot who's behind it. 00:09:33.328 --> 00:09:36.908 Just send me the link. I've just typed the URL tw- five times wrong. 00:09:38.368 --> 00:09:39.308 I sent it on WhatsApp. 00:09:39.868 --> 00:09:42.088 So this-- I'm, I'm very into harnesses right now. 00:09:42.108 --> 00:09:45.208 I'm very interested, um, playing around with some harnesses. 00:09:45.888 --> 00:09:47.408 Um, AWS Strands 00:09:48.488 --> 00:09:53.728 agent SDK has also announced a harness now, so I'm excited to try that one out 00:09:53.788 --> 00:09:56.628 too for like running agents really. 00:09:59.328 --> 00:10:00.948 Cute dog. Cute mascot. 00:10:05.768 --> 00:10:07.008 Well- That's how you get 'em. 00:10:08.848 --> 00:10:10.088 I'm very happy with... 00:10:10.968 --> 00:10:15.808 Like f-for me like alternating between, uh, Claude and Codex 00:10:16.968 --> 00:10:19.828 within Orca is a, is a perfect experience for me. 00:10:20.208 --> 00:10:22.708 And then, of course, I use Hermes w-with... 00:10:23.108 --> 00:10:28.108 via WhatsApp and sometimes I use it through iMessage but I'm just so used to 00:10:28.188 --> 00:10:30.088 using WhatsApp, I just end up using WhatsApp. 00:10:30.648 --> 00:10:34.688 So I'm not too sure what you mean by why you're interested in harnesses. 00:10:34.728 --> 00:10:36.488 They seem commoditized to me. 00:10:36.528 --> 00:10:37.668 They... There's, there's no real 00:10:38.768 --> 00:10:42.368 difference to me. I mean, okay, there's maybe where you run it. 00:10:43.228 --> 00:10:48.028 I personally run my, my stuff locally inside my LAN on a Pi or, or one of my 00:10:48.088 --> 00:10:52.988 ThinkPads. What, what-- why are you so attracted to running on bloody 00:10:54.068 --> 00:10:54.808 Kubernetes? 00:10:57.128 --> 00:10:58.088 So you can have more of them? 00:10:59.468 --> 00:11:00.348 So you can try different things? 00:11:00.428 --> 00:11:03.728 Yeah. I'm, I'm just interested to understand why people create different 00:11:03.768 --> 00:11:07.208 harnesses, right? There's a reason why they, they create those. 00:11:07.248 --> 00:11:08.608 Like Pi.dev is clear. 00:11:08.908 --> 00:11:14.048 He said like this thing had become overly complex, hard to extend, um, and 00:11:14.108 --> 00:11:16.648 Pi.dev is minimal and you... 00:11:16.688 --> 00:11:19.598 and easily extendable. Like that's a very clear message- Yeah ... 00:11:19.668 --> 00:11:20.348 and people love it. 00:11:20.408 --> 00:11:21.788 People build their own on top of it. 00:11:22.148 --> 00:11:25.728 I used it. I'm, I'm still not sure if what I did was the right thing to do, 00:11:26.488 --> 00:11:30.628 um, but I used Pi.dev to, to do like 00:11:31.448 --> 00:11:32.568 I wanted to review, 00:11:33.568 --> 00:11:36.148 um, Terraform plans 00:11:37.468 --> 00:11:40.028 against- Mm ... uh, the actual pull request. 00:11:40.148 --> 00:11:45.388 So I extended Atlantis to, to, to, to do a review, um, 00:11:45.568 --> 00:11:47.788 and I deployed the review through Pi, 00:11:48.808 --> 00:11:49.968 um- Right ... as the harness. 00:11:50.488 --> 00:11:53.588 Yeah. Well, m-maybe I'm getting confused about the terminology here because 00:11:53.628 --> 00:11:54.468 like at work 00:11:55.388 --> 00:11:59.028 we have an-- we call them agents where we-- where they're like, you know, they 00:11:59.068 --> 00:12:04.468 have a couple of skills and maybe even, even an MCP connection and 00:12:04.908 --> 00:12:06.288 such and so forth, and they... 00:12:06.328 --> 00:12:10.628 and, you know, we have a, we have an agent for, for security, we have an agent 00:12:10.688 --> 00:12:15.288 for code review, we have an agent for this, that. 00:12:16.020 --> 00:12:17.600 Is that a, is that a different harness? 00:12:17.780 --> 00:12:18.580 I'm not too sure. 00:12:18.960 --> 00:12:19.980 Uh, does it really matter? 00:12:20.040 --> 00:12:20.720 I'm not too sure. 00:12:21.020 --> 00:12:23.320 But I, I, I get the idea of creating, 00:12:24.660 --> 00:12:28.640 uh, an agent 00:12:30.940 --> 00:12:35.500 personality/identity/ I don't know what you wanna call it, to do a specific 00:12:35.560 --> 00:12:36.520 thing, if that's what you're 00:12:37.340 --> 00:12:38.020 talking about. 00:12:39.420 --> 00:12:42.460 Fa- Yeah. Um, I agree with you that it's a... 00:12:42.520 --> 00:12:45.720 Oh, you, you have it open on the, on the shared screen. 00:12:45.780 --> 00:12:47.200 I didn't know you were sharing your screen. 00:12:47.740 --> 00:12:52.580 Yeah. So I think there is a bit of a confusion between what is a harness versus what 00:12:52.640 --> 00:12:54.840 is, um, what is an agent, 00:12:55.700 --> 00:12:58.840 right? Um, I agree that, 00:12:59.680 --> 00:13:01.940 you know, Claude Code, I would say, is a harness, right? 00:13:02.100 --> 00:13:04.780 It-it's, um, it's an environment that 00:13:06.500 --> 00:13:10.340 provides the LLM with the tools and creates an agent loop. 00:13:11.340 --> 00:13:12.780 Um, but is it an agent? 00:13:12.920 --> 00:13:16.580 I feel like an agent is something that runs more independently. 00:13:16.600 --> 00:13:18.340 You know what? I'm gonna ask ChatGPT. 00:13:20.440 --> 00:13:24.160 What is the difference between- I found myself doing that sometimes. 00:13:24.200 --> 00:13:29.020 Like, a-anytime my, my, my thoughts were not, like, clear, I'm like, "I gotta 00:13:29.080 --> 00:13:32.900 ask ChatGPT." And I feel like maybe I should just spend a little bit more time 00:13:32.940 --> 00:13:35.560 thinking about this and researching it the old way. 00:13:36.780 --> 00:13:38.500 But like, "Nah, I'm just gonna use ChatGPT." 00:13:39.560 --> 00:13:42.500 No, the thing is that I, I feel like I have a very good understanding of, 00:13:42.600 --> 00:13:46.180 like, things. I just want it to, like, help me realign, 00:13:47.060 --> 00:13:50.780 um, my thoughts and see what e- what else it throws up. 00:13:51.720 --> 00:13:54.440 Um- Yeah, like, have I got... 00:13:59.220 --> 00:14:00.740 Yeah. So, 00:14:01.600 --> 00:14:02.620 so this is interesting. 00:14:05.100 --> 00:14:06.540 Yeah, it's very str- Yeah. 00:14:07.060 --> 00:14:10.040 It immediate-- So this is the thing about ChatGPT and its memory, right? 00:14:10.120 --> 00:14:12.240 It immediately linked, um, 00:14:13.300 --> 00:14:18.740 my question with, with the memory of, of what I did, which is I used pi.dev, 00:14:18.880 --> 00:14:24.279 which is a harness, and deployed it into Atlantis instance to act like 00:14:24.320 --> 00:14:27.180 an agent. So is it now an agent? 00:14:27.740 --> 00:14:32.180 Why did it have to be a-- Because if you use Strands Agent SDK or if you use 00:14:32.240 --> 00:14:34.360 Claude Code Agent SDK, right, 00:14:35.400 --> 00:14:37.600 you, you will get... 00:14:38.640 --> 00:14:41.680 It's not Claude, Claude Code Agent SDK, but it is Claude Agent SDK. 00:14:42.060 --> 00:14:43.480 You get the, the core 00:14:44.680 --> 00:14:47.000 capabilities to build an agent, 00:14:47.980 --> 00:14:53.420 meaning you get tools and other things like, um, write file, read file- Mm-hmm 00:14:53.420 --> 00:14:54.790 ... um, you know- There's a lot in it ... 00:14:54.790 --> 00:14:56.020 this type of- Totally. 00:14:56.060 --> 00:14:59.080 Yeah. So- Yeah. Yes. 00:14:59.120 --> 00:15:00.580 So the strengths of SDK- The sandbox. 00:15:02.040 --> 00:15:05.500 Yeah. And also, like, the Strands SDK gives you, like, the ability to, you 00:15:05.540 --> 00:15:10.400 know, spin, spin off, uh, create a graph of agents, um, and do all the things. 00:15:10.680 --> 00:15:13.420 And then what they did recently is they announced 00:15:14.340 --> 00:15:18.140 the AWS Strands harness. Oh, can you, can you see my screen share? 00:15:19.000 --> 00:15:20.260 It's drawing me a diagram. 00:15:21.600 --> 00:15:22.780 Oh, it's drawing you a diagram. 00:15:25.520 --> 00:15:30.200 Did you-- This is Claude. Does it automatically use the Excalidraw or are you using some 00:15:30.240 --> 00:15:31.200 type of MCP plug? 00:15:31.560 --> 00:15:33.220 I-I've, I've got a connector here. 00:15:34.440 --> 00:15:38.800 Ah. Oh, nice. O-O-Opus fi- five point five, baby. 00:15:40.620 --> 00:15:41.300 So you found it. 00:15:43.100 --> 00:15:44.520 Oh, this is on my wife's account. 00:15:44.560 --> 00:15:46.860 It's not on my, on my en- enterprise account yet. 00:15:47.560 --> 00:15:49.520 Oh, yeah. Of course. So digital ha-harness has... 00:15:53.540 --> 00:15:56.540 I'm actually struggling to see the diff- understand this. 00:15:57.520 --> 00:15:58.230 It's not as clear as I'd like it to be. 00:15:58.240 --> 00:16:03.640 It's also interesting because this looks very fam- similar to a picture that, 00:16:03.860 --> 00:16:08.640 like, the Kiro people drew from AWS when they explained the refactoring they 00:16:08.700 --> 00:16:12.580 did. When they, when they did kiro.dev, originally they, they had the Kiro 00:16:12.660 --> 00:16:15.400 CLI, which is like a terminal interface harness. 00:16:15.840 --> 00:16:19.980 Then they have the Kiro IDE web interface for doing speculative development. 00:16:20.380 --> 00:16:21.380 And then they have the... 00:16:21.700 --> 00:16:23.860 Sorry, not web interface, but an, an app. 00:16:24.280 --> 00:16:27.900 And then they have the Kiro Web, which was like three different services and 00:16:27.940 --> 00:16:32.180 three different ways of, of, um, interacting with the agent and the agent loop 00:16:32.300 --> 00:16:33.960 and, and the tools that it has. 00:16:34.379 --> 00:16:38.640 And then they rewrote the whole thing, building the Kiro, I think, what they 00:16:38.700 --> 00:16:43.920 call also the harness or the core capability, um, of Kiro, and then 00:16:44.000 --> 00:16:49.180 building everything, every interface on top of it using ACP, the Agent Client 00:16:49.220 --> 00:16:54.500 Protocol. Mm. So then they built the Kiro, um, I would say harness part, 00:16:55.080 --> 00:16:59.340 and then they built an IDE, like to re-rebuild the IDE experience, the SD 00:16:59.400 --> 00:17:04.420 development experience, uh, using, uh, ACP to, to have basically your Monaco, 00:17:05.160 --> 00:17:09.500 um, VS Code-like interface, and then ACP drives the agent connection. 00:17:09.860 --> 00:17:14.200 And then you have Kiro Crew, which is you have your Discord-like or chat-like 00:17:14.260 --> 00:17:18.700 interface, uh, like your ChatGPT-type interface, and then behind the scene you 00:17:18.740 --> 00:17:22.300 have the Kiro AC- with ACP, it drives the Kiro harness. 00:17:22.580 --> 00:17:27.580 And then you have the Kiro CLI, um, to drive your TUI, your terminal user 00:17:27.620 --> 00:17:32.060 interface, and, and then behind the scene it drives your, um, Kiro harness 00:17:32.120 --> 00:17:36.640 with, with ACP. So I think Mecatl is maybe trying-- kinda doing the same 00:17:36.680 --> 00:17:38.500 thing. Like, what's the difference between Mecatl? 00:17:38.960 --> 00:17:39.840 Ask that in your chat. 00:17:39.880 --> 00:17:44.540 What's the difference between Mecatl and what it does versus Agent Client 00:17:44.600 --> 00:17:49.460 Protocol if you compare it to, like, initiatives like, um, like Kiro? 00:17:49.760 --> 00:17:50.640 And another famous 00:17:51.520 --> 00:17:55.000 promoter, I think, from Agent Client Protocol is Zed, right? 00:17:55.160 --> 00:18:00.000 The, the IDE that have heavily pivoted towards agentic IDE 00:18:00.100 --> 00:18:02.860 using Agent Client. So what's the difference between a-agents, 00:18:04.100 --> 00:18:08.900 uh- ACP and, and, uh, and how it applies to, to Mecatl. 00:18:09.560 --> 00:18:12.680 That's an interesting... Uh, there's definitely a lot more there to explore. 00:18:13.000 --> 00:18:13.680 Uh, unfortunately- Well, the... 00:18:13.740 --> 00:18:17.800 I'm sorry to-- I'm, I'm thinking of agents as like a, like a Docker sandbox 00:18:17.880 --> 00:18:21.780 and you-- and like you're, like you were just saying, you have many different 00:18:21.880 --> 00:18:23.340 UIs on that Docker sandbox. 00:18:23.400 --> 00:18:26.484 You could be Connecting it to your local desktop. 00:18:26.544 --> 00:18:27.804 You could be using it from the web. 00:18:27.904 --> 00:18:30.464 You can be using it from, I don't know, Kubernetes. 00:18:30.704 --> 00:18:32.444 Uh, anyway, that's just the way I was... 00:18:32.784 --> 00:18:33.704 I don't know what happened there. 00:18:34.744 --> 00:18:38.544 Yeah. I think that's a good, that's a good, um, you know, idea on, on 00:18:38.604 --> 00:18:41.904 understanding. It seems like you have, uh, opened the, the, the thing in, 00:18:42.444 --> 00:18:45.584 like, the artifact. You have to close the, the Excalidraw at the top right. 00:18:47.564 --> 00:18:50.364 I think it's a, a good way to think about agent-client protocol. 00:18:50.484 --> 00:18:52.884 Oh yeah, it d- it does men-mention Zed right there. 00:18:53.564 --> 00:18:57.024 Um. Mecatl is a piece of software. 00:18:57.063 --> 00:18:58.184 ACP is a protocol 00:18:59.084 --> 00:19:00.104 where the agent runs. 00:19:00.184 --> 00:19:03.464 Yeah, yeah. Yeah, durability. Yeah. 00:19:07.304 --> 00:19:08.054 Okay. So they- Exactly ... 00:19:08.084 --> 00:19:08.824 like what they're saying... 00:19:09.384 --> 00:19:12.324 Yeah. So basically, Mecatl could be built on top of ACP. 00:19:12.524 --> 00:19:15.294 It just, it's just a protocol- Yeah, exactly ... 00:19:15.294 --> 00:19:16.584 um, separating the front end from the back end. 00:19:16.584 --> 00:19:17.924 But the, but the protocol connects it all up. 00:19:17.984 --> 00:19:19.704 Yeah, I think this is all, this is all coming together. 00:19:20.764 --> 00:19:22.384 So today is the first of October. 00:19:22.504 --> 00:19:25.984 Google just announced, um, Argon, is it? 00:19:26.464 --> 00:19:30.944 A new Gemini Argon, which beats both Astra and Opus and, um- Really? 00:19:31.884 --> 00:19:34.164 Yep. They announced it, like, eleven hours ago. 00:19:34.764 --> 00:19:35.624 Get with the news. 00:19:37.984 --> 00:19:39.184 Jeez. Uh- How can you not be aware? 00:19:39.964 --> 00:19:43.784 I, I was watching this, like, Sky News presentation about all this AI 00:19:43.824 --> 00:19:46.784 investment and this impending bubble bursting, 00:19:47.844 --> 00:19:52.904 and it was pointing out that, like, Google was, like, putting down the most amount 00:19:52.924 --> 00:19:53.764 of cash down. 00:19:54.664 --> 00:19:55.624 And I was thinking to myself, like, 00:19:56.624 --> 00:19:59.344 if they're putting the most amount of cash down, the results don't really 00:19:59.444 --> 00:20:00.764 speak for it, do they? 00:20:02.624 --> 00:20:03.944 They're horrible at delivery. 00:20:04.324 --> 00:20:06.324 Like l- my- Gemini's terrible 00:20:07.144 --> 00:20:10.864 ... I had a Google, I had a Google Cloud with Vertex subscription to access 00:20:10.944 --> 00:20:12.344 Gemini. I would expect... 00:20:12.684 --> 00:20:16.204 I think it was even more expensive than, you know, their promotional bundles 00:20:16.304 --> 00:20:21.084 on, on their, like, subs- like, more user-focused, uh, plans. 00:20:21.604 --> 00:20:25.824 Um, I didn't get access for, like, months, and I canceled my GCP because 00:20:25.884 --> 00:20:28.024 Gemini was just all so horrible. 00:20:28.564 --> 00:20:33.184 They were unable to, like, roll it out smoothly across all of their platforms 00:20:33.644 --> 00:20:38.824 compared to Anthropic, which announces, and it's available the same day, you 00:20:38.864 --> 00:20:41.464 know? Yeah. So is there a pelican? 00:20:41.804 --> 00:20:42.844 How can we judge this thing? 00:20:43.344 --> 00:20:46.284 Pelican. Pelican. 00:20:47.344 --> 00:20:51.124 No pelicans yet because nobody has access to it, probably. 00:20:51.904 --> 00:20:52.584 It's like an announcement. 00:20:52.724 --> 00:20:55.984 Oh, man. I mean, in my opinion, they should just download this. 00:20:56.804 --> 00:21:00.484 If s- So they announced it... 00:21:00.804 --> 00:21:02.084 Oh, for the love of God. 00:21:02.924 --> 00:21:05.224 No, th- this is, that, that, that I find really irritating. 00:21:05.224 --> 00:21:06.624 Look at the benchmark. There's a picture of the benchmark. 00:21:06.964 --> 00:21:08.464 I... That's really irritating. 00:21:09.564 --> 00:21:11.084 No pelican benchmark. Ooh. 00:21:11.104 --> 00:21:12.144 That boils my blood. 00:21:12.764 --> 00:21:16.904 Well, then there's, there was Jev, which I'm, which I didn't get, uh, I w- I 00:21:17.004 --> 00:21:19.604 was a bit slow to move on, so I can't seem to get... 00:21:19.944 --> 00:21:23.244 I'm on the wait list for, like, weeks now, so I guess they're just, it, 00:21:23.524 --> 00:21:26.004 they're, they're doing some sort of exclusive- To have you. 00:21:26.364 --> 00:21:27.174 I mean- ... exclusivity play ... 00:21:27.204 --> 00:21:32.484 there's a ton of what they call system one models that are not doing text 00:21:32.544 --> 00:21:35.444 generation, but taking in the full 00:21:36.324 --> 00:21:41.584 request and sending out a single response in a typed format, which is very 00:21:41.644 --> 00:21:45.284 good, like, sub-millisecond, and, and the price is much cheaper. 00:21:45.864 --> 00:21:48.194 Um- Yeah. I no- I noticed something- There's tons of those now ... 00:21:48.304 --> 00:21:49.004 I was posting something on... 00:21:49.464 --> 00:21:51.464 about using Qwen to do it, and, uh- 00:21:52.384 --> 00:21:55.924 Yeah. So there's people that have taken OpenWeights models and, and retrained 00:21:55.944 --> 00:21:57.434 them to do the same thing and- Yeah ... 00:21:57.464 --> 00:22:02.314 posting similar results. There's Laya that says, "I built this a year ago." 00:22:02.324 --> 00:22:05.264 One thing that hasn't clicked for me is I, I don't, I don't actually 00:22:05.344 --> 00:22:06.524 understand how to, 00:22:07.384 --> 00:22:09.744 to use Jev in anything that I do. 00:22:10.304 --> 00:22:11.084 I don't need to make 00:22:12.684 --> 00:22:14.864 quick decisions about things. 00:22:15.364 --> 00:22:18.244 I mean, have you implemented it in something? 00:22:18.324 --> 00:22:23.584 Like, what I expect Anthropic to do is to replace Claude Code’s, um, 00:22:24.944 --> 00:22:29.464 auto mode, like the classifier, the security classifier, with something like 00:22:29.504 --> 00:22:34.804 that. I try to, uh, look at using Jev or Laya in 00:22:34.884 --> 00:22:37.154 my benchmark suite- But will, will that work- ... 00:22:37.204 --> 00:22:41.044 to, to make judgment calls on, on the output of the agents. 00:22:41.844 --> 00:22:45.124 Um, but- To, to play, to play devil's advocate there, Vincent, 00:22:46.124 --> 00:22:48.764 the auto mode has a quite a bit of like, 00:22:50.104 --> 00:22:54.164 you know, words, tracing, and then you can see why it made a decision. 00:22:54.264 --> 00:22:58.144 Surely, if you, if you cut down auto mode to, like, a yes or no, 00:22:59.144 --> 00:23:00.904 you will... How, how will you get the, the why? 00:23:00.904 --> 00:23:04.964 Oh, Jev is providing on each answer a probability, right? 00:23:05.024 --> 00:23:08.924 Yeah. It's, it's, it's not generating text, but it's giving you yes ninety 00:23:08.964 --> 00:23:09.844 percent, no- 00:23:10.704 --> 00:23:13.104 Yeah, but that's- ... hopefully totals to one hundred percent. 00:23:14.184 --> 00:23:18.724 ... but that's, that's harder to rationalize. That's harder to rationalize over, you know, I thought this was going to... 00:23:19.244 --> 00:23:23.404 I shouldn't be executing something like this because of the Claude MD file, 00:23:23.604 --> 00:23:25.744 and hence I will not permit this. 00:23:27.884 --> 00:23:29.704 Yeah. It depends on what you're trying to do. 00:23:29.764 --> 00:23:34.204 Like, obviously, I think a system one model, um, is way more 00:23:35.344 --> 00:23:39.984 evalu- you can evaluate, uh, it way better if it has a strict set of answers 00:23:40.464 --> 00:23:41.924 and, and a probability. 00:23:42.724 --> 00:23:46.804 I think you can-- It's way more reliable to, to, to, to depend on. 00:23:46.964 --> 00:23:49.744 Like, if it's... It's, it's very niche use case, right? 00:23:49.844 --> 00:23:52.684 It's, it's a, it's a very well-defined problem. 00:23:52.844 --> 00:23:55.564 There's only a certain amount of answers, like a set of answers. 00:23:55.564 --> 00:23:56.904 I, I disagree with the reliability. 00:23:56.964 --> 00:23:59.784 Like, to, to have something reliable, you also need 00:24:00.664 --> 00:24:01.584 to have that- Well, what do you mean? 00:24:01.964 --> 00:24:03.744 I, I was talking about the use case. 00:24:04.584 --> 00:24:05.424 You need logs. No, no, no, no. 00:24:05.444 --> 00:24:09.564 You, you interrupted me in the middle of what I was explaining what it's for, 00:24:09.824 --> 00:24:13.164 and you said you need logs, and you haven't even listened to what I was saying 00:24:13.204 --> 00:24:15.264 what it's for. Well, I was just retorting to the whole thing about 00:24:15.324 --> 00:24:19.104 reliability. You need accountability, reliability, and so, so, so forth. 00:24:19.224 --> 00:24:21.504 Okay. Depends on your use case, right? 00:24:21.964 --> 00:24:24.064 Which I'm trying to explain the use case. 00:24:25.764 --> 00:24:32.888 Right? I was saying that people like Jev so much because of the use cases they use it 00:24:32.948 --> 00:24:36.628 in are there's a very f-very well-defined set of answers. 00:24:37.148 --> 00:24:39.368 Um, they only want one of those answers. 00:24:39.468 --> 00:24:40.908 It needs to be well-structured. 00:24:40.968 --> 00:24:42.508 It needs to respect that structure. 00:24:42.808 --> 00:24:43.588 It needs to be cheap. 00:24:43.628 --> 00:24:45.228 It needs to be fast, right? 00:24:45.628 --> 00:24:48.708 For those use cases, that's where they want the System One model. 00:24:49.068 --> 00:24:52.268 They are doing samples and demos of how powerful it is with like, "Oh, look, 00:24:52.328 --> 00:24:56.088 it can play Tetris," because it gets in the input of the game, and it responds 00:24:56.128 --> 00:24:58.348 with the next move within seconds, right? 00:24:58.808 --> 00:25:01.288 Um, with not within seconds, within milliseconds. 00:25:01.308 --> 00:25:02.948 Yeah. So it's so fast that it can even play a game. 00:25:03.408 --> 00:25:07.928 So i-it is reliable because you have a, a system that needs to run and, and 00:25:07.968 --> 00:25:09.628 you know, an LLM takes time to respond. 00:25:09.668 --> 00:25:13.508 It generates an answer, and then sometimes it doesn't even-- isn't even valid 00:25:13.568 --> 00:25:16.948 JSON. You need to add like a whole bunch of gates and validation around it to 00:25:16.988 --> 00:25:18.948 make sure that it's actually valid JSON and so on. 00:25:19.288 --> 00:25:23.068 So you can't trust the, the response of the LLM, which is a text generator, 00:25:23.488 --> 00:25:24.788 right? So a System One model 00:25:25.968 --> 00:25:28.148 use cases seem very well-defined. 00:25:28.728 --> 00:25:31.968 Now, to come back to the argument that you're making, I said it could work for 00:25:31.988 --> 00:25:35.448 an auto mode. Uh, you said absolutely not because you need audit, you need, 00:25:35.508 --> 00:25:36.528 you need, uh, this and that. 00:25:36.588 --> 00:25:38.868 Okay, fair. Not the right use case. 00:25:39.048 --> 00:25:43.208 Yeah. Okay? I was just saying I'm waiting for Anthropic to, like, employ it 00:25:43.248 --> 00:25:44.368 where it's, where it's appropriate. 00:25:44.548 --> 00:25:46.148 I thought maybe auto mode, you said no. 00:25:46.568 --> 00:25:50.508 And I had a very similar experience when I suggested to Opus, how can we use, 00:25:50.568 --> 00:25:51.488 like, the System One model? 00:25:51.808 --> 00:25:55.488 Because I have a lot of problems with validating the output of the agents. 00:25:55.948 --> 00:26:01.208 I'm running OPA, Reg Val-- um, Reg, Rego, um, to, to identify if the 00:26:01.268 --> 00:26:04.368 agent is building an infrastructure according to certain grades. 00:26:04.928 --> 00:26:09.508 Um, it calls it oracles, and they're at, like, static all the way up to live, 00:26:09.608 --> 00:26:13.448 which means the infrastructure has to be deployed, which means, uh, it mutates 00:26:13.508 --> 00:26:17.008 the account, which means you can't run multiple tests at the same account 00:26:17.068 --> 00:26:18.268 unless you separate them. 00:26:18.608 --> 00:26:23.668 So, so I have these static validation of the, of the, you know, agent's, uh, 00:26:23.748 --> 00:26:27.428 results, and I'm like, "Couldn't I do this better- Mm-hmm. 00:26:27.428 --> 00:26:30.328 -with a System One model?" And Opus was like, "Absolutely not. 00:26:31.208 --> 00:26:32.508 You're gonna have all of these problems. 00:26:32.548 --> 00:26:37.068 It's not gonna make any sense." Um, so again, probably not the right use case. 00:26:37.108 --> 00:26:39.027 Well, it seems, it seems very ni-- okay, well, 00:26:40.068 --> 00:26:44.908 my takeaway is that Jev and this type model responses seem very, very niche 00:26:45.488 --> 00:26:46.308 compared to 00:26:47.508 --> 00:26:51.368 the general use of AI that we're all accustomed to at this point. 00:26:51.988 --> 00:26:56.048 So... I think it makes sense if you need to run agents in production, which is, 00:26:56.448 --> 00:27:00.848 like, if you need-- if you have a system that needs to do some type of intent, 00:27:01.248 --> 00:27:02.988 like needs to take in a lot of context. 00:27:03.328 --> 00:27:07.768 Yeah. It's very hard to do like a pure algorithmic, um, you know, to 00:27:07.808 --> 00:27:12.568 va-evaluate it. It has to have some type of intent or s-sentiment judgment. 00:27:12.748 --> 00:27:14.188 Yeah. And it needs to make a, a quick call. 00:27:14.208 --> 00:27:16.268 Like, like a classifier or something like that. 00:27:16.308 --> 00:27:17.968 Yeah, like email classifiers. 00:27:18.228 --> 00:27:21.648 There's a lot of people coming forward say, "I, I run Jev on my Gmail to 00:27:21.688 --> 00:27:25.168 classify thousands of emails, and within, like, a second, it classified one 00:27:25.228 --> 00:27:29.268 thousand emails- Yeah. -and my email inbox is now handled." We're, we're 00:27:29.348 --> 00:27:33.708 actually-- funny you should mention that, but, like, recently I was impressed. 00:27:34.168 --> 00:27:35.448 I'm always impressed when... 00:27:35.968 --> 00:27:39.308 I know this sounds very disparaging, but, like, you know, like, you work in a 00:27:39.368 --> 00:27:44.768 company and there's a, there's a product owner or there's a scrum master, and 00:27:44.808 --> 00:27:48.448 he's obviously non-- or he or she is obviously non-technical. 00:27:49.688 --> 00:27:52.408 And then they do something with, uh, Claude. 00:27:52.528 --> 00:27:53.948 I'm like, "Well done. 00:27:54.008 --> 00:27:55.368 You did something." 00:27:56.408 --> 00:27:58.908 Um, so, uh, 00:27:59.728 --> 00:28:02.148 I mean, I'm being disparaging, but really it's cool. 00:28:02.168 --> 00:28:05.208 You're being mean. It's, it's-- I'm being, I'm being mean. 00:28:05.468 --> 00:28:09.708 But, but seriously, you know, the, uh, I can't remember what is-- what this 00:28:09.748 --> 00:28:11.068 person's role is exactly. 00:28:11.128 --> 00:28:17.188 This person m-managed to get, you know, the script runner JIRA tickets and put it into 00:28:17.268 --> 00:28:21.848 Claude, create an artifact that, that was interactive in the sense that you 00:28:21.888 --> 00:28:26.128 could, you could change the parameters a little bit, you know, by tag or 00:28:26.508 --> 00:28:30.648 s-some filter, and then, and then it would classify the, the JIRA tickets. 00:28:31.248 --> 00:28:35.348 Much like I think you suggested, uh, to me some time ago, is like it was going 00:28:35.428 --> 00:28:37.928 back in history to sort of like work out, 00:28:38.928 --> 00:28:41.468 you know, the areas where the team has been spending time. 00:28:41.508 --> 00:28:44.728 Like the team has been spend-- you know, twenty percent of tickets are PR 00:28:44.788 --> 00:28:50.107 reviews, twenty percent of tickets are responding to, uh, this team and their 00:28:50.508 --> 00:28:55.228 asks. Twenty percent of the time is done, I don't know, doing, um, 00:28:56.368 --> 00:29:01.108 some other thing. And, uh, I thought that was really interesting and, uh, like 00:29:01.148 --> 00:29:03.208 this person just did it without 00:29:04.388 --> 00:29:05.348 me suggesting it. 00:29:07.088 --> 00:29:08.648 And he- Well, there you go. And the person did a great job. 00:29:08.688 --> 00:29:12.308 And I was like, I was like, I was like, "Yeah, this is the power of AI." 00:29:13.168 --> 00:29:14.688 Yes. Yeah, this is great. 00:29:16.068 --> 00:29:17.778 This is-- th-th-this is the power of AI. 00:29:17.778 --> 00:29:19.388 So the retrospective was a lot better. 00:29:19.428 --> 00:29:20.768 He brought it into a retrospective. 00:29:20.948 --> 00:29:22.668 So like, like, "Oh, what should we work on? 00:29:22.708 --> 00:29:26.188 What can we improve on?" And like, it was the first time, instead of like a 00:29:26.228 --> 00:29:30.508 retrospective being like, you know, "How do you feel today?" It was like we 00:29:30.548 --> 00:29:34.268 had the numbers there, and the, the numbers-- it wasn't cold, hard numbers. 00:29:34.328 --> 00:29:37.348 It was classified numbers, so it was, it was really actionable. 00:29:39.148 --> 00:29:43.648 They, they've gotten better, um, but it's also scary 00:29:44.268 --> 00:29:47.008 to... I, I tell you what my experience is, right? 00:29:47.168 --> 00:29:51.208 I, I, I might have mentioned this last time, but I-- So I basically have 00:29:51.288 --> 00:29:55.948 Atlantis, which is a Terraform automation, uh, and collaboration tool that, 00:29:56.008 --> 00:29:59.568 that runs whenever somebody submits a pull request to this Terraform 00:29:59.608 --> 00:30:02.888 repository. If they make-- or to their product repository, they make a 00:30:02.928 --> 00:30:07.708 Terraform change, Atlantis will run Terraform Plan over it to evaluate the 00:30:07.768 --> 00:30:10.028 impact of that change against the live infrastructure. 00:30:10.488 --> 00:30:12.548 And then that plan can be very lengthy, right? 00:30:12.628 --> 00:30:15.188 Terraform plans can have a lot of information in there. 00:30:15.668 --> 00:30:19.608 Uh, on top of that, if they make a change to a shared module, that triggers, 00:30:19.908 --> 00:30:24.768 um, a plan across all of the instances that module is used, um, if they are, 00:30:25.088 --> 00:30:28.488 you know, trunk-based. So if somebody makes a change, you immediately see the 00:30:28.548 --> 00:30:32.068 impact across the whole infrastructure, um, if that change is gonna cause 00:30:32.088 --> 00:30:35.928 problems. Does it-- It's been a while since I've used Atlas-- Atlantis. 00:30:35.968 --> 00:30:40.468 Does, does Atlantis-- can it run plans in parallel and things like that? 00:30:40.508 --> 00:30:41.688 Yes. Okay. Yeah, yeah. 00:30:42.592 --> 00:30:44.392 But that, that brings its own problems with it. 00:30:44.572 --> 00:30:48.032 Um, and it's also one of the things that- criteria that evaluates on the, on 00:30:48.052 --> 00:30:51.992 the website, um, you know, that, that I built to evaluate Terraform automation 00:30:52.052 --> 00:30:55.752 solutions, and that has been featured on the, uh, Terraform weekly newsletter 00:30:55.792 --> 00:30:57.732 as well last week, which I was like, "Huh?" 00:30:58.612 --> 00:30:59.412 I built it for my own. 00:30:59.472 --> 00:31:02.732 Anyway, the, the thing was I was, I was trying to say is, and where I wanted 00:31:02.792 --> 00:31:07.332 to, to come back to the, the, you know, using, uh, Sonnet or Opus to go 00:31:07.392 --> 00:31:10.592 through a bunch of, uh, re-records and make an analysis. 00:31:11.112 --> 00:31:12.372 Um, basically, 00:31:13.232 --> 00:31:17.852 the plans that Atlantis generates, in some cases, it gets posted across, uh, 00:31:17.972 --> 00:31:19.812 in, in a pull request comment that has a limit. 00:31:20.292 --> 00:31:23.452 Um, so then it gets split, and then you have like five pull request comments. 00:31:23.572 --> 00:31:25.452 Each one have like a part of the plan. 00:31:25.712 --> 00:31:26.632 You need to go through it. 00:31:27.052 --> 00:31:28.472 Um, it's, it's complicated. 00:31:28.512 --> 00:31:29.812 There's a lot of churn sometimes. 00:31:30.232 --> 00:31:35.372 And so we miss things, things that cause problems, that we looked at the, 00:31:35.592 --> 00:31:39.072 the, the Terraform plan, but because it was so much, we, we, we missed the 00:31:39.092 --> 00:31:41.452 detail. Yeah. So, um- Yeah. 00:31:41.492 --> 00:31:45.212 Of course, the obvious thing is, yes, um, I want to put a little... 00:31:45.572 --> 00:31:47.312 You know, Atlantis has a really cool feature. 00:31:47.931 --> 00:31:52.992 Atlantis is, is, is basically an optimized CI/CD, uh, for, for 00:31:53.052 --> 00:31:55.812 Terraform, right? It understands- Yeah ...the concept of state. 00:31:56.092 --> 00:32:00.372 I was, I was gonna say that I'm in this b- a, a very large Terraform estate, 00:32:00.552 --> 00:32:03.572 and we use GitHub Actions, and it's a freaking nightmare. 00:32:03.872 --> 00:32:06.112 But- Yeah. So the, the difference between Atlantis and- Atlantis is way better 00:32:06.132 --> 00:32:09.052 ...and, the difference between Atlantis and the GitHub Actions is that 00:32:09.112 --> 00:32:14.492 Atlantis is aware of like keep, you know, a lot of the details that- Mm ...you 00:32:14.512 --> 00:32:16.752 know, a generic CI/CD doesn't have. 00:32:17.192 --> 00:32:20.832 Yeah. So what I'm trying to get to is that Atlantis is extensible, very much 00:32:20.892 --> 00:32:25.712 like GitHub workflows. You can extend the, the, the built-in workflow, the 00:32:25.732 --> 00:32:29.892 default workflow. The default workflow is plan, apply if approved. 00:32:30.592 --> 00:32:35.732 Um, sorry, plan, do a Conftest, like an OPA evaluation, apply if 00:32:35.792 --> 00:32:40.212 approved, and merge and auto merge the, the PR and, and discard the logs, uh, 00:32:40.292 --> 00:32:43.732 when it's done. You can extend that default one, and so you can add 00:32:44.132 --> 00:32:47.532 pre-workflow, you can do a bunch of things or post-workflow. 00:32:47.892 --> 00:32:49.892 So I extended the post-workflow. 00:32:50.072 --> 00:32:54.072 After the plan has ran, um, it will then look at every 00:32:55.192 --> 00:32:56.172 plan that's on disk. 00:32:56.192 --> 00:33:00.432 Out there. Yeah. And then it will pass all of that into an AI, um, agent. 00:33:01.072 --> 00:33:04.192 Um, I, I'm actually running a little agent, like a harness. 00:33:04.552 --> 00:33:06.412 So it's actually, it has tool calls. 00:33:06.832 --> 00:33:10.772 So it gets a little brief of like, "Hey, here's the, the, the, the pull 00:33:10.812 --> 00:33:15.332 request body. Here's, uh, all of the plan files on disk that with a little 00:33:15.392 --> 00:33:19.612 summary." And, and then it can go ahead and, and go and look and, and find out 00:33:19.652 --> 00:33:23.172 details about, about this plan and identify if there's any concern. 00:33:23.232 --> 00:33:26.472 So it compares the pull request intent against what's actually in the plan. 00:33:26.972 --> 00:33:28.552 And getting to the point 00:33:29.352 --> 00:33:33.772 - No, no, I'm with you ...but the point was, it was running for two or three 00:33:33.832 --> 00:33:36.812 days, and I wanted to know- What? 00:33:37.252 --> 00:33:39.552 Okay. The... It, it was running for two or... 00:33:39.592 --> 00:33:42.352 No, no, like it had been live for two or three days. 00:33:42.592 --> 00:33:45.092 Okay. What I'm saying, not is it keep running for two days. 00:33:45.512 --> 00:33:48.672 I, I put a, a, a two, uh, a turn limit of like fifty turns. 00:33:49.052 --> 00:33:51.732 So there's a limit on how many turns it can take before it posts a comment. 00:33:52.092 --> 00:33:53.212 So it's not gonna run for two days. 00:33:53.252 --> 00:33:54.512 It was live for two days. 00:33:54.952 --> 00:33:59.972 And it... I asked Sonnet, "Go through all of the pull requests over the last 00:34:00.012 --> 00:34:01.052 two days. Look at 00:34:02.132 --> 00:34:03.412 like how is it like... 00:34:03.712 --> 00:34:07.072 Is there any, any, um, sentiment that we can derive? 00:34:07.392 --> 00:34:08.772 Is there any impact?" Mm. 00:34:08.892 --> 00:34:10.492 And, and Sonnet was very, very eager. 00:34:10.532 --> 00:34:13.932 It's like the f- after two days, it was like, "Yes, there's this PR. 00:34:14.432 --> 00:34:19.492 Um, the, the AI comment reviewer was posted, and then we see, uh, right after 00:34:19.552 --> 00:34:23.372 another commit was pushed addressing some of the, uh, the findings." So that 00:34:23.492 --> 00:34:26.912 user saw the, saw the AI review and acted upon it. 00:34:27.192 --> 00:34:29.032 Mm. And the, and, and it has impact. 00:34:29.532 --> 00:34:31.932 Um, so, so I was very happy, right? 00:34:32.132 --> 00:34:33.192 I was, I was like, "Okay, cool. 00:34:33.252 --> 00:34:36.112 I'm gonna, you know, sh-show this to the rest of the team." Um, 00:34:37.232 --> 00:34:41.732 but what I'm, what I wanna come to then is then I made a change, and then I, I 00:34:41.772 --> 00:34:44.492 deployed it, and I let it run two days, and then I asked again Sonnet. 00:34:44.572 --> 00:34:47.192 I thought, "Great, you know, give me one of those beautiful reviews again," 00:34:47.252 --> 00:34:48.932 right? And it says, "No, no impact. 00:34:49.012 --> 00:34:51.852 Actually, people are completely ignoring it." And I was like, "What are you 00:34:51.912 --> 00:34:54.872 talking about? Like two days ago, you said it had a lot of impact." And I 00:34:54.932 --> 00:34:58.712 pointed it to that review, and I gave it some hints of how it did the review, 00:34:59.032 --> 00:35:02.332 and it says, "Oh, you're right." And then it do the review again, and it says 00:35:02.371 --> 00:35:06.232 like, "Oh, actually on this PR, people reacted and like, uh, updated the PR 00:35:06.272 --> 00:35:08.072 body after the review," and blah, blah, blah. 00:35:08.072 --> 00:35:10.452 And I was like, "How can I trust this?" You know? 00:35:10.852 --> 00:35:14.612 You see the problem and like- How are you me-measuring impact when people like 00:35:15.852 --> 00:35:19.992 upvote the AI re- AI, uh- It doesn't, it doesn't upvote. 00:35:20.032 --> 00:35:24.992 But it, it would say like, "At this time, the, the comment was posted. 00:35:25.412 --> 00:35:28.832 At this time, another com-commit was pushed." And it would say, "And the 00:35:28.872 --> 00:35:30.872 commit is addressing the findings from the comment. 00:35:31.212 --> 00:35:35.332 At this time, the PR body- Mm ...was updated." So Sonnet was doing all these 00:35:35.392 --> 00:35:36.212 queries on its own. 00:35:36.472 --> 00:35:37.692 Opus or Sonnet- Yeah ...I can't remember. 00:35:37.732 --> 00:35:41.572 But, but wait, I don't really understand this logic here because maybe, say 00:35:41.612 --> 00:35:44.492 people get better at creating commits first time and then, 00:35:45.352 --> 00:35:47.622 and then the reviews- Yeah, it doesn't matter ...become more and more- It 00:35:47.632 --> 00:35:51.272 doesn't matter. What I'm trying to say is, um, what you could, you could argue 00:35:51.312 --> 00:35:54.972 that I didn't do a very, uh, scientific approach to my analysis of, of the, of 00:35:55.032 --> 00:35:55.912 the, of the impact. 00:35:56.252 --> 00:35:58.312 And I didn't. I asked Opus, "Can you find anything? 00:35:58.412 --> 00:36:01.052 And here's a couple of PRs- Yeah ...and here's a couple of examples of what I 00:36:01.092 --> 00:36:04.772 consider impact." And then Opus came up with a couple of rules and ran those, 00:36:05.092 --> 00:36:08.832 identified all the pull requests where an AI comment had been posted, um, by 00:36:08.892 --> 00:36:11.752 filtering by the name and then, you know, came up with a couple of things. 00:36:12.272 --> 00:36:17.692 My- what I'm trying to get to is that AI can be very convincing at giving 00:36:17.752 --> 00:36:22.232 you a sentiment analysis over the last five days, um, but how much can you 00:36:22.292 --> 00:36:23.332 trust it? Like they- Yeah. 00:36:23.372 --> 00:36:24.572 What I was saying is you can... 00:36:24.932 --> 00:36:26.292 They have gotten a lot better at it, 00:36:27.232 --> 00:36:31.592 but I had some experience where I was like, "I don't know if I can still, you 00:36:31.632 --> 00:36:34.172 know, trust it." Well, actually, I think you, you, you're hitting on a, on an 00:36:34.232 --> 00:36:36.972 interesting point because like I, I usually 00:36:38.112 --> 00:36:41.312 clone out a project and I go like, "How can I improve this?" or something like 00:36:41.372 --> 00:36:45.452 this. And sometimes it, it finds real clangers. 00:36:46.012 --> 00:36:50.312 But obviously in some mature projects where things are pretty under control 00:36:51.136 --> 00:36:54.036 It says, "Oh, you really need to fix this re-- this problem. 00:36:54.196 --> 00:36:57.716 This is your number one priority," which is like, I don't know, a name change 00:36:57.776 --> 00:37:00.736 or something, which, which in reality has no impact. 00:37:01.136 --> 00:37:02.836 Like, it has zero impact. 00:37:03.196 --> 00:37:07.496 Like, I'm trying to get AI to find the most impactful changes, and it's 00:37:07.516 --> 00:37:08.236 suggesting me 00:37:09.056 --> 00:37:10.896 things that really have no impact. 00:37:11.756 --> 00:37:14.996 And, um, and, and that's one of the things that, uh, I feel, 00:37:16.516 --> 00:37:19.176 yeah, it, it becomes like a trust issue, but you have to like... 00:37:21.636 --> 00:37:22.396 You just have to... 00:37:22.456 --> 00:37:28.596 You have to, you have to set the benchmark for AI and, and realize that it's, like, 00:37:28.916 --> 00:37:34.236 just not doing the right thing or not, not having an impactful 00:37:34.316 --> 00:37:36.356 change or just wasting time, I suppose. 00:37:37.396 --> 00:37:38.396 I wanna share something else. 00:37:38.636 --> 00:37:40.576 So we were saying earlier, uh, you were saying that 00:37:41.976 --> 00:37:45.936 the output is not consistent, sometimes very good, sometimes very bad. 00:37:46.496 --> 00:37:50.416 Unrelated to what is on screen right now is a post that I saw on Reddit today, 00:37:50.836 --> 00:37:56.175 which is, uh, a guy who says exactly same prompt one week ago and today, 00:37:56.395 --> 00:37:57.516 and comparing the results. 00:37:57.716 --> 00:37:59.476 And, and today was a lot worse. 00:37:59.756 --> 00:38:04.416 And people are like, "Is this nerfing?" Is Anthropic, um, you know, really 00:38:05.016 --> 00:38:10.016 boosting the performance, um, of the model on, on the release day and then a 00:38:10.076 --> 00:38:14.656 little bit later to, to, to save cost is, like, nerfing it or, or maybe they 00:38:14.696 --> 00:38:19.616 call it quanti-quantis... What is it when they are quantizing it? 00:38:20.136 --> 00:38:22.376 Yeah. Anyway, um, we never know. 00:38:22.416 --> 00:38:25.896 And then people are saying maybe when you, when you select medium on the 00:38:25.956 --> 00:38:27.976 launch day, that's like a hundred effort. 00:38:28.296 --> 00:38:31.636 Yeah. But when you select medium on, on, like, two days later, uh, two weeks 00:38:31.676 --> 00:38:32.806 later- I definitely have that same- It's only fifty ... 00:38:32.806 --> 00:38:35.816 I have that same feeling that they're, they're messing around sometimes. 00:38:36.256 --> 00:38:40.396 Yeah. So, so that's where-- But I'm, I mean, I'm sti- I don't have that, 00:38:40.736 --> 00:38:45.436 like... Okay, avoid these, you know, avoid these frustrations and don't run the same 00:38:45.476 --> 00:38:46.156 task twice. 00:38:46.956 --> 00:38:50.376 Simple. Don't like this. And don't, don't look back. 00:38:50.696 --> 00:38:51.916 Just keep looking forward. 00:38:52.616 --> 00:38:53.696 Then you'll be a lot happier. 00:38:54.156 --> 00:38:55.616 Okay. But this is very interesting. 00:38:55.676 --> 00:38:59.135 Anthropic, uh, did this article, uh, I guess two days ago, right? 00:38:59.456 --> 00:39:02.736 Mm-hmm. Um, where they talk about how Mythos, 00:39:03.536 --> 00:39:08.096 um, you know, was running on a, on a exploitation benchmark. 00:39:08.196 --> 00:39:12.436 The, um, they have an, a benchmark to evaluate 00:39:13.416 --> 00:39:15.536 how the model can find exploits. 00:39:16.136 --> 00:39:17.916 And Mythos was scoring like, 00:39:18.796 --> 00:39:23.876 I don't know, um, there was-- They, they highlighted the score that Mythos got back in 00:39:23.876 --> 00:39:25.616 March. Wait, what, what does GLM mean again? 00:39:25.696 --> 00:39:26.616 Sorry, I should know this. 00:39:26.776 --> 00:39:29.256 GLM 5.3 is a model from Z.ai. 00:39:29.696 --> 00:39:31.516 Um, it's, it's one of the- Oh ... 00:39:31.516 --> 00:39:33.316 like- Oh, so it's benchmarking GLM- Yeah ... 00:39:33.316 --> 00:39:34.736 with, with Mythos. Okay. 00:39:35.536 --> 00:39:40.676 So the latest release of Z.ai is GLM 5.3, and it does better 00:39:40.756 --> 00:39:45.136 today the open weight model that you can run or that Z.ai offers on their 00:39:45.196 --> 00:39:50.296 inference engines, um, and it-- at a lot much cheaper cost than, than Mythos. 00:39:50.576 --> 00:39:51.796 Mythos is not even accessible. 00:39:52.096 --> 00:39:55.816 They say 5.3 today scores better than Mythos in March. 00:39:56.676 --> 00:40:00.776 Um, and, and they didn't release Mythos to the public because it was able to, 00:40:00.876 --> 00:40:05.606 um, bypass safeguard, like able to find exploits in- Ah ... 00:40:05.716 --> 00:40:06.756 um, in like 00:40:08.696 --> 00:40:10.556 software. They actually have... 00:40:11.856 --> 00:40:13.156 All the numbers are in here, you know. 00:40:13.736 --> 00:40:18.336 Um, what is the actual exploits that they're doing, like, uh, OSS, Fuzz and, 00:40:18.376 --> 00:40:19.936 and things like that. And, and 00:40:20.816 --> 00:40:22.076 what they're saying basically is, 00:40:23.076 --> 00:40:26.296 you know, it doesn't matter that we are protecting and not releasing Mythos 00:40:26.956 --> 00:40:29.596 to- Yeah. The other ones are catching up quickly. 00:40:30.496 --> 00:40:32.456 Yeah. Um- That's kinda scary. 00:40:33.496 --> 00:40:37.196 It is scary, but it also means that you know how sometimes when you ask Opus 00:40:37.236 --> 00:40:40.496 to do something and then it drops back because of safety guards, it's like, 00:40:40.776 --> 00:40:44.076 "Oh, you're trying to do something that triggered my safety guards, so don't 00:40:44.116 --> 00:40:47.106 be upset, but we're downgrading you - Yeah ... 00:40:47.256 --> 00:40:51.096 to a lower model." It means that I shouldn't be-- Like, if I have any, any 00:40:51.496 --> 00:40:56.076 cyber or security task, I should be using, uh, Z.ai, right? 00:40:56.256 --> 00:40:58.456 Because- Yeah. It's funny you should mention this because 00:40:59.516 --> 00:41:01.456 I hope this is, uh, a n- 00:41:02.436 --> 00:41:08.136 a hope this is non... I hope this is public information, but, but basically 00:41:09.056 --> 00:41:13.256 Claude has a security product that they want my client to use. 00:41:14.096 --> 00:41:15.256 Yeah. That's also what I said. 00:41:15.396 --> 00:41:17.606 Yeah. They're obviously selling a security product- Yeah ... 00:41:17.616 --> 00:41:18.336 for the premium. 00:41:19.436 --> 00:41:21.156 And now Z.ai is- And, and, and, 00:41:22.176 --> 00:41:24.696 um, you know how it is in big companies, like, "Oh, should we just turn it 00:41:24.796 --> 00:41:27.296 on?" But, like, we have a very large estate, 00:41:28.336 --> 00:41:29.096 and I'm thinking, 00:41:30.696 --> 00:41:37.236 um... And, and, and also we, we, we use Datadog a lot to basically manage, like, a single pane 00:41:37.276 --> 00:41:41.116 of view. And I think the way that the Claude security one works is that they 00:41:41.156 --> 00:41:44.296 have the, their findings in their own whatever panel. 00:41:44.776 --> 00:41:46.216 It's not integrated with Datadog. 00:41:47.136 --> 00:41:50.636 So I was, I was initially proposing that we do the triage 00:41:51.436 --> 00:41:56.536 and, uh, we use-- we basically just use raw Claude and feed into 00:41:56.576 --> 00:41:59.096 Datadog to build a b-- to build a, a picture. 00:41:59.136 --> 00:42:04.036 Like, when I say feed into Datadog, use Datadog MCPs to sort of mute false 00:42:04.096 --> 00:42:05.496 positives and things like this. 00:42:06.076 --> 00:42:07.476 So, so there was... 00:42:07.796 --> 00:42:10.816 What, what I was thinking of is just using MCP and Claude. 00:42:11.956 --> 00:42:14.516 But then Claude is coming across and saying, "If you just switch on this 00:42:14.576 --> 00:42:20.356 button, um, then A, we'll charge you a lot of money, and B, everything will be secure, 00:42:20.556 --> 00:42:23.136 or we find all the securities in their thing." 00:42:23.936 --> 00:42:27.376 Um, I, I don't have control-- I don't have, uh, access to their thing, but I 00:42:27.416 --> 00:42:28.136 just can't... You, you... 00:42:28.816 --> 00:42:32.396 The point I'm trying to make is that I'm just not, I'm not keen to use their, 00:42:32.576 --> 00:42:37.346 their thing really because I feel like, A, it will be expensive, and B, it 00:42:37.376 --> 00:42:40.356 will be disconnected from Datadog and the rest of our operations. 00:42:41.256 --> 00:42:43.296 But- Mm-hmm ... it'd be interesting... 00:42:43.516 --> 00:42:48.496 If anyone's using, uh, Claude Security, comment below and tell me if it's any 00:42:48.556 --> 00:42:54.356 good. Some sunken cost fallacy. 00:42:55.176 --> 00:42:56.696 It's like, has to be good. 00:42:57.536 --> 00:42:58.796 Um- For that price. 00:42:58.836 --> 00:42:59.896 Paying tons for it. 00:42:59.936 --> 00:43:05.216 Yeah. So there's another interesting announcement yesterday, which was, um, Kiro 00:43:05.296 --> 00:43:07.536 announced, uh, Workflows. And 00:43:08.476 --> 00:43:13.536 when I read this post, it really sounds like dynamic workflows from, 00:43:13.996 --> 00:43:20.276 um, from Claude Code. But it seems that they're doing a little bit more, um- Oh, 00:43:21.176 --> 00:43:25.936 yesterday a colleague showed me that he only uses, uh, Claude Code using 00:43:26.016 --> 00:43:27.796 agents, and I didn't even know about this. 00:43:27.956 --> 00:43:29.516 Did you... Do you use agents? 00:43:32.216 --> 00:43:36.536 I mean, every time I ask to, to do something, it spins up a workflow with like 00:43:36.716 --> 00:43:38.536 20 agents. So do I use agents? 00:43:38.796 --> 00:43:40.836 I guess. But, and, and, and do you use it with... 00:43:40.996 --> 00:43:42.716 I think there's like a Worktree option. 00:43:42.796 --> 00:43:44.986 So basically he uses- Yes ... 00:43:45.076 --> 00:43:47.836 Claude like I use with Orca, and I was like, "Oh, I didn't even know you could 00:43:47.896 --> 00:43:49.336 do that in, in Claude." 00:43:51.156 --> 00:43:52.986 Okay, fair. Ah. I'm a bit late to the game. 00:43:52.986 --> 00:43:55.836 Well, I-- Claude Code has an enter Worktree tool. 00:43:56.016 --> 00:43:59.256 So I can tell, for example, I have, I have Fable. 00:43:59.696 --> 00:44:05.036 Sometimes I, I run three different things and I say, "Okay, you spin up, enter 00:44:05.096 --> 00:44:10.536 a workflow, work on this pull request, um, and, you know, run..." 00:44:10.596 --> 00:44:12.136 It runs a dynamic workflow- Okay, so you do that all in Claude. 00:44:12.136 --> 00:44:14.756 Okay. Into a Worktree. I guess- Yeah. I never- Claude Code, yes. 00:44:14.776 --> 00:44:17.636 I never figured out how to do that, so basically I ended up doing it in Orca. 00:44:17.796 --> 00:44:19.976 Oh, there's also this one thing, um, 00:44:20.936 --> 00:44:22.516 that I did use, which you can do... 00:44:22.716 --> 00:44:25.096 And I sh- I talked to you about this before. 00:44:26.736 --> 00:44:27.896 This one, right? Yeah. 00:44:28.096 --> 00:44:29.836 Yeah. That, that's, that's what he showed me. 00:44:30.376 --> 00:44:34.136 So he just- Yeah ... he just bounces between all these agents and like, holy shoot. 00:44:34.236 --> 00:44:35.416 Yeah, I disabled- I didn't know about this. 00:44:35.856 --> 00:44:40.176 I disabled it. Um, first off, what I found very frustrating when I used it, 00:44:40.396 --> 00:44:43.276 you know, months ago, it's that 00:44:45.576 --> 00:44:49.936 I don't know which work that-- like which repository they're working in. 00:44:50.456 --> 00:44:51.516 Like this, 00:44:52.396 --> 00:44:55.636 when they announced it, they have like a little icon of the, the, the little 00:44:55.696 --> 00:44:59.636 crab or whatever it is, using a lasso to pull all of the different sessions 00:44:59.716 --> 00:45:01.656 into one- Mm. ... dashboard, which is this dashboard. 00:45:02.016 --> 00:45:04.956 Now, the problem I have is I don't know where these sessions are running. 00:45:05.376 --> 00:45:06.106 Like- True, true ... 00:45:06.156 --> 00:45:07.076 one session is running over here. 00:45:07.076 --> 00:45:08.556 I think Orca is a lot clearer about that. 00:45:09.076 --> 00:45:10.496 Yeah. Maybe they've improved it. 00:45:10.936 --> 00:45:13.496 Uh, and then another thing happened, and there was a bug. 00:45:13.956 --> 00:45:18.796 Um, normally, if you're inside a session, when you press the right arrow, you 00:45:18.856 --> 00:45:22.856 go back, you go back out to this like overview dashboard of all of the agents 00:45:22.956 --> 00:45:27.636 that are running. And what happened is I had like a massive dynamic workflow 00:45:27.676 --> 00:45:32.536 running for an hour or so, uh, and I pressed accidentally the arrow key, 00:45:33.136 --> 00:45:34.096 and when, when it... 00:45:34.156 --> 00:45:37.316 That happened, it suspended that session and killed the workflow that was 00:45:37.356 --> 00:45:40.736 running in it. And I was like, "Goddamn it." Like, it was a, a workflow that 00:45:40.776 --> 00:45:42.436 was running for hours, and it wasn't done. 00:45:42.756 --> 00:45:44.786 I was very, very pissed. That's annoying. But I actually- But they fixed it. 00:45:44.786 --> 00:45:47.536 I'm actually struggling to understand the relationship between this thing 00:45:47.576 --> 00:45:49.796 you're showing and dynamic workflows. 00:45:49.856 --> 00:45:54.736 Is that something that, it gets spawned from an agent view? 00:45:56.276 --> 00:46:00.536 I mean, every session here, like even, even a session, 00:46:01.516 --> 00:46:03.056 um, that's active- This is so confusing. 00:46:03.096 --> 00:46:04.856 The terminology is not great, is it? 00:46:05.976 --> 00:46:09.436 Every session, like even this session that's here right now, I can go into 00:46:09.476 --> 00:46:13.736 another shell, and I can resume, and then I will have that s- I'll have my 00:46:13.776 --> 00:46:17.656 terminal connected to that session in two different, uh, places. 00:46:17.976 --> 00:46:19.696 So under the hood, they're all running as agents. 00:46:20.016 --> 00:46:22.796 They're... You can all, uh, suspend them, you can all put them in the 00:46:22.836 --> 00:46:23.916 background, and you- Okay. 00:46:23.956 --> 00:46:27.316 I mean, maybe, maybe it gets activated when you run the agents command, but 00:46:27.356 --> 00:46:29.976 for me, each session is an agent on its own. 00:46:30.136 --> 00:46:31.816 Uh- So what does dynamic workflow mean? 00:46:31.876 --> 00:46:35.196 It, it sounds like another breakdown of an a-agent session. 00:46:35.256 --> 00:46:40.216 No. Dynamic workflow is, um, basically it gives 00:46:41.716 --> 00:46:44.556 the main agent the ability to define 00:46:45.356 --> 00:46:49.756 a predetermined step, um, workflow. 00:46:50.176 --> 00:46:53.876 So, well, we can look at some of the dynamic workflows that are running here. 00:46:54.536 --> 00:46:58.256 Um, is it this one? 00:46:58.596 --> 00:47:00.216 This one just, this one just landed. 00:47:00.296 --> 00:47:01.636 So if I go to Workflows, 00:47:03.536 --> 00:47:05.056 um, I have a couple of them here. 00:47:05.636 --> 00:47:08.956 My lord. And so what you can see... 00:47:09.096 --> 00:47:10.296 Not this one, that's not interesting. 00:47:10.336 --> 00:47:15.476 Let's do this one. Okay. So this one ran for one hour and nineteen minutes. 00:47:15.756 --> 00:47:17.256 It ran 17 agents. 00:47:17.816 --> 00:47:21.976 And you can see each one is like Sonnet 5.5 and the number of tokens. 00:47:22.476 --> 00:47:27.656 And it first implemented, this is in a monorepo with like multiple systems, and 00:47:27.716 --> 00:47:29.196 these systems are talking to each other. 00:47:29.536 --> 00:47:32.656 So it implemented the contracts, like the interfaces between the systems 00:47:32.716 --> 00:47:34.476 first, and it did those two in parallel. 00:47:34.716 --> 00:47:38.696 So it did the first contracts, and then it also did a couple of other things. 00:47:38.776 --> 00:47:40.936 There's another one which is super interesting. 00:47:40.976 --> 00:47:41.796 It's probably this one. 00:47:42.096 --> 00:47:43.896 'Cause this one, the contracts were really nice. 00:47:44.216 --> 00:47:48.516 Like it did 12A identity trust contracts, and it did 12, uh, 13A agent 00:47:48.576 --> 00:47:51.136 execution contracts, and then it updated the SDK. 00:47:52.316 --> 00:47:56.456 So just talking aloud here, it looks like dynamic workflows is more of a 00:47:56.576 --> 00:47:59.836 structured way of running your session or something like this. 00:48:00.056 --> 00:48:01.756 Yes, correct. Like, basically, 00:48:02.616 --> 00:48:06.936 you have a main agent like Fable or Opus 5.5 can be your main orchestration 00:48:07.036 --> 00:48:10.916 agent. Then what I usually tell it is like, "You're the main agent, your 00:48:10.956 --> 00:48:14.296 context is precious. You focus on the long-term goal, what we are trying to 00:48:14.336 --> 00:48:18.676 achieve." Mm. "You achieve that goal by breaking down into smaller like 00:48:18.736 --> 00:48:22.816 iterations." And then depending on the iteration size, it can either be a 00:48:22.856 --> 00:48:26.656 background agent that does that iterate, that one iteration, or it can be in a 00:48:26.696 --> 00:48:30.336 dynamic workflow, which will be a large, um, you know, number of things that 00:48:30.356 --> 00:48:31.656 need to be de-de-developed. 00:48:32.156 --> 00:48:33.696 And, and a dynamic workflow, 00:48:34.816 --> 00:48:38.716 the main agent can then decide what needs to be done. 00:48:38.856 --> 00:48:44.156 Like in this case, we are building identity and trust and agent execution 00:48:44.596 --> 00:48:48.956 and then, um, something else, um, as a platform, and we are defining the 00:48:48.996 --> 00:48:53.116 contract between the two systems, then we're implementing the, um, you know, 00:48:53.176 --> 00:48:55.996 the actual implementation behind the interface. 00:48:56.336 --> 00:48:59.626 Then we're integrating across those different, uh, implementations- Mm-hmm ... 00:48:59.656 --> 00:49:02.836 and then we go into a verify, repair, verify, repair, verify, repair. 00:49:02.876 --> 00:49:07.376 Yeah. And then normally it will do a comment linting, uh, step where it 00:49:07.416 --> 00:49:08.556 removes any comments without debate. 00:49:08.556 --> 00:49:11.096 I need to try this, but I th- I think dynamic workflows is disabled on my 00:49:11.156 --> 00:49:17.136 thing. Yeah. So dynamic workflows has been there since like May, and I talked about 00:49:17.176 --> 00:49:21.156 it several times because, uh, it got rid of half of my spec-driven development 00:49:21.576 --> 00:49:26.056 s- things. Because they're basically doing the whole breaking the problem 00:49:26.116 --> 00:49:29.516 down, breaking the plan down into tasks and phases, because that's what 00:49:29.556 --> 00:49:32.596 happening here, right? It took the plan of what do we need to build, and it 00:49:32.636 --> 00:49:35.536 broke us down into different phases, different tasks. 00:49:35.776 --> 00:49:39.036 Yeah. Each agent has a different role and, and in this case, you can see it 00:49:39.096 --> 00:49:42.876 says, "Okay, we're gonna run it, run with the Opus," and it, it, um, it 00:49:42.936 --> 00:49:46.096 verifies, and it then gives a review of like, um- Yeah. 00:49:46.216 --> 00:49:49.406 I, I, I mean, I, I see it more from a- Here are the open questions ... 00:49:49.436 --> 00:49:54.476 the perspective of a human, like as a human looking at this task that you just 00:49:54.536 --> 00:49:55.876 showed me you're working on. 00:49:57.296 --> 00:50:00.056 It has some bullet points or whatever 00:50:01.456 --> 00:50:05.356 to, to see how the problem was broken down and that, that alone, 00:50:06.856 --> 00:50:08.016 from a human perspective, 00:50:09.416 --> 00:50:10.176 has value. 00:50:11.176 --> 00:50:15.416 Never mind, never mind the other way around where the agents were like, 00:50:16.296 --> 00:50:19.176 uh, you know, put to work in an efficient way or something. 00:50:20.216 --> 00:50:23.476 Yeah. And then, and we also talked about like agent, like do you create agent 00:50:23.516 --> 00:50:27.596 profiles? Uh, do you say you're an expert security reviewer or not? 00:50:27.696 --> 00:50:33.956 Blah, blah, all that. Um, I don't because the dynamic workflow kind of on-demand creates a 00:50:34.016 --> 00:50:37.756 prompt that says, "Okay, you're the verifier here. 00:50:38.156 --> 00:50:39.376 Uh, this is what you're gonna get. 00:50:39.816 --> 00:50:43.296 Uh, we're in this thing, you need to focus on this." And, and so it basically, 00:50:43.356 --> 00:50:48.236 the dynamic workflow on-demand prepares an agent, selects the model, and, and 00:50:48.276 --> 00:50:49.356 then gives it a task. 00:50:49.656 --> 00:50:50.476 That's what your role is. 00:50:50.516 --> 00:50:54.996 Like you run a read-only, uh, review, um, you know, step of a dynamic 00:50:55.036 --> 00:50:59.916 workflow. And, and they have really improved the, um, capabilities here 00:50:59.956 --> 00:51:03.756 because these agents can, they can send messages to each other, they can do 00:51:04.176 --> 00:51:07.176 like many things. Like the main agent gets messages from the background 00:51:07.256 --> 00:51:10.176 agents. Um, I mean, dynamic workflow's been in there. 00:51:10.556 --> 00:51:14.136 Uh, I don't think a lot of people use them because they are so expensive when 00:51:14.176 --> 00:51:15.376 you, when you first start. 00:51:15.896 --> 00:51:18.476 Uh, when they first came out, people were like, "I ran one dynamic workflow 00:51:18.516 --> 00:51:22.116 and my five-hour session was gone," 'cause they spun up 20 agents and they 00:51:22.176 --> 00:51:23.076 used so many tokens. 00:51:23.576 --> 00:51:27.296 So, um, yeah, anyway, the reason we talk about dynamic workflows now is that 00:51:27.396 --> 00:51:32.376 Kiro, uh, just introduced it and, um, it has a lot of the same ideas. 00:51:32.476 --> 00:51:35.436 Again, it's a tool. Like there's a recipe. 00:51:35.816 --> 00:51:38.956 The, the main agent knows how to create those dynamic workflows. 00:51:39.376 --> 00:51:43.576 Some of the things that Kiro seems to do different is that they can update the 00:51:43.596 --> 00:51:48.196 dynamic workflow. But even that, like Claude, Claude did it here as well 00:51:48.236 --> 00:51:52.476 because he saw that the loop verify, loop verify was stuck on one thing and he 00:51:52.536 --> 00:51:56.276 paused the workflow, and then he relaunched it and then continued. 00:51:56.696 --> 00:51:59.576 Mm. So, so, so it is dynamically adjusting. 00:51:59.616 --> 00:52:02.496 Like even the main agents can see if the workflow is getting stuck and then 00:52:02.536 --> 00:52:04.856 like stops it and then relaunches it with different steps. 00:52:05.136 --> 00:52:06.836 Okay. Can we go on a little bit of a tangent here? 00:52:06.876 --> 00:52:09.396 'Cause I just, I'm just thinking back 00:52:10.316 --> 00:52:12.836 to reviews and pull requests and things like this. 00:52:13.296 --> 00:52:14.336 So one problem I have, 00:52:15.936 --> 00:52:18.556 uh, it goes back to the Terraform stuff you were t- mentioning before. 00:52:18.696 --> 00:52:23.436 One problem I have at work is that we, we, we make a Terraform change and 00:52:23.456 --> 00:52:26.356 there's a bazillion plans and they're very difficult to go through and 00:52:26.436 --> 00:52:27.136 summarize. 00:52:28.116 --> 00:52:30.816 Um, a- a- and even, it gets even more complex than that. 00:52:30.836 --> 00:52:35.956 Like, like sometimes a Terraform provider up- has an update and there's some 00:52:36.476 --> 00:52:40.056 s- there's some very subtle changes that we need to be aware of when, when we, 00:52:40.416 --> 00:52:42.936 when we approve these, these changes, right? 00:52:43.736 --> 00:52:48.936 Mm-hmm. So what I'm trying to say here is that what is the optimal flow for review? 00:52:48.976 --> 00:52:51.576 Because, because we have a lot of changes in Terraform. 00:52:51.716 --> 00:52:54.876 It's like, it's not like I want to be a s- I mean, I do wanna be a 00:52:54.916 --> 00:52:59.236 superpowered human, but at the same time, I kind of want to make sure that 00:52:59.296 --> 00:53:00.816 the, the CI 00:53:02.256 --> 00:53:06.956 had, has a dynamic workflow itself is, is, is how I'm tying it back to here. 00:53:07.276 --> 00:53:10.516 I don't know if you've ha- have any thoughts about, about, uh, 00:53:12.615 --> 00:53:18.136 orchestration to help. Okay. So what you're talking about here is like basically orchestration of the, of 00:53:18.196 --> 00:53:19.156 the, of the plans. 00:53:20.216 --> 00:53:24.096 I- Yeah ... I'm saying that like work, like workflows are- I mean, plans is 00:53:24.136 --> 00:53:28.736 just one example, but like it, I think a plan, plans is a good example because 00:53:28.776 --> 00:53:30.116 like what happens when you do a plan? 00:53:30.176 --> 00:53:31.856 It sh- shows you what the changes are. 00:53:32.356 --> 00:53:36.176 There could be lots of changes and then you need to synthesize that to, to 00:53:36.216 --> 00:53:37.036 make an approval. 00:53:41.276 --> 00:53:45.376 Okay. So you're saying I want, I want, uh, Terraform cross-state 00:53:45.436 --> 00:53:47.836 orchestration, but I want my agents to review it. 00:53:48.336 --> 00:53:50.736 Yeah. Uh, are you t- A- and basically 00:53:53.056 --> 00:53:55.236 have all that context because I, I know just... 00:53:55.736 --> 00:53:56.476 and like, and 00:53:58.716 --> 00:54:02.876 there is like copilot, uh, review of course and, and all that other stuff, but 00:54:02.976 --> 00:54:06.836 usually it only looks, it only looks at, um, 00:54:08.216 --> 00:54:09.236 the code change, right? 00:54:09.336 --> 00:54:13.276 It doesn't, it might, I think it might look at the actions, I'm not too sure. 00:54:14.376 --> 00:54:15.676 But y- you know what I mean? 00:54:15.716 --> 00:54:17.476 Like the, the, um, 00:54:18.556 --> 00:54:22.776 there's a lot going on and, and I feel like I need to be intentional, I hate 00:54:22.816 --> 00:54:27.756 that word. I need to be intentional but, about building a Terraform review 00:54:27.816 --> 00:54:33.316 bot that's able to know about pr- provider changes, just know about a 00:54:33.396 --> 00:54:38.696 whole bunch of context about the organization and then make these approvals, 00:54:39.896 --> 00:54:42.506 uh, ideally on behalf of a human. 00:54:42.506 --> 00:54:46.636 I think that's why I'm very excited about Stategraph because Stategraph, they 00:54:46.736 --> 00:54:52.236 are really giving agents very interesting context for, for this 00:54:52.276 --> 00:54:53.116 type of, um, 00:54:53.916 --> 00:54:56.336 um, activities. They're able to... 00:54:56.676 --> 00:54:58.996 Because they're not constrained by like, um- 00:54:59.896 --> 00:55:05.236 You know, having one state file per Terraform workspace, they are basically 00:55:05.276 --> 00:55:11.896 having a very large, um, database, and then they allow the agent to do a query 00:55:12.516 --> 00:55:13.736 to, to basically 00:55:15.136 --> 00:55:18.876 identify if I make a change in my networking layer, like what are all the 00:55:19.136 --> 00:55:21.436 resources- Mm. -that actively depend on this? 00:55:21.756 --> 00:55:24.196 What will be the impact across my whole infrastructure? 00:55:24.576 --> 00:55:25.536 So when they, they... 00:55:25.816 --> 00:55:29.896 When Stategraph, when they, uh, broke down the state file and recorded each 00:55:29.936 --> 00:55:35.016 and every resource as a, as a row in a database in Postgres, um, and then they 00:55:35.076 --> 00:55:39.936 added a whole bunch of like, uh, query capabilities, so the agents can run all 00:55:39.996 --> 00:55:42.916 kind of simulations and, and it's in their docs. 00:55:43.456 --> 00:55:48.016 Some of the capabilities that they make, um, possible for, for, um, 00:55:48.616 --> 00:55:50.896 AI are actually very, very exciting. 00:55:51.276 --> 00:55:54.996 Um, in this case, they are showing that you can, you know, generate a token 00:55:55.036 --> 00:55:57.756 that's only read, uh, that can only-- 00:55:58.756 --> 00:56:02.976 that the agent can use for, for doing a-an analysis, um, and it's short-lived. 00:56:03.496 --> 00:56:04.695 Um, but they can do a lot more. 00:56:04.736 --> 00:56:08.256 Like, they're really building this to be more agent-friendly to- Yeah. 00:56:08.256 --> 00:56:12.016 -to be able to query the, the database and, and do these type of things, these 00:56:12.136 --> 00:56:16.976 queries. And I wonder, I guess the, the general practice for secrets in a 00:56:17.036 --> 00:56:21.096 state file is just to not allow the agent to access just that part of the 00:56:21.156 --> 00:56:21.976 state file, right? 00:56:24.016 --> 00:56:26.856 So if you run- I wonder what- If you run- I wonder what guardrails there are 00:56:26.856 --> 00:56:32.216 there. If you run any binary like OpenTofu or Terraform, the agents do 00:56:32.256 --> 00:56:33.436 not see the secrets, right? 00:56:33.736 --> 00:56:36.656 Um, they are masking sensitive values. 00:56:37.216 --> 00:56:38.916 Um, if you say that- Oh, so sorry, I didn't... 00:56:39.096 --> 00:56:41.936 Can you just repeat that? I did, um, my brain- Why would you... 00:56:41.976 --> 00:56:43.516 Why would an agent read a state file? 00:56:44.376 --> 00:56:47.676 You would not. Like, you would run a Terraform binary or an OpenTofu binary, 00:56:47.716 --> 00:56:48.556 and it would mask secrets. 00:56:48.556 --> 00:56:53.236 Right. Right. Well- It would never add into the secret or into the context of the agent. 00:56:53.556 --> 00:56:54.436 Well, uh- 00:56:55.596 --> 00:56:56.336 No, no but. 00:56:59.856 --> 00:57:03.256 I'm, I'm just thinking that agents are really smart, and they might just wanna 00:57:03.316 --> 00:57:04.596 grab it for some reason. 00:57:08.215 --> 00:57:09.576 They should not have access to it. 00:57:10.155 --> 00:57:14.156 Um- But the trouble is they're usually running under a human developer's 00:57:14.916 --> 00:57:16.576 context, which has access. 00:57:18.596 --> 00:57:22.196 Why does... You know what I'm trying to say here. 00:57:22.916 --> 00:57:26.736 Staging and production, in our case, uh, the, they, the humans do not have 00:57:26.796 --> 00:57:30.816 access to it. Like, the only way that a Terraform plan can run in staging and 00:57:30.856 --> 00:57:34.356 production is through a pull request, and then it runs through an Atlantis 00:57:34.416 --> 00:57:38.736 instance that has the ability, yes, it has the ability to decrypt and read a 00:57:38.796 --> 00:57:41.376 token to, to apply the change into the Terraform. 00:57:41.896 --> 00:57:43.436 Um, but the agent, 00:57:44.856 --> 00:57:47.936 I mean, the, the, the instance is isolated. 00:57:48.016 --> 00:57:49.396 No human can run it. 00:57:49.736 --> 00:57:54.076 And then when I'm in-- for my AI reviewer, it actually runs under a separate, 00:57:54.636 --> 00:58:00.076 uh, Linux user isolated by the kernel, and it, um, it doesn't even get access 00:58:00.116 --> 00:58:02.236 to those tokens. Like they are, 00:58:03.256 --> 00:58:07.796 um- Okay. Yeah. I think, I, I think the problem is I probably have too many per... 00:58:08.876 --> 00:58:10.636 I, I have too much permissions. 00:58:10.736 --> 00:58:13.736 I need to, I need to basically- That's what, that's what Terraform... 00:58:13.776 --> 00:58:16.616 Okay, so a lot of people use Terraform- I need to switch my role or something. 00:58:17.316 --> 00:58:23.216 They think when they use Terraform that you don't, like, that, that everything is 00:58:23.256 --> 00:58:26.176 hunky-dory 'cause you can run Terraform Plan and Terraform Apply. 00:58:26.596 --> 00:58:30.956 But they never realized the complexity that you get into when you actually 00:58:30.996 --> 00:58:33.526 need to manage Terraform across a, a large team- Yeah. 00:58:33.526 --> 00:58:35.796 -a large enterprise, and you need to have- Well, it's crazy. 00:58:35.796 --> 00:58:37.976 -role-based access control. Especially with, with the networking, 'cause if 00:58:37.996 --> 00:58:39.696 you F it up, it's a nightmare. 00:58:40.336 --> 00:58:43.896 Yeah. So that's why there's this whole, you know, um, 00:58:45.036 --> 00:58:50.236 ecosystem of, of tools that are built around, um, orchestrating infrastructure 00:58:50.296 --> 00:58:52.056 as code that are obviously not free. 00:58:52.556 --> 00:58:57.916 Um, and that's why I built this website that, um, basically compares all of 00:58:58.056 --> 00:59:00.296 these- Yeah. -solutions. I added a new feature. 00:59:00.476 --> 00:59:01.236 You wanna see the new feature? 00:59:01.676 --> 00:59:02.716 You don't wanna see the new feature. 00:59:03.336 --> 00:59:05.676 No? Uh, first, first, actually a lot of new features. 00:59:05.916 --> 00:59:09.636 Okay. Um, because some of these, they say like they support Slack, but they 00:59:09.676 --> 00:59:11.656 don't support, uh, Microsoft Teams. 00:59:12.096 --> 00:59:17.056 So I added a little toggle because I don't think it's fair to, to grade this, 00:59:17.135 --> 00:59:17.936 um, these... 00:59:18.896 --> 00:59:21.195 If you're not even using Microsoft Teams, you don't care, right? 00:59:21.296 --> 00:59:25.216 So if you're using Slack, that changes the score because, um, some of these 00:59:25.236 --> 00:59:27.376 have- Mm. -better support for Slack than for MS Teams. 00:59:27.696 --> 00:59:31.316 If you're using Bitbucket, a lot of them fall down 'cause they don't have good 00:59:31.356 --> 00:59:33.456 support for Bitbucket. They mostly support GitHub. 00:59:33.956 --> 00:59:35.836 So s- Who, who, who is to judge about... 00:59:36.296 --> 00:59:38.436 I guess you, you need pull requests and... 00:59:38.536 --> 00:59:41.536 Oh, God. What are you talking about? 00:59:42.196 --> 00:59:42.916 Uh, I'm just wondering. 00:59:42.956 --> 00:59:45.916 I was just thinking out loud, like what, what do you mean by good support of 00:59:45.936 --> 00:59:49.016 Bitbucket? I guess y- it's more than just the Git protocol. 00:59:49.056 --> 00:59:52.176 It's- Right on their product page, they say, "We don't support Bitbucket." 00:59:52.296 --> 00:59:54.096 Because again, Terraform automation- I didn't know that. 00:59:54.096 --> 00:59:57.356 -and collaboration solutions are often tightly coupled to your v-version 00:59:57.396 --> 00:59:58.246 control system, right? Mm. 00:59:58.276 --> 00:59:59.376 They might have a GitHub app. 00:59:59.536 --> 01:00:01.136 Right. They need to react onto webhooks. 01:00:01.456 --> 01:00:02.916 They need to react onto pull requests. 01:00:03.236 --> 01:00:07.216 Yeah. So obviously, integration within the, the version control, um, you know, 01:00:07.516 --> 01:00:11.716 platform is very important and is not equal, not equal for every one. 01:00:12.416 --> 01:00:15.336 Okay, so the next feature that I added, sometimes I don't... 01:00:15.556 --> 01:00:20.456 I, I just wanna know who's the best at like, you know, RBAC and SO, uh, single 01:00:20.516 --> 01:00:24.336 sign-on, right? I don't care about like the overall weighted score. 01:00:24.416 --> 01:00:28.496 I just wanna sort it by this one particular criteria, who is the top performer 01:00:28.556 --> 01:00:31.056 in that area. And so here it's showing env0. 01:00:31.476 --> 01:00:34.776 These are all the ones that have, uh, like the highest score for this 01:00:34.816 --> 01:00:39.316 particular criteria. Um, and so that, that was the first, you know, little 01:00:39.996 --> 01:00:42.216 itch that I wanted to scratch so you can- I don't, I don't quite understand 01:00:42.296 --> 01:00:47.996 these little chiclets. So if you, what, if you go, if you go to the zero, does it tell you 01:00:48.036 --> 01:00:50.136 what the partic... I guess you have to hover over it. 01:00:50.876 --> 01:00:55.016 Yeah. Oh. I don't like the, 01:00:56.456 --> 01:00:57.316 the width of this- Yeah. 01:00:57.316 --> 01:00:59.496 -of hover over. Yeah, like normally- You don't have to hover over it. 01:00:59.536 --> 01:01:01.456 You c- if you go down, you can see the full thing. 01:01:01.756 --> 01:01:03.496 Yeah. Um, but still that's not ideal. 01:01:03.556 --> 01:01:05.316 Hold on, hold on, hold your horses. 01:01:05.516 --> 01:01:10.040 Hold it. Hold it. Um, you can also If you really wanna say, "Okay, I've made my selection. 01:01:10.340 --> 01:01:13.980 It looks like I really wanna s- I, you know, I'm most likely, if I look at 01:01:14.020 --> 01:01:18.820 pricing, um, based on my team, the number of seats that I need, the number of- 01:01:18.840 --> 01:01:21.940 Mm-hmm ... resources that exist in my Terraform state and so on, I realize 01:01:22.000 --> 01:01:24.680 that, you know, env0 is extremely expensive. 01:01:25.160 --> 01:01:27.080 It's like $11,000 per month. 01:01:27.520 --> 01:01:31.560 So I decided that I'm not gonna use env0 because it's a top performer here, 01:01:31.640 --> 01:01:35.520 but, uh, actually Spacelift and Stategraph are the two that I wanna compare. 01:01:35.920 --> 01:01:38.620 So I can switch this over to, um, 01:01:39.780 --> 01:01:43.220 Stategraph, and now I can actually really see what matters to me, which is 01:01:43.260 --> 01:01:46.780 like this is the two that I really wanna compare, and I can see which in each 01:01:46.820 --> 01:01:47.850 criteria- Right. Right ... 01:01:47.900 --> 01:01:52.400 why are, is this one considered like better in that criteria, um- I, I mean- 01:01:52.420 --> 01:01:53.140 ... compared to the other one. 01:01:53.420 --> 01:01:59.480 I find this m- like the, the current enterprise I'm working in, we use GitHub Actions. 01:01:59.540 --> 01:02:01.780 I feel like GitHub Actions needs to have its own 01:02:04.880 --> 01:02:09.040 uh, entry here just, just, just so that I can point out- No, it should not. 01:02:09.080 --> 01:02:10.020 Absolutely not ... to my colleagues who don't know better. 01:02:10.060 --> 01:02:13.000 I think I might add a row like GitHub Actions is absolutely not. 01:02:13.360 --> 01:02:16.280 Uh, like it's at the absolute bottom at criteria zero everywhere. 01:02:16.780 --> 01:02:19.900 Yeah. I mean, this is great when you're choosing, but like, like for me in, in 01:02:19.940 --> 01:02:24.080 this enterprise, it's like how do I, how do I get them to use Atlantis a- and 01:02:24.120 --> 01:02:29.380 have... Oh, so what you're saying is I want to show them s- uh, head-to-head, um, what 01:02:29.460 --> 01:02:30.740 GitHub Actions can do and what others- Like just, just to show you what it 01:02:30.780 --> 01:02:33.000 buys you, what the migration is 01:02:33.800 --> 01:02:34.800 between the two things. Okay. Okay. 01:02:36.580 --> 01:02:37.420 Because yeah- Yeah ... 01:02:37.460 --> 01:02:42.080 I'm in this world of, of just using bare bones GitHub, and it's... 01:02:42.960 --> 01:02:44.480 I do miss this tooling around 01:02:45.340 --> 01:02:49.240 'cause, 'cause I s- I, I see the pain that we have whenever we, we're, we're 01:02:49.280 --> 01:02:51.860 trying to rolling out a, a change across the ecosystem. 01:02:53.040 --> 01:02:55.480 Yeah. So this- Is Atlantis in h- in here? 01:02:55.540 --> 01:02:57.740 No. Yes, Atlantis is in here. 01:02:58.180 --> 01:02:59.680 It's like it's $90 01:03:00.600 --> 01:03:04.340 for my self-hosted instance because it only runs during business hours, which 01:03:04.380 --> 01:03:05.620 is also a security feature. 01:03:06.140 --> 01:03:07.930 Um, you can only- Interesting ... 01:03:07.960 --> 01:03:09.430 p- plan or apply during, uh- Interesting ... 01:03:09.460 --> 01:03:10.900 the weekdays. Yeah. I need- Um- 01:03:12.480 --> 01:03:16.340 I need, I need to fix things in this employ- in my client. 01:03:16.400 --> 01:03:17.319 I need to fix things, 01:03:18.220 --> 01:03:21.340 but I don't know if I have the, I don't, I don't know if I have the clout to, 01:03:21.400 --> 01:03:23.320 to get people to move to a better system. 01:03:24.540 --> 01:03:29.420 So I actually have Atlantis as, as a, as a construct, so it's like, um, 01:03:30.600 --> 01:03:33.380 here, but it's not, it's a private repository. 01:03:33.420 --> 01:03:35.100 Mm-hmm. But this is the actual 01:03:35.900 --> 01:03:37.830 AMI and Terraform- Okay ... 01:03:38.060 --> 01:03:39.700 module that I deploy and run. 01:03:40.060 --> 01:03:42.020 So this one costs me $90 a month. 01:03:42.400 --> 01:03:45.060 Um, I, I am also adding the AI reviewer in it. 01:03:45.320 --> 01:03:47.830 That means you're getting, um- Yeah ... 01:03:47.880 --> 01:03:48.740 you're basically getting 01:03:49.820 --> 01:03:52.560 all of the like top-notch stuff, but you can self-host it. 01:03:53.080 --> 01:03:56.760 Um, you get your, you know, you get like a ton out of it. 01:03:57.520 --> 01:04:02.060 This the, this is the, the important use case I feel because everyone's sort 01:04:02.080 --> 01:04:06.380 of bottlenecked by approvals and moving, uh, m-moving 01:04:07.460 --> 01:04:10.260 w-with, with speed. They're getting velocity, right? 01:04:10.780 --> 01:04:11.780 If, if you can 01:04:12.600 --> 01:04:14.840 focus, I'm sure you do, 01:04:16.460 --> 01:04:19.660 on what would basically unblock the pipeline, i.e. 01:04:19.920 --> 01:04:20.740 AI reviews, 01:04:22.200 --> 01:04:24.820 then I feel, uh, you know, what has... 01:04:25.020 --> 01:04:28.800 Which, which system out of all these things would you say on a, on a large 01:04:28.880 --> 01:04:31.360 estate would you get the highest velocity with? 01:04:31.520 --> 01:04:33.340 Like what has the best AI review 01:04:34.200 --> 01:04:37.920 approvals that, that are proven to, to work? 01:04:37.960 --> 01:04:42.300 Okay. So there's quite a few talks that, that been on engineering AI and so 01:04:42.340 --> 01:04:46.440 on, and they talk about AI, SDLC, and velocity and in-increasing velocity. 01:04:46.780 --> 01:04:50.300 And if you ta- if you look at those talks like Uber talking about it or 01:04:50.400 --> 01:04:54.240 others, the first thing that they say to achieve that velocity, they 01:04:55.180 --> 01:05:00.500 adopted monorepos. They spent a year rolling out trunk-based, uh, you know, 01:05:00.840 --> 01:05:04.530 promotion pipelines. Yeah. They do a, a ton- Getting pipelines maturity ... 01:05:04.560 --> 01:05:09.960 of a front platform engineering effort to then, uh, actually achieve 01:05:10.000 --> 01:05:14.520 high velocity. Mm. And, and, and I think somebody was like, "Oh, you know, AI, 01:05:14.640 --> 01:05:16.940 SDLC, dark factories, nobody's talking about it. 01:05:17.020 --> 01:05:17.840 You know, it was a hype. 01:05:17.880 --> 01:05:20.980 They tried it, it didn't work." But if you really look at the talks of the 01:05:21.020 --> 01:05:22.040 people talking about it, 01:05:22.980 --> 01:05:28.440 it's all of the platform and SRE best practices just applied, uh, with u- with 01:05:28.540 --> 01:05:29.680 agents in it, you know. 01:05:29.960 --> 01:05:34.140 'Cause if you have a solid platform engineering practice, you have proper, um, 01:05:34.400 --> 01:05:35.950 you know- Yeah. I, I, I mean- ... 01:05:35.950 --> 01:05:39.320 set up for CI/CD, you're gonna be faster with agents. 01:05:39.400 --> 01:05:41.180 Yeah. Listen, 01:05:42.160 --> 01:05:45.500 I'm, I'm interpreting what you're saying, like it's if, if you adopt one of 01:05:45.540 --> 01:05:49.860 these tacos, like, like if you, if, if you a- if your organization adopts 01:05:49.920 --> 01:05:55.080 Atlantis, you know, nevermind the AI stuff, you're, you're in a better position to do 01:05:55.140 --> 01:05:59.700 the work. Nah. You know how they always say like DevOps as a culture it's about the, 01:05:59.760 --> 01:06:01.040 it's not about the tools and so on. 01:06:01.080 --> 01:06:04.620 What I'm trying to say is if you adopt Atlantis, you're not gonna get it, you 01:06:04.660 --> 01:06:06.030 know, you're not gonna get an immediate- Okay ... 01:06:06.120 --> 01:06:10.300 improvement if you cannot adopt the culture and the abilities to, you know, 01:06:10.440 --> 01:06:14.520 rework your, the way of working to, to make, uh, you know, to, to get the 01:06:14.560 --> 01:06:15.940 benefit out of Atla-Atlantis. 01:06:15.980 --> 01:06:17.800 Like- Okay. So it's, it's- I mean, you could- Yeah. 01:06:17.900 --> 01:06:19.580 I am focusing on the tool, but... 01:06:20.720 --> 01:06:24.740 Yeah. I mean, you could, you could say that we both use a different definition 01:06:24.780 --> 01:06:28.540 of the word adopt because adopt, it's not just install the tool. 01:06:28.920 --> 01:06:31.200 It's adopt means we need to actually, 01:06:32.140 --> 01:06:37.220 um, put in the work of the, of the teams to be able to work with that tool and 01:06:37.260 --> 01:06:40.800 put in place the practices so that the tool can do the work, right? 01:06:41.320 --> 01:06:45.260 Um, so adopting the tool is, is can, can take a lot of effort. 01:06:45.860 --> 01:06:47.200 Uh, and, and only then- 01:06:48.220 --> 01:06:49.840 A little problem ... you can also have agents in the loop. 01:06:50.280 --> 01:06:53.260 I, I, I need to run in a bit, but I just wanted to explain one problem at 01:06:53.300 --> 01:06:55.160 work. I think you might find it funny. 01:06:55.240 --> 01:06:58.520 So we have consolidated a few projects into a monorepo 01:06:59.320 --> 01:07:04.660 around security engineering. But the problem is, is that now our, our security 01:07:04.720 --> 01:07:09.780 gate, which is quite, uh, pedantic, fails because like there's, there's a 01:07:09.820 --> 01:07:15.060 dependency in, you know, some bar project that's nothing to do with my project 01:07:16.188 --> 01:07:17.568 Failing my pipeline. 01:07:19.128 --> 01:07:24.608 So ba- so, so I had to make a pull request for Trivy to ignore someone 01:07:24.668 --> 01:07:29.648 else's CVE so that I could un-- So I could, uh, push out a change in, in my 01:07:29.688 --> 01:07:30.668 part of the monorepo. 01:07:31.668 --> 01:07:34.738 You know what I mean? So the monorepo- Mm-hmm ... 01:07:34.928 --> 01:07:38.048 the security tools don't seem to be aware that there's, like, 01:07:38.908 --> 01:07:41.388 different code owners in, in this project. 01:07:42.228 --> 01:07:46.108 And, you know, my, my, my role- No, the, the security tool doesn't seem to be aware 01:07:46.148 --> 01:07:50.958 that your dependency or declared dependency, um, is- Nothing ... 01:07:50.968 --> 01:07:51.728 like you're not really... 01:07:52.268 --> 01:07:56.068 Like, I mean, from what you just told me, I can understand it either way. 01:07:56.408 --> 01:08:00.408 Either one way is your project is using a base AMI that has a CVE, and that 01:08:00.428 --> 01:08:02.548 needs to be patched, but that you're not the owner of it. 01:08:02.788 --> 01:08:05.868 Yeah. In that case, it's good that Trivy is blocking you or not because you 01:08:05.908 --> 01:08:10.348 can't, you can't rebuild your, your, your project because the base AMI is not 01:08:10.408 --> 01:08:12.478 patched. Uh, so you can't redeploy even though it's running- Yeah, but, um, 01:08:12.478 --> 01:08:13.648 but to be honest, this, this, this... 01:08:14.228 --> 01:08:18.568 Okay, it's not an AMI, but this AMI has nothing to do with my delivery, by the 01:08:18.588 --> 01:08:22.388 way. So, so you're saying you're not even building on top of that AMI, it's 01:08:22.448 --> 01:08:23.528 just monorepo with like- Yeah, exactly. 01:08:23.528 --> 01:08:24.968 Exactly. It's, like, completely separate, 01:08:25.768 --> 01:08:26.808 but it's in the monorepo, 01:08:28.088 --> 01:08:29.748 and I'm blocked because Tri... 01:08:30.228 --> 01:08:34.408 And, and the way that Trivy does is, does is ignores, you just, like, put the 01:08:34.468 --> 01:08:39.108 CVE number. You don't say, like, "Ignore this path." Where. 01:08:39.228 --> 01:08:40.528 Ah. Oh, no, that sucks. 01:08:40.628 --> 01:08:43.348 Like, you're just saying, "We are accepting the risk of this CVE." You're not 01:08:43.408 --> 01:08:45.088 saying where. Hmm. Exactly. 01:08:45.088 --> 01:08:48.468 And, and it- SonarCloud and, and, and Aikido Security, you have to specify- 01:08:48.508 --> 01:08:50.048 Yeah ... in this file, ignore this, this CVE. 01:08:50.068 --> 01:08:54.728 Exactly. The, the... N- no security tool that I've seen, at least the, the one 01:08:54.768 --> 01:08:58.908 that I'm using this, in this client, uh, seem to be aware of the code owners. 01:08:59.928 --> 01:09:02.608 Yeah. So basically, it's all or nothing, and then it, and then everyone's 01:09:02.648 --> 01:09:05.868 getting blocked on each other because there's a few TypeScript projects 01:09:05.928 --> 01:09:10.008 inside, inside here with AWS CDK and all that sort of stuff, and there's, 01:09:10.048 --> 01:09:12.448 there's things ringing off every millisecond. 01:09:12.968 --> 01:09:17.288 But, but, but Kai, open weight models can hack your systems. 01:09:17.608 --> 01:09:19.808 You must... Security is a top priority. 01:09:19.848 --> 01:09:24.187 You can... Uh, it's like, you know, the red bell, the, the p- the, the- Yeah, 01:09:24.368 --> 01:09:25.828 but- ... the conveyor belt stopped. 01:09:26.108 --> 01:09:27.448 It's the emergency brake. 01:09:27.488 --> 01:09:29.898 It's about prioritizing- You have to go patch that system ... 01:09:29.968 --> 01:09:34.068 these gates. I mean, these gates are, are really painful because sometimes, 01:09:35.248 --> 01:09:39.148 sometimes you want to do a gradual incremental change, right? 01:09:39.688 --> 01:09:44.968 And the gates n- never seem to re- realize that what you have out there is, is 01:09:45.008 --> 01:09:45.948 already a problem. 01:09:47.668 --> 01:09:50.568 And you just wanna get the flow to fix things. 01:09:51.188 --> 01:09:52.228 Have you read this article? 01:09:53.168 --> 01:09:58.648 No. February 2020, uh, 2006, not even 2026. 01:10:00.408 --> 01:10:02.528 Continuous integration on a dollar a day. 01:10:03.128 --> 01:10:07.488 The reason I'm talking about the emergency brake, um, because in this thing, 01:10:07.568 --> 01:10:11.488 it's very simple. You have one computer that is used to deploy, and you get a 01:10:11.528 --> 01:10:16.468 rubber chicken, and then everybody can do what they want, but you automate 01:10:16.528 --> 01:10:21.908 your build. But if the build fails, if the build is red, you pull the 01:10:21.968 --> 01:10:26.348 emergency brake, everybody drops what they're doing, and you fix the CI/CD. 01:10:27.608 --> 01:10:29.188 And that's how you do continuous delivery. 01:10:29.628 --> 01:10:31.648 Um, I thought it was an interesting, uh, article. 01:10:31.968 --> 01:10:33.538 Uh, I actually started doing, uh- Well- ... 01:10:33.688 --> 01:10:34.678 DevOps in 2006, so ... 01:10:34.708 --> 01:10:37.048 it ma- it makes sense when you have a good ownership. 01:10:37.088 --> 01:10:37.848 But right now, 01:10:38.928 --> 01:10:43.008 since we've followed your good advice, Vincent, and gone monorepo, it's just 01:10:43.068 --> 01:10:44.328 a... It's insane now. 01:10:44.988 --> 01:10:49.408 Oh, so you were, you were complaining about monorepos, but what I hear is you 01:10:49.488 --> 01:10:51.508 adopted the monorepo without proper tooling. 01:10:54.128 --> 01:10:55.228 And it's not about the tools, 01:10:56.228 --> 01:10:57.448 but it is about the tools. 01:11:00.168 --> 01:11:01.028 Okay, let's end it here. 01:11:01.168 --> 01:11:02.608 There's a few things I need to catch up on. 01:11:02.968 --> 01:11:03.948 Otherwise, um- 01:11:04.888 --> 01:11:08.788 No, but the fact that your security system cannot bu- Like, um, it just sounds 01:11:08.848 --> 01:11:10.448 like your monorepo build is horrible. 01:11:10.748 --> 01:11:11.908 Like, that's why I, I use Turborepo. 01:11:11.908 --> 01:11:12.928 It's not about the tools, Vincent. 01:11:12.968 --> 01:11:13.968 It's about the culture. 01:11:15.988 --> 01:11:17.208 How many times do I have to tell you? 01:11:19.408 --> 01:11:20.688 No tools, only culture. 01:11:20.948 --> 01:11:22.288 Yeah, it- Oh, and, um, 01:11:23.348 --> 01:11:25.628 final note, we're going to Config Management Camp. 01:11:25.688 --> 01:11:29.768 I don't know if it's worth telling our audience be- because your talk got 01:11:29.848 --> 01:11:31.148 accepted and mine hasn't. 01:11:31.248 --> 01:11:33.827 No, mine is only an Ignite, I just realized. 01:11:33.908 --> 01:11:35.408 Well, it's still a talk, man. 01:11:35.488 --> 01:11:38.488 Jeez. No, they... Who, who goes to Ignites? 01:11:39.608 --> 01:11:41.678 I really- Everybody goes for, for lunch during the Ignite ... 01:11:41.678 --> 01:11:42.968 I like the lightning talks. I'm a lightning talk fan. 01:11:43.868 --> 01:11:45.068 I'm all for Ignite. 01:11:45.588 --> 01:11:47.648 Yeah. Um, but you got yours accepted. 01:11:47.708 --> 01:11:49.188 Mine bloody well didn't. 01:11:49.268 --> 01:11:50.648 Did you even submit an Ignite? 01:11:51.168 --> 01:11:57.788 Yeah, I did. Oh, okay. So- I guess, I guess my Ignite and my, my, my, like, 40-minute talk 01:11:57.828 --> 01:12:02.188 were very similar, and I guess they decided that's not worth 40 minutes. 01:12:02.348 --> 01:12:03.168 You get five. 01:12:04.068 --> 01:12:05.088 And I felt upset. 01:12:06.408 --> 01:12:07.318 Well, at least you- And I realized- ... 01:12:07.628 --> 01:12:10.178 at least you got, at least you got something, um, accepted ... 01:12:10.228 --> 01:12:12.828 I was more upset about the fact that they were right. 01:12:13.268 --> 01:12:14.038 And I was like- ... 01:12:14.038 --> 01:12:15.188 "Goddamn it, they're probably right. 01:12:15.228 --> 01:12:17.188 That's probably not worth 40 minutes." Okay. 01:12:17.268 --> 01:12:18.628 So you... Did you buy your tickets? 01:12:18.688 --> 01:12:20.218 We're going, right? I'm gonna buy my Eurostar tickets. 01:12:20.228 --> 01:12:20.888 Yeah, I have to... 01:12:21.428 --> 01:12:23.388 Uh, I, I have to go to FOSDEM as well. 01:12:24.148 --> 01:12:25.488 I wanna go to- Oh, really? 01:12:25.528 --> 01:12:26.908 You, you have some concerns about that. 01:12:26.928 --> 01:12:31.448 Um- I'm not too sure about going to FOSDEM, uh, because, just because I might 01:12:31.488 --> 01:12:32.308 not have time. Just because... 01:12:32.868 --> 01:12:35.488 You, we talked about it last time. But first, second, and third, we're gonna be in Ghent. 01:12:35.548 --> 01:12:36.888 So if anyone's listening- 01:12:38.728 --> 01:12:41.248 You mentioned this last time, and I said, "Are you sure you wanna know? 01:12:41.308 --> 01:12:42.708 What if somebody has a bone to pick?" 01:12:45.628 --> 01:12:47.358 You said Opus was good. 01:12:47.388 --> 01:12:48.168 I'll, I'll, I'll, like... 01:12:48.588 --> 01:12:50.468 No, no, no, it was Vincent that said it. 01:12:52.528 --> 01:12:54.768 You're misremembering. It was all about the- No, we're, we're, we're reading 01:12:54.808 --> 01:12:55.828 this script from Astra. 01:12:55.848 --> 01:12:57.488 It's all about the culture. Astra wrote this script, not us. 01:12:59.448 --> 01:13:01.188 Oh, yeah. Okay. See you. 01:13:01.228 --> 01:13:02.068 Bye. All right.