Episode 47: Agents talking to agents

Can you hear me? Yeah. Hey, how are you, man? You had a whole trip to Kuala Lumpur, was it? Yeah, good. Yeah, Kuala Lumpur was good. Back? It was really good. Right. There was haze, but it cleared. But it was just nice to be outside of England. You know, some geo... What? You were there for the one-on-ones with MK, right? so there's this street basket guy on Instagram called MK and he plays

in America and South America and he was on a tour in Asia and he posts Instagram videos of one-on-one basketball where they are kind of you know punking on each other and being super I don't know disrespect yeah very aggressive and pushing the boundaries and aggressive I trying to tilt your opponent. There was a whole drama happening in Malaysia when he was there because some of the local people felt that he was too disrespectful against some of the players.

Some other guy showed up and there were fights and stuff. You were there for the MK. That was last week or was it two weeks ago? Oh yeah, I was there for them. I mean, I have no interest in basketball. I was interested in the food, just catching up with people. Yeah. And I observed a couple of things that I found interesting. I was in Malaysia like two years ago, and then coming this time, I noticed electric vehicles. like two years ago I don't remember any electric vehicles and now two years later

there's this a ton of them on the road from brands that I've never heard of like Zeekr and Leapmotor and Great Wall and all sorts of crazy brands I've never heard of. BYD Do you know BYD? It's Chinese, I think. I guess it's the same. Oh yeah there's plenty of BYD but we have BYD here in Okay. the UK. We have J.Crew here in the UK. So I wasn't sort of counting things I knew, I was counting things like I've never seen before. And there's a lot of EV change.

It must be the same in Vietnam, right? Quite a lot of BYD. And of course, we have VinFast, which is a famous large car manufacturer in Vietnam and in Vietnam, the Vingroup, actually Reddit and Facebook and all there. Vingroup, they have a lot of properties. It's a big family or it's a big chain of businesses. Alright. There's Vingroup also expanded to America. And there's a lot of chatter about how they Oh yeah. ...chatter

look very successful on paper, but it's actually like, you know, cousins billing each other to, you know, prop up the revenue and things like that. Yeah. ...buffter. There's some huge Vietnamese companies. The IT company always took my eye because it advertises in the Premier League, is it FTP or something like that? FPT. Yeah, FPT is big. I think in terms of, so I used to organize cloud native meetups in Ho Chi Minh City, and that was focused on Kubernetes, you know, back then, and then later, I mean, of course, Grafana and observability.

that I did in Singapore. And then when I came back after COVID, I created, there was no Ho Chi Minh City chapter, so I created one. And initially there were like 80 people showed up. It was very fun, you know, and we started light with Kubernetes. 80? 80? 80? At 8-0, yeah, the first meetup was like, you know, very interesting. very We had a couple of people that were very interested to promote this and host were in their office, which is a digital bank in Vietnam.

And they hosted, like, you know, I had only choice between two offices. So it's either Zylk of Vietnam. I think they helped organize a little bit. And then there was Cake Bank. But then FPT and all, like, they ran the OpenInfra. When they were heard of this campaign, they were Like, I don't know if you know OpenInfra, but it's like an AWS for your data center type of solution, openinfra.org. Like a Vietnamese AWS or something.

No, no, OpenInfra Foundation is a large... I guess it's like if you want to run EC2 on VMware or something like that, it's basically your cloud. It's OpenStack. Sorry, that's the one I was looking for. You know OpenStack? oh yeah I'm just like yeah I've heard of that yeah so so OpenStack which is like you know because we talk about FPT so there was this yeah yeah OpenInfra Foundation and there's this OpenStack that was very popular and when I did cloud native they were quite aggressive initially in approaching like uh you know it should be open

infra why are you doing cloud native and then after one year um they were there on the first meetups but after one year it became like I tried we had a chat where we were organizing together and every time I asked like is there anyone anyone want to speak like there's a Kubernetes 10-year anniversary. Well, long story short, I felt that the OpenInfra Foundation was a bit hostile, and they were mainly backed by FPT and other local data centers. yeah, it just made me step out of the whole thing. Well,

that whole, oh my gosh, what were you saying again? That Red Hat-driven, what was that initiative called for the open infrastructure again? They had their own operating system and everything. Red Hat opening track? Oh, a Flatcar? No. No, no. You mentioned it earlier. OpenShift, no. Governments and IBM are backing it. OpenStack. Yeah, OpenStack. I mean, there was definitely an OpenStack cabal. Yeah.

I don't even know if it's still going. I think it's given up, right? It's like just Kubernetes nowadays, right? Yeah, so they kind of pivoted. They felt like that was their playground. and some of them I mean some of them from the north they are quite I don't know how to say but they're a bit more I don't know against foreigners I would say

I really felt that that was the case that's definitely how they made me feel as I was trying Well, it's very typical in Asia. to organize events so I said I don't need this Actually, that's my first memory of Asia in a way. Like when I was working in Korea, like if it's a foreign company, it's like, you know, you're not gonna fly here, buddy. yeah it's sad because it was all in your Ho Chi Minh City is not like that like in Ho Chi It's very protectionist. Minh City it's very um open and very active like when I when I started doing like the first time I came into this meetup scene there were germans and others

organizing JavaScript meetups, Ruby on Rails meetups. I know in Singapore that was super active, even much longer than in Ho Chi Minh City. Docker kind of really promoted that heavily and then a lot of companies took note of how effective that was, of how big Docker grew by really pushing out swag and support and advertisement on a global level. Oh, yeah, we all have a Docker shirt. clothes on my back thanks to Docker oh yeah

oh yeah I'm still wearing like the third three-year anniversary Docker shirt t-shirt that they they sent um that people when I go on calls go like oh you you're wearing a interesting shirt and then they see it every week and they're like you still still wearing that shirt yeah no anyway like I think Docker showed a lot of large enterprises that this really worked and then Google they also started but then they didn't want to have another platform on it, right? They had their own dev communities that they organized

they kind of split it up it's like Netflix made movies popular and then you get Disney Plus you get Prime and then everything is messed up because they want to own the platform Prime, and it's all split and splintered a bit of politics. I just remembered there was a Vietnamese guy that I met while I was in AWS re:Invent called An Ho Viet. The AWS Vietnam user group and his company is called Awesome. Yeah.

I don't know. Does that ring a bell? Yeah, so again, I am an AWS Community Builder. I talk at AWS Community Days. I reached out numerous times in Vietnam, but when I reached out, there was this conva, and then they say, oh, you know what? AWS in Vietnam, it's really just selling to partners community things not like in Singapore like yeah like they're Oh, so it's not like developer aimed, yeah. Yeah, it's like

they're not running like uh hey you know we're hosting this um engineer who works at the a... telecom company or and and he built something interesting uh on AgentCore or this is how they they they adopt certain patterns cloud patterns and you know this very cool like ...and you know, it's very cool... you know as as a person as an individual um maybe I'm at a job and to play with something, I just want to learn or I want to share my own experience playing with it,

that you can go to even at AWS Singapore. They organize and they host these type of meetups, which is really fun to go to. Yeah, yeah. I mean, you know I was the AWS user group lead for a number of years. Oh, I didn't... Yeah. And I handed it over to Steve. Yes, Steve Teo, right? and the vibe was always to be developer focused and to have tech talks I'm not even sure so you're saying the Vietnamese one was a bit more a bit more like noob

oriented and a bit more a bit more sales oriented It's not noob, it's more like sales. It's more like partners of AWS selling, reselling AWS or some large conferences. All right, that's a bit sad. I mean, that's my experience. And I've reached out many times, even with people from Yeah, yeah. Singapore, like Safras, who was Google Cloud first and then became AWS community lead across the region, Safras. He just said that he just left AWS this Never heard of it.

week, I think. But it's very sad because, I mean, I feel that it's I mean, it's not that sad. so much fun to share. I guess we're just being picky because we're like elite coders and we want, or coders. infrastructure builders and we're looking to, we're on the frontier, right? So we're a little bit picky about what we like to attend. I look at Philippines and Kuala Lumpur because they're close to Singapore so even when I went to community day in Singapore people from the Philippines community groups were there people

from the Kuala Lumpur community groups were there I talked with some guys they said yeah come to Kuala Lumpur they got me on the community day there as well and I followed them on LinkedIn and we're friends and I see the activity and I see how engaging it is. But in Ho Chi Minh City, it's not happening. And even when I try to approach people to get something going, usually the response is like, we don't do that here. We don't, like the people or even the AWS or people in Ho Chi Minh are in the air. Actually, when I was in Malaysia, I talked about City are more focused on selling to business.

community meetups, you know, what's happening with community meetups post-COVID. Hmm. the verdict that I got was that most of them are dead. but there are some diamonds in the rough sort of thing. seem to be yeah Like the one that is pretty active is the Rust meetup in Kuala Lumpur. oh that's awesome yeah that's a great And they meet every couple of weeks. I actually know the organizer, Ivan. And that sounds good. I mean, I would probably in desperation call myself a Rust developer

and attend those meetings because it's got some cool people there talking about cool things. it's it's not about even being a Rust developer it's it's like you know students who want to It's not about learn um you know some you know want to attend and see some interesting projects and and Yeah. a bit weird. that's the same in the community in in kl It is a little bit elitist that it's called Rust because it should be a little bit... There was back in the day a very good meetup in KL called WebCamp KL or something like that, hosted at the Mindvalley offices.

and that was really great because we got all sorts of people there and it was very it was very vibey and it was a nice venue ah so I heard Da Nang in Vietnam is more like, like when I'm in Ho Chi Minh City and there's like a hacker hackathon or something, I was at some hackathon and then people came up came down from Da Nang and there were like foreigners and Da Nang also is famous with like expats and like for people doing nomad lifestyle and I heard several people said come to Da Nang and and I

think there's like a bit of a hacker culture there of you know bootstrapping your startup and a I think and talking and sharing and meeting and and you know I think it lives in in Da Nang and I hope to revive it in Ho Chi Minh City this time at the time I was working remote for the last two years but this time I have an office and one of no you're in you're in some suburb out of Ho Chi Minh, right? For some reason I thought you were in Da Nang for a while I'm in in Ho Chi Minh City and I'm not like but at least now there I'm working for a local

company that has an office and that also says hey we really want to bring back the meetups like the CTO specifically told me like he kind of misses that and he wish that we could do it more and I've been here now like four months we haven't set anything up because I've been crazy with building something, but it's definitely something that I want to start sharing. yeah I think yeah I think meetups are quite hard like the thing that annoys me at my employer They try to do, we do these summits, we do these like get-togethers, these squad days or something like that.

And I think to myself that like, it just could be better, you know, like, I don't know what I'm expecting. I guess my reference point is a bit spoiled because in Singapore, A, it was easy to get to the venue. B, it was a really nice venue. I think in the end we were using the AWS offices, which is like, you know, completely downtown, great bars downstairs, yeah great food. yeah good to catch up after right and talk about all kinds takeaways yeah Yeah, exactly. Easier to get home.

Now it's such a schlep anyway. Oh, more stories from Malaysia. The interesting thing is that my friend in Malaysia was using GLM 5.3 because I think there was some sort of Malaysian investment to... Is it Z.ai? yeah I'm using it as well it's it's like I'm using it for the workloads yeah and he seemed pretty happy with 5.3. Is there... I mean even what's this uh virtual card company

called that's quite famous that a lot of people have like you to issue a digital card with a mean like Visa credit card type. Very popular one. or MasterCard? What the hell? Amex? Yeah, but a startup. Revolut? Yeah, Revolut. Revolut, they just had a blog post on their token, how they managed to cut token costs. And they are also heavily in on GLM. use. Oh yeah, there was an AI engineer talk about that. Yeah. Yeah, so do you have a Z.ai subscription or do you just dabble with OpenRouter?

I have DeepSeek and Z.ai, and I have a wallet. I have Z.ai I don't have a subscription, so I'll just top it up. The reason being, the Anthropic models are extremely expensive, and the enterprise Yeah. budget is kind of like cut. So the alternative is a LiteLLM proxy provided by a platform team that only makes available certain open-weight models. And I think they were evaluating GLM or DeepSeek Pro for

Flash or whatever. And at the time when I was building the platform that I'm building, I just said, because it wasn't available yet, so I set up my Z.ai budget, like my wallet, and I run all of my testing on 5.3. So, yeah, I mean, it's just, yeah, The backup. I have a full sandbox environment, you know, like everything mirrored from that is like, you know, that API endpoints and wallet

wallet. Ivan said it, but I thought it was quite interesting. and version control. He just flips something over in his Claude config, just changes the backend environment. And then he's using, so he uses Claude code with GLM 5.3. I was like, wow, that's interesting. Yeah. You can just, I had a friend, because with SpecLedger, one of the guys building SpecLedger with me and the other guy, he was running his Claude Code against his own endpoints, which at the time I think was DeepSeek.

and um And because of that, when we build the SpecLedger CLI as like it would onboard you, it would set up the repo with like some boilerplate prompts and like skills for SpecLedger itself so they could interact with the API and, you know, log in. And then it would launch a Claude Code or whatever, you know, coding harness that you wanted to use. But honestly, we only really try, like we're using Claude Code ourself. I mean, he would require that launching.

So that user config, he would need it to be supporting search endpoints like the environment variables required because for his setup he needed to launch Claude Code with those configuration settings. So he built that aspect of the launch or user configuration because the user configuration was supposed to be like you can choose Claude Code or you can choose Codex or Gemini CLI or whichever harness that you want and we're just delivering the skills and everything so that all of your alignments, your intent, alignment. Well, after seeing Ivan using Claude Code, I mean, it seems that

are on the platform. Claude Code doesn't discriminate, and it seems to work really well with other models. So why not just, as a harness, settle on Claude Code? I think Anthropic loosened up a lot there. But you think Anthropic messed it up? Yeah, because originally when OpenClaw came around, then Anthropic was very, like, they Because originally, banned OpenClaw the project. They started locking down user accounts that were detected. but they affected They searched for Hermes VM strings or something. There must be some limitations that Ivan didn't show me, like the artifacts must obviously

not work. I think that it changed because they didn't support all of that. And OpenAI went the opposite direction. They hired the guy from OpenClaw. They explicitly said you can use your OAuth subscription with Codex and you can use it with OpenClaw. And with agent client protocol, you can have your own interface on top Cool. of Codex to interact with it, one being OpenClaw and the other one being Kiro Crew. the one that I built but I'm pretty sure Kiro Crew also and now I see Kiro Crew

mentioning support for Claude Code as a harness like ACP as a harness which I thought would get my account banned and another thing is you shared I think or I saw Theo, he shows how his platform is able to launch Gemini or Codex but also Claude Code and he also mentions it's on ACP I didn't check if that's to the end, but So if you have like $200 plan, you get 4k of API tokens. wish I used T3 Code a bit more. I was pretty excited after,

for some reason I have this sort of schadenfreude relationship with Theo. I want him to be wrong, but he's right a lot of the time. I mean, I think he changes his opinion enough to be right a lot of the time. But I was pretty impressed with T3 Code but I think that there were some things that I didn't like about it like the fact that it was a big app and it felt a bit overwhelming and what was the other thing?

I'm pretty addicted to computer use think it supports it and I'm also pretty, it didn't seem to globalize my MCPs. Like in the video it says like it helps with MCP management but when I was using it I couldn't figure out, you know, like I used my Fastmail MCP and I wanted across all the agents I use but I couldn't get it working so I was like oh whatever I'll come back to this another time. So the thing I picked up when it

that was the tail end of my holiday was Rex. Did you see my Rex video? I was just about to say you shared two things that you said you were excited about and I said I think on both that you should have to show me and I know you send a Rex video I haven't had time to look at it. I mean, I could maybe show you now, but it would be a live demo and God knows what would But there was another thing that you were excited about as well, right, happen. that I said?

I'm excited about a lot of things. We're living in a golden age here. It's just got to make the most of it. We got to acknowledge it. That is a beautiful time. Oh, yeah, T3 Code is pretty good, annoyingly good. And I said, you should show me. You said, Rex, I'm going to try Rex, or I really like Rex. And then I said, OK, show me. I thought, yeah, I haven't had time. I feel like I'm kind of stuck in the way that I'm working, although I did make

a change in the way that I'm working a little bit. But I'm stuck on Claude Code. I don't think it's optimal what I'm doing. but I But are you sharing the screen yet? Okay, I have to show you. I'm here to show you Rex then. Not yet, right? You see my... Am I sharing the right screen, hopefully? Yeah, it's blindingly white. I'm, you know, I'm a little bit blinded right now. Yeah, I... I'm

So I was just showing a session that I was just working on on my own personal project. So what is Rex? What is So Rex is basically tmux and Ghostty combined. You use Ghostty, right? Okay. Yeah, I use cmux, which is also built on top of a Ghostty library. Okay, so the cool thing is that you can have remote ones, like for example I have my T14 running Rex, my Linux machine.

But as I mentioned I'm really... Is this Oh, is it from Superlogical? Yeah, it's from Superlogical. Okay, so maybe give that background, because Superlogical is a new company created by Hashimoto, the creator of Terraform and HashiCorp who retired like a year ago and then he Craser, built in his free time Ghostty which is a library that gives you I don't know his focus was it's written in Zig he really he's like one of the two Zig um projects there was Bun and then there's Ghostty and he says Zig allowed him to build like this super fast terminal

emulator um and that was all open source and free and it's built for macOS. And Ghostty terminal, the Ghostty terminal itself is just a demo of what the library can do, right? And then a lot of people built on top of that, such as cmux. So, cmux. And then he started a company with several ex-HashiCorp people after IBM took over. And they were very excited, Superlogical. One of the first videos they introduced, they talk about completely changing the protocol for

tmux is kind of very outdated. It runs over SSH. It's not performant in the way that it handles characters. So they build this dedicated protocol for an advanced multiplexer, terminal multiplexer client server protocol. And if the client supports that protocol, it basically is a super, super fast alternative to tmux. So it's kind of like Ghostty being the super fast terminal emulator that can render 3D character scenes.

really nicely in the terminal and then Rex being now the first you know it being in beta you can be on waiting list so I've heard about it and you got on it got on it I got on by putting my email address in the wait list I also just want to give some context so one of the open source groups that I've been involved in for a long time is called suckless the founders

a good friend of mine, Anselm Garbe, he's, well we're not, we don't keep him touched so much because he's based in Vancouver right now. So the time difference is a bit painful. So like for example in suckless.org, I think I've probably mentioned suckless to you before, though one of the main tools was a window manager. Mm-hmm. Though a number of years ago, So st came about because the community and Anselm recognized that terminals sort of sucked and

they needed to be reinvented. So st came about and it's a very simple terminal. The source code is beautiful in the sense that I think it's like a thousand odd lines as we try to do in suckless. Well, 2 ,000 lines long. and I think it changed the game with the terminals it led to some improvement and Hashimoto I believe studied it but

of course took Ghostty to you know a level of polish that's unbelievable but Hmm. that said like when I went when I'm I remember talking with Anselm. Anselm always had in mind that tmux or you know screen back in the days needed to be reinvented with this terminal so that you would have a so that simple terminal would have a back end so that if you killed the window you could resume it and it would be you know more integrated and it just amazes me that like you know people of course you

know people come up with ideas and the idea is not new but credit credit to Mitchell Hashimoto he's Superlogical and and Rex just in my limited uses of it is just incredible it's just incredible it's a lot of money behind them you know it's a bit like like uh on my on my what's this on my oh yeah on marchie on it's a omachi omakon

my con uh you know you have Mitchell Hashimoto Jack Pearkes from a VP of engineering and R&D at HashiCorp Alasdair Monk, head of experience VP of design at Vercel senior design leader at HashiCorp and Heroku and then uh Hector Simpson who is agentic experience at Poolside but then And the funding is Armon Dadgar from currently, I think, still at HashiCorp. And Guillermo Rauch, who is the Vercel, I think, CTO.

Oh yeah, the Vex, Vex Vercel, yeah. Tobias Lütke, who is the entire.io, like ex-GitHub CTO. There's Dax Raad. Patrick Collison, Tobias Lütke, oh yeah, Mario Red... What? Zechner, that's the pi.dev. How does he fund it? As Have money. Yeah, like, it just shows you what Silicon Valley can do over Europe.

I mean, suckless, I think, is quite a German-European type of setup. up and and You can just see it from the website. you can see that you know exactly it it's like a university website from 2001 can only go so far in a way amazingly Superlogical as with the right

friends but anyway that sort of whatever origin story aside so you use you have Rex you've been using that's the whole background because it kind of that to me is important because it's not some random lib Ghostty you know build project like cmux know, built like C no it's from the actual creators of cmux and there's like the creators of Ghostty and there's no, it's going to be like the I mean, already I'm addicted to Rex and I'm thinking to myself, I significant money behind it yeah wonder what Superlogical are going to charge for this thing. Yeah, no they have in the announcement they have something about free I think they will they will

-Math. their focus will be on the hosted service right the thing that costs And then another thing is that it requires Tailscale to do the sort of connection to remote hosts. So I'm thinking aloud that, Okay. at least my employer doesn't allow Tailscale, you know, it's not a sanctioned VPN. Okay, cool. So what's so good about it? So it's probably not going to fly. Because to me right now, it looks exactly the same like cmux. Okay, it looks like a terminal.

The thing that I kind of stumbled upon it, but the thing that blew my mind was the skills. And I did check with Orca if Orca could do this, and Orca can kind of do this, but it's nowhere near as fast and fluid as Rex. So assuming you have the skill, assuming you're using Rex and assuming you have the

skills installed and the hooks and all that sort of stuff, you can do things like this. Like, so I'm working on a thing, I'm working something on Codex, and then I prompted a bit further on here. can you launch a terminal asking Claude the same question because I wasn't I was just thinking to myself I wasn't too sure the way this was going so with with Rex installed it's able to base I'm sorry yeah but this is a Codex session this is a Codex um session

inside that Codex skill yes I'll use the Rex skill to open up a terminal and independently Okay. investigate the same spike. So then it launched this tab, prompted it Nice. and it goes on from there. X-high effort. Astra, high. Opus 5.5, X-high. What are you doing? Am I, shouldn't I be using X-high? You're on GPT-6. That's one of Theo's latest videos about OpenAI.

Oh dropping it down. Okay so Yeah, you rarely run it on high or extra high. All right, okay, maybe I should and save someone some money. But OK. But you get the point. And then furthermore, I was using Claude in another session and I ran out of usage, like your usage, where we said a half past four. I'm like, oh, damn it. And then all I needed to do is go into Codex and I say, hey, Codex, in my Claude visitor spike, I've run out of

usage credits. Can you resume it here? So with the Rex skill in there, it's able to like suck everything down from the Rex terminal or from where you last left off. And I know, of course, Codex and Claude are smart enough to like look at each other's chat logs. I mean, that's cool. Yeah, they're pretty good at that. But it also has this ability. And I thought that was cool. And I found it just a little bit quicker and faster to work with.

Like, for example, if I had a new session here, and you can choose which like you can say this mac or you can say remotely and I haven't tried remote one, but I could say, like, you know, spawn three new Rex tabs. You know, I could say different work trees, and that would also work. That's also a cool thing. Let me just go into some Git repository.

Okay, so here, sorry. I'm in Codex, I can say, spawn three Rex tabs. Does Rex have a voice integration? No, I don't think so. And also it seems to lack image transfer across, that's a known issue to remote hosts. So that sucks. OK, I'm going to ask some questions. So cmux have similar capabilities. You can tell the agent. It actually wraps the, like, remember I was showing you how I have

these little dashes on the side where I can see the agent turns. And I realized that's a cmux thing, because cmux sees the agent turns and gives this little viewer so you can quickly scroll back to a previous turn, like the little indicators on the left side of my screen, I think, on the last time I showed. So cmux also has a deep integration with the agent. It knows when the agent is waiting for some, you know, it has raised an issue and it will notify you using your MacBook notification as well. Yeah, the notification is also very beautiful here. Yeah. And I don't know how much cmux is doing for me right now, but I

showed cmux to a colleague and then they used cmux and they quickly customized it so that, you know, they have a nice view of all their agents that they were running. And then they also started having the agents interact with, because you have an integrated browser in there as well. So he was running an MCP and then he was running a dashboard. Yeah, this one doesn't have the integrated browser. So cmux, he actually built this whole thing where he was interacting with like a graph, with nodes on the graph with an MCP. But he was also able to ask the agent to look at the browser,

integrated browser window, and then click on buttons and interact with it. interacting with the agent and saying, like, okay, I want you to do this and that. And it would, you know, it would either use the MCP or the browser. And he was doing all of that on cmux, and then he decided that he was hitting some limits. And then he built his own. So he built his own app, macOS app, and then he integrated it completely with, he calls it, I don't know, I'm not going to say the name. Okay, I mean everyone's... But it looks very similar to this and, like, more...

Well, I mean this is very clean. I mean this is just focusing on terminals. Well, this is just a terminal. Like, for people who live in the terminal, like if you run So hopefully you got the point here. Vim. I basically just talked to it, and then I've now got one... Yeah. Well, you don't talk to it because you don't use a voice. But you I just chatted to it with text, and now I've got three different terminals. Yeah. So once... Yeah, like in my video, the big unlock for me with Orca was having separate work trees, and you just click through them.

it's just a lot faster just to talk to it and say like, I want to work tree prompting about OK. this issue and then you just split and you go, yeah, it's OK, yes, I saw you can have multiple tabs. not hierarchical, but it's the way it's organized in tabs, I feel is quite good You showed it twice. I think what would be interesting for me, because this doesn't look interesting to me at all, for my brain. this and more. But what would be more interesting is if you can do this across remote hosts,

if that's the interactivity with multiple nodes, like background agents running. Oh, yeah, yeah. I'm pretty sure you can do the same thing with remote learning. Yeah. So that's something we should explore for the next time, if you can have the same like That's something we Yeah, yeah. Like with with Orca, where you have the app. And I mean, I don't use Orca. It's definitely like Rex is on my radar. I signed up on the wait list. so I'll have ability to play with it. Yeah. Asking an agent to spin up other terminals. I think it's interesting,

Yeah, I mean it's - I just hold on. I just want to say one thing. Hold on. I just want to say something that changed the way I work in the last week, and I think for the better. It's Claude Code's ability, and I don't think it goes across to Codex or others. ability to to have a messaging channel between agents right it's been there for a long time I was aware of it but never used it um the reason was I wanted to control things I wanted to keep track of exactly what prompt goes to which agent and I was having separate like I had I

And I was started with one um you know planning agent that took care of like user alignment and one building agent but then I was hitting deadlines and I wanted to have parallel work so I started having like four agents, but I found that more than four is hard to manage. And I would have the planning agent identify work items for each other agent, and then I would be the messenger between, and I would read the report, and I would forward it to keep control. But I was running out of time again, so I was tired of it.

And I asked the orchestrator, the planning agent, I call him orchestrator, and I had renamed some of the agent sessions. So in Claude, you can slash rename session, and I had called him agent one, two of the project inside my Claude Code so they were like you know they were long-lived sessions and if you have Claude Code by default at some point it enables idle compact so if you before the cache expires which is like I think an hour that the cache is valid the prompt cache um the Claude Code will auto compact so I have a Claude Code running

on my laptop for a week and it's auto compacted like 29 times and and and so I have one agent which has some context of like its name um its role what it has worked on within the project and it has auto compacted so it lost some details but I always ask it to focus on the way of working which is like it keeps a long long term goal and it runs dynamic workflows for large pieces of work and it does small background agents for fix-ups. So what I wanted to get to was that in the last week, and

this is also probably why I exhausted my ultra subscription, I basically removed myself as the messenger. I asked each agent to introduce themselves to the orchestrator, which was a named session, and they found each other. I said, hey, can you introduce yourself to the orchestrator? And it was like, oh, I see the orchestration and session number, blah, blah, blah. And I'm sending the message on the channel and I went to the orchestrator and it showed, received message from agent one. itself and this is what it's been working on because the biggest problem was I lost track of which prompt I had to send to which agent and also I

didn't always inform the orchestrator that the prompt had been sent so the orchestrator was like agent two is still waiting for you to send the prompt but in reality agent two already finished the work so from then on I told each agent to always report back their status to the orchestrator and vice versa the orchestrator receives their reports and would delegate it. So there's... This orchestrator is an agent in Claude, right? It's all Claude Code. So every single one is a single Claude Code session. It's Yeah. Well, this is where... Why I feel Rex is so much better, because with Rex...

How is it better? Because you can, it gives you like a, like you could have Codex, you could have Claude, you can have Claude. The cool thing is that when you use these Rex skills, you can communicate with the Rex CLI or API, I don't know what it's called. So you can ask Claude. And you can say like, you know, like. So that's what you showed earlier. You said like, hey, Claude, my Codex is not out of usage. Yeah. But it went through the message. It didn't send a message to Codex, right?

They're not communicating with each other. You are the messenger. Yeah, but it can work that way. The messaging can work that way. Yeah, so you could be like, yeah, you can say, yeah, Can it? Can they talk to each other? Can Claude talk to Codex? exactly. And you can say like, you can do, I mean, I haven't really really I would like to see that I doubt it okay done it in anger, but you can do things like, you know, I'm in this thing here, like Claude Z, communicate with Claude A, with Codex A. about your work. And so everything's sort of managed by Rex in a way.

I mean, Rex isn't the orchestrator, but it gives you that messaging platform, which I think Does it give you a messaging channel? is really powerful. It does a messaging channel through its API, which is basically just text. Does it? Okay, go into Claude and have it introduce itself to Codex Yeah, that's... on your session right there. Like one of the new ones I started. Oh, yeah. Maybe spawn another Codex session in Claude 1. What do you want me to do?

say maybe slash rename call it agent1 com demo communication demo okay agent1 Okay. Comdemo. Yeah. and then maybe Claude 2 call it agent2 slash rename agent2 Well, I mean each tab has its own name. I mean, we might be confusing it. I mean, I should probably be renaming it here. okay OK. I mean, I don't think it matters at the end of the day.

OK, it doesn't matter. I know that it works if you give them. Okay, I mean, even if you don't give them a name, they will find each other if you have some chat, Yeah, but I think it's more... because you don't have anything in there. And now launch a Codex one, a separate new Codex one, and call it agent3. Okay, let's just get rid of this one. Oops, you ran a prompt rename agent. You didn't. Okay, so this one's called agent one. This one's called agent two. and this one... This one's not. It's messed up. Agent 2 is messed up.

Look, go back to Agent 2. What's wrong? Yeah, you didn't do the slash. Oh sorry, it seemed to understand me. Oh. Okay, now it works. Okay. And then this one is Agent 3. Okay, You're already in code. Yeah. Does rename work here too? Agent 3. Okay, so tell Agent 3, introduce yourself to Agent 2. So like, my name is Bob. Introduce... Why is your name Bob?

Well, I mean, okay. You don't need that. You just say introduce yourself to agent two. And let's see if Codex finds agent two. I just wanted to pass something. Okay, introduce yourself to Yep. Agent 2, which is actually Claude 2 in this. Yeah, and nothing else. Like, no, no, no, no, no, no. So according to you, Rex skills. No, no, now it's using Orca. I got to prompt it a bit better.

I've got the Orca stuff installed right now. Wait, let me OK. just say in Rex. I need to remove the Orca hooks because it's messing up. OK. It looks like it might be able to find it. While this is going, go to agent 2 Claude and say introduce yourself to agent 1.

And don't say no, the other Claude Code, Oh, you mean agent 3? Oh, okay. because it's going to work within a second, unless your skills are messing it up. Oh, it came through. Oh. Introduce yourself to... OK, so they introduced themselves as Codex and Claude Code, and they responded. Yeah, and you can see that the way it does it usually is by this sort of... this API that it does, this block capture thing. Interesting. So that works. OK, so that is nice.

It's very nice. That is what you just said it does. Yeah, I wasn't lying, but I accept the Orca thing is messing up. hmm but you're right, it did communicate a lot quicker, I guess through the Claude harness, but the real power is that... it went fast there right it didn't do anything it didn't have to do because because Claude Yeah exactly, this seemed to be faster, I mean Code has a channel that they can talk to each other directly I didn't measure it, I didn't measure it but it's cool right? I mean this is why I love Rex, just because

It does enable it in a very clean, sort of like non-confusing way. It does enable it. It's independent of the harness. I don't have to rely on Claude to do all the magic. It's just text. It's like as universal as Unix pipes or something. OK. Yeah. OK, that's very interesting. I mean, that's interesting. Yeah. For me, this pattern is interesting because once you've done that, they get into the habits. So with the orchestrator, every time it would just

feed back to me. Agent 1 is waiting. Agent 1 has completed this and it needs an answer on these questions. Agent 1 has ran the AWS validation loop and identified a couple of issues with our previous decisions. And it kind of gives me this one dashboard. But it's not a dashboard. It's still text. and that's where I feel like I'm way behind in actually having a nice interface. I think Terminal isn't it. I really need something else. Terminal is okay.

There's a couple of startups now focused on building, you know, Terminals are sweet. I do realize it does fall down when you're working with web stuff the... and perhaps when you're working with the computer use thing. Did I mention that the Rex iOS client is just amazing? There's an iOS client. What does the iOS client do? So this was a terminal. This is not the iOS client. I'm sorry, I have to connect to bloody my VPN.

Oh, you mean like iPhone? Also, you have a Rex app. Oh, you Yeah, like for example, I can jump into that, you know, that Claude instance that we were just chatting about. Okay. So that's interesting. I mean, like the Claude app is doing that as well, right? Yeah, but the polish on this thing is insane. It's the best terminal emulator I've ever used on iOS. Oh, that's nice. Yeah. Interesting. And it's even got like really nice sort of like things where you can just like swipe left and right. Wow. OK. That's something that Claude, OK.

It's just so smooth. It's crazy. Yeah, because the Claude app with, you know, every time I open it, I have to go to the Codes tab, and I have to find the sessions. And if you name your sessions, then the Claude app and you get nice notifications of orchestrators waiting and agent one has completed this and so on but it's still like a clunky chat app when you're coming from the terminal where you have a lot of commands and you have a nice you know

you have a lot of capabilities but then you're in this app on the phone and it's just like a chat window still I wouldn't want to become like one of those cursor ide guys stuck to my ide or stuck to my terminal and not aware of a better way like I think there's quite a few startups or companies that are building a more intuitive way of working with agents I think kiro crew is one of those um you know visualizations with a gateway um I think the Claude app itself is also trying to be like you know another way of working with

agents I I'm not used to that one yet I I don't like the Claude app it feels it feels I like the lag. When I click on something, the terminal is super snappy and fast. So I'm still stuck in cmux. I use the integrated browser sometimes. It's pretty nice. The one problem I have is my MCP configuration. I really hate when I'm moving between computers, setting up my MCPs all the time. So this is, I'm looking for a solution where my MCPs are always

like available. And I haven't found that yet. So there's still, that's probably the biggest gap for me in my AI workflow. For me, the most annoying thing is that even if I tell the orchestrator that I authorize you to do this, and then the permit, because when you talk about MCP and the reason that it's difficult is sometimes because, you know, it's credentials, it's sensitive. so with me the permission like the security classifier across mission

the agent is sometimes being very annoying like agent one says that the classifier blocked his know. Oh, there's that even. I mean, to be honest, I'm, on a personal note, I ability to do this or that don't do this in my employer. I always run my things, basically, Skip Dangerous or YOLO. I can't be bothered. I always use the classifier and I think because I'm doing it for work the work laptop has pushed, like I know Aikido endpoint security, and then the work computer Yeah, that makes sense. has also pushed a whole bunch of rules that are like, every single time that it runs a

command on AWS and an account ID comes out, it just flips out. It's like, big oops, gotta flag this. Oh yeah. Account code in logs. It's like, I think AWS itself tells them, says publicly, account IDs are not like secrets. They're public information. I remember googling it in the past, not recently What I was doing yesterday was so boring. Cloudflare doesn't, I'm so used to AWS. You know, AWS, I generally use, oh my God, the OIDC thing.

you trust the environment and then you can deploy your Terraform or whatever into that Mm-hmm. environment using that simple trust association. Yeah. Yesterday I was messing around with Cloudflare. Cloudflare doesn't seem to have OIDC and the problem that we had is that in in GitHub there was some Cloudflare API token set. You know, A, I can't read it back out so that's you know problem A, I can't read

out so I don't even know what token is set and even if I could read out the token I don't think I could associate it to like if I log into the Cloudflare dashboard there's a ton of tokens there's tokens attached to different users and it's like in Oh, my God, the Cloudflare interfaces administration panels is horrible. fairness Cloudflare does have this concept I think they call them account keys now so it's not associated with users joy but then in the account keys yeah you create them you know ask you to set an expiration but like you don't want to set an

expiration when you put it in bloody GitHub Actions and then you set the key inside GitHub Actions and again you have the same problem like you don't know what the rotation is or anything like this I mean basically API keys is totally caveman right and that's what a lot of vendors I think Okta is another one that really annoys me which doesn't have a very clean relationship so the minute you you're dealing with API keys you're just setting up yourself a disaster right it makes all the security stuff that you want to do

so much harder okay you just tuned out there That's true. I mean, overall, it was a big rant about Cloudflare being not very user-friendly, but at the same time, Cloudflare has an amazing, I think, new paradigm in the way that you build deploy or run in fresh like applications on the internet which I think is something like we talked about last time I feel like I'm really missing out by not exploring their like platforms more I also heard the same

about Vercel like people telling me that Vercel is their north star and how they built their platform on uh you know how for users to to to use it and I feel like I'm missing out I feel like I'm missing out on on Vercel and Cloudflare and and in that in that happening now, it seems to be that all of these SaaS, like Vercel, which was a CDN, and then became like a next deployment mechanism, then I think they acquired Supabase as well. And then with V0, they began with agentic website builder.

And now they are launching CI and continuous integration. And I think they have version control as well. And then you have the same thing for all of those other continuous integration providers, like depot.dev. They had Cloud Session Store last year. They launched their own CI so that you don't have to worry about Microsoft starting to charge for orchestrating your GitHub runners. for orchestrating Because they started as a GitHub runner, like hosting provider, booting up your depot.dev. Which company is this? Sorry.

They sponsored the open source project, depot.dev. Oh, you just... Okay. They got a big VC investment as well. They completely changed their polish. But now, it's just one sample because you have others like RunsOn and other CI. Well... Basically, what they did for their whole life, for their whole existence, is providing isolated compute to run CI jobs. Guess what? guess what What is the main requirement today is to provide isolated environments to

run LLMs, sandboxes. Did you know just that Dagger went 1.0? yeah so dagger went 1.0 and I think they also launched a cloud for it like a CI engine right they are a CI but yeah again they're the same you need to build all of these CI I I'd completely missed that like all of these startups that were building CI runners are now getting raising six million dollars I But surely it's going to have the same problem that I was just moaning about, the secrets in

your actions and in GitHub where it doesn't solve that security aspect with API tokens does it? think the biggest issue with all of these new CI systems is that I don't know any enterprise I think the biggest issue that's going to be fast to move towards adopting any of those I'm looking at them I was like, Yeah, exactly. They talk oh, my God. They talk about, you know, milliseconds for scheduling job runs. They talk about, you know, all of these capabilities, their version control, you know, when GitHub is having outages left and right, Bitbucket is having outages, and then they come in right. Yeah. Bitbucket based on...

I mean, to be honest, like... and they're like, you know, we are built for agents. ...my current client is a big company, it's a big enterprise, and they're using Cloudflare for zero trust. And sometimes I think to myself, it's not an enterprise solution, because... Oh, my God, it's horrible. I was on a project that was looking at Cloudflare to set up a CDN connection to their AWS-hosted Kubernetes cluster environment. And we had Argo CD for GitOps, and we wanted to have ability to...

ability to... Oh, they wanted... This was a highly regulated company and they wanted from the Cloudflare CDN that they were paying a lot of money for. They wanted to have a private connection to the load balancers in front of their Kubernetes clusters. And it wasn't possible. Cloudflare doesn't do that. You can do direct connect, but it would take three or four months you got to set up. the only thing they would do is like the zero trust with the Cloudflare tunnels, which is not

a sustainable solution for like all of your customer facing traffic. And this was a betting app. So the amount of transactions they were getting is huge. And that was just ridiculous. I was like, why? I really wanted to kick Cloudflare out of the project because I was like, Yeah, it's so this is the problem like you mentioned all these like newfangled solutions you just use Cloudflare and everything works out of the box. But like the trick with enterprises is just solid integrations and

dependability that's hard to... Yeah, but Cloudflare is not a newfangled solution, right? Cloudflare to me is like Akamai. They're this huge CDN company that started as providing... I don't know, they have a sort of a nimble vibe to me. But like, but yeah, when I use it, I think to myself, oh my god, they need to fix this. They need that. They need OIDC. I feel like they have two sides. They have the moneymaker, the big contracts that have They're not AWS. the firewall and, you know, the huge CDN network and the huge core of, you know, data centers

that they built out, which is highly, how you say, inflexible. And when you interact with the dashboards or the interfaces, it's horrible. It's so outdated. It's not UX friendly. and then they have this whole new side of them which You're talking about Cloudflare? is like Durable Objects like the complete Yeah. twist on how you deploy into the cloud at each location no it's horrible I think the Terraform provider is pretty good. Really? oh my god it's so bad because they just released I think a minor version and they just break It's bad.

things there's a lot of problems with upgrading with this we are using Cloudflare and I just They are moving quickly, I must say. did version bumps and it was horrible but yeah I mean they do have a Terraform provider which is already better than some like I don't think Akamai has it Yeah, I mean, I Yeah, I want them to I want them to win but yeah, they definitely not nearly as mature as AWS In fact, I'm wondering to myself. Why doesn't AWS do a zero trust product?

They probably do but it's But it's probably hard to use or something they do have yeah absolutely and it's expensive it's extremely expensive I'm building like this And it's inexpensive egress proxy for agentic so for I know Tailscale is more expensive than Cloudflare, amazingly, anyway. AWS I'm building this agent sandbox environment so it's durable workflows so I'm not actually having long lived agents and I'm giving short lived

bindings when you talk about agents they have like they either have a long lived identity and they are acting on a certain incoming session and then they have a certain identity that they act on behalf of. And this whole identity management of agents is really complicated, but it does matter a lot if you're trying to let an agent interact with external systems. So for this thing that I'm building, I need to have native egress capabilities. Like you can't do get clone on an MCP tool, right?

So we need to have native protocol support. And for that, I need to support the egress proxy pattern that Claude and Anthropic offers. And they have it really nicely documented. So I'm building this fancy egress proxy for credential injection. And you can either generate your root certificate authority and then define intermediate short-lived authorities and then manage the, you know, because you need to TLS terminate connections.

So you need to inject credentials like the network traffic. And you can do it very cheap. But, of course, if you want to do it proper, you don't want to manage a rule, certificate, authority, private key, right? That's a very sensitive piece of infrastructure. So you're looking at AWS Private CA, the ability to run a private key infrastructure on your internal VPC only to do TLS termination. Yeah.

The cost is close to $500 a month. And I'm building this on my mirror environment, which I actually can't Why don't you just use Tailscale or something and fork out $400 to build this. just... How does that fit into as an egress proxy for sandbox and for agents that are living in a sandbox? Well, you can set up your own VLAN with Tailscale.

with Bitbucket? With You can set it up with anything, I'd assume. Really? Really? Yeah. How would you set up an agent to have, without giving the agent an actual real key, a real credential, how would Tailscale solve that problem to then connect to Bitbucket and do a Git clone? I don't get that. Because the Git, a new Bitbucket interaction needs an authentication, right? will solve my Bitbucket authentication problem.

It sounds like Tailscale is just like a virtual private network solution so that you can put machines on the same network without encapsulating on top of the public internet. Am I getting it wrong? I have to get back to you about that one. But like, I was going to extol... Classic consultant response. I was going to extol the tailscar. Because like, let's agree that security in the age of AI is going to get wild, right? We need to heighten that security.

And on the topic of VPNs, not many people... Okay, no public facing API uses VPNs, right? but we're probably going to get to a point with where where everyone has to run something like Tailscale in the sense that every everyone's device has some sort of posture and then and then or some layer so when you connect to that to the public API well it won't be public anymore it'll be protected by something like Tailscale it'll have because

like when you get when Tailscales connect to another Tailscale there's there's like a concept of identity right with packets that you send and then that will be the gateway to use something because I'm trying to make this argument that like having an internet public API is probably not going to fly in the future especially if it's a very sensitive API So, talk about the network separates the machines on it, or the principles, you know, the identities, you're talking more about like,

Yeah. You're it's not like the classic VPN, like WireGuard. It's more about zero trust type of setup. Yeah, these, I guess so. Like, I'm not too sure how I can show this to you, but Tailscale has this like identity sort of part to it. So, for example, I could share my Tailscale network with you, but I would probably control things like, I can only talk

-hmm. to my Raspberry Pi and you can't. Things like that. There's a whole lot of routing and things like that. So that's where you say the BitBucket, if you're able to. So, I mean, this product, like Zero Trust, I rolled out Teleport a few times, and it has this idea of, it's the same that you don't trust a network. Yeah. Every packet carries authentication.

Well, the packet, it's more like in the protocol level. The authentication is sometimes injected by the, like in Teleport, you can have an OIDC provider it can, with SAML, push through your identity. So you log in once, and then your identity is propagated towards any service behind. But you have to set up a SAML integration or a... Yeah, yeah, yeah. There's some sort of, what do you call that?

SAML is the most annoying one because it's XML, but there's the other one, I think it's JWT. But... The most annoying It's OIDC, sorry. It's OIDC, yeah. yeah Yeah. so what you would need then if you want to have an agent running and cloning Bitbucket you would need to have some type of integration with Bitbucket OIDC so if the packet comes in it automatically verifies the signatures and

it knows that it was signed by an authority that it trusts the identity provider and then it allows the activity I'm not sure you can do that with Bitbucket. But yeah, zero trust in an agent age, in the age of agents is probably... But that zero I mean, it was already a pattern that everyone should adopt, but you know how many companies are actually sharing VPN profiles. Even when somebody left the company, they can still log into the VPN and sharing long-lived credentials. That's still such a...

That's I just want to get away from... It's nothing new. I spent hours messing with API keys yesterday. My biggest security win, my immediate security win is just to have better trust Thank you. relationships. I don't know how it can work in terms of zero trust and Tailscale exactly. I've got to investigate. But I don't want to mess around with API keys. I want to say, you know, my role in Cloudflare,

log in with my Tailscale and then that identity is trusted in that context. So I don't have to mess around with API keys. So, but that's the... In security, I think they call it the zero secret, like the very first secret we need to establish to establish the trust. And then there's also, it's very different when you have a machine identity versus if it's a you user identity. Whereas, because AgentCore has this identity component and they talk about

like, you have different identity types When an agent is acting on behalf of a user, then it usually means there's an authentication flow that the agent gets requested for additional approvals before the agent can do those things. So there's this back and forth, like, hey, the agent wants to do these additional things, and there's this additional protocol on handling that. Whereas if you're working with long-lived... Can I give an example? Yeah, when the agent is trying to do things

have the ability to do, but the agent needs to get the permission to do it. I know that in AgentCore gateway and AgentCore identity, there's this inbound and outbound. And when a user flow is involved, it usually means a back and forth because it means the user actually have to go and click yes, go ahead and do that. Maybe there's a security classifier on the user client side that can automatically approve based on the local activity. but there's this additional verification, which you can't do when you talk with machine

identities because there's no user to click approve. When you talk with machine identities, they must have a scope issue. Yeah, yeah. You must have I'm just, yeah, I'm thinking that you can't do approvals in that context. Yeah, so the platform I'm building, it requires that. And also the dream of everything being on short-lived tokens and not ever having an API key or a long-lived secret That is nothing new. That is the battle that I've been fighting in so many organizations that

in a new organization in an ideal scenario, you never have a secret store. you don't Your comparison side, I keep on forgetting the You don't need a password manager. But you always have third-party tenants. name, it's Tarkos. FACOS.guru. need like an infrastructure one where it names and shames people who don't but It's there. Oh, also, there's an... that basically that the names shame people that rely on rely on the API key for the integrations It's called the SSO paywall. There's a website famous for name and shaming organization or

product that puts SSO or short-lived tokens behind a paywall, which is like... It should be basic capability of your product. You wouldn't need like an enterprise cost. Yeah, SSO.tax. Yeah, that's the... Where's Cloudflare? Where's really Cloudflare? Somebody managed... Somebody's buddies' buddies and took them out. Oh, you don't... Semporo Cloud is on it. Well, I mean SSO is probably not what I was... SSO is the first step.

You should at least have SSO. Okay, we need an API key one though. That's an idea. Name and shame integrations that use API keys. Hold on. Maybe I just need to ask Gemini. so chain Chainguard they're very public about fighting long-lived secrets as well and they have several blog articles about how some of their products allow you to like proxy to external systems that do not have that I

think one of them was um OIDC I forgot now but but uh let me just Chainguard. OIDC short-lived credentials. Did we talk about the whole Chainguard, like, this disclosure And what... failure? They're on a... Ah, it's octo-sts. No, this is not... Unwalt? No.

SSO.tax. CVE or Yeah. RCE Yeah. so the have you followed that story about how they started like it's a known issue with the memory management of the PHP To be honest, I ignore the PHP because I don't use PHP and PHP has a billion CVEs,

but yeah. it's not that it has a billion CVEs it's just that So many people use it that it gets abused. And then there's so many projects that are Yeah. portraying to be professional but actually have leaks left and right. Nothing wrong with the PHP itself. It's just that it's so popular. Well, I think PHP is kind of silly to run. And I don't like PHP. But it's silly to run because it's got like an enormous surface area, right? Like if you get in there, you're away.

But so the whole thing was the project maintainers had this long-standing open issue about memory management that they were aware about some problems with it. And then with the advent of AI, some research labs identified an RCE and I think raised it to them in September. pending fixes that they hadn't you know decided yet how they were gonna like implement it so when this RCE came around they say yeah we just you know had to make the decision of the of

the solution and we were able to do that in like I don't know two days or so and they were then building and and testing so they were actually having the fix out very well but they were having a responsible disclosure and they were working with partners and because PHP goes through distributions and people download it so it needs to be a large coordinated effort on uh you know making sure to provide the fixes on time. Mm-hmm. So they're working with all these distribution partners and everyone signed an agreement and they're in this responsible disclosure group. So they have a responsibility on following the rules because you're privileged. Oh yeah.

And then no doubt someone blurted out something. Yeah, so you're privileged to get this early information about, hey, there's an RCE, here's the fix, make sure it's available. We're going to make the announcement of the RCE and the release is available by, I don't know, Thursday, 2 p.m. And what happened is Chainguard, what we were just talking about, they maintain secure images, right? Base images. And they broke the embargo. They released like six hours ahead of the scheduled time, which All right.

forced everyone else to be on an accelerated timeline. and the PHP foundation was quite upset with it because all the other distribution partners in the Scrum within like four hours instead of eight, I think they had to go and push. And a lot of comments were out there like, what Chainguard did deserves them to get kicked out of this privileged information group, right? you shouldn't be informed about RCEs anymore if you're not going to follow embargoes. Yeah, the trust has been broken sort of thing.

Yeah. And then the guys from Chainguard said, look, we had no choice because we saw it actively being exploited. And therefore, we decided it is best for our customers to release the fix prior to the lift of the embargo. Oh really? I thought these things they allow you to release the fix but they don't allow you to announce Yeah. it because I think I'm Mozilla No, it's an agreed point at which they do the release. Really?

yeah so basically I thought that's a reasonable response right but then I read some threads and people are like yeah they talk about they saw exploits but they never say where and when and they have no proof they just say it but no proof and then another thing is like AWS I think AWS jumped on and said we saw exploits like in August or in September right we they kind of Hmm. that story yeah but um yeah the whole PHP thing was quite an you know quite the tea it

was interesting and it's also it's also very interesting because it was in a week of um you know Brotli from Google the compression algorithm having you know they say Google is not having any maintainer on it and there were exploits in it and Brotli is actually implemented the you know the algorithm and the libraries are ported over into many languages and it's integrated into Golang it's Oh, is that where the, oh yeah, the Brotli, yeah. So they said there was not to shell. Is that where? Because in that week, you have so many exploits coming out.

You had the eMudhra intermediate certificate leak. That was hilarious. That was hilarious. Basically, OpenAI or ChatGPT or Claude, I don't know which LLM, was able to reverse engineer the actual prime numbers that made up the private key. And it turns out that the complicated to cast prime number was number four. have all these shirts, and Private Key 4, number 4. So that was hilarious. But that was a wild week. What I find even more hilarious is the xkcd comic.

I mean, this is an old one, right? I don't know how you see when it was released. How do you see when it... Maybe the numbers are serial. Yeah, 221. I mean the latest one is 3319. Go to internet It's like predicted the future somehow. Wow, it's very early Yeah, there's quite a few XKCDs that are applicable But do you think it's just like four?

I mean it's simulating a dice roll, but like it's just so seven different so yeah Coincident it's so amazing Coinkidoink? I think not. So the eMudhra intermediate certificate private key guest being extremely weak. Node2Shell being Node.js needing an urgent patch because you're able to exploit the compression algorithm and then you're able to

get RCE. And then that became Web2Shell because Brotli is implemented or this library lives in a lot of other languages. any web server that supports Brotli is actually exploitable. Mm-hmm. Golang having to, with the certificate revocation, very quickly release a patch because Golang implemented its own certificate revocation list verification or something. Basically, you had to update your Golang. Yeah, that was a wild week, and I don't think it's done. I think it's still going.

I think this is the new future, because we were talking about Chainguard and security and... Yeah, but like if you didn't use PHP, I can't help but think you are setting yourself up for success. I just said there's so many. I said the intermediate certificate leak, there's the because an LLM was able to guess the number. Okay, okay. It's not, you're right. I just said there's the Brotli algorithm. It's not PHP, it's Brotli, it's other things. but like but PHP has that surface area like at the minute you can somehow leverage that

Stop your PHP hate. The H doesn't stand for hate, Kai. well I I It's pre-hypertext processor. all right well my client is using this thing called PrivateBin have you heard of that for sharing secrets and that uses uh that uses Sounds familiar. It's PHP, I guess. PHP and I'm like I'm just like oh my god no no Isn't Facebook still on PHP? Didn't they build their own PHP... Hack or something. I can't remember.

No, engine, like, unoptimized. from Zend, they have their own engine, I think. Yeah, maybe you're right. Maybe I should stop hating on PHP. And you accept JavaScript in your life. Not really. I don't really. Not really, right? But that's the crazy thing now, right? JavaScript is everywhere. Durable Objects, Bun. I'm not a thousand percent sure Go But I believe the linker with Go, it basically only pulls Sorry, it's Deno.

in the functions that that particular app uses. So if you have a Go binary and you try to game it and run Brotli decompress or something, nine times out of ten, that function wouldn't be in the bundle. It wouldn't be in the static binary, so you couldn't exploit it. So this is why I like to use Go because the function surface area is much lower because the distribution doesn't ship up the standard library and then some, right?

You had to bump because every web server in Golang is enabling the Linux kernel and TCP flag, which has also been exploited. Like, if you don't, you have to run go with a Yeah, you certain environment variable to prevent the TCP usage, no matter how it's bundled. You're right. I mean, it doesn't make it immune to those issues, but it does reduce the surface area in my mind, but I'm probably just clinging on to nonsense. Go is so much better. But the thing is, when I build with JavaScript,

you have to bundle, which is exactly what you described. You're not shipping node modules. You're resolving every library's Linking. function that you depend on and bundle it. So bundlers like esbuild, they generate a single program Yeah. Yeah. that only inlines the functions that you depend on and not the whole node modules, like the

whole package that you depend on. But it still runs in this Node.js runtime that has a lot of things in it, right? No? OK. It runs in QuickJS or it runs on the edge. OK, all right. Right now, JavaScript runs everywhere. And I mean, you can hate on PHP, but I think JavaScript is at least as bad.

But it turns out you can write anywhere. you can you know what I got wrong earlier when I said Cloudflare there they didn't acquire Yeah, yeah, yeah. Bun they acquired Deno and Deno had this I mean, yeah, anyone listening, I'm not just being, I guess I'm a bit tribal, like I do have my my ways of doing things because you have to you know and there's so much out there isn't there Vincent and I do have my my way around managing all this complexity so yeah you got to forgive me then when

someone talks about a PHP vulnerability I'm just like batting my eyelids and moving on Yeah, no, the reason I talk about PHP, because there was some tea regarding Chainguard, and The Chainguard has this huge initiative about short-lived secrets, which they pride themselves in Yeah, you're right. That was... And I'm still a bit bemused by that process. security. I thought you're allowed to release fixes, but not announce it. I mean, a lot of companies have released security fixes without telling them what the thing What was this exploit? There was this thing that leaked. is.

I think it was on the Linux kernel. There was this CVE that was announced, and then somebody identified a similar CVE, like in an adjacent code path. And they created a branch, a public branch, to implement a fix. And the name of the branch referenced the family, like family of exploit. So somebody saw the branch, saw the fix being worked on, and tried to identify if

there was another exploit. And then it turned out into a zero day. There was this thing that happened with, I think, the Linux kernel. So when you say you're allowed to announce, I think some of this work is difficult if it happens. that this type of RCEs are being worked on privately. I'm gonna ask chatGBT. Yeah, I actually haven't really spent time to understand this Linux remote exploit. There's some exploit in IPv4 network stack.

I know they created a remote commode. I mean, these are very rare, aren't they? MPTCP, I didn't even know what that is. MPTCP, yeah. What is that? I thought it was like multi-channel TCP or multiplex TCP. Oh, multipath. Yeah. Multipart. Oh,

Interesting. Oh, DirtyFrag. So DirtyCow was public. Everyone had to roll their Linux kernel or roll their Linux machines. And then right after there was... Oh, it was copy fail, sorry. So DirtyFrag was disclosed in May 2026, shortly after copy fail. So the maintainers published a security fix in the public netdev.net.git tree ahead of the planned coordinated disclosure. And that resulted in somebody looking at this,

like, what are they trying to fix? there's a huge, so then it's a zero day, nothing is patched. In your current role, do you Are you like the security guy or you just wear that hat? I think it's everyone's responsibility. It's everyone's responsibility. I think I think I am very interested in like, reading the backstory and the history and how, like the Phineas hack happened against what's it called the security team you Yeah, it's a good idea.

know how the spanish police officers were you know hacked by phineas fisher as well like those are extremely interesting stories when you read how they penetrated the network how they were able to you know move along inside the network and it just it's just important I think to be aware um but to be honest my position is today Yeah, true, true. like when I talked about earlier when I was talking about setting up this egress proxy and managing root you know certificate authority secret material like a private key that's

already like you just don't want that responsibility you don't want to hold on to secret material like that that's the I'm all in on managed solutions there. That's what I mean, I think it's And I think it's the position of the bank. If you've run Linux servers or anything like an EC2 instance with public ingress, today you feel the pain because you need to be constantly On it. rolling. I was joking. It was like, oh, I waited a week and now I can jump ahead.

So it makes it all the more important that people use AWS and cloud I only have to do one rollout, not like three in the last week. solutions because they have a... Yeah. Yes. And that's why I think it's double important that when you build on the cloud today, or That's why that's why all of these initiatives like Vercel, the platform that they offer, they take on that responsibility. the AWS shared ownership responsibility but if you run on AWS Oh really? you should be using these newer cloud managed solutions and way of building where you just

run your program and you don't manage the whole network stack Yeah, you've got to choose your battles, haven't you? because and that's where if you're stuck with plain old HashiCorp language Terraform you're screwed Yeah, you're right. I mean, there's going to be so much transformation happening. I mean, I'm pretty much stuck in the world of enterprises where things move really slowly and it needs to move faster? I'm just actually thinking aloud, how do we get to start moving faster

here? For me, the choice now is very simple. If it's not a managed AWS service, it means it's Because... For me, going to take a long time because it's going to need to go through a whole chain of approvals. it's going to have to go through a whole review, it's going to take so much longer easier who's responsible for patching, who's responsible for intrusion detection, who's responsible Yeah. for all of this so the choice is easy, you want to move fast, you want to move at as high as possible a level you want to be able to ride on their compliance and their

security guarantees but don't use Northflank or Railway Yeah. Yeah. But I mean, there's a serious... Yeah. did you hear also about Northflank and Railway oh we haven't even talked about the Binance But... hack Oh my god. Well, I've washed my hands of crypto long ago. I mean, Binance for me is like crypto is dead, surely. It must be dead now. oh my god did I said the same at Mt. Gox hack back in 2012 I thought the Mt. Gox hack that's you know crypto Bitcoin went down I was like um see I'm right and then right

It's over. It's over, guys. after I think at the time I don't know it was like a hundred dollars I You're a dinosaur if you remember Okay I need to take... remember one guy no there was one more thing I wanted to say about this Binance hack because it accepts me very much Coffeezilla did a breakdown All right are you a crypto guy? I don't know I can't remember. Are you into crypto? no I'm the guy who watches from the sideline and grinds his teeth No I'm Yeah that's me that's me right there yeah I'm the same. because what is the most crazy thing about this Binance hack that Coffeezilla covered so I

read quite a few breakdowns of this hack you know apparently the DevOps engineer was lured by interviews and they sent an assignment to do the assignment on his laptop which included the Oh my goodness, I've heard of things like this, yeah. code that compromised his laptop and he had the credentials to basically talk to the service and hit the hot wallet So someone was interviewing a Binance employee, oh my god. yes but what was the interesting side that Coffeezilla covered not the way that they got in and how they exfiltrated all the money but what happened right after so it was about what was

it 6 billion 5.6 billion of money and Coffeezilla identified that immediately Binance they talk with these there's two types of crypto apparently like you have the ones that are uh you know the ledger rules and then you have another ones where it's it's not really decentralized you still have an authority and they're able to freeze the crypto so the hackers they got a handful of of like a mix of coins whatever was in the hot wallet right they empty it so the hot wallet is the one that's online that's there for doing you know the transactions so Yeah, yeah, how the hell does it have so much money? But yeah, you're going to get to it, I'm

sure. no I I don't know why it had so much I guess because Binance is so big and handles so much traffic and transactions that but its hot wallet probably has so much. Billions! Okay, okay. Binance is huge. So what I was going to say that Coffeezilla highlighted, which was, I think, kind of very farcical, is that of those 5.6 billion, 2.5 billion was frozen by those coins that are, I don't know, freezable by an authority. So what's the point of decentralized currency if you have an authority that can freeze it?

Well, it's a guardrail. Anyway, listen, I have to take my kids to school. I don't really know too much about it. I'm not an authority. watch the Coffeezilla video of the Binance hack and how the hackers only got 2.5 billion. Okay, we can talk about it. Anyway, good catching up with you, Vincent. Oh, we're in for the right. That's just, what, two weeks of hacks. We'll catch up. Okay, yeah, I mean, we're going to have a lot. We We should probably change this to the AI hack or something like that or the security podcast or something like that. I mean it's going to get that way isn't it.

See you, bye.