Multi-agent harness and burning tokens

The agent harness part of Console has changed a lot. I often try things out, either in a moment of inspiration, or because I see someone else do it, and I keep the things that work and throw away the things that don't.

For example, I tried building durable agents that exist in a Paperclip-like org chart, and can talk to each other by delegating tasks downwards and reporting results upwards. My org charts got pretty big!

And at the very top was AL managing everything. This really did not work very well. It was just too complicated. The task system was confusing, delegation felt arbitrary, and this specialisation neither felt more performant nor more ergonomic. I also had a system for parking agent sessions which was necessary as all these Claude Code sessions actually took up a lot of resources, and it was constantly breaking.

The UI as well was just not really that useful. I did find myself wanting to talk to specific agents and it was easier to just jump to them in a command bar than to visualise the hierarchy. It's for this same reason that I didn't feel like my Machete UI in my PKM tab was very useful, I just never really used it.

These felt a lot more like the kinds of things you look at every once in a while to visualise things at a zoomed out level, rather than where you find yourself day-to-day to browse things. So they've gotta go. Because they were so fast to build now, I didn't feel too wedded to them, and the sunk cost was just at experiment level!

What I ended up converging on is a Kanban-like UI as my primary surface. This UI is changing every day, and even in the time between when I took this screenshot to when I made my previous post, you'll notice some things have changed. But the general gist of it is that my PKM is roughly PARA-shaped, and the most important is the P (projects).

Each project generally has one or two agent sessions that are specific to it and know the context from files in its directory. These agents don't do any work besides creating tasks (though I generally prefer to do this myself) and folding the outcomes of those tasks back into its knowledge.

I then have a 4-column Kanban (based on the Obsidian plugin for Kanbans in markdown) with a backlog, in progress, under review, and done, although I don't care about what's in "done", so it's just a little area that we can drag things into for posterity, but they don't show. They're just a record of what was done (the agents update the card as they go along).

On the right is my normal agent chat panel, which I've tweaked and changed over time, but will talk about more another time. The Kanban struck a good balance for me, without an all out Linear-like setup (mainly because I rarely collaborate with humans that way anymore). I plan things out with a wide range of tickets that fit long-lived parallelisable work in the backlog, move them over to in progress (or the main agent does it -- everything is accessible through the con CLI) and that's when the magic happens: the head agent automatically gets forked and given all the context, unless I assign a specific agent. Depending on the task, I usually decide the model for that card (though I rarely go below Sonnet 5).

It will get prodded every once in a while by software, e.g. if a ticket has been in progress for longer than half an hour, or if the card has changed, and it can mark the card blocked. Theoretically they can still talk to other agents if they need to by using the con agent chat command which in the background will create a temporary fork, or con agent send which injects a message directly into a running session. In reality however, they seem to now prefer simply writing into each other's cards. I also prefer this tbh.

There's also a nofork tag (forking is opt-out) and for some projects we don't need a complicated setup with worktrees, like the Console project. For other projects I definitely do, but every agent that runs under Console knows exactly how to use the CLI tools on my machine to manage worktrees.

Then when the work is done and under review, I'll take a look. If there are links in the card, I can click them without opening the card details. There aren't many details except a git-commit like structure where in a text box the first line is the "title" and the rest the description, then the tags and controls. And of course live dictation works everywhere.

The links are quite useful as some projects will create preview builds on Vercel, so I can check the builds for real. Then, sometimes I'll have some follow up conversation with the agents directly, especially when it comes to UI where I need it to make small changes to margins etc.

Finally, when I'm happy, I move the card to done and it gets prompted to clean itself up (worktree etc) and the parent gets a final report so it remains up to date on what went on, but in a very context-efficient manner.

This might all sound a little complicated, but it's really much simpler and more reliable than what I've tried before and/or seen others try. I've found myself getting a LOT more done, especially with Fable 5 which is pretty slow I find, as I can simply launch a dozen parallel tasks and switch between them much more easily (instead of what I used to do which is badly monitor two or three sessions and remember to check in on them between games of online chess).

A lot of issues I used to have with agents doing dumb stuff went away after these changes, and I also found myself much better able to manage a ton of agents. There is one downside however: my token usage went up significantly. Now this is not as painful as it could be, as I'm using AWS Bedrock on startup credits, but I got a warning about my usage:

I went from approx $10k per month across everything (which was still sort of ok as I have $100k for a year) to almost $2.5k per day. This beats my previous record when I tried Fast Mode for a day.

This is clearly not sustainable, and I would finish my credits next month like this. So I think I will need to instead go back to Claude Max, but perhaps I'll need to switch between several subscriptions as I tend to hit the limits quite fast.

That, or maybe it's time to invest in GPUs and run the Chinese models? I need to check the economics on doing that, but perhaps I can write off the electricity as a business expense. With "unlimited" tokens I would start some insane workloads. Things just for fun too like just have an agent think forever and see what happens. It would also be kind of cool to be able to do everything locally so long as the quality doesn't take too much of a hit...