Last time, I wrote about RISC-like tool use for my agent Al. As I'm rebuilding Al from scratch, I've been writing down everything he does so far (albeit not reliably), and everything I need him to be able to do. Everything in the lists below is realistic. For the most part, all he needs are a webhook endpoint and these CLI tools:

• Basic file tools (cat, grep, find, etc)
• curl for APIs and fetching URLs
• gog to access Google apps
• rr (roadrunner) to access the Beeper MCP
• Obsidian CLI to access PKM (not critical given the first point)

That's it! I'll add more over time I'm sure, but this is already a lot.

For now, I'm leaving out smart-home related things, as the tools for that are a bit too complicated, but you can imagine e.g. Al turns the TV on, turns the lights low, and queues up a movie after we decide on one from the movies list. For now, he only needs to do relatively few things with these tools, some of which are not even that critical. Let's expand a bit:

• The ability to communicate
  • Send/receive WhatsApp messages
    • Own phone number
  • Send/receive emails
    • Own email inbox with hooks for incoming email
  • Send/receive Slack messages
    • Own Slack app
  • Send/receive voice notes
    • Own OpenAI key
  • Make phone calls
    • Own Twilio key and number -- I tried other providers as I hate Twilio, but unfortunately Twilio is the best for this right now. Might use something else in the future.
• The ability to schedule tasks for the future
  • Own calendar to schedule events that trigger hooks
• A place to take notes
  • Own folder in my Obsidian vault (symlinked out)
  • Ability to store and modify own files
  • A place to log mistakes (via Obsidian CLI)
  • Own self-improvement Kanban (via Obsidian CLI)
• The ability to check RSS feeds (cURL)
  • https://kill-the-newsletter.com/ to turn newsletters into feeds
• The ability to keep tabs on me
  • The ability to monitor my own emails and Drive files (via gog CLI)
  • And messages (Beeper MCP via roadrunner CLI)
  • A way to check my GPS coordinates (and orientation?) on demand (via OwnTracks API)
• The ability to gather information from the internet
  • Fetch arbitrary URLs
  • Access to Google Maps API
• Ability to modify some of my own files (via Obsidian CLI)

So that's well and all, but what would it do with all those powers? These are relatively few powers, but you'd be surprised at how that's enough to seriously change how you operate. I can't wait for my Pebble Index to arrive as I'll be issuing commands via that primarily.

Here's a full breakdown of my use cases so far, organised into general categories.

Self-improvement

Al needs to be able to improve himself by maintaining and modifying his prompts and code. Al does not modify these directly, but by making suggestions.

• Watch for OpenClaw updates or use cases that may be interesting and proactively make feature suggestions to own project if it may be useful
• Regularly review own chat log and log mistakes
• Regularly look at list of mistakes, brainstorm and research ways to address, create improvement tasks
• Create new workflow files to learn new abilities over time after I've shown how it's done the first time

Helping with work

Al needs to be able to help with my startup Artanis, and in some cases communicate with my colleagues over Slack or whatever other medium.

• Maintain notes on Artanis in memory/artanis/
  • Keep tabs on our Monthly Update emails (there's an RSS feed)
  • Keep tabs on all relevant Google Docs/Sheets
  • Keep tabs on our calendars and Slack (especially the #planning channel)
  • Keep tabs on our Linear workspace
• Complete specific workflows
  • Create Gmail forwarding rules for invoice forwarding
  • Help me track down missing invoices for bookeeping, including asking my team
  • Schedule call review slots in my calendar when Sam sends a call recording
  • Turn long written agendas (e.g. offsite schedule, or an event) into calendar events in my calendar
  • Modify my working location on my work calendar so colleagues can know if I'm WFH or WFO or somewhere else on any given day
• The morning of when a Team Lunch is scheduled, sort out the logistics
  • Ask my team what they feel like having for lunch
  • If external guests are joining (it will say on the calendar event), ask them too
  • Find out if we're eating out or getting takeaway
  • If they suggest a cuisine rather than a restaurant, find an appropriate restaurant
    • Give the option between a past place (so remember what we eat) and a new place when found
    • If takeaway, find based on delivery time and if listed or not on either Uber Eats or Deliveroo
    • If eating out, find based on walking distance from office and rating
  • If takeaway, initiate a group order (both Uber Eats and Deliveroo support this) and send the link. Only add things to the order when explicitly asked. When everyone is done adding, check out.
  • If eating out, make the booking for the time on the calendar event

Filter through the noise

Al should act as a first line of defence against digital comms noise.

• Notify me of any important emails or messages when they come in
• Reply to emails on my behalf when asked
• Unsubscribe from emails on my behalf when asked
  • Where there's no way to unsubscribe, this includes replying with a request to be taken off their list, from the same email address
• Archive emails on my behalf when asked

Help me stay on top of things

Al should keep me up to date on what he's doing, what the world is doing, and what I should be doing.

• Send daily morning reports
  • Summary of schedule for the day (5 calendars)
  • Any emails/messages that I haven't responded to in longer than 2 working days
  • HackerNews front page picks, with links, based on my interests
  • Any open threads that might need my attention from our projects
  • Summary of overnight work or research
  • Summary of any conversations with others
  • Bookmarks of the day, which I can decide to keep or delete
• Send weekly report (Monday report is special)
  • In addition to the above, also check all DNS and server statuses
    • If ISP have rotated my IP, run workflow to update IPs
    • If a server is down, flag immediately
• When I flag a grocery list item that needs restocking, add it to the calendar event for ordering groceries
• Initiate weekly Sainsbury's grocery order every Sunday
  • Check if browser use is still authenticated (no API)
  • Check what time the delivery should be scheduled for in the calendar
  • Start an order, and if the time slot is not available, look for alternate times in the calendar, then ask me if it's ok to move it. Once we've found a time, make sure the calendar event is at the right time and book a delivery slot
  • Never order new items, only items from past orders or in Favourites. Follow the workflow file to understand how to use this UI
  • Add all the regular items to the order and the additional ones from the calendar event
  • Get it over the minimum charge and check out the order
  • Send a message to notify for amendments before the cut-off date

Communicate with others

Al should take some comms weight off me by responding to my contacts when they message directly.

• Gatekeep people asking to meet
  • Find out what the purpose/agenda is
  • Short-circuit / reject if appropriate
  • Determine if it can be done in messages instead and ping me
  • Help them find a virtual/physical time/place to meet if appropriate based on my availability and preferences
• Follow up with them when needed when they haven't responded
• Figure out information from them on my behalf when they're being too obtuse
• Allow my mother to find my general location in an emergency
• Immediately inform me of new conversations and odd requests
• Remember things about my contacts and practice information hygiene by only loading the person's file that is in conversation

Pay others

Al should pay others on my behalf, for personal payments. As this is a sensitive action, these use cases are not a subset of examples, but the total set of what Al can do here.

• Pay my driving instructor Jameel £67.50 the morning I have a driving lesson scheduled and text him to say I paid and when I'll meet him
• On receiving an email from Kate on scheduling changes, modify the standing order, then respond to her confirming the exact changed that was made and when to expect payments.
  • As this doesn't work via the Monzo API, Al should also just initiate the payments manually on the due dates in received invoices and pay exactly £45
• As Al is limited to only paying those exact two amounts to those exact recipients and a limited frequency, Al must notify me immediately if a payment fails, why, and suggest what needs to change (e.g. a limit increase)

Help manage my schedule

Al should help scheduling and rescheduling events in my calendar and use external tools to help.

• Book my gym classes the moment they become available via provided URLs. Right now, this means checking every Wednesday morning and booking classes for 2 weeks into the future. Where I have clashes, ask me what to do
• Schedule gardening tasks into my calendar based on seasons and inventory
• Put travel itineraries in my calendar with the correct timezones
• Help me reach physical locations that I need to be at
  • Finding e.g. the nearest pharmacy
  • Sending directions straight to my phone
  • Informing me of ETAs
  • Informing others of ETAs

Help manage my personal knowledge base

I have a lot of notes, journals, etc (this blog is part of it). Al should not ever edit anything I've written, or write anything on my behalf, however can help me a lot with organising these.

• Enrich and append items to my lists
  • Reading list
  • Video games list
  • 2x movie lists
  • Writing ideas
  • Blog drafts
  • Dream logs
  • Stray thoughts
  • Tagged bookmarks (one file per bookmark, unlike other lists)
• Help me work through those lists by
  • Making suggestion when I'm looking for something to read or watch, and updating the columns in those lists accordingly. When appropriate, asking for a review from me to add that to the list as well
  • Providing writing prompts, reminding me to finish a blog, or finding connections between thoughts/notes
• Surface old notes for me to decide if they should be kept, pruned, or recombined
• Surface old files that may be interesting to revisit or write about

Help me research, learn, grow

Al should gather information off of the internet and present it to me in an appropriate way depending on if it's for research to answer specific questions, or to learn new concepts.

• Analyse codebases
  • Clone an open source codebase into ~/src
  • Launch a Claude Code session to analyse stack, architecture etc
  • Produce a short report to be shared with me
  • If the codebase is too large, then Claude code should not clone and simply browse the remote repo (e.g. on GitHub)
  • Wait until I have no more questions before deleting the repo
• Help me plan tasks or software projects
  • Research libraries or tools for me
  • Launch a Claude Code session to plan a software project (architecture, functional decomposition, etc)
  • Produce Kanbans in the PKM
• Help me learn new things
  • Research a topic online and gather learning materials
  • Organise those materials into a course with scheduled blocks
  • Tutor me through those materials. Potentially create engaging podcasts timed to fit within e.g. train journeys
• Act as a therapist and coach
  • Analyse my notes and journals at a regular cadence, comparing them to my growth goals and progress
  • In timed blocks, act as a therapist to dissect that information, and a coach to help me towards my goals

More to come, stay tuned!

I've completely changed my digital right hand to use OpenClaw for most of February, and while it was quite fun (and sometimes dangerous -- ask my colleagues about the Monzo API), I quickly found the cracks. These were not related to security as people might think, but rather reliability. I started logging mistakes not long after the switch in a MISTAKES.md, and the vast majority are related to the agent ignoring my instructions. This was usually caused by spotty memory (which is exacerbated by the agent forgetting to remember things as instructed) as well as some deeper architectural issues.

Overall, I think this is mostly fixable, but I would have to just rebuild it from scratch, in a much simpler and more principled way, based on what I've learned. I'm a big believer in the future where single-user apps will proliferate, so I think it's more important to align on the right principles than implementation-specific things like what integrations you use.

1: Text is king

I've written about this before, so I'll summarise with the Unix Philosophy:

Write programs that do one thing and do it well. Write programs to work together. Write programs that handle text streams, because that is a universal interface.

An LLM's whole thing is text, so we should lean heavily into that.

2: Code is text

Therefore code is king too. Any abstraction over taking actions is unnecessary, beside the ability to use a CLI. General-purpose agents work better with a reduced instruction set. We don't need plugins and tools and MCPs etc (although it's great that the MCP hype is pushing people to make their products machine-accessible).

An agent with a CLI can use cURL to talk to your REST API. It can learn new CLI tools faster than you can, through a man page or --help. It can write scripts to solve hard tasks. Code is already a problem-solving language, and there was a hell of a lot of training data, so agents are really good at this. They can also write tests and fix bugs iteratively. Don't kneecap it by inventing some new protocol because you think you're making things better.

3: Use human products

OpenClaw did one thing very well, which is to have human chat apps as first-class integrations. You should not invent yet another web chat UI -- take the conversation to where humans already are and where they talk to other humans. Yes, WhatsApp doesn't have text streaming and markdown table rendering etc, but I suspect over time these spaces will be more agent-friendly.

However, OpenClaw did not go all the way. You should not use cron jobs for scheduling, you should use a calendar. You should not use a markdown file in some internal workspace directory to plan, you should use a Kanban board.

"But Yousef", you lament, "didn't you just say they should live in the CLI?". Yes, but they should use human products from the CLI. The products should be the same, but not necessarily the interfaces, even if over time interfaces are shifting towards conversation. They can be different for humans and agents. Agents can interact with these products via their API, not browser use. Some examples:

• My OpenClaw instance has its own Google Calendar, with software that fires hooks when a new event starts. I have access to this calendar too for visibility, but I can also easily modify it.
• I can see all the OpenClaw workspace files as they're symlinked out of my Obsidian vault. This means I get markdown rendering/editing as well as backups for free. I use an Obsidian plugin too that can render markdown files in a certain format as a Kanban board, which OpenClaw barely needed any information to know how to use.
• The future of building software is not in a Claude Code terminal, but in your project management software. I use Linear, which already supports assigning tickets to agents. You should talk to these agents in the Linear comments, like you would a human engineer, and you can review their work in PRs on GitHub, again just like you would a human engineer. Linear has an MCP server, but it's not needed, as my agents know the linearis CLI tool. They create new tickets for me through that all the time.

4: Policy beats memory

An agent's memory should just be the chat log. By all means store that, but you should never need to check it unless its relevant to the task at hand, just like a human will only search their chat history to get something, but it's a terrible way to stay organised or remember things. Chat should be ephemeral by nature.

Agents should act on anything important immediately and not rely on chat. This means remembering a new workflow for example. Agents should have their own permanent notes, just like a human may have a personal knowledge base. Just like humans, quality beats quantity, so the Digital Garden should remain curated. It's not a journal or a blog.

Both the agent and the human should maintain policy together. Policy can exist in many forms, but for a general-purpose agent, it's good to organise these so that they're pulled on demand. An example of this is Claude's Agent Skills where each skill contains detailed descriptions and other resources for a specialised workflow.

The crucial part is that the main prompt should inform the agent where these skills live, how to unpack one, and most importantly, under which conditions to unpack one. The goal is to solve scaling a sprawl of skills (say that 10 times fast). A single skill may be as thin as "here's a new command line tool that does X", it just doesn't need to be in context all the time.

Full disclosure: at my startup this is the exact problem we're solving. Getting policy (e.g. some niche knowledge about your domain) out of your head and into your agent, and keeping this internally consistent without ambiguity or contradiction. Every mistake that your agent makes is an opportunity for policy refinement.

There are other problems that need to be solved, e.g. access control: can you guardrail your agents in a guaranteed way (software) that is granular/flexible enough but still easy to use and doesn't ask you for a million permissions? These are all subsets of strong policy.

I'm starting to believe that anything additional to this (e.g. creating "planning modes" or orchestrator agents, or allowing the agent to spin up sub-agents, or even just compacting chat history instead of truncating) just gets in the way and often backfires. I suspect all these attempts at juicing agents will just become less and less useful over time as the models get better at a more foundational level.

It's been a while since I've written about the bot formerly known as Sentinel. It has continued to evolve, with the most significant change being a switch from Node-RED to n8n. Over the past couple weeks, there has been an even more fundamental shift however: a brain transplant. One that has prompted me to rename Sentinel to Al (that's Al with an L).

The fact that it looks like AI with an i is a coincidence -- it comes from the Arabic word for "machine", and I also decided to start referring to Al as "him" rather than "it" for convenience. Rather than an "assistant", I position Al as an digital extension of myself and the orchestrator of my exocortex^[1], while I'm his meatspace extension.

The big shift is that I've jumped on the OpenClaw bandwagon! People have mixed views on OpenClaw, but I can say that overall I find it quite exciting. I, like many people, have fully embraced the use of Claude Code, and have been trying to retrofit it to do more than simply build software. OpenClaw looks like the beginning of an ecosystem that allows us all to do just that without all rolling our own disparate versions.

OpenClaw embraces a concept that I find so fundamental as to be laughably obvious: the interface to chat bots should be existing chat apps. This is why Sentinel and Amarbot had their own phone numbers. While I have used Happy (and later HAPI, self-hosted on https://hapi.amar.io) in order to access my Claude Code sessions from mobile without the insanity of a mobile terminal, even those felt like anti-patterns. You can read more on why I think this is in my post on why we should interact with agents using the same tools we use for humans, as well as my post on agent chat interfaces in general.

For the record, I think these mode of interaction are inevitable, not a preference. While AI will interface with things via APIs, raw text, or whatever else, the bridge between AI and human will be the same tools as between human and human. This is also why I think the UIs purporting to be the "next phase" of agentic work where you manage parallel agents (opencode, conductor, etc) are the wrong path. The people who got this absolutely bang-on correct are Linear, with Linear for Agents, and I don't just say that because I love Linear (I do). I should assign tickets to agents the same way I would a human, and have discussions with them on Linear or Slack, the same way I would a human.

So, that brings me to what I think OpenClaw is currently bad at. First, cron should not be used as a trigger. Calendars should. Duh. One of the first things I did was give Al his own calendar and set up appropriate hooks. We need to be using the same tools, and I never use cron. This gives me a lot of visibility over what's going on that is much more natural, and I can move around and modify these events in the way they're supposed to be: through a calendar UI, not through natural language conversation.

On the topic of visibility, workspace files need to be easily viewable and editable. By this I mean all the various markdown files that form the agent's memory and instructions. While I may not need to edit these and Al can do that on his own, if we do want to explicitly edit, it should be easy to do so, and natural language conversation is not the way (this is not code)! To solve this, I've put Al's brain into my Obsidian vault (yes, the very vault from which these posts are published!) and symlinked it into the actual OpenClaw directory. So now I have can browse these files with a markdown editor heavily optimised for me! An added bonus of this is that Al's brain is now replicated across all my devices and backups for free, using Syncthing which already covers my vault.

It's still early days, so I'm still finding out the best ways to collaborate with Al. There are still a lot of issues, mainly related to Al forgetting things, that I'm working through, but it's been great! He can do everything that Sentinel already could, including talk to other people independently and update my lists. The timing is perfect, as I pre-ordered a Pebble Index 01^[2] and this will very likely be the primary way that I communicate with Al in the future when it comes to those one-way commands. For two-way, I still use my Even G1 smart glasses, but I suspect that I may go audio-only in the future (i.e. my Shokz OpenComm2 as I don't like having stuff in my ears).

I've scheduled some sessions with Al where we try and push each other to grow. For him, this means new capabilities and access to new things and various improvements. For me, this means learning about topics that Al has broken down into a guided course or literal coaching. I'll post more as I go along! In the meantime, you can also chat with him.

Sentinel, my AI personal assistant has evolved a bit since I last wrote about him. I realised I hadn't written about that project in a while when it came up in conversation and the latest reference I had was from ages ago. The node-red logic looks like this now:

1. Every morning he sends me a message (in the future this will be a status report summary). The goal of this was to mainly make sure the WhatsApp integration still works, since at the time it would crap out every once in a while and I wouldn't realise.
2. I have an endpoint for arbitrary messages, which is simply a URL with a text GET parameter. I've sprinkled these around various projects, as it helps to have certain kind of monitoring straight to my chats.
3. He's plugged in to GPT-4-turbo now, so I usually just ask him questions instead of going all the way to ChatGPT. He can remember previous messages, until I explicitly ask him to forget. This is the equivalent of "New Chat" on ChatGPT and is controlled with the functions API via natural language, like the list-adder function which I already had before ("add Harry Potter to my movies list").

As he's diverged from simply being an interface to my smart home stuff, as well as amarbot which is meant to replace me, I decided to start a new project log just for Sentinel-related posts.

Edit: this post inspired me to write more at length about chat as an interface here.

I mentioned recently that I've been using OpenAI's new functions API in the context of personal automation, which is something I've explored before without the API. The idea is that this tech can short-circuit going from a natural language command, to an actuation, with nothing else needed in the middle.

The natural language command can come from speech, or text chat, but almost universally, we're using conversation as an interface, which is probably the most natural medium for complex human interaction. I decided to use chat in the first instance.

Introducing: Sentinel, the custodian of Sanctum.

No longer does Sanctum process commands directly, but rather is under the purview of Sentinel. If I get early access to Lakera (the creators of Gandalf), he would also certainly make my setup far more secure than it currently is.

I repurposed the WhatsApp number that originally belonged to Amarbot. Why WhatsApp rather than Matrix? So others can more easily message him -- he's not just my direct assistant, but like a personal secretary too, so e.g. people can ask him for info if/when I'm busy. The downside is that he can't hang out with the other Matrix bots in my Neurodrome channel.

A set of WhatsApp nodes for Node-RED were recently published that behave similarly to the main Matrix bridge for WhatsApp, without all the extra Matrix stuff in the way, so I used that to connect Sentinel to my existing setup directly. The flow so far looks like this:

The two main branches are for messages that are either from me, or from others. When they're from others, their name and relationship to me are injected into the prompt (this is currently just a huge array that I hard-coded manually into the function node). When it's me, the prompt is given a set of functions that it can invoke.

If it decides that a function should be invoked, the switchResponse node redirects the message to the right place. So far, there are only three possible outcomes: (1) doing nothing, (2) adding information to a list, and (3) responding normally like ChatGPT. I therefore sometimes use Sentinel as a quicker way to ask ChatGPT one-shot questions.

The addToList function is defined like this:

{
	name: "addToList",
	description: "Adds a string to a list",
	parameters: {
		type: "object",
		properties: {
			text: {
				type: "string",
				description: "The item to add to a list",
			},
			listName: {
				type: "string",
				description: "The name of the list to which the item should be added",
				enum: [
					"movies",
					"books",
					"groceries",
				]
			},
		},
		required: ["text", "listName"],
	},
}

I don't actually have a groceries list, but for the other two (movies and books), my current workflow for noting down a movie to watch or a book to read is usually opening the Obsidian app on my phone and actually adding a bullet point to a text file note. This is hardly as smooth as texting Sentinel "Add Succession to my movies list". Of course, Sentinel is quite smart, so I could also say "I want to watch the first Harry Potter movie" and he responds "Added "Harry Potter and the Sorcerer's Stone" to the movies list!".

The actual code for adding these items to my lists is by literally appending a bullet point to their respective files (I have endpoints for this) which are synced to all my devices via the excellent Syncthing. In the future, I could probably make this fancier, e.g. query information about the movie/book and include a poster/cover and metadata, and also potentially publish these lists.