Everything Al does

Last time, I wrote about RISC-like tool use for my agent Al. As I'm rebuilding Al from scratch, I've been writing down everything he does so far (albeit not reliably), and everything I need him to be able to do. Everything in the lists below is realistic. For the most part, all he needs are a webhook endpoint and these CLI tools:

• Basic file tools (cat, grep, find, etc)
• curl for APIs and fetching URLs
• gog to access Google apps
• rr (roadrunner) to access the Beeper MCP
• Obsidian CLI to access PKM (not critical given the first point)

That's it! I'll add more over time I'm sure, but this is already a lot.

For now, I'm leaving out smart-home related things, as the tools for that are a bit too complicated, but you can imagine e.g. Al turns the TV on, turns the lights low, and queues up a movie after we decide on one from the movies list. For now, he only needs to do relatively few things with these tools, some of which are not even that critical. Let's expand a bit:

• The ability to communicate
  • Send/receive WhatsApp messages
    • Own phone number
  • Send/receive emails
    • Own email inbox with hooks for incoming email
  • Send/receive Slack messages
    • Own Slack app
  • Send/receive voice notes
    • Own OpenAI key
  • Make phone calls
    • Own Twilio key and number -- I tried other providers as I hate Twilio, but unfortunately Twilio is the best for this right now. Might use something else in the future.
• The ability to schedule tasks for the future
  • Own calendar to schedule events that trigger hooks
• A place to take notes
  • Own folder in my Obsidian vault (symlinked out)
  • Ability to store and modify own files
  • A place to log mistakes (via Obsidian CLI)
  • Own self-improvement Kanban (via Obsidian CLI)
• The ability to check RSS feeds (cURL)
  • https://kill-the-newsletter.com/ to turn newsletters into feeds
• The ability to keep tabs on me
  • The ability to monitor my own emails and Drive files (via gog CLI)
  • And messages (Beeper MCP via roadrunner CLI)
  • A way to check my GPS coordinates (and orientation?) on demand (via OwnTracks API)
• The ability to gather information from the internet
  • Fetch arbitrary URLs
  • Access to Google Maps API
• Ability to modify some of my own files (via Obsidian CLI)

So that's well and all, but what would it do with all those powers? These are relatively few powers, but you'd be surprised at how that's enough to seriously change how you operate. I can't wait for my Pebble Index to arrive as I'll be issuing commands via that primarily.

Here's a full breakdown of my use cases so far, organised into general categories.

Self-improvement

Al needs to be able to improve himself by maintaining and modifying his prompts and code. Al does not modify these directly, but by making suggestions.

• Watch for OpenClaw updates or use cases that may be interesting and proactively make feature suggestions to own project if it may be useful
• Regularly review own chat log and log mistakes
• Regularly look at list of mistakes, brainstorm and research ways to address, create improvement tasks
• Create new workflow files to learn new abilities over time after I've shown how it's done the first time

Helping with work

Al needs to be able to help with my startup Artanis, and in some cases communicate with my colleagues over Slack or whatever other medium.

• Maintain notes on Artanis in memory/artanis/
  • Keep tabs on our Monthly Update emails (there's an RSS feed)
  • Keep tabs on all relevant Google Docs/Sheets
  • Keep tabs on our calendars and Slack (especially the #planning channel)
  • Keep tabs on our Linear workspace
• Complete specific workflows
  • Create Gmail forwarding rules for invoice forwarding
  • Help me track down missing invoices for bookeeping, including asking my team
  • Schedule call review slots in my calendar when Sam sends a call recording
  • Turn long written agendas (e.g. offsite schedule, or an event) into calendar events in my calendar
  • Modify my working location on my work calendar so colleagues can know if I'm WFH or WFO or somewhere else on any given day
• The morning of when a Team Lunch is scheduled, sort out the logistics
  • Ask my team what they feel like having for lunch
  • If external guests are joining (it will say on the calendar event), ask them too
  • Find out if we're eating out or getting takeaway
  • If they suggest a cuisine rather than a restaurant, find an appropriate restaurant
    • Give the option between a past place (so remember what we eat) and a new place when found
    • If takeaway, find based on delivery time and if listed or not on either Uber Eats or Deliveroo
    • If eating out, find based on walking distance from office and rating
  • If takeaway, initiate a group order (both Uber Eats and Deliveroo support this) and send the link. Only add things to the order when explicitly asked. When everyone is done adding, check out.
  • If eating out, make the booking for the time on the calendar event

Filter through the noise

Al should act as a first line of defence against digital comms noise.

• Notify me of any important emails or messages when they come in
• Reply to emails on my behalf when asked
• Unsubscribe from emails on my behalf when asked
  • Where there's no way to unsubscribe, this includes replying with a request to be taken off their list, from the same email address
• Archive emails on my behalf when asked

Help me stay on top of things

Al should keep me up to date on what he's doing, what the world is doing, and what I should be doing.

• Send daily morning reports
  • Summary of schedule for the day (5 calendars)
  • Any emails/messages that I haven't responded to in longer than 2 working days
  • HackerNews front page picks, with links, based on my interests
  • Any open threads that might need my attention from our projects
  • Summary of overnight work or research
  • Summary of any conversations with others
  • Bookmarks of the day, which I can decide to keep or delete
• Send weekly report (Monday report is special)
  • In addition to the above, also check all DNS and server statuses
    • If ISP have rotated my IP, run workflow to update IPs
    • If a server is down, flag immediately
• When I flag a grocery list item that needs restocking, add it to the calendar event for ordering groceries
• Initiate weekly Sainsbury's grocery order every Sunday
  • Check if browser use is still authenticated (no API)
  • Check what time the delivery should be scheduled for in the calendar
  • Start an order, and if the time slot is not available, look for alternate times in the calendar, then ask me if it's ok to move it. Once we've found a time, make sure the calendar event is at the right time and book a delivery slot
  • Never order new items, only items from past orders or in Favourites. Follow the workflow file to understand how to use this UI
  • Add all the regular items to the order and the additional ones from the calendar event
  • Get it over the minimum charge and check out the order
  • Send a message to notify for amendments before the cut-off date

Communicate with others

Al should take some comms weight off me by responding to my contacts when they message directly.

• Gatekeep people asking to meet
  • Find out what the purpose/agenda is
  • Short-circuit / reject if appropriate
  • Determine if it can be done in messages instead and ping me
  • Help them find a virtual/physical time/place to meet if appropriate based on my availability and preferences
• Follow up with them when needed when they haven't responded
• Figure out information from them on my behalf when they're being too obtuse
• Allow my mother to find my general location in an emergency
• Immediately inform me of new conversations and odd requests
• Remember things about my contacts and practice information hygiene by only loading the person's file that is in conversation

Pay others

Al should pay others on my behalf, for personal payments. As this is a sensitive action, these use cases are not a subset of examples, but the total set of what Al can do here.

• Pay my driving instructor Jameel £67.50 the morning I have a driving lesson scheduled and text him to say I paid and when I'll meet him
• On receiving an email from Kate on scheduling changes, modify the standing order, then respond to her confirming the exact changed that was made and when to expect payments.
  • As this doesn't work via the Monzo API, Al should also just initiate the payments manually on the due dates in received invoices and pay exactly £45
• As Al is limited to only paying those exact two amounts to those exact recipients and a limited frequency, Al must notify me immediately if a payment fails, why, and suggest what needs to change (e.g. a limit increase)

Help manage my schedule

Al should help scheduling and rescheduling events in my calendar and use external tools to help.

• Book my gym classes the moment they become available via provided URLs. Right now, this means checking every Wednesday morning and booking classes for 2 weeks into the future. Where I have clashes, ask me what to do
• Schedule gardening tasks into my calendar based on seasons and inventory
• Put travel itineraries in my calendar with the correct timezones
• Help me reach physical locations that I need to be at
  • Finding e.g. the nearest pharmacy
  • Sending directions straight to my phone
  • Informing me of ETAs
  • Informing others of ETAs

Help manage my personal knowledge base

I have a lot of notes, journals, etc (this blog is part of it). Al should not ever edit anything I've written, or write anything on my behalf, however can help me a lot with organising these.

• Enrich and append items to my lists
  • Reading list
  • Video games list
  • 2x movie lists
  • Writing ideas
  • Blog drafts
  • Dream logs
  • Stray thoughts
  • Tagged bookmarks (one file per bookmark, unlike other lists)
• Help me work through those lists by
  • Making suggestion when I'm looking for something to read or watch, and updating the columns in those lists accordingly. When appropriate, asking for a review from me to add that to the list as well
  • Providing writing prompts, reminding me to finish a blog, or finding connections between thoughts/notes
• Surface old notes for me to decide if they should be kept, pruned, or recombined
• Surface old files that may be interesting to revisit or write about

Help me research, learn, grow

Al should gather information off of the internet and present it to me in an appropriate way depending on if it's for research to answer specific questions, or to learn new concepts.

• Analyse codebases
  • Clone an open source codebase into ~/src
  • Launch a Claude Code session to analyse stack, architecture etc
  • Produce a short report to be shared with me
  • If the codebase is too large, then Claude code should not clone and simply browse the remote repo (e.g. on GitHub)
  • Wait until I have no more questions before deleting the repo
• Help me plan tasks or software projects
  • Research libraries or tools for me
  • Launch a Claude Code session to plan a software project (architecture, functional decomposition, etc)
  • Produce Kanbans in the PKM
• Help me learn new things
  • Research a topic online and gather learning materials
  • Organise those materials into a course with scheduled blocks
  • Tutor me through those materials. Potentially create engaging podcasts timed to fit within e.g. train journeys
• Act as a therapist and coach
  • Analyse my notes and journals at a regular cadence, comparing them to my growth goals and progress
  • In timed blocks, act as a therapist to dissect that information, and a coach to help me towards my goals

More to come, stay tuned!