Log #life

#all #projects #meta #ai #life #entrepreneurship #dev #hacks #writing

This page is a feed of all my #life posts in reverse chronological order. You can subscribe to this feed in your favourite feed reader through the icon above. You can also get a weekly digest of all of my posts via email by subscribing here:


Amar Memoranda > Log (life)

Thoughts on interfaces, AI agents, and magic

In UI design, Skeuomorphism is where UI elements look like their physical counterparts. For example, a button might have highlights/shadows, you might adjust time through slot-machine-like dials, or hear a shutter sound when you take a photo. I quite like skeuomorphic design.

I pay special attention to icons. My younger sister is young enough to have never used a floppy disk and therefore only knows this symbol 💾 to mean "save" but not why. You see it everywhere: icons (like a phone handset), language (like an "inbox"), and other tools (like the dodge and burn tools in photo editors, which stem from physical retouching on film).

Sometimes, words have gone through several layers of this, where they're borrowed over and over again. For me, one area where I see this a lot is in networks. In the days of radio and analogue electronics, we got a lot of new words that were borrowed from other things that people were already familiar with. Once computer networks came along, suddenly "bandwidth" adopted a different meaning.

The key here is this idea of familiarity. When something is new, it needs to be rooted in something old, in order for people to be able to embrace it, let alone understand it. Once they do, only then do you see design trends cut the fat (for example, the shiny Web 2.0 style made way for the more flat design we have today). If a time traveller from 20 years ago were to visit, of course they would find modern design and affordances confusing.

Take this a step further however: what about the things that never had a physical counterpart or couldn't quite be connected to one? Well, it seems we latch on to the closest available concept or symbol! For example, what exactly is "the cloud"? It never substituted water vapour in the sky; it was something new. Why is this ☰ the symbol for a hamburger menu? Because it sort of looks like items in a menu. Not to mention, why did we call it a hamburger menu? Because the symbol sort of looks like a hamburger.[1]

Anyway, why do I bring all this up? Because I noticed new words and icons showing up in the AI space, as AI is becoming more ubiquitous. AI assistance built into tools are becoming "copilots". The symbol for "apply AI" is becoming magic sparkles that look a bit like this ✨. I find this very interesting -- people seem to not quite have a previous concept to connect AI to other than "magic", and the robot emoji might be a little too intimidating 🤖 (maybe I should change the Amarbot triggering reaction to sparkles instead).

A couple days ago, this was trending on HackerNews, and sparked some conversation in my circles. As you might know, I have some interest in this space. It seemed to have some overlap with gather.town, a 2D virtual environment for work. This category really took off during covid. This product in particular has some big name backers (though not a16z ironically enough).

This got me thinking... AI agents would truly be first-class citizens in environments like these. You would interact with them the same way you interact with a human colleague. You could tell them "go tell Bob to have the reports ready by 2pm" and the agent would walk over to Bob's virtual desk, and tell them using the same chat / voice interface that a human would use.

How would agents interact with the outside world? LLMs already have an understanding of human concepts baked in. Why hack a language model to execute code (Code Interpreter) when you could use the same skeuomorphism that humans are good at, in an environment like this? If there's a big red button in the corner of your virtual office called "server restart button", a human as well as an AI agent can easily interact with that. Neither may ever know that this magic button does something in a parallel universe.

It might be some ways off before we're all working out of the metaverse, but I believe that the only way for that to happen is if it becomes more ergonomic than real life. It just so happens that this is great for humans as well as AI agents! There are already a class of tools that make you more productive in AR/VR than on a normal monitor (think 3D CAD). However when it comes to day-to-day working, organising your thoughts, communicating, etc, we still have some ways to go. To cross that bridge, we most likely need to embrace skeuomorphic design in the first instance.

What might that look like? Certainly storing information in space. Your desk top (and I don't mean "desktop", I mean literally the surface of your desk) can go 3D, and you can perhaps visualise directory trees in ways you couldn't before. Humans have excellent spatial reasoning (and memory) as my friend working on virtual mind palaces will tell you.

You could of course have physical objects map 1:1 to their virtual counterparts, e.g. you could see a server rack that represents your actual servers. However, instead of red and green dots on a dashboard, maybe the server can catch on literal fire if it's unhealthy? That's one way to receive information and monitor systems! A human as well as an AI agent can understand that fire is bad. Similarly, interactions with things can be physical, e.g. maybe you toss a book into a virtual basket, which orders a physical version of it. Maybe uploading a photo to the cloud is an actual photo flying up to a cloud?

Or maybe this virtual world becomes another layer for AI (think Black Mirror "White Christmas" episode), where humans only chat with a single representative that supervises all these virtual objects/agents, and talks in the human's ear? Humans dodge the metaverse apocalypse and can live in the real world like Humane wants?

Humans are social creatures and great at interacting with other humans. Sure, they can learn to drive a car, and no longer have to think about the individual actions, rather the intent, but nothing is more natural than conversation. LLMs are great at conversation too of course (it's in the name) and validates a belief that I've had for a long time that conversation may be the most widely applicable and ergonomic interaction interface.

What if my server was a person in my virtual workspace? A member of my team like any other? What if it cried if server health was bad? What if it explained to me what's wrong instead of me trawling through logs on the command line? I'm not sure what to call this. Is this reverse-skeuomorphism? Skeuomorphic datavis?

I might have a fleet of AI coworkers, each specialised in some way, or representing something. Already Sentinel is a personification of my smart home systems. Is this the beginning of an exocortex? Is there a day where I can simply utter my desires and an army of agents communicate with each other and interact with the world to make these a reality?

(Most) humans are great at reading faces (human faces that is, the same way Zebras can tell each other apart). This concept was explored in data visualisation before, via Chernoff faces. There are reasons why it didn't catch on but I find it very interesting. I was first introduced to this concept by the sci-fi novel Blindsight. In it, a vampire visualises statistical data through an array of tortured faces, as their brains in this story are excellent at seeing the nuance in that. You can read the whole novel for free online like other Peter Watts novels, but I'll leave the quote here for good measure:

A sea of tortured faces, rotating in slow orbits around my vampire commander.

"My God, what is this?"

"Statistics." Sarasti seemed focused on a flayed Asian child. "Rorschach's growth allometry over a two-week period."

"They're faces…"

He nodded, turning his attention to a woman with no eyes. "Skull diameter scales to total mass. Mandible length scales to EM transparency at one Angstrom. One hundred thirteen facial dimensions, each presenting a different variable. Principle-component combinations present as multifeature aspect ratios." He turned to face me, his naked gleaming eyes just slightly sidecast. "You'd be surprised how much gray matter is dedicated to the analysis of facial imagery. Shame to waste it on anything as—counterintuitive as residual plots or contingency tables."

I felt my jaw clenching. "And the expressions? What do they represent?"

"Software customizes output for user."

There are so many parallels between language and programming. For example, Toki Pona (a spoken language with a vocabulary of only 120 words) is like the RISC of linguistics. You need to compose more words together to convey the the same meaning, but it's quite elegant how you can still do that with so few words. It seems like languages don't need that large a vocabulary to be "Turing complete" and able to express any idea. Or maybe because language and thought are so tightly coupled, we're just not able to even conceive of ideas that we don't have the linguistic tools to express in the first place.

You can create subroutines, functions, macros in a program. You can reuse the same code at a higher level of abstraction. Similarly, we can invent new words and symbols that carry a lot more meaning, at the cost of making our language more terse. A language like Toki Pona is verbose because ideas are expressed from elementary building blocks and are context-dependent.

I imagine a day where abstractions layered on top of abstractions disconnect us from the underlying magic. You see a symbol like the Bluetooth icon and it has no other meaning to you except Bluetooth. In your virtual world, you interact with curious artefacts that have no bearing on your reality. You read arcane symbols as if they were ancient runes. You cast spells by speaking commands to underlings and ambient listeners that understand what you mean. Somewhere along the way, we can no longer explain how this has become a reality; how the effects we see actually connect to the ones and zeros firing. Is that not magic? ✨


  1. This is sometimes called a drawer menu too, but the point still stands, as it slides out like a drawer. Other forms of navigation have physical counterparts too, like "tabs" come from physical folders. One you start noticing these you can't stop! ↩︎

Aug 18, 2023 • #ai #life

Two new puzzles

I was sent two small puzzles by two separate friends recently. The first was from a friend who is currently visiting Gothenburg, Sweden, and spotted one of the programming recruitment puzzles. I might have seen this exact one through an ARG community, but couldn't quite remember.

We were chatting on the phone at the time, so I tried solving it in my head talking through it out loud, and got the right URL first try! If you can't be bothered to solve it, the URL takes you here, and as expected, it's a recruitment funnel.

Then, a more traditional puzzle:

After scratching my head for a bit, my solution was as follows (spoilers!):

  • Simplify the situation by instead assuming Zippy starts 400m away from the gate, and you start at the gate, and you're running towards each other (it's the same)
  • Zippy runs at 3x your speed, so by the time you meet, he will have covered 3x the distance as you
  • Chop the 400m into quarters, give Zippy 3 and yourself 1
  • 400/4 = 100m, so you meet when you're halfway into the park!
Jul 26, 2023 • #life

Impossible password requirements

Recently, there was this password game trending (and was also on the front page of HN). It goes quite deep, and I really like the variety of the challenges. I ragequit after my chicken died of overfeeding (you'll know what I mean when you reach it), especially because the game is different every time you play it, so you can't just copy-paste, but have to restart from scratch.

Anyway, coincidentally, as I was sorting through some old bookmarks, I came across this link: https://www.troyhunt.com/partnerships/. I was really confused as to why I bookmarked this. The automatic summariser summary was very innocuous.

troyhunt.com seeks partnerships for cross-platform solutions and value-added web services. To start the process, an account must be created.

Troy Hunt is the guy who made "Have I Been Pwned". My confusion ended once I actually tried to make an account. I remembered that he made this page to send scammers to and have them waste time trying to satiate impossible password requirements!

I mentioned many times that I like these sort of puzzles (and I still need to write about my own various attempts at similar ones), but some of these password ones are really quite clever and a great inspiration!

Jul 5, 2023 • #life

Bonsai diversification

My Red Maple and Wisteria seeds haven't sprouted yet, but I was left with all this extra soil! So I decided that I ought to plant the other species too. The remaining seeds I have are for Black Pine, Cherry Blossom, and Japanese Cedar. This is what they look like respectively:

I only had three Cherry Blossom seeds, and unlike the Red Maple, I decided to only plant one seed in that pot. Besides that, I've largely only used half of the seeds I have of each species so far, and I'm thinking that even that is unnecessary, but let's see!

As I was soaking them for 48 hours, they kind of got mixed up a bit, and I had a bit of a challenge separating the Black Pine from the Cherry Blossom, but I think I got there in the end. To better keep track of everything, as I was really starting to forget which is which, I put in some little wooden sticks:

The soil had dried quite a bit, so I made it wetter, maybe even a little too wet, as it was soaking the cotton on the bottom and created some condensation on the plastic. I also used tap water, which I didn't do for the first two, as it's pretty hard / rich in calcium. For my tomato plant, the effects of this were soon obvious as calcium residue was visible on the top of the soil and edges of the soil where it meets the pot. I didn't want to have the same for these plants, but it should hopefully all be OK.

If you'd like to learn some more about each species, here are their sections in my little book:

So now we have 5 different pots stratifying -- let's see which sprout first!

Apr 25, 2023 • #life #projectsParent project

Bonsai sowing and stratification

My bonsai seeds have soaked for 48 hours and I'm ready to move on to the next step! The reveal: I picked Wisteria and Red Maple. They ticked all the right boxes for me as my first try.

The Wisteria seeds are the small ones and the Red Maple are the two big ones. I used half of the seeds that I had of each species.

I assembled the "Auto Irrigation Growing Pot" and tried to ignore the conflicting instructions. I think you're not meant to fill the reservoir with any water at all until after the Stratification step (which I'll explain in a sec), and it's ambiguous how deep the seeds should go beyond "same depth as the size" (the size of what, the seeds?), so I just used my best judgement.

It turns out that I actually have a lot of soil. I didn't even use up a full peat disc so far. I have three more pots, so I'm considering getting some more seedlings started in the meantime and increase the chances of success...

At any rate, I sowed the current seeds and sprinkled a tiny bit of water into the soil to keep it moist, as it had dried out a bit in the meantime. I don't think the instructions should have the soil bit as step 1 if you're then going to soak the seeds for 48 hours after that, it should really be the second step.

And now that they're sown, I put them in the fridge. In the fridge, the one on the left is the Red Maple (this is more of a note to myself -- I should label them really; there are little wooden sticks for that in the kit). Putting them in the fridge is the first part of the Stratification step, which is meant to simulate winter conditions, then spring, so that they can germinate as they would in nature.

I'll be checking on them every few days and keeping the soil damp. Hopefully in two or three weeks they will start sprouting and I can remove them from the fridge. I set some calendar events. So now we wait!

Apr 12, 2023 • #life #projectsParent project

Banzai, bonsai!

I finally decided to start on my bonsai project. To read more about what this is all about, check out the project page. I haven't written anything about the tomato project, or any of the other (failed) horticulture projects, but I will eventually, since documenting failures is important too! This is the first log of what is probably going to be rather perennial chronicles.

The kit that I'm using to get a start with bonsai is really quite neat. It comes with 5 different species of seeds: Japanese Wisteria, Cherry Blossom, Japanese Cedar, Red Maple Tree, and Black Pine Tree.

This is a great set of tools in such a small package and I'm quite excited! The instruction booklet goes into a decent amount of detail, though I already know a bunch from YouTube and other places as I had a general interest in bonsai before deciding to try myself.

It came with two peat pucks that you put in some water and watch as they slowly grow while they absorb the water.

I decided to do both of them, as I wanted to try multiple species at the same time, and they grow to about 3x their original size! It's actually quite a lot of soil.

I then decided on two species that I wanted to grow. The next step was to put some of the seeds in warm water for 48 hours, such that they can soften, which makes it easier for the seedling to break through the shell. The two that I picked had seeds that looked very distinct from each other!

If you would like to know what species I picked, check back in 48 hours when I document the sowing process! I'll give you a hint: I didn't pick the mainstream choice (Black Pine).

Apr 9, 2023 • #projects #lifeParent project

Forums and data-hoarding

A few days ago, I wrote a post where I reminisced about the online forums of yesteryear. I mention tracking down and reaching out to a webmaster of a forum that meant a lot to me. Well, I have an update: she responded, and it was indeed her! I considered asking her if she might be open to digging up any backups she might have had, but then I thought about it, and I figured that perhaps some things are better left in the past. I don't remember the contents of those posts, but I do remember the positive emotions, and I think that's enough.

This leads me onto a topic that I've wanted to write about for a while: data-hoarding. I personally struggle with the concept of entropy in general. This manifests itself in many ways, but a clear one is information loss. If I were to leave this unchecked, I could see myself easily becoming a data-hoarder. The impulse is much stronger for unique data that I created (personal data), and in fact this is probably a strong motivator for my note-taking, as I see writing as a form of "backing up my brain". I've barely scratched the surface, but I reckon with enough data, I could even be resurrectable.

In my personal notes, I have a directory called "maxims", where I reason about a set of principles that I live my life by. There are however a certain set of mental tools that help me cope with life in general, but aren't quite at the level of certainty of a "maxim". I decided to start writing these down, and for now I'm putting them in a separate folder called "meditations" (kind of inspired by Marcus Aurelius' writing) until I come up with a better name.

The reason I bring this up is because there's a useful tool that helps me mitigate this urge to hoard personal data, namely picturing the notion that, for all we know, physics seems to indicate that our universe is time-reversible (except when you're dealing with black holes, but let's not get into that). In other words, if you know the end state of a system, and its evolution laws, you can simulate it backwards and determine a previous state, regardless of how chaotic it may be. Of course, it's one thing to calculate where a thrown ball originated from, another to un-burn a book, but physically it's all the same.

Similarly, relativity seems to indicate that if you travel at the speed of light, then all of time can exist at once, bringing up the concept of a Block Universe, and our perception of past and present is more of a side-effect of our mode of existence. To that end, I like to imagine that if certain data has existed at some point, then it is "stored", and theoretically retrievable, in the past, or the Akashic Records to use the term I learned from Ra. If not through "time-travel", then if someone were to take a perfect snapshot of our universe and simulate physics in reverse.

If you find this kind of thing interesting, I recommend Sabine Hossenfelder's book "Existential Physics", which she signed for me after a talk at the Royal Institute!

Before I get too carried away, let me write down one more story. When I was little, we had a Win 98 computer. I knew that machine inside and out. I remember all the games I used to play on it with my brothers (anyone remember the Worms games?) and I remember making little games with the old versions of PowerPoint. We made a mouse-maze game at one point, called "The A-maze-ing Maze", and in my head I can still hear the voice-over recording that I asked my brother to make for the instruction slide of the game, and his inflection of the word "maze".

I kept it in good working condition probably until I went off to university. Some people are amazed at how well-kept my projects from the early 2000s are, but that computer truly had even the earliest projects I ever worked on on it.

My mother didn't like that I always had electronics and hardware lying around in my room. She often threatened to throw my things out. I told her that this old computer was especially important to me, and I put it in my closet so that it's not in the way.

At some point, I probably came back home from uni to visit, and the computer was gone. My mother told me that she had it scrapped, and it was long gone at that point. I can't remember how I reacted, but I often remember the feeling, and I'm not sure if I'll ever get over that loss, as silly as that may sound. It truly felt like losing a part of myself.

I have a good relationship with my mother, and I've brought this up several times since then, but I don't think she quite understands what it meant to me, and she never apologised. Usually, she says that she assumed that I had already pulled out the hard drive, as I had a very tall stack of what she assumed were hard drives (they were actually CD-ROM drives).

Anyway, I'm not writing this to roast my mum! In fact, allow me to add one more anecdote (I lied, sorry) to offset the above story somewhat. One of my earliest memories of losing progress that hadn't saved was when playing the game "Amazon Trail" (a somewhat more modern spin-off of the well-known Oregon Trail). I made hours of progress on that game, and lost everything to a crash. My mother was the one who was there to comfort me as I cried.

I'm sharing this to put into words a different kind of loss, and a means of managing it. Like with the death of a person, you can imagine that they exist on in your memories. I like to think that the things I lose exist in a much more concrete way, in space-time itself, and that the loss was deterministically inevitable.

While that might not yet enable me to let go of certain losses, I can at least avoid obsessing over hoarding other data, and allowing certain things to be forgotten. Perhaps that can help someone else too!

Apr 6, 2023 • #life

Urban planner puzzle

Veronica recently sent me this puzzle:

In English: the grid is a city and you must place buildings in the cells. Buildings can have a height between 1 and 5 floors inclusive. Rows and columns have Sudoku rules; you can't have a building of the same height be on the same row/column. The numbers on the edges are how many buildings are visible from that vantage point.

So for example, for a row of 13254, the number on the right would be 2 (you can only see the buildings 4 and 5) and the number on the left would be 3 (you can only see 1, 3, and 5).

Give it a go then check against my solution!

My solution
24135
35241
52413
41352
13524
Apr 3, 2023 • #life

DreamViews birthday wishes

When I was much younger, after the internet had already picked up mainstream steam, but before social media, I spent a lot of time on online forums. The communities were small (and even if they were big, the "regulars" were a small community) and everyone knew everyone. Most of these no longer exist and the chances that I can reconnect with the friends I made are very slim, which is a shame as they've strongly influenced who I am today. I hear that similar communities exist on Mastadon and in some pockets of the internet but I already know it just won't be the same.

I'd get home from school, sit at the family computer, and check for new posts in threads I was part of. For the smaller ones, I would check all new posts on the entire forum and I engaged in a lot of discussions. I remember runescape-tip.com with waves of nostalgia as I just looked it up on the Wayback Machine (we're talking mid-2000s). I also remember the mutual support between friends on teenforum.tv, under the administration of the 21 year old webmaster and MySpace-layout-maker Nora, who seemed so old and wise at the time. Some friendships survived the forums' demise, but somewhere between transitions from MSN to Skype and beyond, it all fizzled away.

One forum that survived to this day however is dreamviews.com, although it looks quite different to how I remembered it. In typical vBulletin fashion, every year I get a birthday email from there, and every year I remember the friends I made on there. We obviously spoke a lot about lucid dreaming, a topic I was very interested in (though barely had any), but also a lot of off-topic stuff. Judging by the length of time the emails go back, I was active on there "only" as far back as 2010, which is later than the others.

I remember checking DreamViews one day, a long time ago, after I had already been inactive for quite some time (probably as I was at uni and busy with life) and I saw a post with one of the more veteran users on the site, reminiscing about all the "old" active users that had gone inactive. He listed usernames that I recognised, and mine was among them! That was the first time I considered that I too might have had an impact on all these people who had an impact on me, and that I wasn't just a random internet stranger to them. By the time I saw that post, the veteran user had also already moved on, so I decided to leave it there, and preserve these memories in my journaling as best I could.

I once tracked Nora down on a different platform, in 2014 (6 years after teenforum.tv died) and she had responded, according to my email notifications. I don't know what that response was anymore, because even that different platform (some kind of design community) is now dead too. Today, I did some sleuthing, and tracked her down on LinkedIn. I messaged her 5 minutes ago, and who knows, maybe she still has a backup of those forums and I can reminisce over the conversations? If anything comes of it, I'll be sure to post!

Apr 2, 2023 • #life

Statue inpainting

I can't imagine I'm the first to try this, but new hobby acquired:

  1. Go to the British Museum (other museums with statues will work too!)
  2. Find broken statues
  3. Take a photo
  4. Erase the gaps (DALL-E 2 lets you upload and edit on the fly)
  5. Write the name of the piece as the prompt, with the date
  6. Use image inpainting to fill in the rest of the statue

I ran the ones below on the spot and it was quite fun. Before this, whenever I visited the British Museum (a few times a year), I didn't really give most of those statues a second glance.

An exercise for the reader (this one's interesting because they put a reference of what it could have look like if it were complete based on a different statue):

And another bust of good old Caesar (might be interesting as there's so much reference material, and it's so broken):

Try it and have fun! I'll try another batch the next time I go.

Mar 16, 2023 • #ai #life

Cancelling digital minds

Recently, people whose work I admire made me have to confront the "art not artist" dilemma once more. In this case, Nick Bostrom with racism, and Justin Roiland with domestic abuse.

Thinking about it, more generally, I guess it comes down to:

  1. I can no longer consume a creator's work work without it being tainted by the context of their negative actions, so it became worse for me
  2. I do not want to a give a voice/reach to their views by making them more famous
  3. I do not want to do 2 indirectly by financially supporting them
  4. More selfishly, I do not want to be associated with them or for people to think I support their views/actions by supporting their work
  5. I do not wanting to signal to others that they can normalise these kinds of views, or behave a certain way, without consequences

However, it makes me think about the question: what if an AI were to be in a similar situation? Done something good and also done something bad. The current vibe seems to be that AI is a "tool" and "guns don't kill people, people kill people". But once you assign agency to AI, it starts opening up unexplored questions I think.

For example, what if you clone an AI state, one goes on to kill, the other goes on to save lives, in what way is the other liable? It's a bit like the entanglement experiment that won the 2022 Nobel physics prize -- you're entangling across space (two forks of a mind) vs time (old "good" version of a celebrity vs new "bad" version of a celebrity) where all versions are equally capable of bad in theory. To what extent are versions of people connected, and their potential?

It also reminds me of the sci-fi story Accelerando by Charles Stross (which I recommend, and you can read online for free here) where different forks of humans can be liable for debts incurred by their forks.

On a related note, I was recently reading a section in Existential Physics by Sabine Hossenfelder titled "Free Will and Morals". Forgive the awful photos, but give it a read:

So it doesn't even have to be AI. If someone is criminally insane, they are no longer agents responsible for their own actions, but rather chaotic systems to be managed, just like you don't "blame" the weather for being bad, or a small child for making mistakes.

Then, what if in a sufficiently advanced society we could simply alter our memories or reprogram criminal intent away? Are we killing the undesirable version? The main reasons for punishment are retribution, incapacitation, deterrence, and rehabilitation, but is there research out there that has really thought about how this applies to AI?

There's a fifth reason that applies only to AI: Roko's Basilisk (warning: infohazard) but it's all connected, as I wonder what majority beliefs we hold today that future cultures will find morally reprehensible. It might be things like consuming animals or the treatment of non-human intelligence that is equivalent to or greater than humans by some metric. At least we can say that racism and domestic violence are pretty obviously bad though.

Jan 29, 2023 • #ai #life

Do NOT use Twilio for SMS

Twilio used to be a cool and trustworthy company. I remember when I was in uni, some CS students (I was not a CS student) built little SMS conversation trees like it was nothing, and suddenly SMS become something you could build things with as a hobby.

Over the past month, my view of Twilio has completely changed.

The attack

Ten days ago (Jan 19th) at around 7am UTC, I woke up to large charges to our business account from Twilio, as well as a series of auto-recharge emails and finally an account suspension email. These charges happened in the span of 3 minutes just before 5am UTC. My reaction at this point was confusion. We were part of Twilio's startup programme and I didn't expect any of our usage to surpass our startup credits at this stage.

I checked the Twilio dashboard and saw that there was a large influx of OTP verification requests from Myanmar numbers that were clearly automated. I could tell that they're automated because they came basically all at once, and mostly from the same IP address (in Palestine). At this point, I realised it was an attack. I could also see that this was some kind of app automation (rather than spamming the underlying API endpoint) as we were also getting app navigation events.

After we were suspended, the verifications failed, so the attack stopped. The attacker seemed to have manually tried a California IP after that some hours later, probably to check if they've been IP blocked, and it probably wasn't a physical phone (Android 7). Then they stopped.

I also saw that our account balance was more than £1.5k in the red (in addition to the charges to our bank account) and our account was suspended until we zero that balance. The timing could not have been worse as we were scheduled to have an important pitch to partners at a tier 1 VC firm. They could be trying the app out already and unable to get in as phone verification was confirmed broken.

Our response

We're on the lowest tier (as a startup) which means our support is limited to email. I immediately opened a ticket to inform Twilio that we were victims of a clear attack, and to ask Twilio for help in blocking these area codes, as we needed our account to be un-suspended ASAP. They took quite a long time to respond, so after some hours I went ahead and paid off the £1.5k balance in order for our account to be un-suspended, with the hope that they can refund us later.

I was scratching my head at what the possible motive of such an attack could be. I thought it must be denial of service, but couldn't think of a motive. We're not big enough for competitors to want to sabotage us, so I was expecting an email at any point from someone asking for bitcoin to stop attacking us, or a dodgy security company coming in and asking for money to prevent it. But Twilio sent an email saying that this is a case of toll fraud.

I recommend reading that article, but in essence, those numbers are premium numbers owned by the attacker, and every time Twilio sends them a verification SMS, they make money, and we foot the bill.

Twilio's response

Twilio seemed to follow a set playbook that they use for these situations. Their documentation names a set of countries as the one where toll fraud numbers most likely come from and recommend are blocked (I suppose it's easy to get premium numbers there): Bangladesh, Sri-Lanka, Myanmar, Pakistan, Uzbekistan, Azerbaijan, Kyrgyzstan, and Nigeria.

I immediately went and blocked those area codes from our side, though Twilio also automatically blocked all countries except the US and the UK anyway, so it didn't really make a difference. Also, the attacker tried again using Indonesian numbers after that, so clearly a blocklist like that is not enough. Later I went and one by one selectively allowed only countries we actually serve.

Beyond this, Twilio's response was to try and do everything to blame this on us. They wash their hands of the responsibility to secure their own APIs, and instead the onus is on us to implement our own unreasonable security measures.

I told a friend about this, and through that friend found out that this is actually a very common problem that people have been having with Twilio, because Twilio dropped the ball. Apparently, out of all of those cases, we got pretty lucky (some people lost 6 figures). For me, the main issues are:

  • Why aren't risky countries blocked by default? Worse, why are all countries in the world allowed by default?
  • Why isn't "Fraud Guard", one of the switches (free) that Twilio told us we should have turned on, already turned by default?
  • I had set up an auto-recharge rule (charge to £20 if balance goes below £10) just in case the startup credits ever ran low. This backfired dramatically as the account kept auto-recharging in an infinite loop until we were suspended (the reason for the huge charges to our bank account). Why?!
  • We were already using the default rate-limiting that applies to individual numbers (something like 5 verification requests every 10 minutes), and our server had some general global rate-limiting per-IP (this probably already protected us quite a bit from what could have been). How is it reasonable to expect your clients to put a global rate limit, across IPs and numbers, for specifically the endpoint that asks for Twilio verification? Just have the rate limit on your side maybe? It's not our responsibility to think of these kinds of things; that's why we're using third party provider!

Their email was incredibly patronising, like others have reported, and they acted like they're doing us a huge favour by blessing us with a partial refund in account credits (not even real money). But we need to explain to them first how we promise to be better and not do a silly mistake like this again!

The refund

Twilio tries to push you into agreeing not to dispute the bank charges (see the link above for why they do this). I refused to agree to this, and first wanted to know exactly how much they would refund us, and if they would refund us in real money, not account credits (they agreed to "prioritize" this).

They told us that their finance team is who decides the refund amount, based on the information we provide on how we'll do better and a breakdown of the charges. I told them exactly what we did to combat this, and what the charges were. We had lost a few hundred in startup credits, then just over £2k in real money.

Instead of telling me how much they would refund (remember, I still haven't agreed not to dispute the charges, which they "required" in order to issue a refund), they went ahead and refunded us £847 and some change immediately.

I believe this to be a ploy to try and prevent us from disputing the original charges, because if we dispute now, we would have more back than what they charged.

What now?

I sought some advice, with mixed opinions, but it seems quite clear that if we dispute these charges, at the very least it would mean that we can no longer use Twilio for SMS anymore (which I don't want to anyway). But, this means switching to a different provider before disputing.

It would be relatively easy to switch, as they all tend to work the same way anyway, but would still require:

  • Researching other providers (that don't use Twilio in the backend)
  • Reading their documentation
  • Swapping out the libraries and a dozen or so lines of code
  • Making sure we leave no room for another round of toll fraud
  • Testing and deploying

This is not difficult, but time and effort that I don't have right now, as well as a distraction from our actual core product. I don't know if £1.1k is worth that "labour", or any extra stress that may come if Twilio decides to make a stink about this and pass us on to collections etc.

All I know is: Twilio, never again. I will advise people to not use Twilio for the rest of my life and longer depending on how that advice may spread and how long this article survives.

Jan 29, 2023 • #life #dev #entrepreneurship

New York Pass? Don't do nothin'!

My brother's in New York and I was reminded of a scam we fell for there once. This wasn't the typical Time's Square Elmo-league stuff, but seemed quite legitimate! I wanted to recount the story in case it might help someone.

We were planning to visit the Empire State building (which by the way, wasn't that great, especially that foggy day) and when we arrived there we were shocked to see a queue going all around the block and across several streets. We were approached by a man named DeShawn Cassidy selling the New York Pass.

"You can leave. Your Wallet. At home," he says. "You can laugh at aaaaall these people," as he points to the massive queue, telling us we can skip it with the glorious New York Pass. It's fast-lane entry and cheaper tickets into the Empire State building and a bunch of other attractions around New York within a certain time period.

He was a very convincing and charismatic salesman. We asked him why the people in the queue aren't cleaning him out if it's so good. He threw his hands up and said, "It behooves me!" misunderstanding what that word means.

We paid him $80 for 5 passes I believe, which was a great deal. He rubbed his hands like a fly about to have a meal as we were taking the money out, and gave us a receipt, staking his name and reputation on it, "DeShawn Cassidy", and that we can call him at any time if we need anything.

Of course, you know how the rest of the story goes. DeShawn was all but erased from existence, and we didn't have the opportunity to "laugh at all these people" as the security made us queue like everyone else. The special entrances were only for people who actually worked in the building.

We thought that maybe there's a faster queue inside, after clearing the building queue, and at least we don't need to get new tickets. Wrong again! The man at the till took one look at our little plastic cards, and in the strongest New York accent that still rings in my mind to this day, said the infamous words:

New York Pass? Don't do nothin'!

Jan 20, 2023 • #life

Digging in to my DNA

A while ago I dug into my DNA via a number of services. I had the uncommon opportunity of being able to compare the results of two services (while only really paying for one). Now I finally got around to writing this up and might update it over time as I do more genealogy-related things. https://yousefamar.com/memo/notes/my/dna/

Jan 12, 2023 • #life

Marshmallow test puzzle

My friend Selvan sent me this puzzle:

Feel free to give it a try before revealing my thought process and solution! Also, in case you're wondering, the sticks do have to have marshmallows on both ends, and they're straight, and marshmallows can't be in the same position or at infinity. Also, the sticks can cross (this doesn't violate the "2D" requirement). None of this was obvious to me!

My solution

At first, I looked at this as a graph. The graph is undirected and the vertices unlabelled. There are two possible edge weights, and the graph is not allowed to violate the triangle inequality. Intuitively, whenever edge weights are involved, I think of force-directed graphs (like a spring system with different length springs) that relax into a configuration where there's no tension in the springs.

Anyway, if you think about it as a graph, you'll realise that topologically, the first configuration is exactly the same as a square with an X in it. In fact, it's not possible for any other configuration to exist, as a graph with 4 vertices and 6 edges is completely connected. This means that we can't play around with topology, only the edge weights (or rather, move the vertices around, if you think of it that way).

There is no alternative layout where a fourth vertex is inside a triangle like the example, so the vertices *must* be in a quadrilateral layout. If you then build a trapezium using three long sticks and one short stick, you'll quickly see that there's a layout at which the shorter ones are all the same length. I made a visualisation to help illustrate this:

Afterwards, Selvan prompted me to realise that the distance between the bottom left corner and the point of intersection in the middle of the X should be the same as the red line distance, answering at which point exactly the vertices along the red lines are equidistant from each other!

Dec 21, 2022 • #life

Pringle overload

Almost exactly 6 years ago, I ate too many Pringles, as reminded by my photo app throwback. My brother won a contest where the prize was crates of Pringles and he gave me all the sour cream and onion ones. I ate too many of them in too short a time and since then I kind of lost my taste for them. The same thing happened to me with peanuts — I used to love them and now I basically never eat them.

Oct 21, 2022 • #life

Oyster vs contactless

When I was a student, I got an oyster photocard for commuting with a discount. Eventually I also had my railcard added to this (though IIRC, the discounts aren't cumulative). I had it renewed right at the last possible moment before expiry and aging out, and the new card was meant to expire on the 31st of Jan 2020. It never did and I've been using it since — maybe expiry meant the discount?

Eventually the outermost plastic layers peeled off (the layer with my name and photo on it) leaving an ominous blank card.

The card number was also peeled off, so when I had an incomplete trip one day, while getting that sorted, a friendly TFL employee let me know what it was on a receipt of my past few journeys. Only then did I really think about what the point of using an oyster card is (since I'm not getting discounts anymore) over a contactless credit card.

It seems there isn't really much of a benefit for me, so I'll probably just let it run out and stop using it. I might draw a little picture in that empty spot.

I had a normal oyster card many many years ago (before the first photocard) that I at some point added to the online dashboard with 60p still on it. I had given this oyster card to a homeless lady thinking there was more than that on it and she probably tossed it. I reckon if I plan my last trip in such a way that the balance goes to -60p, then never top it up again, then my overall balance with TFL should be... well, balanced!

Oct 20, 2022 • #life