When I was in uni, one of my engineering lecturers would grade us based on the quality of our lab books. He would insist that it is vital you write down everything you do such that if you were hit by a bus, a competent engineer could inherit your lab book and seamlessly pick up where you left off. He taught us the term "Bus Factor", which is the number of engineers that would have to disappear for a project to stall. When you work on a project solo, you're a single point of failure, so your lab book should act the vehicle for knowledge transfer. The higher the bus factor, the better!

Later, in my journey as a founder, I came across Reed Hastings' thoughts on team building at Netflix. To maintain a high talent density, the average talent should only go up with each addition to your team, which in turn attracts even more talented people who want to be surrounded by the best. Reed coined the "Keeper Test" where you ask yourself if you would fight to keep a team member leaving to go to another job. I instead consider this loosely equivalent to imagining how screwed you'd be if a member of your team got hit by a bus. If it would be a disaster, they pass the bar and should be retained. If not, they should be working elsewhere; you're a professional sports team, not a family.

I agree with both philosophies, but you can see how they would be difficult to reconcile. Do you build a resilient team, or one with zero redundancy? Surely a team of irreplaceable players is a fragile one?

As always the optimum lies somewhere in the middle. You do not want to build a culture where nobody can take a holiday without the company grinding to a halt. And similarly, you do not want to have fungible members on your team warming the bench.

I believe the way out if this conundrum is to have overlapping secondary skills. If your startup is small (<10 people) the magic number is Bus Factor 2. At any given point in time, two people should be able to perform the same role in your business. This doesn't mean hire like you're Noah, but rather make sure that every person on your team has a backup (in skill if not capacity). It's a continuous process; keep others filled in on what you're doing and write clear documentation that can fill them in more in your absence. If possible, whenever you name a project owner, also name their backup or shadow.

The backup might not be as good (e.g. maybe a back-end dev can do a bit of front-end if needed) but this at least means that tasks under a certain role won't be hard blocked the day the primary team member is unavailable. It's ok if it hurts to have your defender play goalie, that means the goalie passes the keeper test (I swear no pun intended!).

Hire T-shaped people in a startup as their generalism will allow your business to still run smoothly without a full complement temporarily, like a melted RAID array before you save it. But nobody should ever be twiddling their thumbs because they're focusing on their spike and not a spare wheel -- never have two goalies! Or worse: a goalie that's worse at being a goalie than your defender.

With that out of the way, time for some pontification! How does this relate to making customers deeply dependent on your product or service? Superficially, this sounds like a good thing: job security and building something people need! Notice I said "need" and not "want". On reflection, taken at face value, this leads to perverse incentives.

On a macro scale, this is how you get monopolies and highly inelastic goods/services. In the long term, this leads to technological stagnation and inefficiency, and in the short term businesses can engage in price gouging and other unethical practices. No competition, no innovation.

On a micro scale, I want to provide goods/services that people buy because it's truly their best option, not because they're forced to. Just as I wouldn't want to sell something because I had no other option and needed the money. I also want to have the option to cut a bad-fit client/customer loose without the moral weight of condemning them to bankruptcy. The power balance between buyer and seller should be such that a genuinely fair arrangement/price can be reached. The market usually does a good job of making things fair.

You can see how this relates to employees; you're still just exchanging money for services. If you cannot afford to lose your job and have no union, your employer can exploit you. If your company cannot survive without a key person, the power dynamic is reversed. Even if the relationship is amicable, imagine not being able to leave a struggling startup for greener pastures because you know that doing so would send it to the grave along with the livelihoods of the rest of the team. Or creating busywork to avoid firing your friends. Not a nice spot to be in, so try and keep the power balanced!

By giving customers an easy way to migrate off of your product/service, not only will they trust you more, but you'll know that they're staying with you for the right reasons and you're actually giving them something they want. If you try and lock them in with dirty tactics (e.g. a Walled Garden ecosystem, predatory contracts, high migration or knowledge transfer cost, etc) you will fail in the long term, because they'll jump ship the moment they can elope with a less toxic spouse. This is how incumbents get killed.

It's not enough to avoid these tactics, you also need to take steps to allow them to stop working with you if they need to. E.g. the ability to export their data in a standard format, or well-written documentation on what you've built for them.

As always, independence is a virtue not just for yourself, but to grant to others as well.

Twilio used to be a cool and trustworthy company. I remember when I was in uni, some CS students (I was not a CS student) built little SMS conversation trees like it was nothing, and suddenly SMS become something you could build things with as a hobby.

Over the past month, my view of Twilio has completely changed.

The attack

Ten days ago (Jan 19th) at around 7am UTC, I woke up to large charges to our business account from Twilio, as well as a series of auto-recharge emails and finally an account suspension email. These charges happened in the span of 3 minutes just before 5am UTC. My reaction at this point was confusion. We were part of Twilio's startup programme and I didn't expect any of our usage to surpass our startup credits at this stage.

I checked the Twilio dashboard and saw that there was a large influx of OTP verification requests from Myanmar numbers that were clearly automated. I could tell that they're automated because they came basically all at once, and mostly from the same IP address (in Palestine). At this point, I realised it was an attack. I could also see that this was some kind of app automation (rather than spamming the underlying API endpoint) as we were also getting app navigation events.

After we were suspended, the verifications failed, so the attack stopped. The attacker seemed to have manually tried a California IP after that some hours later, probably to check if they've been IP blocked, and it probably wasn't a physical phone (Android 7). Then they stopped.

I also saw that our account balance was more than £1.5k in the red (in addition to the charges to our bank account) and our account was suspended until we zero that balance. The timing could not have been worse as we were scheduled to have an important pitch to partners at a tier 1 VC firm. They could be trying the app out already and unable to get in as phone verification was confirmed broken.

Our response

We're on the lowest tier (as a startup) which means our support is limited to email. I immediately opened a ticket to inform Twilio that we were victims of a clear attack, and to ask Twilio for help in blocking these area codes, as we needed our account to be un-suspended ASAP. They took quite a long time to respond, so after some hours I went ahead and paid off the £1.5k balance in order for our account to be un-suspended, with the hope that they can refund us later.

I was scratching my head at what the possible motive of such an attack could be. I thought it must be denial of service, but couldn't think of a motive. We're not big enough for competitors to want to sabotage us, so I was expecting an email at any point from someone asking for bitcoin to stop attacking us, or a dodgy security company coming in and asking for money to prevent it. But Twilio sent an email saying that this is a case of toll fraud.

I recommend reading that article, but in essence, those numbers are premium numbers owned by the attacker, and every time Twilio sends them a verification SMS, they make money, and we foot the bill.

Twilio's response

Twilio seemed to follow a set playbook that they use for these situations. Their documentation names a set of countries as the one where toll fraud numbers most likely come from and recommend are blocked (I suppose it's easy to get premium numbers there): Bangladesh, Sri-Lanka, Myanmar, Pakistan, Uzbekistan, Azerbaijan, Kyrgyzstan, and Nigeria.

I immediately went and blocked those area codes from our side, though Twilio also automatically blocked all countries except the US and the UK anyway, so it didn't really make a difference. Also, the attacker tried again using Indonesian numbers after that, so clearly a blocklist like that is not enough. Later I went and one by one selectively allowed only countries we actually serve.

Beyond this, Twilio's response was to try and do everything to blame this on us. They wash their hands of the responsibility to secure their own APIs, and instead the onus is on us to implement our own unreasonable security measures.

I told a friend about this, and through that friend found out that this is actually a very common problem that people have been having with Twilio, because Twilio dropped the ball. Apparently, out of all of those cases, we got pretty lucky (some people lost 6 figures). For me, the main issues are:

• Why aren't risky countries blocked by default? Worse, why are all countries in the world allowed by default?
• Why isn't "Fraud Guard", one of the switches (free) that Twilio told us we should have turned on, already turned by default?
• I had set up an auto-recharge rule (charge to £20 if balance goes below £10) just in case the startup credits ever ran low. This backfired dramatically as the account kept auto-recharging in an infinite loop until we were suspended (the reason for the huge charges to our bank account). Why?!
• We were already using the default rate-limiting that applies to individual numbers (something like 5 verification requests every 10 minutes), and our server had some general global rate-limiting per-IP (this probably already protected us quite a bit from what could have been). How is it reasonable to expect your clients to put a global rate limit, across IPs and numbers, for specifically the endpoint that asks for Twilio verification? Just have the rate limit on your side maybe? It's not our responsibility to think of these kinds of things; that's why we're using third party provider!

Their email was incredibly patronising, like others have reported, and they acted like they're doing us a huge favour by blessing us with a partial refund in account credits (not even real money). But we need to explain to them first how we promise to be better and not do a silly mistake like this again!

The refund

Twilio tries to push you into agreeing not to dispute the bank charges (see the link above for why they do this). I refused to agree to this, and first wanted to know exactly how much they would refund us, and if they would refund us in real money, not account credits (they agreed to "prioritize" this).

They told us that their finance team is who decides the refund amount, based on the information we provide on how we'll do better and a breakdown of the charges. I told them exactly what we did to combat this, and what the charges were. We had lost a few hundred in startup credits, then just over £2k in real money.

Instead of telling me how much they would refund (remember, I still haven't agreed not to dispute the charges, which they "required" in order to issue a refund), they went ahead and refunded us £847 and some change immediately.

I believe this to be a ploy to try and prevent us from disputing the original charges, because if we dispute now, we would have more back than what they charged.

What now?

I sought some advice, with mixed opinions, but it seems quite clear that if we dispute these charges, at the very least it would mean that we can no longer use Twilio for SMS anymore (which I don't want to anyway). But, this means switching to a different provider before disputing.

It would be relatively easy to switch, as they all tend to work the same way anyway, but would still require:

• Researching other providers (that don't use Twilio in the backend)
• Reading their documentation
• Swapping out the libraries and a dozen or so lines of code
• Making sure we leave no room for another round of toll fraud
• Testing and deploying

This is not difficult, but time and effort that I don't have right now, as well as a distraction from our actual core product. I don't know if £1.1k is worth that "labour", or any extra stress that may come if Twilio decides to make a stink about this and pass us on to collections etc.

All I know is: Twilio, never again. I will advise people to not use Twilio for the rest of my life and longer depending on how that advice may spread and how long this article survives.

Yesterday evening I had a call with three founders looking for some advice on specific things. Something that came up was how to make a proper pitch deck. My advice is usually to go to Slidebean and check out the pitch decks of some well-known companies. There's a clear pattern to how these are structured, depending on who the target of the deck is.

But recently, a different founder sent me a pitch deck asking for feedback and he used a platform called Tome^[1], and his slides were pretty cool, and when viewed on that platform could even have little video bubbles where he explains the slide. At first I though this was a GPT-3-based slide generator (similar to ChatBA (formerly ChatBCG)) but it seems to be more than that and looks like it could be a great tool for putting together a pitch deck on a whim!

I'm letting day-nft.com expire. This was an experiment with 3 other people where we minted simple NFTs that each correspond to a different date going back something like 10 years. The technical part was relatively straightforward, but we realised that the whole thing is just one big hype game, and in order for it to succeed we would need to do things that we weren't comfortable with morally, so we abandoned the project. At that point I had already done some research and analysis on NFT marketplaces (which I intent to publish at some point) that helped me cement the current views I hold about this space.