​

​

​Podcast, YouTube, Blog​

Dear founder,

​

This week, I ran into two very interesting instances of platform risk in my own software business. I will share what happened, what I did, and how I intend to combat them in the future.

​

And before I dive into the nitty gritty details, be aware that this episode might get a bit technical. Don’t worry, there’s something to be learned from this even if you don’t know what an AWS RDS blue/green deployment is.

Because the problem of platform risk is universal.

Every software business has to run on somebody else's computer. Every business, for that matter, depends on other businesses to function: suppliers, vendors, resellers — they all impact our success. And that is particularly true for software founders.

Few of us get to self-host our servers in our basements or in conveniently located data centers. Almost all parts of a software business stack are either run or maintained by somebody else, and Podscan, my own business, is no different.

Over the last couple of weeks, I ran into two very distinct but still pretty dangerous instances of platform risk. The first instance started two weeks ago when I found an email from Amazon Web Services (AWS) hidden in my to-be-read-later stash. I use hey.com for email, and it was in my transactional email stream, so I didn't see it for a couple of days.

I’m glad I found that email, because it warned me that AWS thought there was unauthorized access to my account and, if I didn't respond within five days, they might close the account.

I found the email three days in, so I still had some time to respond, but I was pretty upset. This email could have cost me my entire business because I run Podscan's database on AWS using their managed database service, RDS. All of my data, including backups, is on AWS, and they thought there was unauthorized access.

They needed me to respond quickly, or they would delete my account. With an extremely short ultimatum.

And of course, it happened at the worst possible time: just hours before I left for a long weekend vacation. I was faced with the possibility that within a couple of days, all my data could be restricted or, at worst, deleted. Now, had I been at home, I could have devised a backup extraction strategy, but try doing that from a moving train or the hotel wifi. We’re talking about just under a terabyte of data here.

But I didn’t panic — I just sighed and got to work.

I immediately reached out to AWS customer support to figure out what was going on. I looked into all the logs in my account and found no illegal access. But I did find what triggered their internal alert. I had set up a new server for my search engine, MeiliSearch, using an old AWS access key created when I started Podscan, half a year ago. It went unused then, and me suddenly using it to provision a beefy server caused AWS to decide this was unauthorized access and to restrict my account.

In a way, I really like their security approach: They restricted it immediately after suspecting tampering, causing a full password reset and limiting my account so I couldn't create any new instances or security keys before I’d talk to their security team. This led to an email back-and-forth over several days, with AWS telling me what they thought had happened and me proving it wasn't illicit access.

Throughout my vacation, I responded to their requests, and by the last day of my vacation, everything was resolved. I had to go through every single service and deployment on AWS and describe what it’s for, who accesses it, from where, and if it’s legitimate. Once they had all their needed, restrictions were lifted and all was back to normal.

But that could only happen because I found that email and acted swiftly.

This situation made me realize that my data is not entirely safe if AWS can just turn it off whenever they please. I need to back up my sizable data to other accounts and have a local copy.

So I’m putting systems in place to back up and restore my data from AWS without being entirely dependent on them. My database is critical to Podscan, so I need to protect it with external backup systems — think weekly dumps on my local computer as long as I can still manage the database size, and uploading into non-AWS backup locations.

The second instance of platform risk happened a couple of days ago, and in a way, is related to the cause of AWS’s false positive detection. The new server I created on AWS the day before AWS restricted my account started malfunctioning. I'm using MeiliSearch on there, a very fast search engine that allows sub-50 millisecond responses to search queries. It’s effectively a copy of my existing relational database, optimized for quick search retrieval. Now, judging from my performance logs, that server started malfunctioning ever so slightly over the last few days, and then it completely stopped working. It would just block requests for over two minutes, making both searching and keeping the index current impossible.

With search being an essential tool for my API and UI, I needed to fix it.

And that was quite the odyssey. I found that nobody else seemed to have this problem, except for one or two people on Stack Overflow. And the solution they found was to delete the search index and repopulate it from scratch, which, at my database size, takes almost half a day.

There was no way to repair a broken index, which this issue suggested was the cause of the problem. Meilisearch is a great piece of software, but it does occasionally run into issues at scale that it itself can’t reliably detect, warn about, or even report as an error. It just locked up on me.

So I went for the great reset. I had no alternative to this solution, which meant the system was performing subpar for a few hours, as a completely new index was created and slowly populated with data.

This made me realize that I’m quite dependent on this system as well, so while my index was growing, I started looking for solutions to take Meilisearch out of the picture or at the very least work with my existing database and have multiple search engines running simultaneously.

As I was waiting for things to catch up, I dove into PHPStorm and built a frontend search feature that works entirely in MySQL, utilizing a full-text index on the fields my customers search in, which is podcast names, episode names, and full transcripts of those shows.

Of course, this isn’t a quick thing to deploy. The code itself is just a few queries, but the index is another story — because my MySQL server does not yet have a full-text index on all the transcripts. There are about 6 million transcripts, each many thousands of words. Creating such an index takes a long time. Different databases handle index creation differently, and I learned that a full-text index in MySQL can only be done in shared mode, meaning there is a write lock on the table while the index is being built. Applications can read from it, but they can’t add new rows or update existing ones. This is problematic for an active application.

I tried creating the index, but it took five hours and still wasn’t done. And all of a sudden, my application went down because the backlog of writes overwhelmed the SQL server. I thought it could at least keep up with reads, but that was a bit too optimistic.

So I stopped the whole thing and my system quickly recovered. Then, after being interrupted by a power outage of all things (which is a whole other example of platform risk but at least gave me some time to mow the lawn and get my mind off things), I learned that there was a way to get my MySQL full-text index without slowing down the system: I needed to create a read replica, turn off the read-only protection, create the index on the replica, and then promote it to the read-write server, effectively switching out database connections of the fly.

I will have a full-text index on all my podcast episodes and transcripts soon. This will allow me to implement search more reliably without needing an extra search server. I’ll still use it, but not rely on it exclusively. I plan to build a backup system for quick search through MeiliSearch on my UI and API, with a fallback to my own database using MySQL’s full-text index.

Phew.

Building a software business is never without its surprises, and most of them are externalities you just can’t be fully prepared for.

All you can do is have the courage to face these challenges every single time they rear their ugly heads and build processes to prevent them from happening again.

This week taught me that I need to have reliable backup mechanisms to reduce platform risk. I’m working on implementing these solutions to ensure the critical functionality of Podscan can always work as long as it’s connected to a functional and well-maintained database.

If you have to work with platforms, you have to know the risks they pose.

I'll share a few updates about my SaaS on the pod, and I'd love to know what you think about them! Please leave a voice message at podline.fm/arvid 🥰

​

And if you want to track your brand mentions on podcasts, check out podscan.fm!

​

Classifieds

I recently launched The Bootstrapper's Bundle, which contains Zero to Sold, The Embedded Entrepreneur, and Find your Following. If you want to start a bootstrapped business and build a validated product and a personal platform while doing it, check out this bundle. It contains all eBooks, audiobooks, video courses and extra materials I ever created. It's just $50, for now.

​

​

Thank you for reading this week’s essay edition of The Bootstrapped Founder. Did you enjoy it? If so, please spread the word and ​share this issue on Twitter.​

If you want to reach tens of thousands of creators, makers, and dreamers, you can ​apply to sponsor ​an episode of this newsletter. Or just reply to this email!

​