Future of Discovery
20 July 2021 – Jacob O’Bryant

I finally finished the big refactoring for The Sample. Now its recommendations are generated by the same API server that’s been driving recommendations for Findka Essays for the past several weeks. I am extremely relieved. The new code base is much easier to work with, and scaling shouldn’t be an issue for a long time. My brother Marshall is taking over the API server (he’s a data science person), and after I finish cleaning up a few things, I’ll focus again on growth. Exciting times.

I also submitted our application to Y Combinator yesterday since it was their early deadline. I’ve been feeling a little meh about Y Combinator lately. This is my sixth time applying (second time applying with The Sample). Now that we have something working, I’m tempted to forget about YC and wait until we’re ready to hire a team before I worry about raising any money. But I guess if they help us get to that stage faster, it could be worth it. In any case it doesn’t hurt to apply, and then if they accept us we can figure out if we really want to do it or not.

The article this week is longer than usual but also more interesting than usual. It goes over various metrics for The Sample and how they interact. There’s a lot more to it than just daily active users!

The Sample Roadmap

View online

Last week I talked about my vision for Findka and how it looks like we may actually have a shot at realizing it. This week I’m writing up more details about how we’re planning to get there.

Later on we’ll branch out, but right now we need to turn The Sample, our first product, into an effective business. There are three essential pieces, all of which are in place. We just need to scale them up.

Recommendation quality

We need to be great at picking which newsletters to forward. There are a few ways we can measure this. I think the best one will probably be “# of 1-click subscribes per week.” If you subscribe to a newsletter, that seems like a pretty clear signal that you like it.

For the uninitiated: The Sample includes a “subscribe in 1 click” link with every newsletter we forward, similar to Amazon’s 1-click buy button. I get an email alert whenever someone clicks it, and then I fill out the signup form for them. I originally added this feature purely to increase newsletters’ subscribe conversion rate. But perhaps even more important, it gives us a way to measure how many subscriptions we generate. So I think this will be our north star metric.

In the past week, we had 469 1-click subscribes. The conversion rate for open email -> 1-click subscribe is 4%. So I’m happy about that.

Open rate and click-through rate (standard email metrics) are also useful, although thanks to Apple, open rate is going to get a lot noisier. Currently we’re at 57% and 26%, respectively. These are basically email-specific measures of retention. New signups measure how interesting the product looks, but retention measures how well it actually delivers. (Ask me how I learned that some time).

Finally, there’s the rating distribution: how many 5-star ratings do we get, 4-star ratings, etc? What’s the average? You might think this would be the most obvious metric for recommendation quality, since people tell us directly how good they think our recommendations are. However there are some complicating factors. First, you’re more likely to rate stuff that you don’t like than stuff that you do like. Our average rating is 2.7 stars, which seems kind of low—but only 13% of emails get rated. How many times do people like the newsletters they receive but don’t rate them? I have no idea.

Beyond that, a higher average rating is not necessarily better. It comes down to the fundamental explore-vs-exploit trade-off. People want stuff they’ll like, but they also want variety. If you run an A/B test and one arm results in a higher average rating, is that because that arm’s recommendations were better, or is that arm just being conservative? Conversely, the more you explore and try to introduce people to new kinds of things, the more likely you’ll be to get it wrong.

The best approach here might be to use 1-click subscribes, open rate, and click-through rate as an overall measure of quality, and then use ratings to balance exploration and exploitation. For example, look at the most active users and see what their average rating is. Say it’s 3 stars. Then if a particular user has an average rating of, say, 4.5 stars, we should give them a larger variety of recommendations, until their average rating is about 3 stars. And vice-versa. And of course, try all that in an A/B test to see if it actually increases 1-click subscribes.

That being said, ratings have one big advantage: they’re a per-recommendation metric instead of a per-user metric. Instead of assigning users to specific arms of an A/B test, you assign individual recommendations. Since each user gets many recommendations, the tests will reach statistical significance much faster. Since The Sample is still young, we’ll rely for now mainly on rating data to optimize the recommendation algorithm. Once we hit a certain size, we’ll start evaluating tests on the per-user metrics.

Growth via referrals

Again, as I wrote last week, our main growth strategy going forward is cross-promotion. If a newsletter author refers people to The Sample (say, by including a link in one of their emails), then we forward their newsletter more often in return. The metric here would be “new referred subscribers per week.” So far referrals have been about 5% of all our new subscribers. Now that we have a lot of subscribers, I want to scale this up and hopefully grow week-over-week via referrals.

There are several parts to this. One is cold outreach: I ask newsletter authors if they want to cross-promote with The Sample. CrowdMagnet has been handy here. The nice thing about The Sample is that it can cross-promote with any newsletter; it’s not restricted to a certain topic.

We also get inbound cross-promotion, because whenever someone submits their newsletter, I send them a referral link. We got about 30 - 40 submissions from the signup spike last week. So when possible I’ll try to cross-promote with people who have newsletter authors in their audience. I’ll also try to find more ways to reach authors. For example, I’m thinking of creating (yet another) public directory for newsletters, including links for advertising pages and info about cross-promotion. Like a combination of CrowdMagnet and Swapstack, except the purpose would be simply to increase exposure for The Sample among newsletter authors. In any case, I’m going to spend a lot more time talking to people who have submitted newsletters already, and I’ll look for additional ways I can help them.

The next part is making sure we actually deliver on the cross-promotion. 15% of our recommendations are dedicated to cross-promotion. For these, instead of taking a particular user and guessing “which newsletter would they give the highest rating to?”, we take a particular newsletter and guess “which user would give this the highest rating?”

Currently we assign each newsletter author a certain amount of credit based on how many subscribers they refer to us and how active those subscribers are. Then we compare each author’s share of the credit with their share of the recommendations. For example, if an author has 5% of the credit, then 5% of the promoted recommendations should go to them. If they’ve had only 3% of the promoted recommendations so far, then we boost the probability of their newsletter being promoted until we reach equilibrium.

Ideally we would give everyone back at least as many subscribers as they get for us, but it is outside our control. If we tried to make credit proportional to 1-click subscribes instead of just number of forwards, then we’d give too many forwards to newsletters that have a low conversion rate (e.g. because they’re more niche then usual, or because they just aren’t very good). So we guarantee forwards, but not subscriptions.

But I do still monitor the number of subscriptions we get vs. the number of subscriptions we give. Total we’ve received 203 referral subscriptions and we’ve given back 196 1-click subscribes (counting only subscriptions for newsletters that have referred at least one subscriber to us). So the average payback rate is 97%. The median payback rate is 102%.

(I would’ve expected to have more/bigger outliers who got more subscriptions then they referred, which would make the average higher than the median. However it turns out there’s a bug somewhere, and there’s a newsletter that gave us 42 subscriptions (!) but hasn’t been forwarded at all. So I’ll be fixing that once I finish writing this…)

Depending on what the median payback rate is going forward, we’ll likely adjust the number of recommendations we dedicate to promotion (currently 15%, as I mentioned above). If we increase the promotion rate, it’ll increase the median payback rate in the short-term, but it’ll likely reduce recommendation quality, which is bad for retention. So for the referral system to work, our algorithm needs to be efficient enough so it can get a good payback rate without needing the promotion rate to be too high.

Finally, I need to take some of those metrics and put them on the subscribe page, instead of just having it say “get more subscribers.” That’ll impact the number of people who share referral links, and it’ll also impact how prominently people share those links. If we’re convincing, people will be more likely to e.g. share the link at the top of their email and on social media instead of just at the bottom of an email.

Advertising revenue

Finally, monetization. You can buy an ad for $15 and then we keep running it until it’s been clicked by at least 20 different people. We’ve made $90 from it so far. Our ad click-through rate is about 1%.

However, this isn’t a focus right now. Recommendation quality and referral growth are the top concerns. But at some point, we’ll optimize this system too. There are some obvious things we’ll do: increase ad click-through rate, get more people to buy ads, and switch to a bidding system instead of charging a flat rate per click.

Beyond that, we might try out an ad network for other newsletters. For example, if you buy an ad on The Sample, we’ll run it first ourselves—and then after we’ve collected some performance data, we’ll calculate which other participating newsletters might be a good fit and see if they want to run the ad too (in which case we’d take a cut).

From my own experience in buying ads in newsletters, finding newsletters to advertise in is a secondary problem. The main difficulty is that it’s really hard to know which newsletters will give you good ROI. So if we could abstract that away—just buy an ad in The Sample and we’ll deal with figuring out where to run it—I think that would be a big win for everyone involved.

There are other ways we could monetize besides ads. We could charge our users a paid subscription. However that would be a terrible idea because we have network effects. The recommendation quality and our ability to drive subscriptions for other newsletters are directly impacted by the number of subscribers we have, so advertising is a much better fit.

I’ve also thought about affiliate links for newsletters. These don’t exist as far as I’m aware, but theoretically, it might work if we could take a cut of subscription revenue that we generate for authors. For example, if we get 10 subscribers for an author and one of those upgrades to a $5/month plan, they could pay us either a one-time fee or an ongoing percentage of revenue from that subscriber. That might be an option later on, if we’re large enough to get some email providers to support it. Or we could make our own newsletter service and take a cut of paid subscriptions that way, like Substack.

However that puts us in the same predicament as ecommerce recommender systems: do you optimize for the buyer or the seller? The recommendations that make us the most money are not necessarily the same ones that will be most appreciated by the users. This is another reason that I really like advertising: it puts us under less pressure to dilute recommendation quality for the sake of revenue.

That’s it for The Sample. Next week I’ll go over the post-Sample plans.

View discussions