Social vs. Science Experiments

Or, why progress curves in AI and web3 look different

Oct 10

Welcome to the 1,333 newly Not Boring people who have joined us since last Monday — I need to take breaks more often! If you haven’t subscribed, join 156,749 smart, curious folks by subscribing here:

🎧 For the audio version, click the play button or listen on Spotify or Apple Podcasts

Listen now · 18:06

Today’s Not Boring is brought to you by… Secureframe

This is How They Tell Me Secureframe Saves the World

Secureframe is the leading, all-in-one platform for security and privacy compliance. Secureframe makes it fast and easy to get and stay compliant so you can focus on what matters: Scaling your business, customers, and revenue.

With Secureframe, hypergrowth organizations can:

Get SOC 2 audit ready in weeks, not months
Stay compliant with the latest regulations
Scale compliance with your business to meet ISO 27001, GDPR, HIPAA, PCI, and other requirements

Plus, Secureframe helps sales teams respond to RFPs and security questionnaires quickly and easily with AI so they can close more deals, faster.

Click here to set up a demo. Mention “Not Boring” during your demo to get 20% off your first year of Secureframe. Promotion available through October 31, 2022.

Hi friends 👋,

Happy Monday! I hope you had a great weekend.

We talk a lot about progress here in Not Boring. Progress can be difficult and messy and winding, and importantly, progress moves at different speeds and follows different paths for different kinds of products. Today’s essay is my thinking on two types: science experiments and social experiments.

Successful science experiment products clearly move the world forward; social experiment products are less obviously beneficial and messier in the short-run, but I think they’re every bit as important in the long run, especially in combination with science experiment products.

This is a “thinking out loud post.” I would love to hear your thoughts and feedback.

Let’s get to it.

Science Experiments vs. Social Experiments

Some cutting edge tech products are the result of science experiments, others are the result of social experiments.

Science experiment products face a lot of technical risk early on, and getting a product to market can take a lot of time and capital. But when a product is finally objectively good enough, they can face less market risk. Launches are often preceded or accompanied by research papers. Think categories like AI, biotech, energy, chips, and space.

Social experiment products face less technical risk and can get to market in months instead of years, but once launched, face the very real risks associated with relying on people as key components of the product. These products enter the market partially formed, and evolve with their users. Think categories like web3, social media, and marketplaces.

This distinction is one of those ideas that feels too obvious to type now that I’ve typed it, but a lot starts to make sense to me when I look at the world through this framework. We often lump everything into the “technology” pile and that causes a bunch of confusion.

That confusion has been top of mind recently as crypto has dipped while AI has soared. Living through both waves, the progress curves feel very different.

For AI, a science experiment, there’s nothing, nothing, nothing… then BOOM. A fully formed butterfly emerges from the chrysalis. For web3, a social experiment, little flints spark adoption, and adoption shapes the products. Nothing, something, more, more, “probably nothing”… then KABOOM. Something unexpected breaks, and it’s back to the drawing board.

Social experiment products are forged in the public eye, with the inevitable ups and downs on display for the world to see. Science experiment products are forged in private, and enter public consciousness with most of the kinks worked out. Of course, the technology isn’t always perfect. Try generating an image with hands or eyes in DALL•E 2.

*“A man screaming into the void with his hands in the air,” DALL•E 2*

But we went from radio silence to “holy shit that’s magical” in the blink of an eye, and now that Stability AI has open sourced models like Stable Diffusion and OpenAI followed suit by open sourcing Whisper, it’s up only. There’s something new literally every day. Last week, Google’s DeepMind demonstrated an AI that can make algorithms for matrix multiplications. AI making AI.

It’s worth noting that many science experiments can fail once they hit the markets, either because they don’t actually work yet (see: Theranos), because they were too early (see: many, many failed science experiments), or because scaling experiments to commercial levels is difficult (see: hardware is hard). A certain product category may not be technically or economically feasible for a long time, even while researchers work in the expectation that when this cost curve reaches that point, or this input becomes cheaper, everything will snap into place. Some companies get a little overexcited, rushing to market before the science or economics are quite right.

But while individual efforts fail often, the experiments they run push the category forward, one technological innovation building on the next until someone’s product is good enough or the market is finally ready.

Zooming out, progress in science experiment categories like AI looks something like this:

Fitting that line looks something like an S-Curve: Slow Growth, Fast Growth (AI is here now), and then Slow Growth again. It’s a smooth line up that never dips: a new innovation sets a new technological floor that future products build from. Growth slows when all of the juice has been squeezed and a new technology trigger is needed to set off a new curve.

People working on science experiments have the benefit of the relative obscurity of the lab. They can tinker, mess up, keep tinkering, progress, take ten steps backwards, and have a breakthrough, all in the privacy of their own four walls. Obviously, private tinkering comes with its own challenges – getting funding might be harder, you can’t point to a million users, the problems are really hard to solve and dead-ends are frequent – but at the end of the day, the biggest question has an objective answer: did you build the thing or not?

People building social experiments have no such luxury. While technology plays an important role – blockchains, token standards, zero-knowledge proofs, and the like are all major technical innovations – people are a necessary component of the product itself. And people are messy, flighty, scammy, fickle, excitable, and all of the other wonderful things that make us people. Plus, most crypto products are money technologies too, and money enhances all of those human attributes.

That means that while people building social experiments can tinker with their tech in private, they can’t simulate the full product experience in a lab and release a perfect product with all of the kinks worked out. They need to release a product into the public lab that is the real-world and let the chips fall as they may.

Worse, they often need to build hype ahead of the product’s capabilities because they need to stand out and attract people in a crowded market so that the product can actually do what they think it might be able to do in the first place. Even with the best intentions, they’ll be called useless at best, a scam at worst.

Look at the early days of Twitter (“Why would I want to read what people are eating for lunch?”), all of the commentary around Clubhouse, a small startup whose app got so popular that thousands of grownups cheered its downfall, or practically any web3 product. All of the triumphs and defeats play out on the stage of public opinion, causing violent spikes up and down in entire categories.

This is not meant to excuse the actual scams as fun experiments. Au contraire, the actual scams are doubly-destructive, because they harm their own users and they cast a shadow over the well-intentioned projects. They make a hard thing harder. Fuck ‘em.

Zooming out, progress in social experiment categories like web3 looks something like this:

Fitting that line looks more like a rollercoaster that trends up but has some stomach-churning drops along the way. People rush in when things look good, and rush out when they don’t, which happens with any technology, but the products themselves get more useful when people rush in and less useful when they rush out. Attaching money and prices amplify this effect.

That social experiments will be messy and rocky is so obvious that AI already understands it, yet whenever that messiness and rockiness actually play out in reality, we’re surprised anew, wondering if this downturn spells the “end of X.”. When I fed a prompt for this essay into GPT-3 yesterday, days after I drew those curves, it accurately predicted the shape of the two curves (GPT-3 output in green):

GPT-3, the result of a science experiment, relies on the unwitting past participation of millions of people who have posted things to the internet that now feed the models. The single-player experience that any user has now that it’s out in the wild, however, is valuable whether anyone else uses it or not. In fact, if I were the only person in the world using GPT-3, it would be even more useful to me: y’all would think I was a super genius robot for churning out superhuman quantities of good material.

On the other hand, if a shadowy super coder built a technically perfect blockchain, one that solved the scalability trilemma, never went down, was easier to build on than web2 infrastructure, all of the good things, but I were the only person using it, it would be exactly useless to me.

Metcalfe’s Law, which states that the value of a network is proportional to the number of people using it, cuts both ways for social experiments that need to establish network effects. It’s really hard to get the right people to use the right product in the earliest stages, and most network businesses fail because of this Cold Start Problem.

That’s why hype is often a necessary ingredient to a social experiment product, and why progress in social experiment categories looks like a bunch of Hype Cycles strung together.

The y-axis on the Gartner Hype Cycle is Visibility, not Progress, but since social experiment products need users for the products to have any value, there’s a direct correlation between the two. There’s also an indirect relationship, as higher prices attract more people who build more products, as a16z describes in the Crypto Price-Innovation Cycle.

The relationship between Visibility and Progress in science experiment categories is almost entirely indirect, but it exists. Hype and Visibility attract more people and resources, which speeds up the pace of progress and often brings down costs, in a virtuous cycle. That’s the steep part of the S-Curve, and why actual progress in science experiment categories looks less like a bunch of Hype Cycles (progress doesn’t decline, just decelerates), and more like stacked S-Curves:

That’s not to say that Hype doesn’t impact public perception of science experiments. AI was famously always “ten years away” and “just linear regressions with a fancy name”... until it wasn’t. But AI’s success didn’t depend on getting a lot of people to use not-yet-good-enough chatbots; it depended on keeping a relatively small number of nerds excited enough by the progress to keep making progress for the sake of solving really hard and important problems.

One of the coolest things happening in the world right now is that a lot of science experiment categories are blooming simultaneously, often in concert with each other. In addition to AI, techbio, robotics, and renewable energy all seem to be jumping out of the lab and into the real world with products better than people would have expected and cost structures better than experts would have predicted. AI is genuinely useful and fun instead of frustrating. Moderna got a mRNA-based COVID vaccine to market in record time and saved millions of lives. Now, AI and biotech are combining in amazing ways, as exemplified by DeepMind’s AlphaFold and Atomic AI. Solar costs are plummeting and opening up new use cases and demand sources.

After a bunch of tedious, incredibly difficult work in relative privacy, these products are so obviously useful that adopting them is becoming a no-brainer. That’s one of the benefits of quietly doing science experiments in the lab before commercializing.

An observant reader might note that solar benefits from learning curves and Wright’s Law, and that improvements in cost and efficiency are a direct result of more solar panels being produced. So adoption is important. Adoption is important and necessary for any business once it becomes a business; it’s the whole point. Once science experiments come out of the lab and into the market, they often benefit from Scale Economies.

But adoption isn’t the product itself, as it is with social experiments. A billion people buying solar panels brings down costs faster, and makes it possible to get closer to the edge of physical limitations more quickly, but it doesn’t change the science. A billion people believing in Bitcoin directly makes Bitcoin stronger and more useful in ways that it wouldn’t be if only ten people believed in it.

Social experiments live and die by Network Effects, the prize for dealing with all of the human messiness upfront for the few social experiments that make it to the other side. Network effects can become so strong that it doesn’t matter how bad the bits a company produces are, people will stick around (ahem, Facebook).