Last March we (ACX and Manifold Markets) did a test run of an impact market, a novel way of running charitable grants. You can read the details at the links, but it’s basically a VC ecosystem for charity: profit-seeking investors fund promising projects and grantmakers buy credit for successes from the investors. To test it out, we promised at least $20,000 in retroactive grants for forecasting-related projects, and intrepid guinea-pig investors funded 18 projects they thought we might want to buy.

Over the past six months, founders have worked on their projects. Some collapsed, losing their investors all their money. Others flourished, shooting up in value far beyond investor predictions. We got five judges (including me) to assess the final value of each of the 18 projects. Their results mostly determine what I will be offering investors for their impact certificates (see caveats below). They are:

For more on what each of these are, see https://manifoldmarkets.notion.site/ACX-Minigrants-Sep-2023-Reports-5f049d731a664ef3a73a280aae38c335. Base Rate Times and AI Safety Treaty were "control group" entries not funded by this contest.

We’ll be buying back impact certs at the value on the MEDIAN column - so, for example, we’ll pay $300 for 100% of the certs for the Crystal Ballin’ Podcast.

You might notice that every project lost its investors money except Project 4, which 25-tupled its original investment. I’m delighted that, armed with nothing but a spreadsheet and a cool idea, we’ve successfully recreated the rapacious winner-take-all nature of capitalism and applied it to the charitable sphere!

But this was a test run, and we noticed that some projects we thought were great failed to make back their original investment. And there’s nothing about capitalism that says you’re not allowed to overpay for a security - heck, in some parts of capitalism, like crypto, people rarely do anything else! So Austin and I have decided to outbid ourselves for two projects that we especially liked. Here are the final final results, changes in red:

Now three out of eighteen projects made their investors money on net. This rate (16%) is still lower than real VC investments (35%), but such is life in the cut-throat forecasting-related-charity industry. I warned everyone ahead of time that I would spend between $20K and $40K on this project. We ended up spending $31K. Investors still (collectively) chose to bet as if they valued these projects at $45K, so I don’t know what you were expecting. Maybe you were expecting to selflessly contribute to the forecasting ecosystem - in which case, good job.

Top Five Projects

…in fact, very good job! Some of these projects really impressed us. I want to highlight the five projects that our judges rated as most impactful, and which gave their investors some of the highest rates of return.

1: Max Morawski’s Rationality And Effective Altruism Education At University Of Maryland. Someone named Benjamin Cosman bought 100% of equity in this project at a valuation of $300. Our judges ended up valuing it at $7500, giving Benjamin a 25x RoI. Max planned to lead some workshops on rationality/EA ideas at the university where he taught, and expected to get about 10 students. Instead, he got 60, and keeps gaining more as time goes on. His students seem excited and have encouraged him to have an “advanced class” where he keeps teaching the students from last semester’s session, as well as continuing the “introductory track”. He says:

UMD is more of a party school than I thought, so I've been slowly carving out a social enclave of the nerdy students who aren't comfortable with that while introducing people to basic rationality / EA ideas. Which makes me think the niche I'm trying to occupy is more stable than I thought, because it serves an existing need.

This was a hard grant to value. One way to value it is something like: suppose that he keeps doing this x 3 years, and 5% of his students become long-term committed rationalists/EAs. That’s 9 new committed rationalist/EAs. Suppose half of those would have counterfactually found our community anyway; that’s about 4 new ones. Suppose of those four, one takes the GWWC pledge to donate 10% of lifetime income, and another goes into direct work and has a good career in some useful institution. Each of those people could plausibly generate $100K in charitable value. So maybe we should value this at $100K.

Thinking in these terms makes all education/movement-building grants vastly better than all other types of grants. That might be true. Or it might mean we’re overestimating commitment levels and benefits and failing to discount properly. Also, I discounted this because although I’m sure Max talks about forecasting in his classes, it seems marginal as a “forecasting” grant. Still, most judges valued this somewhere between $5,000 and $10,000, so Benjamin is getting 25x his money back.

My only regret with this grant is that Max undervalued himself, so all his money is going to his investor. Max, please get in touch with me at scott@slatestarcodex.com about having me retroactively recoup some of your other expenses if you want.

2: OPTIC Forecasting Tournament

This is like Science Olympiad or Academic Decathlon or other college tournaments in that class, but for forecasting.

Their pilot event last April attracted 39 individual competitors, but they’re already planning three more tournaments this fall.

Tyler Cowen likes to say that grants are as much about investing in people as in projects, so I also wanted to compliment the founders: Jingyi, Saul, and Tom, from (respectively) Brandeis, Brandeis, and Harvard. Saul has since gone on to be the lead organizer for Manifest, a forecasting conference with 200+ attendees and talks by Nate Silver, Robin Hanson, and many more. Jingyi is an officer of Brandeis Effective Altruism. And I remember Tom as being a top performer in the 22-and-under age bracket in last year’s Prediction Contest. Good luck to all of them in their remaining college and future endeavors.

Saul kept 22% of equity in OPTIC, and the remainder was bought by three investors, with Domenic Denicola holding the majority share. Domenic will turn his $3306 investment into $6072, and Saul will earn $2024 for his hard work (will he share with Jingyi and Tom? That's for him to decide, but cheating co-founders is a venerable startup tradition!)

3: Manifolio: Kelly Bets For Manifold

The Kelly Bet is a formula for calculating what percent of your portfolio you should invest in any given investment opportunity to optimally balance exploiting good opportunities with not going broke off bad ones. It’s achieved notoriety recently as the thing Sam Bankman-Fried refused to do, with predictable results.

William Howard created Manifolio, a “tool for making Kelly-optimal bets on Manifold”. This is a remarkably user-friendly, financial-ignoramus-friendly site; I was able to use it immediately.

For example, I have “life savings” of 44,000 mana, Manifold’s play money. Suppose I see this market on Manifold:

…and I think it’s mispriced: the real probability is 33%. I put my account name, the link to the market I’m interested in, and my probability into Manifolio:

And it returns the Kelly-optimal amount to bet (ℳ207). If I gave it my API key, it would even place the bet for me!

Judges agreed this was a well-made app that addressed a real need (there’s also a Chrome extension version). We would have evaluated it higher, except that’s for a play-money betting site with limited user base, and so far it hasn’t gotten much traction. I’m hoping that highlighting it here changes that. And Austin has mentioned the possibility of integrating it with Manifold itself at some point.

William kept 25% of equity, and six other shareholders (including Domenic Denicola and Manifold owner Austin Chen) paid a total of $4000 for the other 75%. William stands to earn $2300, and the investors stand to turn their $4000 investment into $6900.

Awkwardly, Austin Chen is an investor, judge, and final donor on this project, as well as a major beneficiary. I decided not to worry about this - partly because this is a test run, but mostly because when we add up all the different hats Austin is wearing, on net he’s still mostly donating money to other people on this one.

4: Superforecaster Predictions Of Long-Term Impacts

David Rhys-Bernard turned this into a paper, Forecasting Long-Run Causal Effects. He investigates "25,000 forecasts from 1,400 respondents" on "the effects of seven randomized experiments" and finds that (by some metrics) "expert forecasters outperform academics" and "better forecast calibration is what drives this", but also that "forecasters strongly overestimate the strength of the relationship between short and long-run outcomes".

I won't claim to have 100% read the paper, but I think it's valuable to get academics engaged with these topics, and the project is part of Rhys-Bernard’s PhD dissertation which he’ll be defending soon. David is a researcher at Rethink Priorities and helping him get his PhD will hopefully improve his ability to contribute to this excellent organization. (Possibly premature) congratulations, David!

Four people (including Domenic again?) invested $3000 in David's project. They’ll make back $2000 of that (sorry!). For some reason (maybe a software bug?) David kept 0.033% of the equity in his own project, meaning we owe him 66 cents. Hopefully this will help pay for grad school.

5: Devansh Mehta’s Impact Assessment Of Social Programs

Oh man, this one.

Devansh Mehta has some kind of cool idea. I cannot say with confidence exactly what the cool idea is. It seems to be using blockchain-based hypercertificates to create an impact market in investigative journalism in India - based in Auroville, a New Age holy city which has transcended money in favor of giant gold golf balls.

I appreciate Auroville's aesthetics and have written about it before at https://www.astralcodexten.com/p/model-city-monday-11821

Many people like Devansh’s idea. He was selected as a “Next Billion Fellow” by the Ethereum Foundation and has received previous grants from Gitcoin Grants and the Plurality Institute. He has clearly put a lot of work into this. He has some great videos and explainers talking about what he’s doing:

Still, four of the five judges, including me, recommended relatively low amounts of retroactive funding. Partly this is because the project isn’t finished. But partly it’s because we’re confused and dubious. We had a hard enough time starting this tiny unambitious impact market in the ACX/rat/EA community, where everyone is rich, plugged-in, and loves weird institutional design. Devansh wants to build a much more ambitious impact market, in India, for investigative journalism in particular. And he’s trying to run it on arcane blockchain technology which must take special programmer expertise and whose justification I find a little flimsy. I acknowledge the vast ambition of this project, its impressive backers, and the large amount of work that’s already been done. I just don’t feel like funding a bet at these odds, and three of the remaining four judges agreed with me.

But one of the four, Marcus Abramovitch - who earned his judge position by being Manifold’s top-ranked forecaster - recommended $50,000. This is more than twice as much as the rest of us assigned to all 18 projects combined; Marcus really believes in this. He wrote:

This is extremely impressive! I think it is <10% of what it could eventually be, but I think this has the potential to be worth millions of dollars if well executed - and thus far, it has been. Devansh has essentially created the first impact market where people can create impact certificates, provably have only one impact certificate per project (not totally sure about this) and have people/funders can come in and fund the resultant impact. Right now, this is just for small projects in India and the impact certs are often written in Indian dialects that are hard to translate and it doesn’t have a lot of funders coming in, but this is a fully fleshed out impact market ecosystem that seems to be working and could be grown out.

I think this will either be worth a lot (if it succeeds) or nothing at all, but if I were a charity venture capitalist, this project has been de-risked and gone from its seed stage to being ready for a series A investment and has become worth a hell of a lot more than when it started. I hope Devansh continues this and expands it further and wider.

Unfortunately for Devansh and his investors, we took the median recommendation of all five judges, which was $1,000.

I feel like we’ve failed Devansh here - and, more important, failed the concept of impact markets. We hoped impact certs would free us from having to play prophet, instead letting us reward good work at its full value after the fact. But Devansh’s project isn’t finished yet, so we had to judge it prospectively - with all our normal human tendencies to doubt bizarre and hyper-ambitious things.

So I will make Devansh and his main investor Carl Saldanha an offer: you can either take our current offer of $1,000 right now. Or you can keep your certs and try to sell them to me later, after this project has gone as far as it’s going to go. If it fails, I will pay you $0, and you will lose your otherwise-guaranteed $1,000. If it succeeds, I will pay you whatever I value its success at, up to a maximum of $10,000 (half the total budget we originally allotted to this round).

In any case, good luck!

In the interests of space, I’m only discussing the top five projects here, but you can find the others here. And you can read the judges’ recommendations and comments on each project here.

What Did We Learn About Impact Markets?

I made the project leads rate their success on a score of one to ten. If I had to do the same, I would give this impact market trial run a 4.

The good: We ran the market successfully! Nobody found a loophole that forced us to give them infinite money! Projects got funded! Some of them were good! So far, nobody has said they feel cheated by the results! The response to “can we do this at all?” is unequivocally yes!

The bad: Austin and I were skeptical of investors’ choices. Partly this is just our subjective judgment. But since the goal of an impact market is that it should fund good charities by the subjective judgment of the final oracular funders (us) - the fact that we overall disagreed with funding decisions is pretty damning.

For example, the best-funded project was “subsidize real money prediction markets on Polymarket”. The project worked, it was fine, nothing went wrong. But the judges (including me) had trouble figuring out why this was better than the many other real money prediction markets that happen on Polymarket all the time, and felt like we already had good data on how real-money markets compare to play-money ones. This was something we didn’t really want, but our investors spent $8,000 on it.

On the other hand, nobody invested in the Base Rate Times, a newspaper-styled prediction market dashboard. But project leader Marcel pursued it anyway, with his own money, and we ended up valuing it at $6,000 - if it had been in our market, it would have been the second-most-successful project out of 18. No mystery why people didn’t invest in it - Marcel valued it at $20,000, nobody else thought it was worth that much, and if they had invested they would have (at our $6,000 valuation) lost money. If this had been a normal grants round, Marcel could have asked for a grant, received only however much it cost to make it (up to $6,000), and had it gone well. It still sort of went well, since Marcel made it anyway, but I feel like this is another mark against our methods.

Investors spent $33,220 to produce what we judged as $21,125 - $31,125 in charitable value, making the market inefficient. I don’t know exactly how to think about this: I wouldn’t want us to pay the investors much more than they invested, or I’d feel like we were being ripped off. Still, it seems bad.

And this market was in many ways fake. Who would want a stock market where there was only one person who bought stocks, all stocks had to be sold exactly six months after buying them, and the final buyer could only spend $20K total no matter how well companies did? We set ourselves the least ambitious possible task, and we succeeded only by the least ambitious possible standard.

So the next step is to scale things up to something more real. And we need to find a way to do that makes it less negative-sum. My tentative plan for version 2.0 is to talk to a few serious charities and ask them to agree to consider “buy my impact certificate” a reasonable grant to make, at the same funding schedule as any other prospective grant they consider. I’d also like to do this myself via ACX Grants. Since investors can look at charities’ track records and see what they’ve previously funded, that will make them more confident in their ability to predict likely returns. I’ll talk more about this once I get this year’s ACX Grants set up (current ETA: late November).

While you’re waiting, here are some other interesting things you can do:

• Check out Manifund, which continues to raise money for interesting projects. There’s no functioning impact market for now, so you shouldn’t think of this as buying an impact certificate. But if you want to look at interesting projects and fund them out of the goodness of your heart, check them out.

• Bid on impact certs on the secondary market. If a would-be oracular funder wants to outbid me for one of the certs mentioned in this post, by all means, go for it! And if a cert-holder wants to hold onto their cert in the hopes that it will become worth more in the future, go for that too! Just remember this is a one-time offer to buy them on my part, and there may never be another functioning impact market.

• Enjoy the public goods we’ve produced. The Crystal Ballin’ Podcast has one episode and is hoping to make more (as are their competitors, the Market Manipulation Podcast). OPTIC is looking for participants and volunteers. You can still use Manifolio to make Kelly bets, the Telegram bot for Telegram-based prediction markets, and the browser extension to see what Manifold markets people are betting on. And although it’s not technically one of ours, I still like The Base Rate Times.

Thanks again to everyone who participated, whether as project leader, investor, or judge. And thanks especially to Austin Chen, Rachel Weinberg, and the rest of the Manifund and Manifold teams for the technology and organization that made this possible. If we owe you money, expect an email from Manifund sometime in the next two weeks. If you don’t get it, email me at scott@slatestarcodex.com.