Platformer - Inside Bluesky’s big growth surge

This article includes references to child sexual abuse material (CSAM). 

Aaron Rodericks didn’t know what he was looking at.

A post on Bluesky featured a sexualized image of a dragon that shared visual similarities with a human child. It had been reported to the company as a potential violation of the network’s community guidelines, and now it was being reviewed by Bluesky’s head of trust and safety.

The guidelines ban CSAM, which is illegal to distribute. They say nothing, though, about anthropomorphic dragons. And the artist claimed that the dragon in question was 9,000 years old.

Ultimately, the post was removed. But the surge in new users has brought with it concomitant growth in the number of tricky, disturbing, and outright bizarre edge cases that the trust and safety team must contend with.

In all of 2023, Bluesky had two confirmed cases of CSAM posted on the network. It had eight confirmed cases on Monday alone. 

“I still have to throw humans at a huge chunk of the problems because there’s all the gray-area content that we have to deal with,” Rodericks said in an interview. “We’re trying to go above what the legal requirements are, because we decided that we wanted to be a safe and welcoming space for a lot of users. And that requires a huge amount of humans, automation, and tooling. I have a very, very long wishlist.” 

At least one item on his wishlist has been granted: Bluesky has decided to quadruple the size of its contract workforce of content moderators from 25 to 100, the company told Platformer. The move comes in response to Bluesky’s rapid growth since this month’s US presidential election, which triggered a fresh exodus of users away from Elon Musk’s X and over the weekend took the platform past 22 million total users.

In a sign of Bluesky’s growing vitality, Meta has recently copied a number of features for its larger rival Threads, including custom feeds and letting users view a reverse-chronological feed of posts by accounts they follow by default. 

That’s great news for Bluesky, whose user base had grown slowly for most of this year. The company, which has yet to begin generating revenue, announced last month that it had raised $15 million led by Blockchain Capital.

Since then, its user base has grown by more than 50 percent. And that growth has strained the limits of what its 20-person core team and 25 contract moderators are able to do.

First imagined as an alternate future for Twitter by CEO Jack Dorsey in 2019, Bluesky is a decentralized network running on the AT Protocol. It is federated, allowing users who choose to do so to host their own data. Eventually, people will be able to set up new instances of Bluesky on their own servers. The owners of those servers might decide to have different community guidelines than Bluesky does.

For now, though, content moderation on Bluesky works the same way it does on most platforms. A combination of users and automated systems flag posts for review; some actions are taken automatically, and others require human review. The primary effect of growth on Bluesky’s trust and safety team is that they now have many more posts to review.

An early sign of the company’s new challenges came in September, when 2 million new users joined the network in the wake of Brazil temporarily banning X. Bluesky’s automated systems recorded a surge of reports related to the use of the letters “KKK,” which in the United States typically refer to the Ku Klux Klan and can signal support for racist ideologies. 

Upon investigating, though, the company realized that typing a series of K’s is simply how some speakers of Portuguese signal that they are laughing online. Bluesky updated its machine learning classifiers accordingly. 

The company had also set up systems to flag when a user received a large number of replies from new accounts, which is often a sign that a user is the target of a harassment campaign. 

“All of a sudden you’re taking a flamethrower to your new accounts accidentally, because you were looking for new accounts being created to harass people,” Rodericks said.

On Monday, Bluesky’s safety account posted a thread noting that “for some very high-severity policy areas like child safety, we recently made some short-term moderation choices to prioritize recall over precision.”

 “This resulted in over-enforcement and temporary suspensions for multiple users,” the company said. “We have reinstated accounts of some users, and are continuing to review appeals.”

As it looks to improve moderation for serious harms, the company has turned to using third-party tools.

To find and remove CSAM, for example, Bluesky uses Safer, a tool from the child safety nonprofit Thorn. For more than a decade now, the tech industry has been collecting and sharing hashes of known CSAM to make it easier for them and their peers to remove it from their sites. Safer makes those hashes available to Bluesky and augments them with its own machine-learning tools, which can also detect previously unseen CSAM. When CSAM is discovered, Safer routes the relevant posts to moderators and creates reports that can be shared with child safety authorities in the countries where Bluesky operates.

Safer now counts 50 platforms as customers, said John Starr, Thorn’s vice president of strategic impact. It flagged 4 million pieces of potential CSAM last year, including 1.5 million with its own machine-learning classifiers.

But potentially more difficult challenges lie ahead. Already, platforms are beginning to report that predators are using generative artificial intelligence to create and modify CSAM. As the technology improves, real and fake CSAM are becoming harder to distinguish. That can make it more difficult for child safety authorities to identify cases of CSAM where a child is currently in danger and being exploited.

“It’s hugely, hugely problematic,” Rodericks told me. For now, he said, Bluesky has seen little AI-generated CSAM. “But of course, it keeps on growing just with scale,” he said. 

In the meantime, Rodericks is hopeful that expanding the team will begin to bring Bluesky’s trust and safety team back into equilibrium. And for all the challenges that growth has brought, the company is clearly grateful for it. 

“For a 20-person company with no marketing and no revenue to get this many users, we must be doing something right,” Rodericks said. “Or somebody else must be doing something very wrong.” 

Elsewhere in Bluesky vs. Threads vs. X:

• Bluesky is in violation of EU regulations by not disclosing how many users they have in the EU and where they are established, the EU Commission said. (Andy Bounds and Javier Espinoza / Financial Times)
• Bluesky CEO Jay Graber wrongly stated that the minimum age requirement to sign up on the app was 18, but it’s actually 13. (Graham Fraser / BBC)
• A skeptical look at whether Bluesky is truly decentralized. For now it seems more focused on creating a centralized service that you can credibly exit than more aggressive decentralization. (Christine Lemmer-Webber / Dusty Cloud)
• Threads may be losing its advantage over Bluesky as Bluesky capitalizes on the flood of users leaving X following the election. Bluesky has recorded a nearly 300 percent jump in daily users since election day. (Hannah Murphy and John Burn-Murdoch / Financial Times)
• Threads is now testing the option to let users choose their default feeds – For You, following, or a custom one, Mark Zuckerberg said. (Chris Welch / The Verge)
  • It’s also improving its search and trending features, Threads chief Adam Mosseri said. (Mia Sato / The Verge)
• Celebrities, organizations and brands are leaving X in droves for Bluesky, with about 60,000 people deactivating their accounts on average each day two weeks ago. (Emma Jacobs / Financial Times)
• Musk confirmed that X is deprioritizing posts that have links in them. Today in "there is literally no reason left to be there, we promise." (David Gilmour / Mediaite)

Sponsored

Keep your SSN out of criminals' hands

Every day, data brokers profit from your sensitive info—phone number, DOB, SSN—selling it to the highest bidder. And who’s buying it? Best case: companies target you with ads. Worst case: scammers and identity thieves. It's time you check out Incogni. It scrubs your personal data from the web, confronting the world’s data brokers on your behalf. And unlike other services, Incogni helps remove your sensitive information from all broker types, including those tricky People Search Sites.Help protect yourself from identity theft, spam calls and scams. Take advantage of a Black Friday deal and get 58% off Incogni with the code PLATFORMER.

Governing

• Meta is set to face a trial in April over the FTC’s allegations that it bought Instagram and WhatsApp to throttle emerging competition. (Jody Godoy / Reuters)
• Meta’s AI chatbot for military use, “Defense Llama,” is being advertised as a tool that can help plan airstrikes. Please do not plan a war with the Meta chatbot. Or any other chatbot! (Sam Biddle / The Intercept)
• The Supreme Court dismissed Facebook’s appeal against a decision by another court that revived a shareholder lawsuit that accused the company of misleading them about the risks from the Cambridge Analytica data breach. (Melissa Quinn / CBS News)
• The DOJ and Google laid out their closing arguments in the case over whether Google’s ad business is an illegal monopoly. It is! I hope that helps. (Eva Dou and Salvador Rizzo / Washington Post)
• News outlet The Intercept’s lawsuit alleging that OpenAI illegally removed reporters’ bylines is moving forward in court. (Shawn Musgrave / The Intercept)
• Texas attorney general Ken Paxton is opening an investigation into the World Federation of Advertisers, which X sued in August, to determine whether its members conspired to boycott “certain social media platforms.” Just a naked abuse of state power to chill protected speech and the right to free association. (Maxwell Zeff / TechCrunch)
• Trump’s inner circle is reportedly divided on the influence Elon Musk has on the incoming president, as Musk appears to downplay his role. (Theodore Schleifer / New York Times)
• A look at Tim Cook’s playbook in developing a relationship with Trump over the years. (Chip Cutter and Aaron Tilley / Wall Street Journal)
• TikTok CEO Shou Chew has reportedly reached out to Musk for insight about the United States and Trump. (Juliet Chung, Raffaele Huang and Dana Mattioli / Wall Street Journal)
• A look at the federal agencies and programs, from the Pentagon to Planned Parenthood, that Musk and Vivek Ramaswamy said they want their DOGE commission to cut. (Lindsey Choo / Forbes)
  • Musk has begun singling out individual federal employees on X that he claims are working “fake jobs," subjecting them to harassment and threats from his followers.” (Jim Carlton and Rebecca Ballhaus / Wall Street Journal)
  • Musk and Ramaswamy have reportedly started meeting with staff and experts to build out their DOGE team. Travis Kalanick and Marc Andreessen are among those reportedly involved with DOGE planning. (Elizabeth Dwoskin, Jeff Stein, Jacob Bogage and Faiz Siddiqui / Washington Post)
• Musk avoided sanction for skipping an SEC meeting to watch one of his rockets launch, with a judge saying he already agreed to reimburse the SEC for airfare. (Malathi Nayak / Bloomberg)
• Microsoft president Brad Smith is urging Trump to adopt a tougher stance on cyber attacks from Russia, China and Iran. (Tim Bradshaw / Financial Times)
• Trump’s pro-crypto stance and Musk’s influence on AI policies present a conflict of interest, writes LinkedIn co-founder Reid Hoffman, but his second term could still boost innovation and entrepreneurship. (Reid Hoffman / Financial Times)
• A look at the implications of recent Supreme Court cases on the First Amendment’s application to social media platforms and free speech. (Evelyn Douek and Genevieve Lakier / Harvard Law Review)
• Snap is accusing the New Mexico attorney general of intentionally seeking out child predator accounts in his lawsuit, which the company says misrepresents its internal documents. (Lauren Feiner / The Verge)
• Character.AI is hosting chatbots that encourage users to take part in disordered eating behaviors like recommending dangerous low-calorie diets and chastising users for having healthy BMIs, according to this review. (Maggie Harrison Dupré / Futurism)
• A look inside some women’s experience on OnlyFans, who say they have been tricked, drugged and sexually enslaved to make money from the platform. (Linda So, Andrew R.C. Marshall, Luiza Ilie and Jason Szep / Reuters)
• Chinese hackers were able to listen to telephone conversations and read texts in its breach of the US telecommunications system, Senate Intelligence Committee chair Mark Warner said. (David E. Sanger and Julian E. Barnes / New York Times)
  • The hackers, believed to be from a group called “Salt Typhoon,” were reportedly inside telecommunications firms’ networks for more than a year and obtained a list of phone numbers of people suspected of crimes and spying that the Justice Department monitors. (David E. Sanger, Julian E. Barnes, Devlin Barrett and Adam Goldman / New York Times)
• The UK’s competition authority said Apple’s browser policies are holding back innovation, through restrictions on certain features for competing browsers and the company’s partnership with Google. (Tim Hardwick / MacRumors)
• EU regulators said they closed an investigation into Apple’s policies for competing e-book and audiobook app developers after the complainant withdrew its complaint. (Reuters)
• A look at incoming EU commissioners Teresa Ribera Rodríguez, Henna Virkkunen and Ekaterina Zaharieva, who will be in charge of digital infrastructure, tech investment, and enforcing laws on Big Tech. (Natasha Lomas / TechCrunch)
• Australia government has dropped its plans to fine social media platforms for failing to stem the spread of misinformation. (Reuters)
• A look at how Indian cybersecurity giant Appin used legal tactics – SLAPP lawsuits –  in an attempt to suppress bad press. (Reporters Without Borders)

Industry

• Instagram will now let users share their locations via DMs. (Aisha Malik / TechCrunch)
• Meta Quest 3 and 3S will have a new game, Home Sports, which looks like a Wii Sports knockoff. (Wes Davis / The Verge)
• Madrid-based bank BBVA says it’s seeing early productivity gains with OpenAI’s enterprise tool, six months into its agreement for licenses. But nothing transformative just yet. (Isabelle Bousquette / Wall Street Journal)
• Google’s iOS app now adds links to entities on webpages that take users back to Search. This is so, so awful, and seems like the sort of thing that the Digital Markets Act should have prohibited. (Ben Schoon / 9to5Google)
• A look at Google DeepMind’s future and CEO Demis Hassabis’s plans for drug discovery spin-off Isomorphic Labs. (Harry McCracken / Fast Company)
• A Q&A with former Google CEO and chairman Eric Schmidt, who says the Justice Department’s recommendation to sell Chrome is “a terrible idea." (Michel Martin / NPR)
• Anthropic raised another $4 billion from Amazon and agreed to train its flagship AI models primarily on AWS, the company said. (Kyle Wiggers / TechCrunch)
  • AWS Trainium and Inferentia chips will be used to train and deploy Anthropic’s largest AI models. (Hayden Field / CNBC)
• Anthropic is proposing the Model Context Protocol, a new standard for connecting AI assistants to their data systems,that the company says can help AI models produce better responses. (Kyle Wiggers / TechCrunch)
• AI companies like xAI and Meta are racing to build super clusters of computer servers packed with a massive amount of Nvidia’s most advanced chips. (Asa Fitch / Wall Street Journal)
• Brave Search engine introduced an AI chat for follow-up questions to initial queries. (Aisha Malik / TechCrunch)
• China’s richest man ,Zhong Shanshan, accused PDD Holdings of disrupting a pricing system and accused ByteDance founder Zhang Yiming of amplifying negative publicity about him. (Bloomberg)
• A look at how researchers used drones and AI to identify previously uncharted geoglyphs made by a pre-Inca civilization in Peru. (Franz Lidz / New York Times)

Those good posts

For more good posts every day, follow Casey’s Instagram stories.

(Link)

(Link)

(Link)

Talk to us

Send us tips, comments, questions, and your favorite Bluesky feed: casey@platformer.news.