Sponsorship

The Reshape

Stay up to date with AI 🤖 & Data Science 📊 by reading top-notch articles. We are the first to spot hot news (data proven!). At the same time, we scour the Internet to find the most overlooked publications. Get a package of both in a concise form delivered straight to your inbox for free 📬.

Not subscribed to Console? Subscribe now to get a list of new open-source projects curated by an Amazon engineer in your email every week.

Already subscribed? Refer 10 friends to Console and we’ll donate $100 to an open-source project of your choice!

Share Console

Projects

unison

Unison is a new programming language, currently under active development. It’s a modern, statically-typed purely functional language, similar to Haskell, but with the ability to describe entire distributed systems with a single program.

language: Haskell, stars: 3950, watchers: 116, forks: 190, issues: 615

last commit: July 10, 2021, first commit: September 19, 2013

https://www.unisonweb.org/slack

rustpad

Rustpad is an efficient and minimal collaborative code editor that is self-hosted, with no database required.

language: Rust, stars: 1492, watchers: 19, forks: 36, issues: 3

last commit: June 24, 2021, first commit: May 31, 2021

https://www.ekzhang.com/

hypercore

Hypercore is a secure, distributed append-only log.

language: JavaScript, stars: 1877, watchers: 65, forks: 154, issues: 66

last commit: May 05, 2021, first commit: December 20, 2015

https://twitter.com/hypercoreproto

rustc_codegen_gcc

The Rust GCC back-end was officially accepted into the Rust compiler this week!

An Interview With Eric Zhang of Rustpad

Hey Eric! Thanks for joining us! Let’s start with your background. How did you learn how to program and where have you worked in the past?

I started programming about 10 years ago with Python, immediately took a liking to building fun toy apps in middle school, and have been learning since then. My current interests are in programming languages, graphics, machine learning, and systems. Previously I researched computer vision at Nvidia, built infrastructure for Scale AI’s machine learning team, and did programming languages research at Harvard.

What was it like working at a start-up and how did it compare with research or academia?

Scale moves very fast, and on the ML team, there was a lot more emphasis on finishing pressing features to improve metrics and reduce manual labor. In academia you care about writing papers; in industry it’s more about training functional models & building the data pipelines around them.

Who or what are your biggest influences as a developer?

Open source has had a tremendous impact on my life, both in using the software and learning from it. I’ve personally gained a ton of insight from reading into tricky parts of open source code, taking inspiration from the design of popular projects, and interacting with successful project governance models.

What's an opinion you have that most people don't agree with?

I started programming before there was any social or monetary incentive (didn’t even know what having a “career” meant, back then). I’d continue building, even if all programming jobs disappeared tomorrow.

If I gave you $10 million to invest in one thing right now, where would you put it?

Developer tools, research, anything that accelerates progress in the industry.

How do you separate good project ideas from bad ones?

Good projects, for me, are ones that I can finish quickly and produce a meaningfully distinct result from everything that currently exists out there. I want to have different ideas and inspire others to come up with their own.

Why was Rustpad started?

I had a need to build a better centralized collaborative text editor than the ones currently available pre-packaged.

Who, or what was the biggest inspiration for Rustpad?

Firepad, ot.js, and ShareDB.

Are there any overarching goals of Rustpad that drive design or implementation?

Low-latency editing, seamless collaboration (live cursors), correctness, horizontal/vertical scalability, and efficiency (low server cost).

If so, what trade-offs have been made in Rustpad as a consequence of these goals?

Rustpad does not persist state. In a production deployment, you would definitely want to persist state, but this is something that would need to be appended on the app. For short-term, one-off editing sessions of the kind that Rustpad targets, this is a fine tradeoff, and since everything is stored in RAM, it is very fast.

What is the most challenging problem that’s been solved in Rustpad, so far?

The biggest unknown factor for me, when starting the project, was the operational transformation control algorithm. This is tricky because it involves concurrent distributed systems logic, communicating with multiple clients, as well as async Rust (which is difficult). See https://github.com/ekzhang/rustpad/blob/main/rustpad-server/src/rustpad.rs for the eventually consistent algorithm. There’s a lot of race conditions and subtle liveness issues that could arise if not thinking carefully about this.

Are there any competitors or projects similar to Rustpad? If so, what were they lacking that made you consider building something new?

Firepad was tied to Firebase. Not customizable, uses outdated web programming practices, and therefore not a viable alternative for me. ShareDB was perhaps a little too heavyweight and aimed at a more general problem.

What was the most surprising thing you learned while working on Rustpad?

For me, it was that I could buy a tiny $10 DigitalOcean droplet and still handle all of the WebSocket traffic from thousands of users clicking from the front page of Hacker News, editing documents simultaneously, while using less than 3% CPU capacity. Rust is really fast. Even if Rustpad grew a ton, I could still vertically scale really easily (just click a button) before having to set up more complex horizontal options like sharding, which would require writing code.

What is your typical approach to debugging issues filed in the Rustpad repo?

Reproduce, assign myself, then fix. For example, one recent issue had to do with support for emojis, which include some of the rare Unicode code points that are encoded as UTF-16 surrogate pairs. It turns out that Rust encodes strings as UTF-8 and can extract characters as scalar values, while JavaScript takes the approach of saying that the length of a string is its UTF-16 length. This leads to client/server inconsistencies in surrogate pairs (very exotic: emojis, ancient Greek text, cuneiform), and I had to fix this and write good tests.

What is the release process like for Rustpad?

Continuous, done through CI, which is possible and quite simple because the app is stateless (so we don’t have to worry about backwards compatibility). This is good for me as a single maintainer who doesn’t want to spend too much time worrying about release correctness.

Is Rustpad intended to eventually be monetized if it isn’t monetized already?

No, not intended to be monetized.

Do you have any other project ideas that you haven’t started?

Yep, I’m constantly thinking about new project ideas and letting them stew!

Care to elaborate?

Right now I’m building a collaborative math typesetting system called Slate, beta is at https://slate.rs. :) Other ideas for smaller web projects: an interactive playground for creating new parsers and languages, web dashboards for exploratory data analysis that can import from arbitrary JSON files and include shareable visualizations (powered by SQL in the browser), and a CLI tool to easily broadcast your terminal with other people in a pinch.

Do you have any suggestions for someone trying to make their first contribution to an open-source project?

Make sure you talk to the maintainer first! Most open source maintainers have poured many hours into their projects and would be happy to chat. By talking to them, you front-load the effort of aligning values and gain clarity into how the project is structured.