• Information Theory and The Digital Age [PDF]
  In this paper, we explore how these themes and concepts manifest in the trajectory of Information Theory. It begins as a broad spectrum of fields, from management to biology, all believing Information Theory to be a 'magic key' to multidisciplinary understanding. As the field moved from this initial chaos, various influences narrowed its focus. Within these established boundaries, external influences such as the space race steered the progress of the field. Through it all, the expansion of Information Theory was constantly controlled by hardware – indeed, the lack of such technology caused the ‘death’ of Information Theory, and its widespread availability is behind its current overwhelming success...

• Competitive programming with AlphaCode
  As part of DeepMind’s mission to solve intelligence, we created a system called AlphaCode that writes computer programs at a competitive level. AlphaCode achieved an estimated rank within the top 54% of participants in programming competitions by solving new problems that require a combination of critical thinking, logic, algorithms, coding, and natural language understanding...

• Musings on typicality
  If you’re training or sampling from generative models, typicality is a concept worth understanding. It sheds light on why beam search doesn’t work for autoregressive models of images, audio and video; why you can’t just threshold the likelihood to perform anomaly detection with generative models; and why high-dimensional Gaussians are “soap bubbles”. This post is a summary of my current thoughts on the topic...

A Message from this week's Sponsor:

Data Science Articles & Videos

• Can Robots Follow Instructions for New Tasks?
  Existing robotics research has made strides towards allowing robots to generalize to new objects, task descriptions, and goals. However, enabling robots to complete instructions that describe entirely new tasks has largely remained out-of-reach...In “BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning”...we present new research that studies how robots can generalize to new tasks that they were not trained to do...The resulting system can perform at least 24 novel tasks, including ones that require interaction with pairs of objects that were not previously seen together.

• The State of AI Ethics Report (Volume 6)
  Now in its sixth cycle, this edition of the State of AI Ethics Report comes to you with a wide array of topics and contributions from leading lights in the field. For the first time, we have a Spanish text contribution in the report in our endeavor to produce multilingual content for the community to consume. We’ve added a new chapter on Trends that highlights subtle and not so subtle changes taking place in the AI ethics landscape. This one is a must-read for anyone who is planning on bring AI ethics meaningfully into their organizations, or pursuing research and looking for ideas on which areas to make an impact in...

• Predicting Predictions with Datamodels
  What drives machine learning (ML) models’ predictions?...This question is rarely an easy one to answer. On one hand, we know that predictions are a product of training data and learning algorithms. On the other hand, it is often hard to characterize exactly how these two elements interact...In our latest work, we introduce datamodels—a step towards acquiring a more fine-grained understanding of how learning algorithms use training data to make predictions. This post introduces the datamodeling framework, describes its simplest, linear instantiation, and illustrates its success in modeling data-to-prediction mapping for deep neural networks...

• Everything Gets a Package: My Python Data Science Setup
  I make Python packages for everything. Big projects obviously get a package, but so does every tiny analysis. Spinning up a quick jupyter notebook to check something out? Build a package first. Oh yeah, and every package gets its own virtual environment...Let’s back up a little bit so that I can tell you why I do this. After that, I’ll show you how I do this. Notably, my workflow is set up to make it simple to stay consistent...

• OpenAI GPT-3 Text Embeddings - Really a new state-of-the-art in dense text embeddings?
  This week, OpenAI announced an embeddings endpoint (paper) for GPT-3 that allows users to derive dense text embeddings for a given input text at allegedly state-of-the-art performance on several relevant tasks. In this post, I will be reviewing how good these new GPT-3 embeddings really are. Are they really a new state of the art?...

• Percy Liang on Machine Learning Robustness, Foundation Models, and Reproducibility
  An interview with Percy Liang, an Associate Professor of Computer Science at Stanford University and the director of the Center for Research on Foundation Models...

• Introduction to Probability for Data Science
  From what I see from the tsunami of data science books, there are essentially two categories: a) The first type of books are written for programmers and b) The other type of books are the classical probability textbooks written for mathematicians...When you look at the two ends of this spectrum, I hope you can see the gap  —  We need a book that balances the theory and practice...From over than half a decade of teaching the course, I have distilled what I believe to be the core of probabilistic methods. I put the book in the context of data science, to emphasize the inseparability between data (computing) and probability (theory) in our time...

• Building Machine Learning Infrastructure at Netflix and beyond
  Podcast with Savin Goyal on Metaflow and the state of ML infrastructure...Savin Goyal is CTO and co-founder of Outerbounds, a startup building infrastructure to help teams streamline how they build machine learning applications. Prior to starting Outerbounds, Savin and team worked at Netflix, where they were instrumental in the creation and release of Metaflow, an open source Python framework that addresses some of the challenges data scientists face around scalability and version control...

• How Not To Draw An Owl
  I've been thinking a lot lately about how to effectively learn new skills and technologies. I was recently studying data testing with Great Expectations. They have solid documentation, a human-readable CLI, automatically generated and narrated notebooks, and so much more. Data teams could hardly expect a better foundation upon which to learn how to test their data...This experience did, however, remind me that as tools become more composable, difficulties may emerge with onboarding because tools don't exist in a vacuum....

• Advancing AI trustworthiness: Updates on responsible AI research
  This year in review of responsible AI research was compiled by Aether, a Microsoft cross-company initiative on AI Ethics and Effects in Engineering and Research, as outreach from their commitment to advancing the practice of human-centered responsible AI. Although many of the papers’ authors are participants in Aether, the research presented here expands beyond, encompassing work from across Microsoft, as well as with collaborators in academia and industry...

Forum*

Jobs

• (Senior) Analytics Engineer - Fabulous - Remote
  
  Fabulous is a mobile app helping thousands of people every day to change their lifestyles by integrating healthy habits into their lives. Fabulous is using a behavioral economics lens to help everyone achieve their fullest potential. We work closely with researchers based at Duke University and our advisor is Dan Ariely, author of NYT bestseller Predictably Irrational. We are looking for an experienced Analytics Engineer to consolidate the Data Science team and lead the development and enrichment of our Data Pipelines. We have a modern Data-Stack based on Fivetran, dbt, BigQuery, Amplitude, Metabase...

Training & Resources

• TensorFlow-JAX Tutorials
  Learn about TensorFlow and JAX but in a different way! ✅ Meant for everyone (from novice to advanced users) ✅ Deep dive into the fundamental building blocks of the frameworks ✅ Aim to improve your understanding and to some extent the mental model of the API ✅ Not a typical documentation-type tutorial ✅ Runnable notebooks available ❌ Not a replacement of the documentation...

• 6 steps to train any model [Twitter Thread]
  I spent 500+ hours on Kaggle competitions last year and just became a Kaggle Master...Over those many hours, I learned a systematic process you can use to train any model on any dataset...6 steps to train any model 🧵...

• Current State of JAX vs Pytorch? [Reddit Discussion]
  Any thoughts about this? How do their inner workings differ and what should I consider before learning one?...I read that the documentation on JAX Is lacking in other reddit comparison posts, but they are already a year old, is this still the case? And is JAX still growing or does it look as if Google might abandon it?...

Books

• Hands-On Machine Learning with scikit-learn and Scientific Python Toolkits

  
  Integrate scikit-learn with various tools such as NumPy, pandas, imbalanced-learn, and scikit-surprise and use it to solve real-world machine learning problems...
  
  For a detailed list of books covering Data Science, Machine Learning, AI and associated programming languages check out our resources page.
   

  ---

  
  P.S., Enjoy the newsletter? Please forward it to your friends and colleagues - we'd love to have them onboard :) All the best, Hannah & Sebastian