Issue #548
May 23, 2024

Hello!

Once a week, we write this email to share the links we thought were worth sharing in the Data Science, ML, AI, Data Visualization, and ML/Data Engineering worlds.

And now…let's dive into some interesting links from this week.

Editor's Picks

• Show and tell: Stats communicators share their stories
  One of the biggest challenges facing any statistician is communicating their work to people who don’t eat, sleep and breathe data to the extent they do…how do we help the non-expert get their head around complex statistics, and at the same time find our voices as messengers? In this special feature, we ask three of the best stats communicators working today [Daniel Parris, Alli Torban, Tom Chivers] to tell us what they do, and how and why they do it. Because rarely does the data speak for itself…
  

• Histograms for faster boosting
  Gradient boosted machines can take a while to train, but there is an internal trick that we can pull off to make it a whole lot faster to train trees. It turns out that a histogram may be all we need!..
  

• Predicting type 1 diabetes in children using electronic health records in primary care in the UK: development and validation of a machine-learning algorithm
  Children presenting to primary care with suspected type 1 diabetes should be referred immediately to secondary care to avoid life-threatening diabetic ketoacidosis. However, early recognition of children with type 1 diabetes is challenging. Children might not present with classic symptoms, or symptoms might be attributed to more common conditions. A quarter of children present with diabetic ketoacidosis, a proportion unchanged over 25 years. Our aim was to investigate whether a machine-learning algorithm could lead to earlier detection of type 1 diabetes in primary care…
  

A Message from this week's Sponsor:

Is your A/B testing system reliable?

There are seven core components of an A/B testing stack, but if they’re not all working properly, it can mean your company isn’t making the right decisions. Meaning teams aren’t shipping features that are actually helping customers, the org is leaving money on the table, and you’re likely doing lots of manual work to validate experiment results.

Now imagine using a reliable experimentation system like the ones developed by Airbnb, Netflix, and many other experimentation-driven companies. You’d be able to:

• Work with the most accurate and up-to-date metrics, completely in your own data warehouse

• Leverage business metrics (revenue, activations, and more) over shallow metrics (clicks, signups, etc.) easily and quickly

• Save time in setting up, running, and analyzing experiments

• Help your non-data teams to self-serve experiment set up and analysis

Most successful tech companies rely on the same seven building blocks of an experimentation platform, and we put together a white paper that walks through these core components and the opportunities and failure modes they each present.

Download the white paper to see if you have all seven, and if you don't, what you could be missing.

* Want to sponsor the newsletter? Email us for details --> team@datascienceweekly.org

Data Science Articles & Videos

• Spatial ML: Predicting on out-of-sample data
  Using spatially derived features can greatly improve model performance. But predicting on out-of-sample data can be tricky. 3️⃣ approach i can think of…These features can include, but not limited to:

  • the spatial lag (neighborhood average) of a variable

  • counts of neighboring features

  • most common category nearby

  • spatial embedding via principle coordinate analysis…

• When do you prefer SQL or Python for Data Engineering? [Reddit]
  When do you prefer to use SQL vs Python, what usually are the main determining factors?…
  

• Guiding LLM Output with DSPy Assertions and Suggestions
  Assertions in DSPy allow you to define strict rules and constraints that the LLM's output must (or maybe that you just want) to adhere to. For example, you can use assertions to ensure that the generated text doesn't contain certain "bad" words, or that the output conforms to a specific structure or format…Suggestions, on the other hand, provide a more flexible way to guide the LLM's output. Instead of hard-failing when a constraint is not met, suggestions offer feedback and guidance to the model, allowing it to refine its response and try again…In this blog post, we'll walk through practical examples of how to implement assertions and suggestions in your own DSPy-powered applications…
  

• NULL BITMAP Builds a Database #1: The Log is Literally the Database
  Today we are starting a new series…We are going to make an LSM-based storage engine (Log-structured storage engine) piece by piece. I think a really lovely thing about LSMs is how they lend themselves to a piece by piece implementation: they're fundamentally a bunch of little components that fit together in intricate ways that can be improved independently and swapped out, so I think it's the perfect project for a piecemeal implementation…
  

• Cleaning tables with merged cells - Walkthrough with table in .docx file
  Picking up my work on data cleaning with R: Here's a brief post on working with data from tables with merged cells, often shared in PDF or Word documents and based on real-world examples…


• What is it like to dislike data?
  This year’s Joint Statistical Meetings have me as a panelist for the memorial session of a well-respected professor of statistics. He devoted his work to the human aspect of statistical consulting. He was also my major advisor for the master’s program in statistics. I have been reflecting on the question of what I identify as his biggest influence on me. As many things that have to do with humans, it is a little hard to articulate…
  

• Superlative mechanical energy absorbing efficiency discovered through self-driving lab-human partnership
  Energy absorbing efficiency is a key determinant of a structure’s ability to provide mechanical protection and is defined by the amount of energy that can be absorbed prior to stresses increasing to a level that damages the system to be protected. Here, we explore the energy absorbing efficiency of additively manufactured polymer structures by using a self-driving lab (SDL) to perform >25,000 physical experiments on generalized cylindrical shells. We use a human-SDL collaborative approach where experiments are selected from over trillions of candidates in an 11-dimensional parameter space using Bayesian optimization and then automatically performed while the human team monitors progress to periodically modify aspects of the system…
  

• What is prompt optimization?
  Prompt optimization is the process of improving the quality of prompts used to generate content. Often by using few shots of context to generate a few examples of the desired output, then refining the prompt to generate more examples of the desired output…
  

• Aurora: A Foundation Model of the Atmosphere

  We introduce Aurora, a large-scale foundation model of the atmosphere trained on over a million hours of diverse weather and climate data. Aurora leverages the strengths of the foundation modeling approach to produce operational forecasts for a wide variety of atmospheric prediction problems, including those with limited training data, heterogeneous variables, and extreme events. In under a minute, Aurora produces 5-day global air pollution predictions and 10-day high-resolution weather forecasts that outperform state-of-the-art classical simulation tools and the best specialized deep learning models…
  

• Let's talk about LLM evaluation
  Since my team works on evaluation and leaderboards at Hugging Face, at ICLR 2024 (2 weeks ago) a lot of people wanted to pick my brain about the topic (which was very unexpected, thanks a lot to all who were interested). Thanks to all these discussions, I realized that a number of things that I take for granted evaluation wise are 1) not widely spread ideas 2) apparently interesting…
  

• What’s your achievements in Data Engineering [Reddit]
  What's the project you're working on or the most significant impact you're making at your company at Data engineering team…
  

• Optimize a RAG DSPy Application
  How many samples are necessary to achieve good performance with DSPy?…
  In this tutorial, you will:

  • Build and optimize DSPy modules that use retrieval-augmented generation and multi-hop reasoning to answer questions over AirBnB 2023 10k filings dataset,

  • Instrument your application using Parea AI,

  • Inspect the traces of your application to understand the inner works of a DSPy forward pass.

  • Evaluate your modules

  • Understand how many samples are necessary to achieve good performance on the test set…
    

A Message from this week's other Sponsor:

Join Us to Unlock Generative AI for the Enterprise

Generative AI is poised to transform the world – IF data privacy solutions can keep up. Join us in San Francisco for Confidential Computing Summit, two eye-opening days on June 5 & 6 that will bring together top business and technology leaders to evaluate the latest solutions in secure and trustworthy AI, explore confidential data use cases, and get you up-to-speed on what’s now and what’s next. Get $200 off with promo code DSW —> https://bit.ly/4bvJvnk 

* Want to sponsor the newsletter? Email us for details --> team@datascienceweekly.org

Training & Resources

• Practical Statistics in Medicine with R

  The primary emphasis of this textbook is to introduce students to the basic ideas of medical statistics using R. It can also be used as a support for self-directed learning for students and researchers in biomedical field to analyze data. Additionally, bridging the gap between theory and practice, it may be useful for (under)graduate students with a science background (engineering, mathematics) who want to move towards biomedical sciences and develop essential R skills…
  

• Princeton’s ECE524: Foundations of Reinforcement Learning
  Interested in learning the mathematical foundations of Reinforcement Learning (RL)? Now is a good time! This semester, we will make videos and lecture notes from my graduate-level RL theory course at Princeton available to the public…
  

• Shiny apps for demystifying statistical models and methods
  The purpose of these apps is to demonstrate how statistical methods and models work, using data simulation and visualization. Most of them simulate data according to the rules of a model, plot the simulated data, and report statistical results obtained by fitting the simulated data to the model from which they were generated. The user can adjust values of model parameters or change features of the model, then see what happens to the plotted data and to the statistical results. Most apps have an option to change the random seed, which will generate a new (pseudo-) random set of data…
  

Last Week's Newsletter's 3 Most Clicked Links

• Databases for Data Scientist - And why you probably don’t need one

• Data Wrangler Extension for Visual Studio Code

• 100 Exercises To Learn Rust

* Based on unique clicks.
** Find last week's issue #547 here.

Cutting Room Floor

• Hot Take: "Data are" is grammatically incorrect even if the guide books say it's right

• Tell the LLM to "Repeat the Question". An Interesting Observation

• A whole-slide foundation model for digital pathology from real-world data

• Optimizing Databricks LLM Pipelines with DSPy

• Deep Dive into Transformers by Hand

• NASA Swift Satellite and AI Unravel the Distance of the Farthest Gamma-Ray Bursts

• Indexify - Build Production-Grade LLM Applications that React to Your Data

• Interactive Density-based Clustering with DBSCAN

• Quantum Many-Body Physics Calculations with Large Language Models

• Behind the scenes scaling ChatGPT - Evan Morikawa at LeadDev West Coast 2023

Whenever you're ready, 2 ways we can help:

1. Looking to get a job? Check out our “Get A Data Science Job” Course
   It is a comprehensive course that teaches you everything related to getting a data science job based on answers to thousands of emails from readers like you. The course has 3 sections: Section 1 covers how to get started, Section 2 covers how to assemble a portfolio to showcase your experience (even if you don’t have any), and Section 3 covers how to write your resume.

2. Promote yourself/organization to ~62,000 subscribers​ by sponsoring this newsletter. 35-45% weekly open rate.

Thank you for joining us this week! :)

Stay Data Science-y!

All our best,
Hannah & Sebastian