📝 Guest post: The Original Open Source Feature Store - Hopsworks*
Was this email forwarded to you? Sign up here In TheSequence Guest Post our partners explain in detail what machine learning (ML) challenges they help deal with. This article reintroduces the core concepts of a Feature Store; the dual storage logic (online and offline) and how data scientists and ML practitioners can use APIs to establish a DevOps process for feature pipelines. IntroFeature stores for machine learning have become a key piece of data infrastructure for putting models in production. As with all key data infrastructure, there is demand for open source feature stores, and Feast has gained much attention as a framework for building your own feature store. But, if you need an open-source feature store that, out of the box, comes with its own offline and online stores, a user interface, enterprise-level security, high availability, and scalability, Hopsworks might be what you are looking for. Stand Alone Feature StoreHopsworks is widely used as a stand-alone feature store. A feature store provides Offline and Online Stores for large volumes of historical feature values and real-time access for current feature values. A feature store also provides an API for creating, reading, updating, and deleting feature values, and retrieving training data and feature vectors. Offline StoreHopsworks comes with its own offline store, based on Apache Hudi, but you can also use your own data warehouse/lake as an offline store. The offline store contains historical feature values and is used to create training data and provide features for offline batch scoring. Features stored in Hudi are known as cached features, while features stored in external stores are known as on-demand features, and the Hopsworks offline feature store can have a mix of cached and on-demand features. Maybe you already have all of your features in Snowflake or Redshift or Delta Lake, then you just need to mount them in Hopsworks as on-demand features. Or maybe you prefer the low-cost, high-performance cached features, that are stored in object stores in the cloud (S3, ADSL, GCS) or on commodity servers on-premises using the award-winning, HDFS-compatible, HopsFS file system. Online StoreHopsworks online feature store is built on RonDB, the only database optimized for feature store use cases. The online store contains the latest feature values and is used to provide feature vectors to deployed models at runtime. Any historical or context data that is used by your models to make predictions is typically stored as pre-engineered features in the online store, and retrieved using entity ids. For example, you might retrieve a customer’s recent history using a customer or session ID. Hopsworks Online Store provides sub-millisecond feature vector lookups, high throughput (scales to 10s of millions of reads or writes per second), high availability (with 7 nines availability in the telecom space), disaster recovery, elastic scalability, and MySQL support. The Hopsworks Feature Store (HSFS) APIYou interact with the Hopsworks Feature Store using the Hopsworks Feature Store (HSFS) API, with implementations in Python, PySpark, and Spark (Scala and SQL). For programs that run outside Hopsworks, you need to know the endpoint for Hopsworks, which feature store you are connecting to, and an API key to be able to connect. Python developers can create features in Pandas and write the Dataframe directly to the feature store. You can create training data by joining features together with Pandas-like commands, ensuring point-in-time correctness. When you have a deployed model, you can retrieve individual feature vectors from the feature store with a Python or Scala method call. If you have large volumes of data, you can use Spark to engineer your features and write them to the feature store as dataframes. If your features reside in a table in an external data warehouse or data lake, like Snowflake or Parquet on S3, respectively, you can use SQL to compute your features as aggregations. And if you need very fresh features, you can use Spark Streaming or Flink to compute your features on real-time data, HSFS provides a data-scientist-friendly Python API to create training data by joining features together (with no data leakage), filtering feature values, and applying transformations on features from the feature store before they are returned as training data. The transformation functions, defined in Python, are re-used both to create training data to transform feature vectors retrieved with the Online API when serving a model online. DevOps for Feature PipelinesOne of the challenges in writing feature pipelines is testing both the feature computation logic and also validating the data that passes through the pipeline. The HSFS API enables you to develop your features on any Python or Spark environment, and test those features in a local development environment as well as test them with a continuous integration platform, like Github Actions or Jenkins. Feature pipelines commonly include three different types of tests:
Open Source InstallationHopsworks consists of a number of open-source services, including HopsFS, RonDB/MySQL, Apache Spark, Apache Flink, Conda, Docker, Apache Hive, OpenSearch, Kafka, Prometheus, and Grafana. The recommended way to install Hopsworks is with a bash script that downloads all the software and configures the services. Hopsworks currently runs on Linux (Ubuntu, RHEL/Centos). If you have access to GCP or Azure command-line utilities, and you have privileges to create virtual machines, you can also create the virtual machines and install the platform in a single script. This script also allows you to stop, start, and suspend (GCP-only) your Hopsworks cluster, so you only pay for the compute you use. Managed PlatformLike most open-source projects, Hopsworks is also available as a managed platform on AWS, Azure, and soon GCP at www.hopsworks.ai. It is the quickest way to get started and try out Hopsworks. *This post was written by Hopsworks’s team. We thank Hopsworks for their ongoing support of TheSequence.You’re on the free list for TheSequence Scope and TheSequence Chat. For the full experience, become a paying subscriber to TheSequence Edge. Trusted by thousands of subscribers from the leading AI labs and universities. |
Older messages
🟢 ⚪️Edge#158: A Deep Dive Into Aporia, the ML Observability Platform
Thursday, January 27, 2022
Read it without subscription
📝 Guest post: Data Labeling and Its Role in E-commerce Today – Recent Use Cases*
Wednesday, January 26, 2022
No subscription is needed
🔄🔄 Edge#159: MLOps Full Recap
Tuesday, January 25, 2022
Dive in!
👷♀️🧑🏻🎓👩💻👨🏻🏫 The MoE Momentum
Sunday, January 23, 2022
Weekly news digest curated by the industry insiders
📌 Learn from 40+ AI experts at mlcon 2.0 ML dev conf <Feb22-23>
Friday, January 21, 2022
Our partner cnvrg.io is hosting another incredible virtual conference mlcon 2.0! It is FREE
You Might Also Like
Scoop: Tiger Global-backed Innovaccer in talks to raise $250M
Wednesday, May 1, 2024
Plus: An update on Google's layoffs and the social platform X didn't see coming View this email online in your browser By Christine Hall Wednesday, May 1, 2024 Welcome to TechCrunch PM. Today,
🖥️ Why I'm Never Going Back to a Windows PC — Tips Before You Buy a Smart Ring
Wednesday, May 1, 2024
Also: How to Clear the Moisture Detected Warning on Samsung Phones, and More How-To Geek Logo May 1, 2024 Did You Know A single 1 oz shot of espresso only has approximately 40 mg of caffeine, whereas a
Daily Coding Problem: Problem #1428 [Hard]
Wednesday, May 1, 2024
Daily Coding Problem Good morning! Here's your coding interview problem for today. This problem was asked by Microsoft. Given an array of positive integers, divide the array into two subsets such
Top Tech Deals 👀 Samsung Gaming Monitor, Pixel Watch 2, MacBook Air, and More
Wednesday, May 1, 2024
Get a discounted M3 MacBook Air or expand your Xbox storage. How-To Geek Logo May 1, 2024 Top Tech Deals: Samsung Gaming Monitor, Pixel Watch 2, MacBook Air, and More Get a discounted M3 MacBook Air or
Infographic | Visualizing Global Gold Production in 2023 🏅
Wednesday, May 1, 2024
Gold production in 2023 was led by China, Australia, and Russia, with each outputting over 300 tonnes. View Online | Subscribe Presented by: Access European benchmarks with a trusted 25-year history
⚙️ GPT-5 may be releasing sooner than expected
Wednesday, May 1, 2024
Plus: Amazon rebrands AI branch
Noonification: How to Create a CI/CD Pipeline Using GitHub and AWS EC2
Wednesday, May 1, 2024
Top Tech Content sent at Noon! Get Algolia: AI Search that understands How are you, @newsletterest1? 🪐 What's happening in tech today, May 1, 2024? The HackerNoon Newsletter brings the HackerNoon
Arc for Windows is better than Chrome
Wednesday, May 1, 2024
Adobe bug bounty; Rabbit's first R1 software update; Dream podcaster mic -- ZDNET ZDNET Tech Today - US May 1, 2024 placeholder Arc browser is now available for Windows and it's so much better
Is TikTok trying to get banned from the App Store early?
Wednesday, May 1, 2024
TikTok is offering some users a way to buy its in-app tipping tokens outside of Apple's App Store. View this email online in your browser By Alex Wilhelm Wednesday, May 1, 2024 Good morning, and
Get Compliant in 2024 - Download Ultimate PAM Policy Template Today
Wednesday, May 1, 2024
Privileged Access Management Policy Template What are your PAM policies for 2024? Get ready for the New Year Is your approach to Privileged Access Management as current and effective as it could be? In