A breakdown of what reproducible machine learning is, why it is important, and what goes into making your ML reproducible.
by Hamza Tahir
A feature store differs from a data warehouse in that it contains a low-latency database, for serving recent feature data for an entity, and a scalable database, for storing and accessing large volumes of historical feature values.
by Jim Dowling
Transition from batch to real time with an integrated feature store.
by Adi Hirschtein
The first fully open-source Feature Store, based around Dataframes (Spark/Pandas), Hive (offline), and MySQL Cluster (online) databases. Supports model training/management/serving and Provenance.
Other content:
Pydata London meetup 2019
The Feature Store - Jim Dowling
Logical Clocks Webinar
The Feature Store for Machine Learning
Spark AI 2019
End to End Spark ML Pipelines with Delta and Hopsworks Featurestore - By Kim Hammar
Scale By the Bay 2019
Bay Area AI: Jim Downing, The Feature Store: the missing API between Data Engineering Science?
USENIX OpML '20
Time Travel and Provenance for Machine Learning Pipelines
AirBnB use Zipline for Feature management as part of their BigHead platform for ML.
Other content:
Spark AI 2020
Zipline - A Declarative Feature Engineering Framework
Scale By the Bay 2019
Varant Zanoyan, Zipline at Airbnb
Spark AI 2018
Zipline: Airbnb’s Machine Learning Data Management Platform - Nikhil Simha and Varant Zanoyan
Comcast have had 2 iterations of their Feature Store, and as of early 2020, appear to be using Redis as their online Feature Store. They have previously used Flink for online feature computation and its queryable state API.
Galaxy is Pinterest’s incremental dataflow-based Feature Store on AWS. It includes a DSL for Feature Engineering, Linchpin.
Aluxio Inc
Pinterest – Big Data Machine Learning Platform at Pinterest
Wix’ Feature Store is based on storing feature data in protobufs, with batch processing using SparkSQL on parquet files stored in S3 and online serving based on HBase/Redis. It provides a Python API for accessing training data as Pandas Dataframes.
The Bigabid Feature Store contains thousands o features and is a centralized software library and documentation center that “creates a single feature from a standardized input (data)”. Read more here:
https://www.bigabid.com/blog/data-the-importance-of-having-a-feature-store
Overton is Apple’s platform for managing data and models for ML. There is a publication about it: Overton: A Data System for Monitoring and Improving Machine-Learned Products
StreamSQL have built a Feature Store as a commercial product based on Apache Pulsar, Cassandra, and Redis.
GoJek/Google released Feast in early 2019 and it is built around Google Cloud services: Big Query (offline) and Big Table (online) and Redis (low-latency), using Beam for feature engineering.
Tecton are developing a managed feature store for AWS that manages both features and feature computations.
ScribbleData have developed a feature store for ML.
Intuit have built a feature store as part of their data science platform. It was developed for AWS and uses S3 and Dynamo as its offline/online feature serving layers.
Amazon launched the Amazon SageMaker Feature Store during their annual Re:invent keynote.
Documentation: Create, Store and Share Features with Amazon SageMaker Feature Store
DoorDash created a large storage capacity and high read/write throughput Feature Store using Redis.
Building a Gigascale Feature Store with Redis (engineering blog)
The first Feature Store (by Uber) that provides a DSL and is heavily built around Spark/Scala with Hive (offline) and Cassandra (online) databases. It is now called Michelangelo Palette.
See also this talk about Michelangelo Palette at InfoQ:
https://www.infoq.com/presentations/michelangelo-palette-uber/
Netflix uses shared feature encoder libraries in their MetaFlow platform to ensure consistency between training and serving, and S3 for offline features and microservices for serving online features. There are shared feature engineering libraries, written in Java. Runway, their model mgmt platform, builds on Metaflow.
Not much is known about Facebook’s Feature Store, cursory information is given here.
Early work on sharing feature computation jobs by Databricks.
Twitter decided to build a library, not a store. It is a set of shared feature libraries and metadata, along with shared file systems, object stores, and databases.
Zomato have used Flink to compute features in real-time and then integrate their real-time feature store with their applications. They note that the real-time feature store needs high throughput read and write at low latency (>1m writes/min). These use manged ElastiCache/Redis on AWS for the online feature store.
A Feature Store for AWS that has both an offline and an online database.
A Feature Store for KubeFlow on GCP.
If feature engineering code is not the same in training and inferencing systems, there is a risk that the code will not be consistent, and, therefore, predictions may not be reliable as the features may not be the same. One solution is the have feature engineering jobs write their feature data to both an online and an offline database. Both training and inferencing applications need to read their features when they make predictions – online applications may need low latency (real-time) access to that feature data. The other solution is to use shared feature engineering libraries (ok, if your online application and training application are both able to use the same shared libraries (e.g., both are JVM-based)).
“Given these events in the past what were the feature values of the customer during the time of that event” Carlo Hyvönen
Time-travel is not normally found in databases – you cannot typically query the value of some column at some point in time. You can work around this by ensuring all schemas defining feature data include a datetime/event-time column. However, recent data lakes have added support for time-travel queries, by storing all updates enabling queries on old values for features. Some data platforms supporting time travel functionality:
Michelangelo added a domain-specific language (DSL) to support engineering features from raw data sources (databases, data lake). However, it is also popular to use general purpose frameworks like Apache Spark/PySpark, Pandas, Apache Flink, and Apache Beam.
Training data for models can be either streamed directly from the feature store into models or it can be materialized to a storage system, such as S3, HDFS, or a local filesystem. When multiple frameworks are used for ML – TensorFlow, PyTorch, Scikit-Learn, then materializing train/test data into the native file format for the framework (.tfrecords for Tensorflow, .npy for PyTorch) is recommended.
Common file formats for ML frameworks:
Models may have been trained with hundreds of features, but online applications may just receive a few of those features from an user interaction (userId, sessionId, productId, datetime, etc). The online feature store is used by online applications to lookup the missing features and build a feature vector that is sent to an online model for predictions. Online models are typically served over the network, as it decouples the model’s lifecycle from the application’s lifecycle. The latency, throughput, security, and high availability of the online feature store are critical to its success in the enterprise. Below is shown the throughput of some key-value and in-memory databases that are used in existing feature stores.
