Feature Store for ML

Subscribe to the newsletter!

Thank you! Your submission has been received!
Oops! Something went wrong while submitting the form.
Data scientists are duplicating work because they don’t have a centralized feature store. Everybody I talk to really wants to build or even buy a feature store......if an organization had a feature store, the ramp-up period [for Data Scientists can be much faster].

Harish Dodi, O'Reilly Podcast

Featured from the blog

Feature Store Milestones

A Summary of the most important Feature Store milestones.

by Nathalia Ariza

Feature Store vs Data Warehouse

Beyond Brainless AI with a Feature Store

Evolve your models from brain-free AI to Total Recall AI with the help of a Feature Store.

by Jim Dowling

Data Lineage Doesn’t Have To Be Hard

Organizations now have the desire, and the regulatory requirement, to keep track of their data and how it is used inside of their company.

by Jack Ploshnick


The first fully open-source Feature Store, based around Dataframes (Spark/Pandas), Hive (offline), and MySQL Cluster (online) databases. Supports model training/management/serving and Provenance.

Visit Hopsworks

Amazon Feature Store

Amazon launched the Amazon SageMaker Feature Store during their annual Re:invent keynote.

Documentation: Create, Store and Share Features with Amazon SageMaker Feature Store


AirBnB use Zipline for Feature management as part of their BigHead platform for ML.


Comcast have had 2 iterations of their Feature Store, and as of early 2020, appear to be using Redis as their online Feature Store. They have previously used Flink for online feature computation and its queryable state API.


Galaxy is Pinterest’s incremental dataflow-based Feature Store on AWS. It includes a DSL for Feature Engineering, Linchpin.

Aluxio Inc

Pinterest – Big Data Machine Learning Platform at Pinterest


Wix’ Feature Store is based on storing feature data in protobufs, with batch processing using SparkSQL on parquet files stored in S3 and online serving based on HBase/Redis. It provides a Python API for accessing training data as Pandas Dataframes.


The Bigabid Feature Store contains thousands o features and is a  centralized software library and documentation center that “creates a single feature from a standardized input (data)”. Read more here:


Overton is Apple’s platform for managing data and models for ML. There is a publication about it: Overton: A Data System for Monitoring and Improving Machine-Learned Products


StreamSQL have built a Feature Store as a commercial product based on Apache Pulsar, Cassandra, and Redis.


GoJek/Google released Feast in early 2019 and it is built around Google Cloud services: Big Query (offline) and Big Table (online) and Redis (low-latency), using Beam for feature engineering.


Tecton are developing a managed feature store for AWS that manages both features and feature computations.


ScribbleData have developed a feature store for ML.


Intuit have built a feature store as part of their data science platform. It was developed for AWS and uses S3 and Dynamo as its offline/online feature serving layers.

Google MLops Platform with a Feature Store (Vertex AI)

Google released VertexAI, a managed platform for developing and operating AI applications with its own feature store.

Documentation: Vertex AI, Feature Store

Door Dash

DoorDash created a large storage capacity and high read/write throughput Feature Store using Redis.

Building a Gigascale Feature Store with Redis (engineering blog)

Databricks Feature Store

Databricks announced their feature store as a part of their Machine learning platform during the 2021 Data and AI summit's keynote.

Splice Machine

Michelangelo Palette

The first Feature Store (by Uber) that provides a DSL and is heavily built around Spark/Scala with Hive (offline) and Cassandra (online) databases. It is now called Michelangelo Palette.

See also this talk about Michelangelo Palette at InfoQ:


Achal Shah presentation of Michelangelo (2018)


Netflix uses shared feature encoder libraries in their MetaFlow platform to ensure consistency between training and serving, and S3 for offline features and microservices for serving online features. There are shared feature engineering libraries, written in Java. Runway, their model mgmt platform, builds on Metaflow.


Not much is known about Facebook’s Feature Store, cursory information is given here.


Twitter decided to build a library, not a store. It is a set of shared feature libraries and metadata, along with shared file systems, object stores, and databases.


Zomato have used Flink to compute features in real-time and then integrate their real-time feature store with their applications. They note that the real-time feature store needs high throughput read and write at low latency (>1m writes/min). These use manged ElastiCache/Redis on AWS for the online feature store.

Survey Monkey

A Feature Store for AWS that has both an offline and an online database.



A Feature Store for KubeFlow on GCP.

Concepts & Articles

Feature Store Articles

Feature Store Concepts

Consistent Features – Online & Offline

If feature engineering code is not the same in training and inferencing systems, there is a risk that the code will not be consistent, and, therefore, predictions may not be reliable as the features may not be the same. One solution is the have feature engineering jobs write their feature data to both an online and an offline database. Both training and inferencing applications need to read their features when they make predictions – online applications may need low latency (real-time) access to that feature data. The other solution is to use shared feature engineering libraries (ok, if your online application and training application are both able to use the same shared libraries (e.g., both are JVM-based)).

Time Travel

“Given these events in the past what were the feature values of the customer during the time of that event” Carlo Hyvönen

Time-travel is not normally found in databases – you cannot typically query the value of some column at some point in time. You can work around this by ensuring all schemas defining feature data include a datetime/event-time column. However, recent data lakes have added support for time-travel queries, by storing all updates enabling queries on old values for features.  Some data platforms supporting time travel functionality:

Feature Engineering

Michelangelo added a domain-specific language (DSL) to support engineering features from raw data sources (databases, data lake). However, it is also popular to use general purpose frameworks like Apache Spark/PySpark, Pandas, Apache Flink, and Apache Beam.

Materialize Train/Test Data?

Training data for models can be either streamed directly from the feature store into models or it can be materialized to a storage system, such as S3, HDFS, or a local filesystem. When multiple frameworks are used for ML – TensorFlow, PyTorch, Scikit-Learn, then materializing train/test data into the native file format for the framework (.tfrecords for Tensorflow, .npy for PyTorch) is recommended.

Common file formats for ML frameworks:

  • .tfrecords (TensorFlow/Keras)
  • .npy (PyTorch, Scikit-Learn)
  • .csv (Scikit-Learn, others)
  • .petastorm (TensorFlow/Keras, PyTorch)
  • .h5 (Keras)

Online Feature Store

Models may have been trained with hundreds of features, but online applications may just receive a few of those features from an user interaction (userId, sessionId, productId, datetime, etc). The online feature store is used by online applications to lookup the missing features and build a feature vector that is sent to an online model for predictions. Online models are typically served over the network, as it decouples the model’s lifecycle from the application’s lifecycle.  The latency, throughput, security, and high availability of the online feature store are critical to its success in the enterprise. Below is shown the throughput of some key-value and in-memory databases that are used in existing feature stores.

Feature Store Comparison

Feature Engineering
Supported  Platforms
TimeTravel /
Point-in-Time Queries
Training Data
MySQL Cluster
DB Tables, Elasticsearch
(Py)Spark, Python
AWS, GCP, On-Prem
SQL Join or
Hudi Queries
.tfrecords, .csv, .npy, .petastorm, .hf5, etc
Spark, DSL
SQL Join
Streamed to models?
Apache V2
DB Tables
Beam, Python
SQL Join
Streamed to models
Kafka/ Cassandra
Protocol Buffers
Shared libraries
KV Store
KV Entries
Flink, Spark, DSL
Streamed to models?
HDFS, Cassandra
Kafka / Redis
Flink, Spark
Kafka & S3
Kafka & Microservices
Spark, shared  libraries
Strato / Manhatten
Scala shared feature libraries
Scala DSL, Scalding,  
shared libraries
Yes, no details
Yes, no details
Yes, no details
Yes, no details
DSL (Linchpin), Spark
Yes, in mem database
Yes, no details
Spark, Python, Nuclio
AWS, Azure, GCP, on-prem
Yes, native time series or SQL
Yes, no details