The feature stores evaluated were chosen because they were (1) reproducible - you can create an account and re-run the code to reproduce the results, (2) they have a ready-to-use feature store (not a virtual feature store). We gladly receive contributions for new feature stores, DeWitt Clause permitting. See how to contribute, below, for how to add a new feature store.
The benchmark results presented here should follow these database benchmarking principles:
Although new benchmarks for AI systems have recently appeared (such as TPCx-AI), these cover a very wide array of use cases, including video and images. In contrast, feature stores are designed primarily to manage structured data that comes from databases, data warehouses, and files.
In this context, the feature store community developed a first set of benchmarks for common usage patterns of feature stores. So far, two benchmarks have been published:
Feel free to create a PR to add a new feature store or benchmark. Be sure to include all the hardware setup and software version numbers, that should be as close as possible to existing benchmarks to ensure apple-to-apple comparisons. For virtual feature stores, include the automated complete setup of the feature store plus online and offline stores.