The Killer Feature Store: Orchestrating Spark ML Pipelines and MLflow for Production

The ‘feature store’ is an emerging concept in data architecture that is motivated by the challenge of productionizing ML applications. The rapid iteration in experimental, data driven research applications creates new challenges for data management and application deployment. These challenges are complicated by production ML pipelines with interdependent modeling and featurization stages. Large tech companies have published popular reference architectures for ‘feature stores’ that address some of these chal