Databricks has acquired Fennel to enhance feature engineering pipelines for batch, streaming and real-time data, boosting model performance. Fennel improves the efficiency and data freshness of feature engineering pipelines for batch, streaming and real-time data by only recomputing the data that has changed.
Integrating Fennel’s capabilities into the Databricks Data Intelligence Platform will help customers quickly iterate on features, improve model performance, and provide GenAI models with personalized and real-time context.
Machine learning models are only as good as the data they learn from. That’s why feature engineering is so critical: features capture the underlying domain-specific and behavioral patterns in a format that models can easily interpret. Even in the era of generative AI, where large language models are capable of operating on unstructured data, feature engineering remains essential for providing personalized, aggregated, and real-time context as part of prompts.
Despite its importance, feature engineering has historically been difficult and expensive due to the need to maintain complex ETL pipelines for computing fresh and correctly transformed features. Many organizations struggle to handle both batch and real-time data sources and ensure consistency between training and serving environments — not to mention doing this while keeping quality high and costs low.
Fennel addresses these challenges and simplifies feature engineering by providing a fully-managed platform to efficiently create and manage features and feature pipelines. It supports unified batch and real-time data processing, ensuring feature freshness and eliminating training-serving skew.
Fennel helps reduce the complexity and time required to develop and deploy machine learning models and helps data scientists focus on creating better features to improve model performance rather than managing complicated infrastructure and tools.
As part of the acquisition, the Fennel team will join Databricks’ engineering organization.
