black smartphone showing icons with icons

Revolutionizing Machine Learning Workflows: The Waze Journey with TFX and Vertex AI

Waze has long been at the forefront of combining community-driven insights with cutting-edge technology to redefine how we navigate our roads. It stands as the largest community-based traffic and navigation app globally, leveraging real-time data to help users avoid both literal and metaphorical roadblocks. Beyond its core mobile navigation capabilities, Waze has expanded its technological horizons to include a web platform, a dedicated carpool app, comprehensive partnership services, and a dynamic advertisement platform. This diverse portfolio presents a multitude of technological challenges and use cases, each with its unique demands.

At the heart of Waze’s innovation is its reliance on machine learning (ML) to enhance its services. These applications range from accurately predicting estimated times of arrival (ETA) and matching riders with drivers in the Carpool service, to optimizing ad targeting to ensure relevancy and engagement. However, developing and deploying ML solutions that are robust and production-ready is no small feat. It requires a sophisticated infrastructure capable of handling complex data, ensuring low-latency real-time processing, and managing the scalability needed to support Waze’s vast user base.

The transition to embracing machine learning at Waze wasn’t without its hurdles. Initially, the company faced a chaotic environment due to the adoption of multiple ML frameworks and a disconnect between machine learning operations (MLOps) and the rest of the development workflow. This led to a fragmented development cycle, where data scientists were bogged down with operational and monitoring tasks rather than focusing on model development and data analysis. Recognizing the need for change, Waze embarked on a journey to streamline its ML workflows, aiming for a model where data scientists could seamlessly take their projects from research to a production-grade service without the overhead of managing infrastructure.

To achieve this, Waze adopted a full-cycle data science philosophy, emphasizing automation and streamlined processes to accelerate development and enhance quality. This approach has empowered data scientists at Waze to focus on what they do best: building models and analyzing data, with the added advantage of robust infrastructure and operational support available out of the box.

Waze’s machine learning stack is built on Google Cloud Platform (GCP), utilizing TFX (TensorFlow Extended) for managing ML pipelines and Vertex AI for a comprehensive suite of managed services that support ML operations. Vertex Pipelines, in particular, play a crucial role by simplifying the complexity of deploying TFX pipelines, ensuring that data scientists can work efficiently and effectively.

Despite the sophisticated technology stack, making these tools accessible and easy to use for data scientists required additional effort. Waze developed “glue code” to integrate various components and abstract complexities, resulting in an easy-to-use Python package named “waze-data-tfx”. This package eliminates boilerplate code and provides pre-implemented models and utility functions, allowing data scientists to concentrate on feature engineering and modeling. The result is a dramatically simplified development process, where deploying a production-ready ML pipeline can be achieved with minimal code changes.

The impact of these changes has been profound. Data scientists at Waze can now deploy full-fledged production pipelines in a fraction of the time it used to take, enabling rapid iteration and deployment of new features and improvements. Looking ahead, Waze plans to further enhance its ML workflows by incorporating additional components for model validation and fairness checks, making these advanced features available to all projects automatically.

In summary, Waze’s journey in integrating machine learning into its platform showcases the power of combining innovative ML frameworks with a focus on automation and streamlined development processes. By empowering its data scientists to work more efficiently and effectively, Waze continues to push the boundaries of what’s possible in traffic navigation and beyond, all while maintaining its commitment to community and user satisfaction.