Eagle - Big Data Platform on Google Cloud

Case Study

Eagle - Big Data Platform on Google Cloud

eSolutions has implemented, in just 6 months, the Big Data Platform - Eagle for Carrefour Romania, which simplifies and democratizes access to data, so that the retailer can optimize and streamline its analysis and reporting processes. 

The Eagle platform aims to aggregate information from a multitude of IT systems in Google Cloud, ensuring easy and consistent data analysis and allowing data standardization according to common models, usable in all countries where Carrefour is present. The platform also develops data models addressing local market's needs, in order to build data science analyses and integrates the possibility to implement artificial intelligence algorithms. By analyzing the data from the platform, the retailer can now observe shopping patterns or can segment customers, information that will enable a  data-driven decision-making process.

The Client

With over 10.100 stores in 34 countries, the Carrefour Group is the second retailer in the world and leading in Europe. Daily, over 10 million customers visit Carrefour stores across the world, enjoying a wide variety of products and services at fair prices.

In Romania, the Carrefour Group offers its customers multiple shopping possibilities, both in physical shops as well as online, through the unique portal at www.carrefour.ro or through the delivery service BRINGO.

Business Challenge

The main technical difficulties that had to be overcome derived from the very large volumes of data, the need to process them in a unitary format, standard at the Carrefour group level. All these challenges led to considerable difficulties in data analysis and in implementing a data-driven decision-making process. Another challenge was the development of a replicable solution for all countries in which the retailer is present, which would allow easy implementation of new machine learning and AI algorithms.

Therefore, the platform represents the foundation for implementing algorithms that substantiate the pricing, loyalty, customization, or supply policies.

Solution delivered

  • The Eagle platform is based on Lambda Architecture, so that it supports both batch and streaming data ingestion, stores structured, semi-structured, and unstructured data, and processes very large amounts of data.
  • The platform was developed using the big data components available in Google Cloud Platform, such as BigQuery for storage, Dataflow for batch and stream data processing, Cloud Composer for pipeline orchestration.
  • The process of loading and transforming data is done using technologies such as Google Cloud Dataflow, Apache Beam, Scio, Scala, SQL, Cloud Function, Pub/Sub, and BigQuery.
  • The entire infrastructure of the platform is developed using Terraform and parameterized for multiple deployment environments using Terragrunt. Infrastructure-as-code was used to be able to rebuild or replicate the platform at any time.
  • Platform monitoring and alerts are implemented using the monitoring, logging, and alerting components from Google Cloud Platform.

Results

Eagle, Carrefour's big data platform in Google Cloud, centralizes daily information from the retailer's main data sources, leading to a data-driven vision of the company. There are millions of records that are processed and managed every day. All the aggregated data is suited to be processed through AI algorithms. Therefore, the process will highlight insights about interaction patterns and customer segmentation, enhancing the efficiency of promotional campaigns In the following stages of the project, new data sources, machine learning, and AI algorithms will be implemented, for actual data processing.

The Team

The project was implemented in just 6 months by a team of big data experts from eSolutions, including three Google Cloud Certified Platform Architects, a Google Cloud Certified Professional Data Engineer, a Big Data Developer, and a Project Manager.

Technologies

Google Cloud Platform tools (Pub/Sub, Cloud Functions, Airflow, BigQuery, GCP Monitoring), Terraform, Terragrunt, Scala, Apache Beam, Scio.