In Short
Operation at this scale demands agility to align with business growth and goals.
Tradera’s data warehouse was on-premise, and a decision was made to re-platform it to run on BigQuery instead.
The solution? Data warehouse modernisation. Their goal was to migrate to the cloud, so Tradera got in touch with Devoteam G Cloud in August of 2019 to assist in their legacy migration.
We enjoyed working with the team from Devoteam G Cloud, who were knowledgeable, responsive and professional during our data warehouse project together.
About Tradera
Tradera, a leading online commercial marketplace, hosts more than 2.5 million members offering millions of listings for products in categories like collectables, electronics, clothing and art. These listings can be purchased via auctions or fixed prices.
Tradera sees more than 4 million visits weekly to their site. Though most of their members are Swedish, Tradera is accessible to members around the world. The headquarters of Tradera is in Sweden.
The Problem
Whole Tradera’s data warehouse was on-premise. They made a decision to re-platform it to run on BigQuery instead. Previously having conducted a project on BigQuery, they discovered the advantages of using a scalable and serverless data warehouse.
The Goal
The goal was to migrate to a scalable and serverless data warehouse.
The Solution
The first step was to run an investigation on their needs and present their findings to Tradera through a proof-of-concept. According to their Data Warehouse Modernisation methodology developed in tandem with Google, they first prepared a Technical Analytics Design Document, detailing how they would deliver the solution covering all the differences in programming syntax between Microsoft SQL and BigQuery.
In their check-up they covered topics such as Backup and Restore, Migration, Transformation, Monitoring, Data Access Management and forged an agenda to develop this solution.
The plan was to implement different ETL pipelines on Google Cloud and transform the data in Big Query without sacrificing data accuracy.
Using the new setup on BigQuery and Cloud Composer, ETL performance was quadrupled. ETL pipelines were also easily monitored via Grafana and source controlled for increased transparency.
Methodology
Experts from Devoteam G Cloud performed the migration of the infrastructure, stored the data and maintained the visualisation layer after the transformation. Some of the steps included re-designing the ETL pipeline and translating MS SQL to represent the same logic in BigQuery.
The Result
The project was successfully completed in mid-April, and now Tradera runs most of their workloads in Google Cloud, using BigQuery to analyse their business.
“Using the new setup on BigQuery and Cloud Composer, ETL performance was quadrupled. ETL pipelines were also easily monitored via Grafana and source controlled for increased transparency.”
-Jason Quek, CTO, Devoteam G Cloud
