Travel Marketplace: Data Warehouse Migration from GCP to Yandex Cloud
Migrated a travel marketplace's data infrastructure from GCP to Yandex Cloud – ETL pipelines and data models rebuilt to meet import substitution requirements, with no drop in data quality.
Where the data comes from
Sources we pull
A travel marketplace aggregating package tours ran its analytics stack on GCP: BigQuery as the primary warehouse, DBT Cloud for SQL transformations, and Cloud Run with Docker and Meltano for ETL pipelines. Russian import substitution legislation and mounting sanctions risk made continuing with Google Cloud a compliance issue the business could no longer defer.
What we do with it
The data layer
Replacing BigQuery with a single DBMS was economically impractical: daily recalculations create burst compute demand for several hours, after which the cluster sits idle. A fixed-size instance either falls short under peak load or sits over-provisioned the rest of the time, and most DBMS options cannot rescale without a restart. So we designed a two-tier warehouse: Yandex Object Storage (S3) for raw and intermediate data, ClickHouse for pre-aggregated serving marts. Compute runs on Managed Kubernetes pods that spin up only during ETL execution, keeping costs proportional to actual usage. Data flows through four processing layers: Stage (deduplication and source merging), CTE (business logic transformations and aggregations), DDS (non-aggregated marts), and CDM (final aggregated marts loaded into ClickHouse). Airflow, deployed in the same K8s cluster, orchestrates all DAGs. Meltano handles ingestion from backend databases, CRM APIs, Google Sheets, and BigQuery. Transformations are implemented in Python using Datapipe, an in-house pipeline library built for efficient S3-based data processing.
- 01 Raw data ingestion via Meltano and Datapipe connectors
- 02 Stage layer: deduplication and source merging on S3
- 03 CTE layers: business logic transformations and aggregations
- 04 DDS layer: non-aggregated data marts on S3
- 05 CDM layer: final aggregated marts
- 06 ClickHouse load for BI access
Stack
What comes out
What you see
Import substitution requirements met with no degradation in data quality or processing cost. The pipeline currently runs 40+ transformations on hundreds of GB of raw data, completing the daily recalculation in approximately 20 minutes of compute time. S3-based intermediate storage costs substantially less than equivalent ClickHouse disk capacity, and pod-based K8s execution eliminates idle compute spend. During initial migration, Google Analytics export volumes exceeded available node memory; the team resolved this by introducing temporary intermediate files and processing data in smaller batches, trading peak memory usage for longer but memory-efficient runs.