Cost-Effective TV Royalty Data Processing on GCP

TV royalty analytics startup Standard
Taken to
  1. Seed
  2. PoC
  3. MVP
  4. Production
  5. Chasing perfect

Incremental GCP pipeline for a TV royalty analytics startup: metrics refresh every 15 minutes across several terabytes of broadcast data.

15 min
Metric refresh cycle
< 1 hour
New data reaction time
$1,000-1,600
Monthly GCP cost
Five-stage incremental GCP data pipeline: ingest from external bucket, parse delta in GCS, load to BigQuery, calculate metrics, export to client databases

Where the data comes from

Sources we pull

The client tracks viewing metrics for films, TV shows, and broadcast content to calculate royalties for actors and everyone involved in TV production.

The source is broadcast data: large, delivered daily, and always growing. Before the rebuild, the team ran PHP scripts and SQL queries against relational databases on a third-party cloud, recalculating the full history on every run.

That approach exhausted the available compute as the data grew, and the system eventually failed under the load entirely.

What we do with it

The data layer

We rebuilt the system as an incremental GCP pipeline on Datapipe, our own Python orchestration framework.

It tracks which source records have changed and recalculates only the affected metrics and data marts, so a daily run no longer reprocesses the whole history.

Data lands from an external bucket into Cloud Storage, gets parsed and normalized, then loads into BigQuery with deduplication and feeds incremental metric and mart calculations.

The scope grew over time to cover new metric definitions, enrichment calculations, optimized reporting marts, and exports to the client's ClickHouse instance.

  1. 01 External bucket ingestion into GCS
  2. 02 Delta parsing and format normalization in GCS
  3. 03 GCS-to-BigQuery import with deduplication
  4. 04 Incremental metric and mart calculation in BigQuery
  5. 05 Export to client databases and GCS output bucket

Stack

BigQueryCloud RunCloud StorageArtifact RegistryDatapipeClickHouse
Two rows of daily data cells. The old full-reprocessing setup recomputes the entire history on every run, so cost grows until it hits a compute ceiling and fails. The incremental pipeline recomputes only the newest cell, the daily delta, so cost stays flat as the archive grows.
Full reprocessing recomputes the whole history each run; the incremental pipeline touches only the daily delta, so cost stays flat.

What comes out

What you see

The client now processes several terabytes of broadcast data on a stable monthly GCP budget of $1,000-1,600, a level the previous relational database setup could not reach at any budget.

Metrics refresh every 15 minutes, and the pipeline reacts to incoming data within one hour. Both targets were out of reach before. The system has run in production for over two years and is extended incrementally on request. That is a longer run than many of the TV shows it bills royalties for stayed on air.

Related

serviceAnalytics & Datause caseCost-Effective Data Processing

Want this on your line?

Discuss a pilot → Get a budget range