Building an End-to-End Medallion Architecture on GCP with PySpark and Dataproc Serverless

In modern enterprise data platforms, organizing raw, semi-structured, and structured data into a reliable, scalable structure is essential. The Medallion Architecture—pioneered by Databricks and widely adopted across cloud platforms—provides a logical framework for incrementally cleaning, structuring, and enriching data as it flows through three distinct tiers: Bronze (Raw), Silver (Cleansed), and Gold (Curated/Analytics).

In this deep dive, we will walk through building an end-to-end Medallion pipeline on Google Cloud Platform (GCP) using Google Cloud Storage (GCS), Dataproc Serverless (PySpark), and BigQuery.

1. Architecture Overview

Key Architectural Layers

1. Bronze Layer (Raw Storage): Stores raw source files (e.g., CSV, JSON) as received without alteration. This preserves the historical source state for auditing and re-processing.

2. Silver Layer (Cleansed Storage): PySpark processes the raw files by filtering bad records, handling missing values, standardizing column names, casting data types, and adding pipeline metadata (ingestion_timestamp). Data is written back to GCS in an optimized, compressed Parquet format.

3. Gold Layer (Analytics & Reporting): The cleansed Silver Parquet dataset is ingested directly into BigQuery dimensional tables (dim_customers), making it instantly available for downstream BI dashboards, data science models, and ad-hoc analytics.

2. Infrastructure Setup & Environment Configuration

Before running the transformation job, initialize the GCP environment by enabling the required APIs, creating storage buckets, setting up the BigQuery dataset, and configuring IAM service account permissions.

Step 1: Enable GCP Services

gcloud services enable dataproc.googleapis.com \
                       bigquery.googleapis.com \
                       storage.googleapis.com \
                       --project=<YOUR_PROJECT_ID>

Step 2: Create Storage Buckets and BigQuery Dataset

# Create GCS Buckets for Bronze and Silver tiers
gcloud storage buckets create gs://<YOUR_BUCKET_NAME>-bronze --location=us-central1
gcloud storage buckets create gs://<YOUR_BUCKET_NAME>-silver --location=us-central1

# Create Gold Dataset in BigQuery
bq mk --location=us-central1 <YOUR_PROJECT_ID>:gold_analytics

Step 3: Grant IAM Permissions

Dataproc Serverless runs batch jobs under the default Compute Engine service account. Grant it the necessary Dataproc Worker role to manage internal cluster allocations and storage access:

gcloud projects add-iam-policy-binding <YOUR_PROJECT_ID> \
    --member="serviceAccount:<PROJECT_NUMBER>-compute@developer.gserviceaccount.com" \
    --role="roles/dataproc.worker"

3. The Core Transformation Pipeline (transform_customers.py)

The PySpark script reads raw customer data from Bronze, cleanses and standardizes the schema, writes the Parquet output to Silver, and streams the curated dataset into BigQuery.

from pyspark.sql import SparkSession
from pyspark.sql.functions import col, current_timestamp, trim, upper

# 1. Initialize Spark Session configured for Dataproc Serverless
spark = SparkSession.builder \
    .appName("CustomerMedallionPipeline") \
    .getOrCreate()

# GCS Resource Paths
BRONZE_PATH = "gs://<YOUR_BUCKET_NAME>-bronze/raw/customers/customer.csv"
SILVER_PATH = "gs://<YOUR_BUCKET_NAME>-silver/processed/customers/"
PROJECT_ID = "<YOUR_PROJECT_ID>"

# ---------------------------------------------------------
# BRONZE LAYER: Read Raw Data
# ---------------------------------------------------------
df_raw = spark.read \
    .option("header", "true") \
    .option("inferSchema", "true") \
    .csv(BRONZE_PATH)

# ---------------------------------------------------------
# SILVER LAYER: Data Cleansing & Schema Standardization
# ---------------------------------------------------------
df_silver = df_raw \
    .dropDuplicates() \
    .dropna(subset=["customer_ID"]) \
    .withColumnRenamed("customer_ID", "customer_id") \
    .withColumnRenamed("Name", "customer_name") \
    .withColumnRenamed("City", "city") \
    .withColumn("customer_name", trim(upper(col("customer_name")))) \
    .withColumn("city", trim(upper(col("city")))) \
    .withColumn("ingestion_timestamp", current_timestamp())

# Write Cleansed Parquet to Silver Storage Tier
df_silver.write \
    .mode("overwrite") \
    .parquet(SILVER_PATH)

# ---------------------------------------------------------
# GOLD LAYER: Load into BigQuery Analytics Table
# ---------------------------------------------------------
df_silver.write \
    .format("bigquery") \
    .option("table", f"{PROJECT_ID}.gold_analytics.dim_customers") \
    .option("writeMethod", "direct") \
    .mode("overwrite") \
    .save()

print("Medallion Pipeline Executed Successfully!")

4. Submitting the Job via Dataproc Serverless

Dataproc Serverless allows you to execute PySpark scripts without creating or managing persistent clusters.

1. Upload the raw CSV file to your Bronze storage path:

   gcloud storage cp customer.csv gs://<YOUR_BUCKET_NAME>-bronze/raw/customers/
2. Upload the PySpark transformation script to GCS:

gcloud storage cp transform_customers.py gs://<YOUR_BUCKET_NAME>-bronze/scripts/

       3. Submit the batch job using gcloud dataproc batches submit:

gcloud dataproc batches submit pyspark gs://<YOUR_BUCKET_NAME>-bronze/scripts/transform_customers.py \
    --region=us-central1 \
    --version=2.1 \
    --jars=gs://spark-lib/bigquery/spark-bigquery-with-dependencies_2.12-0.32.2.jar \
    --project=<YOUR_PROJECT_ID>

5. Pipeline Verification & Data Audit

After execution, verify each layer to ensure data integrity across the platform:

• Verify Bronze Raw Storage:

gcloud storage ls -l gs://<YOUR_BUCKET_NAME>-bronze/raw/customers/

• Verify Silver Parquet Output:

gcloud storage ls -l gs://<YOUR_BUCKET_NAME>-silver/processed/customers/

• Query Gold BigQuery Analytics Table:

bq query --use_legacy_sql=false "SELECT * FROM \`<YOUR_PROJECT_ID>.gold_analytics.dim_customers\`"

6. Key Enterprise Takeaways & Best Practices

1. Zero Infrastructure Overhead: Using Dataproc Serverless eliminates idle cluster costs and manual auto-scaling configurations.

2. Open Data Formats: Storing the Silver tier in open Parquet format prevents vendor lock-in and allows other compute engines (e.g., Presto, DuckDB, Trino) to query the lake directly.

3. Optimized Warehouse Writes: Utilizing the Spark-BigQuery Connector with direct write methods bypassing temporary GCS staging speeds up gold table refreshes significantly.