Multi-Cloud Data Management

---

1 - Data Gravity

1.1 The concept

Data Gravity: Massive data attracts workloads. It is more cost-effective and performant to move compute to the data than the other way around.

Factor	Impact
Egress costs	$0.08-0.12/GB between clouds
Latency	10-100ms cross-cloud
Compliance	GDPR requires localization
Bandwidth	Limited between clouds

1.2 Strategies

Strategy	Description	Use Case
Colocation	Apps near the data	Analytics
Caching	Local copy of data	Read-heavy
Replication	Bidirectional sync	Active-Active
Event streaming	Asynchronous sync	Real-time

---

2 - Database replication

2.1 Cross-Cloud PostgreSQL

-- On the primary (AWS RDS)
-- Create a publication
CREATE PUBLICATION my_publication FOR ALL TABLES;

-- On the replica (Azure)
-- Create a subscription
CREATE SUBSCRIPTION my_subscription
CONNECTION 'host=rds-primary.aws.com dbname=mydb user=repl password=xxx'
PUBLICATION my_publication;

2.2 Terraform configuration

# AWS RDS Primary
resource "aws_db_instance" "primary" {
  identifier        = "primary-db"
  engine            = "postgres"
  engine_version    = "15"
  instance_class    = "db.r6g.large"
  allocated_storage = 100
  
  # Enable logical replication
  parameter_group_name = aws_db_parameter_group.logical_replication.name
  
  publicly_accessible = true  # For cross-cloud (VPN recommended)
}

resource "aws_db_parameter_group" "logical_replication" {
  family = "postgres15"
  name   = "logical-replication"
  
  parameter {
    name  = "rds.logical_replication"
    value = "1"
  }
}

2.3 MySQL with Debezium

# Debezium CDC for MySQL
apiVersion: kafka.strimzi.io/v1beta2
kind: KafkaConnector
metadata:
  name: mysql-source
spec:
  class: io.debezium.connector.mysql.MySqlConnector
  tasksMax: 1
  config:
    database.hostname: mysql.aws.example.com
    database.port: 3306
    database.user: debezium
    database.password: ${MYSQL_PASSWORD}
    database.server.id: 184054
    database.server.name: aws-mysql
    database.include.list: mydb
    schema.history.internal.kafka.bootstrap.servers: kafka:9092
    schema.history.internal.kafka.topic: schema-changes.mydb

---

3 - Cross-Cloud Object Storage

3.1 Architecture

3.2 Replication with rclone

# rclone configuration
# ~/.config/rclone/rclone.conf
[aws]
type = s3
provider = AWS
access_key_id = xxx
secret_access_key = xxx
region = eu-west-1

[azure]
type = azureblob
account = myaccount
key = xxx

[gcp]
type = google cloud storage
project_number = 123456789
service_account_file = /path/to/sa.json

# Bidirectional sync
rclone sync aws:my-bucket azure:my-container --progress

# Continuous sync
rclone sync aws:my-bucket gcp:my-bucket --checksum --progress

3.3 Multi-Cloud MinIO Gateway

# docker-compose.yml
version: '3'
services:
  minio:
    image: minio/minio
    command: gateway s3 https://s3.amazonaws.com
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: password
    ports:
      - "9000:9000"

---

4 - Event Streaming

4.1 Cross-Cloud Kafka

4.2 Kafka MirrorMaker 2

# mm2.properties
clusters = source, target

source.bootstrap.servers = msk.aws.example.com:9092
target.bootstrap.servers = kafka.azure.example.com:9092

source->target.enabled = true
source->target.topics = .*

replication.factor = 3
checkpoints.topic.replication.factor = 3
heartbeats.topic.replication.factor = 3

offset-syncs.topic.replication.factor = 3

sync.topic.acls.enabled = false

4.3 Confluent Cluster Linking

# Create a link between clusters
confluent kafka link create azure-link \
    --cluster source-cluster-id \
    --destination-cluster destination-cluster-id \
    --destination-bootstrap-server kafka.azure.example.com:9092

# Create a mirror topic
confluent kafka mirror create my-topic \
    --link azure-link \
    --cluster destination-cluster-id

---

5 - Cross-Cloud Backup

5.1 3-2-1 strategy

3 copies - 2 storage types - 1 offsite (another cloud)

5.2 Velero for Kubernetes

# AWS Backup Location
velero backup-location create aws-backups \
    --provider aws \
    --bucket velero-backups-aws \
    --config region=eu-west-1

# Azure Backup Location
velero backup-location create azure-backups \
    --provider azure \
    --bucket velero-backups-azure \
    --config storageAccount=myaccount

# Backup to both
velero backup create daily-backup \
    --storage-location aws-backups \
    --snapshot-volumes

# Copy to Azure
velero backup copy daily-backup \
    --destination-storage-location azure-backups

5.3 Database Backup Script

#!/bin/bash
# backup-db.sh

DATE=$(date +%Y%m%d)
DB_NAME="production"

# Dump PostgreSQL
pg_dump -h rds.aws.example.com -U admin $DB_NAME | gzip > /tmp/$DB_NAME-$DATE.sql.gz

# Upload to AWS
aws s3 cp /tmp/$DB_NAME-$DATE.sql.gz s3://backups-aws/db/

# Upload to Azure
az storage blob upload \
    --account-name backups \
    --container db \
    --file /tmp/$DB_NAME-$DATE.sql.gz \
    --name $DB_NAME-$DATE.sql.gz

# Upload to GCP
gsutil cp /tmp/$DB_NAME-$DATE.sql.gz gs://backups-gcp/db/

# Cleanup
rm /tmp/$DB_NAME-$DATE.sql.gz

---

6 - Data Compliance

6.1 GDPR and localization

Requirement	Solution
Data in the EU	EU regions only
Portability	Export in standard formats
Erasure	Multi-cloud procedures
Audit trail	Centralized logs

6.2 Encryption at rest

# AWS S3
resource "aws_s3_bucket_server_side_encryption_configuration" "main" {
  bucket = aws_s3_bucket.main.id
  
  rule {
    apply_server_side_encryption_by_default {
      kms_master_key_id = aws_kms_key.main.arn
      sse_algorithm     = "aws:kms"
    }
  }
}

# Azure Blob
resource "azurerm_storage_account" "main" {
  # ...
  
  blob_properties {
    versioning_enabled = true
  }
  
  identity {
    type = "SystemAssigned"
  }
}

# GCP Storage
resource "google_storage_bucket" "main" {
  name     = "my-bucket"
  location = "EU"
  
  encryption {
    default_kms_key_name = google_kms_crypto_key.main.id
  }
}

---

Summary

In this chapter, we learned:

• The concept of Data Gravity
• Database replication
• Cross-cloud object storage
• Multi-cloud event streaming
• Backup strategies
• Data compliance

---

Next step

In the next chapter, we will look at Multi-Cloud Monitoring.

→ Next chapter: Multi-Cloud Monitoring

---

← Back to table of contents