Skip to main content

Kubernetes Multi-Cloud

1 - Multi-Cluster approaches​

1.1 Patterns​

PatternAdvantagesDisadvantages
FederationNative K8s abstractionComplex, less mature
Service MeshTraffic managementNetwork overhead
GitOpsSimple, declarativeNo cross-cluster routing

2 - Service Mesh with Istio​

2.1 Multi-Cluster architecture​

2.2 Istio Multi-Cluster installation​

# Cluster 1 (AWS)
istioctl install --set profile=default \
    --set values.global.meshID=mesh1 \
    --set values.global.multiCluster.clusterName=cluster1 \
    --set values.global.network=network1

# Create the remote secret for cluster 2
istioctl x create-remote-secret --name=cluster1 > cluster1-secret.yaml

# Cluster 2 (Azure) - Apply the secret
kubectl apply -f cluster1-secret.yaml

# Cluster 2 - Installation
istioctl install --set profile=default \
    --set values.global.meshID=mesh1 \
    --set values.global.multiCluster.clusterName=cluster2 \
    --set values.global.network=network2

2.3 Cross-Cluster Service Entry​

# Expose a service from cluster 1 to cluster 2
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: external-api
spec:
  hosts:
    - api.cluster1.local
  location: MESH_INTERNAL
  ports:
    - number: 80
      name: http
      protocol: HTTP
  resolution: DNS
  endpoints:
    - address: api.cluster1.example.com
      ports:
        http: 80

3 - Multi-Cluster GitOps with ArgoCD​

3.1 Architecture​

3.2 Add clusters to ArgoCD​

# Add the AWS cluster
argocd cluster add aws-eks-production --name aws-prod

# Add the Azure cluster
argocd cluster add azure-aks-production --name azure-prod

# Add the GCP cluster
argocd cluster add gcp-gke-production --name gcp-prod

# List the clusters
argocd cluster list

3.3 Multi-Cluster ApplicationSet​

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: my-app-multicloud
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: aws-prod
            url: https://eks.aws.example.com
            region: eu-west-1
          - cluster: azure-prod
            url: https://aks.azure.example.com
            region: westeurope
          - cluster: gcp-prod
            url: https://gke.gcp.example.com
            region: europe-west1
  template:
    metadata:
      name: 'my-app-{{cluster}}'
    spec:
      project: default
      source:
        repoURL: https://github.com/myorg/my-app.git
        targetRevision: main
        path: 'kubernetes/overlays/{{cluster}}'
      destination:
        server: '{{url}}'
        namespace: my-app
      syncPolicy:
        automated:
          prune: true
          selfHeal: true

4 - Rancher Multi-Cluster​

4.1 Installation​

# Install Rancher with Helm
helm repo add rancher-latest https://releases.rancher.com/server-charts/latest

helm install rancher rancher-latest/rancher \
    --namespace cattle-system \
    --create-namespace \
    --set hostname=rancher.example.com \
    --set replicas=3

4.2 Importing clusters​

# Via the Rancher interface:
# 1. Cluster Management > Import Existing
# 2. Generate the kubectl command
# 3. Run it on the target cluster

# Example of a generated command
kubectl apply -f https://rancher.example.com/v3/import/xxxxx.yaml

4.3 Fleet for GitOps​

# fleet.yaml
defaultNamespace: my-app
helm:
  releaseName: my-app
  chart: ./charts/my-app
  values:
    replicaCount: 3

targetCustomizations:
  - name: aws-production
    clusterSelector:
      matchLabels:
        env: production
        cloud: aws
    helm:
      values:
        replicaCount: 5
        
  - name: azure-dr
    clusterSelector:
      matchLabels:
        env: dr
        cloud: azure
    helm:
      values:
        replicaCount: 2

5 - Karmada Federation​

5.1 Architecture​

5.2 PropagationPolicy​

apiVersion: policy.karmada.io/v1alpha1
kind: PropagationPolicy
metadata:
  name: my-app-propagation
spec:
  resourceSelectors:
    - apiVersion: apps/v1
      kind: Deployment
      name: my-app
  placement:
    clusterAffinity:
      clusterNames:
        - aws-cluster
        - azure-cluster
    replicaScheduling:
      replicaDivisionPreference: Weighted
      replicaSchedulingType: Divided
      weightPreference:
        staticWeightList:
          - targetCluster:
              clusterNames:
                - aws-cluster
            weight: 2
          - targetCluster:
              clusterNames:
                - azure-cluster
            weight: 1

6 - Global Traffic with Submariner​

6.1 Connect the clusters​

# Install Submariner on each cluster
subctl deploy-broker --kubeconfig /path/to/management/config

# Join the clusters
subctl join broker-info.subm --kubeconfig /path/to/aws-cluster \
    --clusterid aws-cluster

subctl join broker-info.subm --kubeconfig /path/to/azure-cluster \
    --clusterid azure-cluster

6.2 ServiceExport​

# Export a service to make it accessible cross-cluster
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceExport
metadata:
  name: my-service
  namespace: my-app

6.3 ServiceImport​

# Import a service from another cluster
apiVersion: multicluster.x-k8s.io/v1alpha1
kind: ServiceImport
metadata:
  name: my-service
  namespace: my-app
spec:
  type: ClusterSetIP
  ports:
    - port: 80
      protocol: TCP

7 - Multi-Context kubectl configuration​

# View the contexts
kubectl config get-contexts

# Switch context
kubectl config use-context aws-production

# Use a specific context
kubectl --context=azure-dr get pods

# kubectx to make it easier
kubectx aws-production

# Useful aliases in .bashrc
alias kaws='kubectl --context=aws-production'
alias kazure='kubectl --context=azure-dr'
alias kgcp='kubectl --context=gcp-analytics'

Summary​

In this chapter, we learned:

  • The multi-cluster approaches
  • Istio multi-cluster Service Mesh
  • ArgoCD multi-cluster GitOps
  • Rancher and Fleet
  • Karmada Federation
  • Submariner for networking

Next step​

In the next chapter, we will look at Multi-Cloud Networking.

→ Next chapter: Multi-Cloud Networking

← Back to table of contents