Skip to main content

Cloud Monitoring and Logging

1 - Cloud Operations Suite​

Cloud Operations (formerly Stackdriver) is GCP's observability suite.

2 - Cloud Monitoring​

2.1 Automatic metrics​

GKE and Cloud Run automatically send metrics:

  • CPU, Memory, Network
  • Request count, latency
  • Error rates

2.2 Custom metrics​

# Python
from google.cloud import monitoring_v3

client = monitoring_v3.MetricServiceClient()
project_name = f"projects/mon-projet"

series = monitoring_v3.TimeSeries()
series.metric.type = "custom.googleapis.com/my_metric"
series.resource.type = "global"
series.metric.labels["environment"] = "production"

point = monitoring_v3.Point()
point.value.double_value = 42.0
point.interval.end_time.seconds = int(time.time())

series.points = [point]

client.create_time_series(name=project_name, time_series=[series])

2.3 Dashboards​

# Create a dashboard
gcloud monitoring dashboards create --config-from-file=dashboard.json
{
  "displayName": "My Application",
  "gridLayout": {
    "widgets": [
      {
        "title": "CPU Utilization",
        "xyChart": {
          "dataSets": [{
            "timeSeriesQuery": {
              "timeSeriesFilter": {
                "filter": "metric.type=\"kubernetes.io/container/cpu/core_usage_time\""
              }
            }
          }]
        }
      }
    ]
  }
}

3 - Alerting​

3.1 Create an alert​

gcloud alpha monitoring policies create \
    --display-name="High Error Rate" \
    --condition-display-name="Error rate > 1%" \
    --condition-filter='resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class="5xx"' \
    --condition-threshold-value=1 \
    --condition-threshold-comparison=COMPARISON_GT \
    --notification-channels=projects/mon-projet/notificationChannels/123

3.2 Notification channels​

TypeDescription
EmailEmail notification
SlackSlack webhook
PagerDutyPagerDuty integration
WebhookCustom HTTP POST
SMSSMS notification
Pub/SubPub/Sub message

3.3 Example YAML policy​

# alerting-policy.yaml
displayName: "High Latency Alert"
combiner: OR
conditions:
  - displayName: "P99 latency > 1s"
    conditionThreshold:
      filter: |
        resource.type = "cloud_run_revision" AND
        metric.type = "run.googleapis.com/request_latencies"
      aggregations:
        - alignmentPeriod: 60s
          crossSeriesReducer: REDUCE_PERCENTILE_99
          perSeriesAligner: ALIGN_DELTA
      comparison: COMPARISON_GT
      duration: 300s
      thresholdValue: 1000
      trigger:
        count: 1
notificationChannels:
  - projects/mon-projet/notificationChannels/123

4 - Cloud Logging​

4.1 Write logs​

# Python with structuring
import json
import google.cloud.logging

client = google.cloud.logging.Client()
logger = client.logger("my-app")

# Structured log
logger.log_struct({
    "message": "User logged in",
    "userId": "12345",
    "severity": "INFO",
    "httpRequest": {
        "requestMethod": "POST",
        "requestUrl": "/api/login",
        "status": 200,
        "latency": "0.5s"
    }
})

4.2 Log Explorer​

# Query logs via CLI
gcloud logging read \
    'resource.type="cloud_run_revision" AND severity>=ERROR' \
    --limit=100 \
    --format=json

4.3 Log filters​

FilterDescription
severity>=ERRORErrors and above
textPayload:"error"Contains "error"
jsonPayload.userId="123"Specific JSON field
timestamp>="2024-01-01"After a date
resource.labels.service_name="my-app"Specific service

4.4 Log-based Metrics​

# Create a metric from logs
gcloud logging metrics create error_count \
    --description="Count of error logs" \
    --log-filter='severity>=ERROR'

5 - Cloud Trace​

5.1 Automatic instrumentation​

Cloud Run and GKE automatically send traces.

5.2 Manual instrumentation​

# Python with OpenTelemetry
from opentelemetry import trace
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

# Setup
tracer_provider = TracerProvider()
cloud_trace_exporter = CloudTraceSpanExporter()
tracer_provider.add_span_processor(BatchSpanProcessor(cloud_trace_exporter))
trace.set_tracer_provider(tracer_provider)

tracer = trace.get_tracer(__name__)

# Usage
with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("user.id", "12345")
    # ... code

5.3 Trace analysis​

  • Latency distribution
  • Span breakdown
  • Error analysis
  • Service dependencies

6 - Error Reporting​

6.1 Automatic configuration​

Error Reporting automatically captures unhandled exceptions in Cloud Run and GKE.

6.2 Manual reporting​

from google.cloud import error_reporting

client = error_reporting.Client()

try:
    # Code
    pass
except Exception:
    client.report_exception()

7 - Uptime Checks​

7.1 Create an uptime check​

gcloud monitoring uptime-checks create http my-uptime-check \
    --display-name="My App Health Check" \
    --resource-type=uptime-url \
    --monitored-resource-labels=host=my-app.example.com \
    --path=/health \
    --check-interval=60s \
    --timeout=10s

7.2 Verification regions​

  • USA (multiple regions)
  • Europe
  • Asia-Pacific
  • South America

8 - SLO Monitoring​

8.1 Define an SLO​

# slo.yaml
displayName: "Availability SLO"
serviceLevelIndicator:
  basicSli:
    availability: {}
goal: 0.999  # 99.9%
calendarPeriod: MONTH

8.2 Create the SLO​

gcloud slo create \
    --service=my-service \
    --config-from-file=slo.yaml

Summary​

In this chapter, we learned:

  • Cloud Monitoring and metrics
  • Alerts and notifications
  • Cloud Logging and filters
  • Cloud Trace for distributed tracing
  • Error Reporting
  • Uptime Checks and SLO

Next step​

In the next chapter, we will look at Best practices.

→ Next chapter: Best practices

← Back to table of contents