Platform Engineering

What Makes a Good IDP?

An Internal Developer Platform is the sum of all the technology and tooling that a platform team provides to developers. It's not a single product you install. It's an integrated set of capabilities that, taken together, create a self-service developer experience. The best IDPs share common characteristics: they are self-service (no tickets needed), opinionated but flexible (golden paths with escape hatches), composable (built from best-of-breed components), and observable (you can see what's happening at every stage).

Think of an IDP as having five layers: the developer portal (the UI), service catalog (what exists), golden paths (how to create things), infrastructure orchestration (how things are provisioned), and observability integration (how to see what's happening). Each layer builds on the ones below it.

Backstage Deep Dive

Backstage, originally developed at Spotify, is the most widely adopted open-source framework for building developer portals. It was born from Spotify's experience managing 2,000+ microservices with hundreds of engineering teams. The core problem: nobody could find anything. Who owns this service? Where are the docs? What's the deployment status? What APIs does it expose? Backstage answers all of these questions through a unified portal.

Architecture

Backstage is a React frontend with a Node.js backend. It uses a plugin architecture that's core to its extensibility. The backend connects to a PostgreSQL database for the software catalog and can integrate with virtually any external system through plugins.

The Software Catalog

The software catalog is Backstage's most important feature. It provides a centralized registry of all software components, APIs, resources, and their relationships. Every entity in the catalog is described by a YAML descriptor file that lives alongside the code:

The Scaffolder (Golden Path Templates)

The Scaffolder lets platform teams define templates that developers use to create new services, libraries, or infrastructure. This is where "golden paths" come to life. Instead of copying an old repo and deleting half the files, developers fill out a form and get a fully configured, production-ready service.

Build vs. Buy: The IDP Decision Matrix

Should you build your IDP on Backstage, buy a commercial solution like Port or Humanitec, or build something entirely custom? This decision depends on your team's capabilities, timeline, and specific needs.

IDP Maturity Model

Not every organization starts at the same level. Here's a maturity model for IDP adoption:

1. Level 0 - Ad Hoc: No standardized tooling. Each team manages its own CI/CD, infrastructure, and deployment. Knowledge lives in individual heads. New service creation takes days or weeks.
2. Level 1 - Standardized: Common CI/CD pipeline exists. Basic deployment automation. Shared infrastructure templates. Service creation takes hours to a day.
3. Level 2 - Self-Service: Developer portal exists. Golden path templates for common patterns. Self-service infrastructure provisioning. Service creation takes minutes.
4. Level 3 - Optimized: Full IDP with software catalog, automated compliance, cost visibility, and integrated observability. Developer satisfaction is measured and improved. Service creation is measured in minutes with full production readiness.
5. Level 4 - Autonomous: Platform capabilities are AI-augmented. Automated remediation, predictive scaling, intelligent routing. The platform continuously improves itself based on usage patterns.

The Service Catalog: Your Single Source of Truth

A service catalog answers the most fundamental questions in a microservices architecture: What services exist? Who owns them? What do they depend on? Are they healthy? Where are the docs? Without a catalog, these questions devolve into Slack threads, tribal knowledge, and archaeological digs through GitHub.

An effective service catalog must be automatically populated (not manually maintained), always up-to-date (synced from source control and runtime), relationship-aware (shows dependencies and dependents), and actionable (links directly to dashboards, runbooks, and deployment tools). If developers have to manually update the catalog, it will rot within weeks. Automate everything.

Golden Paths in Practice

A golden path is more than a template. It's a complete, supported workflow that takes a developer from "I have an idea" to "my service is running in production." A well-designed golden path includes: a project template with sensible defaults (logging, health checks, metrics instrumentation, Dockerfile, CI/CD config), automated repository creation with branch protection rules, CI/CD pipeline configuration, infrastructure provisioning (database, cache, message queue), observability setup (dashboards, alerts, SLO definitions), documentation scaffolding, and catalog registration. The key is that all of this happens in one click. The developer fills out a form (service name, team, language, database type) and everything else is automated.

The Modern CI/CD Landscape

CI/CD is the backbone of any platform. It's the system that transforms developer intent (a git push) into running software (a production deployment). Getting this right has an outsized impact on developer productivity. A slow or unreliable pipeline is a daily tax on every engineer in the organization.

The CI/CD landscape has consolidated significantly. GitHub Actions has become the dominant player for GitHub-hosted repositories, with GitLab CI leading for GitLab shops. Jenkins, while still widely deployed, is increasingly viewed as legacy. CircleCI and Travis CI have lost significant market share. The trend is clear: CI/CD is moving toward tighter integration with source control platforms.

GitHub Actions Deep Dive

GitHub Actions is the most popular CI/CD platform for good reason: it's integrated directly into GitHub, has a massive marketplace of pre-built actions, supports matrix builds, reusable workflows, and composite actions, and offers generous free-tier resources.

Production-Ready Workflow

Composite Actions vs Reusable Workflows

GitHub Actions provides two mechanisms for code reuse, and understanding the distinction is important for platform teams. Composite actions combine multiple steps into a single action that can be referenced from any workflow. They're ideal for encapsulating a common sequence -- like "set up our Python environment with the right version, install dependencies, configure our internal PyPI mirror." Reusable workflows are entire workflow definitions that can be called from other workflows. They're better for standardizing complete pipelines -- like "our standard Python CI pipeline with linting, testing, security scanning, and artifact publishing."

The key architectural difference: composite actions run inline within the calling job (sharing the same runner, filesystem, and environment variables), while reusable workflows run as separate jobs (with their own runner and isolated filesystem). For platform teams, the typical pattern is to provide reusable workflows for standard pipelines and composite actions for common setup sequences.

Reusable Workflows

One of the most powerful features of GitHub Actions for platform teams is reusable workflows. Instead of duplicating CI/CD configuration across hundreds of repositories, you define workflows once in a central repository and reference them from individual repos. This is how you enforce organizational standards without mandating them.

Deployment Strategies

Choosing the right deployment strategy is a critical platform decision. Each strategy trades off between speed, safety, and complexity.

CI/CD Security: Supply Chain Protection

Your CI/CD pipeline is one of the most privileged systems in your organization. It has credentials to deploy to production, access to secrets, and the ability to modify running services. Securing it is non-negotiable.

Pipeline Security Checklist

• Pin action versions by SHA, not by tag. Tags can be moved; SHAs cannot. Use actions/checkout@abc123def instead of actions/checkout@v4.
• Use OIDC for cloud authentication instead of long-lived secrets. GitHub Actions supports OIDC federation with AWS, GCP, and Azure.
• Scan dependencies with Dependabot, Snyk, or Trivy in every PR.
• Sign artifacts with Sigstore/cosign. This creates verifiable provenance for your container images.
• Limit permissions using the permissions key. Default to read-only and explicitly grant write access only where needed.
• Require code review for workflow file changes. A compromised workflow can exfiltrate secrets.

Monorepo CI Challenges

If your organization uses a monorepo (or is considering one), CI/CD becomes significantly more complex. The core challenge: when a PR modifies files in /services/payment/, you don't want to build and test every service in the repository. You need affected-target detection.

Tools like nx affected, Turborepo's pipeline caching, and Bazel's build graph analysis solve this problem at different levels of sophistication. The platform team's job is to provide this infrastructure so that individual teams don't need to think about it. A well-configured monorepo CI should build only what changed, cache everything that didn't, and run tests in parallel across as many machines as needed.

The practical challenge goes deeper than just "build what changed." You need to handle transitive dependencies -- if the shared payment-types library changes, every service that imports it must be rebuilt and retested. This requires building a dependency graph of your repository. Nx and Bazel do this natively. For other setups, you can build a lightweight dependency graph using tools like madge (JavaScript), pydeps (Python), or custom scripts that parse import statements. At Shopify, their monorepo CI system analyzes Ruby dependency trees to determine which test suites to run, reducing average CI time from 45 minutes to under 8 minutes.

Secrets Management in CI/CD

Secrets in CI/CD pipelines are a common attack vector. The best practices have converged around a few key patterns. First, use OIDC federation instead of long-lived credentials. GitHub Actions can assume AWS IAM roles directly via OIDC, eliminating the need to store AWS access keys as repository secrets. Second, use environment-level secrets in GitHub Actions. A secret scoped to the "production" environment requires manual approval before a workflow can access it, creating a human gate for production deployments. Third, never echo secrets in logs -- GitHub Actions masks known secrets, but dynamic secrets assembled at runtime can leak. Fourth, rotate secrets regularly and use short-lived tokens wherever possible. A leaked CI secret that expires in 1 hour is far less dangerous than one that lasts forever.

Pipeline Performance Optimization

Pipeline speed directly correlates with developer productivity. Every minute added to a CI pipeline is multiplied by the number of PRs per day across the organization. At 100 PRs/day, a 5-minute improvement saves over 8 hours of developer wait time daily.

• Dependency caching: Cache pip, npm, Maven, and Gradle dependencies between runs. This alone can save 2-5 minutes per build.
• Docker layer caching: Use BuildKit cache mounts and GitHub Actions cache backend. Rebuilding only changed layers can reduce Docker builds from 10 minutes to 30 seconds.
• Test parallelization: Use pytest-xdist, Jest workers, or Go's built-in parallel testing. Split test suites across multiple CI runners.
• Selective testing: Only run tests for changed packages and their dependents. Tools like Nx and Bazel provide this natively.
• Self-hosted runners: For large organizations, self-hosted runners on dedicated hardware can be 2-5x faster than GitHub-hosted runners due to better CPU, more RAM, and local caching.

Why IaC Is Non-Negotiable

Infrastructure as Code is the practice of managing and provisioning infrastructure through machine-readable definition files rather than physical hardware configuration or interactive configuration tools. In 2026, this isn't a nice-to-have. It's table stakes. If your infrastructure isn't defined in code, it's defined by whoever clicked buttons in the AWS console last, and you'll never know exactly what state it's in.

IaC provides three critical capabilities: reproducibility (create identical environments), auditability (every change is a PR with review and history), and scalability (provision 100 environments as easily as one).

Terraform Deep Dive

Terraform remains the dominant IaC tool, with the largest ecosystem of providers and the deepest community knowledge. Despite HashiCorp's controversial BSL license change in 2023 (and the resulting OpenTofu fork), Terraform continues to be the default choice for most organizations.

State Management

Terraform state is the most important (and most dangerous) concept in the Terraform ecosystem. The state file is a JSON document that maps your Terraform configuration to real-world resources. It's how Terraform knows that aws_instance.web in your config corresponds to i-abc123def in AWS.

Module Design Patterns

Well-designed Terraform modules are the building blocks of a scalable IaC practice. A good module encapsulates a reusable infrastructure pattern with sensible defaults and clear input/output contracts.

Workspace and Environment Strategy

Terraform workspaces are one of the most misunderstood features. They allow you to maintain multiple state files within the same configuration directory, typically used for managing identical infrastructure across environments (dev, staging, production). However, many teams have been burned by workspaces because they make it easy to accidentally apply production changes when you thought you were in dev.

The recommended pattern for most organizations is to use directory-based separation rather than workspaces. Have a separate directory (and therefore separate state file) for each environment. This is more verbose but eliminates an entire class of operator errors. Workspaces still make sense for certain patterns, like managing per-tenant infrastructure in a multi-tenant SaaS application, where the configuration is truly identical and the blast radius of a mistake is limited to a single tenant.

Drift Detection

Infrastructure drift occurs when the actual state of your cloud resources diverges from what's defined in your Terraform configuration. Someone clicked a button in the AWS console. An auto-scaling policy modified instance counts. A security team manually updated a security group. Drift is inevitable in organizations of any size, and ignoring it erodes the value of IaC entirely.

Set up scheduled terraform plan runs (via CI/CD or tools like Spacelift, env0, or Terraform Cloud) that detect drift and notify your team. At Shopify, infrastructure drift alerts are treated like production incidents -- if Terraform state doesn't match reality, someone investigates within 24 hours. The fix is either to update the Terraform config to match reality or to run terraform apply to force reality back to match the config.

Atlantis: PR-Based Terraform Workflows

Atlantis is an open-source tool that automates Terraform via pull requests. When a developer opens a PR that modifies Terraform files, Atlantis automatically runs terraform plan and posts the output as a PR comment. Reviewers can see exactly what infrastructure changes will be made before they approve. Once approved, a comment like atlantis apply executes the changes.

This workflow is powerful because it brings the same code review rigor to infrastructure changes that we already apply to application code. No more "I ran terraform apply from my laptop and something went wrong." Every change is planned, reviewed, and applied through a controlled process.

Pulumi: Real Programming Languages for IaC

Pulumi takes a fundamentally different approach from Terraform: instead of a domain-specific language (HCL), you write infrastructure in TypeScript, Python, Go, or C#. This means you get loops, conditionals, functions, classes, type checking, IDE support, and the full power of a real programming language.

GitOps with ArgoCD

GitOps is the practice of using Git as the single source of truth for declarative infrastructure and applications. ArgoCD is the most popular GitOps operator for Kubernetes. It continuously monitors a Git repository and automatically synchronizes the desired state (defined in Git) with the actual state (running in the cluster).

Infrastructure Testing

Infrastructure code deserves the same testing rigor as application code. Terratest (Go-based) and the Terraform testing framework (built-in since 1.6) let you write automated tests that provision real infrastructure, validate it works, and tear it down. Policy-as-code tools like OPA (Open Policy Agent) and HashiCorp Sentinel let you define organizational policies (e.g., "all S3 buckets must have encryption enabled") and enforce them at plan time.

Kubernetes: The Operating System of the Cloud

Kubernetes has won the container orchestration war. It's the default platform for running containerized workloads at scale, and understanding how to operate it in production is essential for any platform engineer. But let's be clear: Kubernetes is not always the right answer. If you have fewer than 10 services, your team doesn't have K8s expertise, and you're running straightforward web applications, managed services like AWS ECS or even Heroku might be better choices. K8s shines when you have many services, need sophisticated scheduling, or require portable workloads across clouds.

Managed vs. Self-Hosted

Unless you're Google, run a managed Kubernetes service. EKS, GKE, and AKS handle the control plane, etcd management, upgrades, and HA for you. The operational burden of self-hosted Kubernetes is enormous and rarely justified.

Resource Management

Misconfigured resource requests and limits are the number one cause of K8s performance issues and cost overruns. Every container in Kubernetes can specify CPU and memory requests (guaranteed minimum) and limits (hard maximum).

Autoscaling: HPA, VPA, and KEDA

HPA (Horizontal Pod Autoscaler) adds or removes pod replicas based on CPU, memory, or custom metrics. It's the default autoscaling mechanism and works well for stateless services. VPA (Vertical Pod Autoscaler) adjusts the CPU and memory requests of individual pods based on historical usage. It's useful for right-sizing, but be cautious in production since it restarts pods to apply changes. KEDA (Kubernetes Event-Driven Autoscaling) extends HPA with event-driven scaling based on external metrics like SQS queue depth, Kafka lag, or Prometheus queries. It's essential for queue-based workloads.

A critical production lesson: always configure scaleDown stabilization windows. Without them, your pods will flap up and down in response to short traffic spikes. The behavior section lets you control how aggressively the HPA scales in each direction. Scale up quickly (double capacity every 60 seconds if needed) but scale down slowly (remove only 10% every 60 seconds, after a 5-minute stabilization window). This prevents premature scale-down during traffic fluctuations.

Namespace Strategy and Multi-Tenancy

Namespace design is an underappreciated platform decision. The two main approaches are namespace-per-team and namespace-per-service. Namespace-per-team is simpler to manage and aligns with organizational boundaries -- the payments team gets the payments namespace. Namespace-per-service provides better isolation but creates more administrative overhead. Most organizations start with namespace-per-team and graduate to namespace-per-service as they grow. Regardless of your choice, enforce resource quotas on every namespace. Without quotas, a single runaway service can starve the entire cluster of CPU or memory.

Helm Charts vs. Kustomize

Helm and Kustomize represent two philosophies for managing Kubernetes manifests. Helm uses Go templating to generate manifests from parameterized charts. It's powerful, has a massive ecosystem of community charts, and supports lifecycle management (install, upgrade, rollback). The downside: Helm templates can become unreadable spaghetti of {{ if .Values.something }} conditionals. Kustomize uses overlay-based patching -- you write plain YAML and apply patches for different environments. It's simpler, produces readable output, and is built into kubectl. The downside: complex customizations require many small patch files. The pragmatic choice for most platform teams: use Helm for third-party applications (ingress controllers, monitoring stacks, databases) where community charts save enormous effort, and Kustomize for your own applications where you control the base manifests.

Networking: Ingress and Service Types

Kubernetes networking is one of the most confusing aspects for newcomers. Every pod gets its own IP address. Pods can communicate with any other pod directly (flat networking). Services provide stable DNS names and load balancing across pods. But getting traffic from outside the cluster to your services requires an Ingress resource and an Ingress controller.

The three main Ingress controllers are NGINX Ingress Controller (the default, battle-tested, handles most use cases), Traefik (auto-discovers services, great for dynamic routing), and AWS ALB Ingress Controller (provisions actual AWS ALBs, best for AWS-native shops). For production, configure TLS termination at the ingress level using cert-manager with Let's Encrypt for automatic certificate provisioning. Use Network Policies to restrict pod-to-pod communication -- by default, any pod can talk to any other pod in the cluster, which is a significant security risk. At minimum, deny all ingress traffic by default and explicitly allow only the connections each service needs.

Service Mesh: Do You Actually Need One?

A service mesh (Istio, Linkerd) provides traffic management, observability, and security at the network layer without modifying application code. It sounds great in theory. In practice, it adds significant complexity, latency, and operational overhead. The honest answer: most organizations under 50 services don't need a service mesh. Start with standard K8s networking and add a mesh only when you have concrete requirements (mTLS everywhere, canary routing, circuit breaking) that can't be met by simpler solutions.

If you do need a mesh, Linkerd is the simpler choice. It's lightweight (Rust-based proxy), has minimal configuration, and provides the core features (mTLS, observability, traffic splitting) without the complexity of Istio. Istio is more powerful but significantly harder to operate -- its control plane has multiple components, configuration is verbose, and debugging mesh issues requires deep expertise. At Lyft, switching from Istio to Envoy-based direct configuration reduced their service mesh operational incidents by 70%.

Secrets Management in K8s

Kubernetes Secrets are base64-encoded, not encrypted. Anyone with API access can read them. For production workloads, integrate with an external secret manager: AWS Secrets Manager, HashiCorp Vault, or GCP Secret Manager. Use the External Secrets Operator to sync external secrets into K8s automatically. Never commit secrets to Git, even encrypted ones. Use sealed-secrets or external-secrets-operator instead.

K8s Cost Optimization

Kubernetes cost management is a discipline unto itself. Common waste patterns include: over-provisioned resource requests (CPU requested but never used), idle namespaces from abandoned services, persistent volumes attached to stopped pods, and load balancers for internal-only services. Tools like Kubecost, Opencost, and cloud-native cost analyzers can surface these issues. The platform team should provide cost visibility per team and per service, ideally integrated into the developer portal.

Beyond Monitoring: What Observability Actually Means

Monitoring tells you when something is wrong. Observability tells you why. In a distributed system with hundreds of services, you can't predict every failure mode. You need systems that let you ask arbitrary questions about your infrastructure and applications without deploying new instrumentation. That's observability.

The three traditional pillars are metrics (numerical measurements over time), logs (discrete events), and traces (request flows across services). Modern observability adds a fourth pillar: profiling (continuous CPU and memory profiling to understand performance at the code level).

OpenTelemetry: The Standard

OpenTelemetry (OTel) has become the de facto standard for instrumentation. It provides vendor-neutral APIs, SDKs, and tools for generating telemetry data. The key insight: instrument once with OTel, and send data to any backend (Datadog, Grafana, Honeycomb, Jaeger). This decouples your instrumentation from your observability vendor, which is critical for avoiding lock-in.

Datadog vs. Grafana Stack

SLOs, SLIs, and Error Budgets

Service Level Objectives (SLOs) are the foundation of reliability engineering. An SLO is a target for a Service Level Indicator (SLI). For example: "99.9% of HTTP requests should return a non-error response within 200ms." The SLI is the measurement (request success rate). The SLO is the target (99.9%). The error budget is the inverse: if your SLO is 99.9%, your error budget is 0.1% -- that's about 43 minutes of downtime per month.

Alerting Best Practices

Alert fatigue is real. If your on-call engineers get 50 alerts a night and ignore most of them, your alerting system is worse than useless -- it's actively harmful because it trains people to ignore alerts. The fix is SLO-based alerting: alert on error budget burn rate, not individual metrics. If your payment service's error budget is burning at 10x the sustainable rate, that's an alert. A single 500 error is not.

Grafana Dashboard Design

A well-designed Grafana dashboard is worth more than a thousand alerts. The problem is that most Grafana dashboards are terrible -- 40 panels crammed onto one page, half of them showing metrics nobody understands, with no clear hierarchy of information. The best dashboards follow the USE method (Utilization, Saturation, Errors) for infrastructure and the RED method (Rate, Errors, Duration) for services.

Structure your dashboards in layers. The top-level dashboard is the service overview: request rate, error rate, latency percentiles (P50, P95, P99), and SLO burn rate. One glance tells you if the service is healthy. Below that, have a detailed dashboard with per-endpoint breakdowns, dependency health, resource utilization, and cache hit rates. Finally, have a debugging dashboard with low-level metrics like GC pauses, thread pool utilization, and connection pool stats. Most on-call engineers should never need to go past the overview dashboard unless something is actually wrong.

Log Aggregation at Scale

Logs are the most expensive telemetry signal to store and query. At 100+ services each emitting structured JSON logs, you can easily generate terabytes per day. The platform team must make deliberate decisions about log retention, sampling, and indexing. Loki's approach (index only labels, store log lines in cheap object storage) is dramatically cheaper than Elasticsearch's full-text indexing, but it trades query flexibility for cost. Most organizations find that Loki's label-based queries ({service="payment",level="error"}) cover 90% of debugging use cases.

Establish logging standards across the organization: use structured JSON, include request IDs for correlation, include trace IDs for linking to distributed traces, and use consistent severity levels. The platform team should provide logging libraries that enforce these standards automatically, so individual developers don't need to think about log format.

Distributed Tracing Patterns

In a microservices architecture, a single user request might traverse 10+ services. Without distributed tracing, debugging performance issues or errors across service boundaries is nearly impossible. OpenTelemetry's trace context propagation (W3C Trace Context headers) is the standard. Every service in the request path adds its span to the trace, creating a complete picture of the request's journey through the system.

The platform team should provide auto-instrumentation for common frameworks (Express, Flask, Spring Boot, gRPC) so that developers get tracing out of the box without writing instrumentation code. OpenTelemetry auto-instrumentation libraries make this straightforward for most languages.

The trace above immediately reveals that the Stripe API call is the bottleneck -- 65ms of a 250ms request. Without tracing, you'd be guessing. With it, you know exactly where to optimize. The key implementation detail: make sure your trace sampling strategy is right. Tracing 100% of requests in production is prohibitively expensive. Sample 1-5% of normal traffic, but always trace 100% of error responses and slow requests (those exceeding your P99 target). This ensures you have traces for every interesting request without drowning in data.

Observability-Driven Development

The emerging practice of observability-driven development (ODD) means thinking about observability before writing code, not after. When designing a new feature, ask: "How will I know if this is working correctly in production? What metrics, logs, and traces do I need?" This shifts observability from an afterthought to a design requirement. Platform teams enable ODD by providing easy-to-use instrumentation libraries, standardized dashboards that auto-populate when new services register, and SLO templates that give teams a starting point for defining their reliability targets.

SRE Principles: The Google Way, Distilled

Site Reliability Engineering, as defined by Google, is "what happens when you ask a software engineer to design an operations function." The core principles are: embrace risk (100% reliability is neither possible nor desirable), eliminate toil (manual, repetitive work should be automated), simplicity (simpler systems are more reliable), and error budgets (use the error budget to balance reliability and velocity).

The most important SRE concept for platform leaders is the error budget. It fundamentally changes the reliability conversation from "we must never have outages" to "we have a budget for failures, and we spend it on shipping features." When the error budget is healthy, teams should ship faster. When it's depleted, teams should focus on reliability work. This creates a self-correcting system.

Toil: Measuring and Reducing It

Toil is work that is manual, repetitive, automatable, reactive, and devoid of enduring value. Every SRE team should measure the percentage of time spent on toil. Google's target is no more than 50% toil; the rest should be engineering work that reduces future toil.

Common Sources of Toil in Platform Engineering

• Manually provisioning databases, namespaces, or certificates
• Manually scaling services during traffic spikes
• Manually rotating secrets or certificates
• Responding to alerts that don't require human judgment
• Manually running deployment pipelines or rollbacks
• Copy-pasting configurations between environments

Each of these is a candidate for automation. The platform team's explicit goal should be to systematically eliminate these toil sources through self-service tooling, automation, and better defaults.

On-Call Rotation Design

A well-designed on-call rotation is critical for team sustainability. Poorly designed on-call leads to burnout, attrition, and worse reliability (tired engineers make worse decisions).

Incident Management Framework

When things go wrong (and they will), having a structured incident management process is the difference between a 30-minute resolution and a 6-hour fire drill. The Incident Commander (IC) model, popularized by PagerDuty and Blameless, assigns clear roles during an incident.

Incident Severity Classification

Blameless Postmortems

The postmortem is the most important artifact of any incident. Done well, it prevents recurrence and builds organizational knowledge. Done poorly (or not at all), the same incidents repeat endlessly.

A blameless postmortem focuses on systems and processes, not people. "The deployment went out without adequate testing" is blameless. "John deployed without testing" is not. The distinction matters because blame discourages honesty, and dishonest postmortems don't prevent recurrences.

Postmortem Template

• Title and date: Descriptive title, incident date, and severity
• Summary: 2-3 sentence description of what happened and the impact
• Timeline: Minute-by-minute account from detection to resolution
• Root cause: Technical root cause (often multiple contributing factors)
• Impact: Quantified impact (users affected, revenue lost, duration)
• What went well: Things that worked during the response
• What went poorly: Things that slowed down the response
• Action items: Specific, assigned, time-bound tasks to prevent recurrence. Each action item must have an owner and a due date.
• Lessons learned: Broader takeaways for the organization

SRE Team Structures

There are three common models for organizing SRE teams, and each has tradeoffs. The centralized model has a single SRE team that supports all product teams. This works well at smaller organizations (under 200 engineers) because it concentrates expertise and creates consistent practices. The downside: the SRE team becomes a bottleneck, and they lack deep context on any individual product. The embedded model places SREs directly on product teams. They build deep domain knowledge and have strong relationships with developers. The downside: practices diverge across teams, and SREs can become glorified ops people who just respond to pages. The hybrid model (used by Google, Netflix, and most large platform organizations) has a central SRE platform team that builds shared tooling and defines standards, plus embedded SREs on critical product teams who apply those standards with domain-specific knowledge.

Chaos Engineering

Chaos engineering is the practice of deliberately injecting failures into your system to verify that your resilience mechanisms work. Netflix pioneered this with Chaos Monkey (randomly killing production instances), but the field has matured significantly. Modern chaos engineering is hypothesis-driven: "We believe that if we kill 2 of 3 payment service replicas, the remaining replica will handle traffic without errors." You define the steady state, introduce the perturbation, and measure whether the steady state was maintained.

Tools like Litmus (Kubernetes-native), Gremlin (commercial), and AWS Fault Injection Simulator make it straightforward to run chaos experiments. Start with non-production environments, graduate to production during low-traffic windows, and eventually run continuous chaos in production. The most common first experiment: kill a random pod and verify that the service recovers within 30 seconds without user impact. If that fails, you've found a real problem that would have eventually caused an outage.

Runbooks That Actually Get Used

A runbook is a documented procedure for handling a specific operational scenario. Most runbooks are written once and never read because they're outdated, hard to find, or too vague to be useful. Effective runbooks are: linked directly from alerts (every PagerDuty alert should have a runbook URL), tested regularly (during game days or chaos engineering experiments), maintained by the on-call rotation (the person who used the runbook updates it), and structured for a stressed human at 3am (numbered steps, copy-pasteable commands, clear decision trees).

Integrate runbooks into your IDP. When a developer clicks on an alert in the developer portal, the relevant runbook should be one click away. Tools like Backstage TechDocs, Notion, or Confluence can host runbooks, but the key is discoverability and freshness.

Why Build Systems Matter for Platform Engineering

The build system is the foundation of the developer inner loop: edit, build, test, repeat. Every second added to the build-test cycle is multiplied by every developer, every iteration, every day. At scale, build performance isn't a nice-to-have -- it's a competitive advantage. Google, Meta, and other large tech companies invest heavily in build infrastructure because they understand that developer productivity compounds.

Bazel Deep Dive

Bazel, originally developed at Google (where it's known as Blaze), is the most powerful build system available for large codebases. It's designed for correctness, reproducibility, and speed. Bazel's key features: hermetic builds (builds depend only on declared inputs, ensuring reproducibility), content-based caching (only rebuild what changed, based on file contents not timestamps), remote execution (distribute builds across a cluster of machines), and multi-language support (Java, Go, Python, C++, TypeScript, and more through rules).

When to Use Bazel

Bazel makes sense when your codebase exceeds what language-specific tools can handle efficiently. If your Go monorepo takes 20 minutes to test, if your Java project has 500+ modules, or if you have a polyglot monorepo with shared protobuf definitions, Bazel is likely the right choice. For smaller codebases, the learning curve and configuration overhead may not be justified.

Remote Caching and Remote Execution

Remote caching is the single most impactful Bazel feature for developer productivity. When developer A builds a target, the build outputs are stored in a shared cache (typically backed by a service like Buildbarn, BuildBuddy, or Google's Remote Build Execution). When developer B builds the same target with the same inputs, the outputs are fetched from cache instead of rebuilding. This can reduce build times from minutes to seconds.

Turborepo for JavaScript/TypeScript Monorepos

If your organization runs a JavaScript or TypeScript monorepo, Turborepo (acquired by Vercel) is the standard choice. It provides task scheduling with dependency awareness, local and remote caching, and parallel execution. It's significantly simpler than Bazel for JS-only codebases.

Monorepo vs. Polyrepo

Developer Productivity Metrics

You can't improve what you don't measure. The DORA metrics (Deployment Frequency, Lead Time for Changes, Change Failure Rate, Mean Time to Recovery) measure software delivery performance at the team level. The SPACE framework (Satisfaction, Performance, Activity, Communication, Efficiency) provides a more holistic view of developer productivity that includes subjective experience.

Developer Environment Standardization

Reproducible development environments eliminate the "works on my machine" problem. Dev Containers (VS Code) define development environments as Docker containers with all dependencies pre-installed. GitHub Codespaces provide cloud-hosted development environments that spin up in seconds. Gitpod offers a similar experience with more flexibility on hosting. The platform team should provide pre-configured dev environments for every supported stack, so a new developer can go from git clone to running tests in under 5 minutes.

The critical insight about dev environments: the platform team owns the base images. Individual teams customize on top with service-specific dependencies, but the platform team maintains the language-runtime base images with approved compiler versions, security patches, and standard tooling pre-installed. This creates a layer of consistency while preserving team autonomy. At Shopify, their dev container images are rebuilt nightly with the latest security patches and pushed to an internal registry. When a developer opens a codespace, they automatically get the latest secure environment.

Reducing Inner Loop Time: A Practical Guide

The developer inner loop (edit, build, test, see result) is where developers spend most of their time. Every optimization here has outsized impact. The key strategies:

• Hot reload: For interpreted languages (Python, Node.js, Ruby), use file watchers that automatically restart the server on file changes. Tools like nodemon, uvicorn --reload, and guard eliminate the manual restart step entirely.
• Incremental compilation: For compiled languages (Go, Rust, Java), use incremental build tools. Go's build cache is excellent by default. For Java, Gradle's incremental compilation is much faster than Maven. For Rust, use cargo watch with incremental compilation enabled.
• Local test filtering: Run only the tests affected by your change. pytest -k "test_payment" runs only matching tests. jest --onlyChanged runs tests for changed files. go test ./services/payment/... runs tests only in the changed package.
• Mock external dependencies: Calls to external APIs, databases, and message queues should be mockable locally. Docker Compose files that spin up local Postgres, Redis, and localstack (mock AWS) in seconds are table stakes for a modern platform.
• Preview environments: For frontend changes, deploy preview environments on every PR (Vercel, Netlify, or custom K8s-based preview envs). This eliminates the need to run the full stack locally for UI changes.

Why Feature Flags Are a Platform Capability

Feature flags decouple deployment from release. You deploy code to production continuously, but you control who sees new features through flags. This is transformative because it means deployment becomes routine (just merging code) and release becomes a business decision (flipping a flag). For platform teams, feature flags are infrastructure that enables progressive delivery, experimentation, and safe rollouts.

Without feature flags, your deployment pipeline is your release process. Every deployment is a release. Every release is all-or-nothing. With feature flags, you can deploy ten times a day and gradually roll out features to 1%, then 5%, then 25%, then 100% of users, monitoring metrics at each stage.

Feature Flag Architecture

LaunchDarkly vs. Unleash vs. Flagsmith

Flag Implementation Pattern

Types of Feature Flags

Not all flags are created equal. Understanding the different types helps you manage them properly:

• Release Flags: Enable/disable new features during rollout. Short-lived (days to weeks). Should be removed after full rollout. Example: new-checkout-flow.
• Operational Flags: Control system behavior in production. Long-lived. Used for graceful degradation. Example: enable-recommendation-engine (turn off during high load).
• Experiment Flags: A/B testing variants. Medium-lived (weeks to months). Tied to analytics. Example: pricing-page-variant (A/B/C test different pricing layouts).
• Permission Flags: Gate features by user segment. Long-lived. Example: premium-feature-dashboard (only for paid users).

Progressive Delivery with Feature Flags

Progressive delivery combines feature flags with automated analysis to create self-healing releases. The pattern: deploy new code with a flag defaulting to off. Enable the flag for 1% of traffic. Automatically compare error rates, latency, and business metrics between the flag-on and flag-off populations. If metrics are healthy, automatically increase to 5%, 25%, 50%, 100%. If metrics degrade, automatically roll back the flag to 0%.

This is the holy grail of continuous delivery: fully automated, metrics-driven releases that require zero human intervention for successful deployments and automatically protect users from bad deployments. Tools like Flagger (for Kubernetes), LaunchDarkly (with metric integration), and custom solutions can implement this pattern.

The implementation detail that matters: statistical significance. You can't compare a 1% sample to the 99% control group and draw meaningful conclusions about error rate differences. You need enough traffic at each stage to achieve statistical significance before promoting. For a service handling 10,000 requests per minute, 1% gives you 100 requests per minute -- enough to detect a 5% error rate increase with confidence within 10-15 minutes. For lower-traffic services, you may need to keep the canary running for hours before promoting. Build this math into your progressive delivery automation.

Trunk-Based Development with Flags

Feature flags are the enabling mechanism for trunk-based development at scale. Without flags, long-lived feature branches are the only way to hide incomplete work. With flags, developers merge to main every day (or multiple times per day), and incomplete features are simply gated behind flags that default to off. This eliminates merge conflicts, reduces integration risk, and keeps the main branch always deployable.

The workflow is straightforward: developer creates a feature flag for their new capability, wraps new code paths in flag checks, merges small increments to main daily, and when the feature is complete and tested, gradually rolls out via the flag dashboard. The feature branch never existed -- all work happened on main, protected by the flag. This is how Netflix, Google, and most high-performing engineering organizations operate, and it's a night-and-day improvement over gitflow for teams that adopt it.

Percentage Rollouts and Canary Patterns

The power of percentage rollouts is that they let you validate changes with real traffic before committing to a full release. The key technical detail: percentage rollouts must be sticky. If user Alice falls into the 5% cohort, she should consistently see the new experience on every request, not randomly flip between old and new. This is typically implemented by hashing the user ID with the flag name and using the hash to determine the cohort. This ensures consistency without requiring server-side session state.

Kill Switches for Incidents

Operational flags serve as kill switches during incidents. When your recommendation service is overloaded and dragging down the checkout page, a single flag toggle can disable recommendations without a code deployment. This is faster, safer, and more reversible than any code change. Every external dependency should have a kill switch. Your platform should make it trivial to add one.

Why Database Migrations Are the Hardest Problem

Every other topic in this course has a reasonable "undo" button. Bad deployment? Roll back. Bad config? Revert. Bad feature flag? Toggle it off. But database migrations? If you've dropped a column, that data is gone. If you've locked a table with 500 million rows, your application is down until the migration completes. Database migrations at scale are the highest-stakes operation in software engineering, and they deserve the deepest respect.

The fundamental tension is this: your database schema needs to evolve (new features require new columns, tables, indexes), but your database is a shared, stateful resource that serves live traffic. Any schema change that takes locks, rewrites data, or changes constraints can potentially bring down your application. The techniques in this module exist to resolve that tension.

The Expand-and-Contract Pattern

The expand-and-contract pattern (also called "parallel change") is the gold standard for zero-downtime schema migrations. Instead of making a breaking change in one step, you break it into three safe steps:

1. Expand: Add the new column/table alongside the old one. Both coexist. Application writes to both.
2. Migrate: Backfill existing data from old to new. Application reads from new, writes to both.
3. Contract: Remove the old column/table once all code paths use the new one. This is the only step that removes something, and by this point, nothing depends on it.

MySQL: pt-online-schema-change and gh-ost

MySQL's ALTER TABLE can lock the table for the duration of the change on older versions and certain operations. For tables with millions of rows, this can mean minutes or hours of downtime. Two tools solve this problem:

pt-online-schema-change (Percona Toolkit) creates a new table with the desired schema, copies data in chunks, uses triggers to capture changes made during the copy, and then swaps the tables atomically. It's battle-tested and widely used, but the trigger-based approach adds overhead to write operations during the migration.

gh-ost (GitHub Online Schema Tool) takes a different approach: instead of triggers, it reads the binary log to capture changes. This is less intrusive because it doesn't modify the source table's write path. gh-ost also supports throttling, cut-over control, and testing migrations without actually executing them.

PostgreSQL: pg_repack and Logical Replication

PostgreSQL handles many ALTER TABLE operations without exclusive locks (adding a column with no default, adding an index with CREATE INDEX CONCURRENTLY). However, some operations still require a full table rewrite: changing a column type, adding a NOT NULL constraint with a default to an existing column (pre-PG11), or doing a full table VACUUM.

pg_repack reorganizes tables online without exclusive locks. It's useful for reclaiming space (after deleting many rows) and for schema changes that would otherwise lock the table.

Migration Tools: Flyway vs. Liquibase

Rails and Django Migration Pitfalls

Framework-level migration tools (Rails ActiveRecord migrations, Django migrations) are convenient but dangerous at scale. They hide the SQL being generated, making it easy to accidentally create a migration that locks a table for minutes.

Data Backfills at Scale

After adding a new column (the "expand" phase), you need to backfill existing rows. For a 500M-row table, this is itself a significant operation. Key principles for safe backfills:

• Batch processing: Update rows in batches of 1,000-10,000, not all at once. A single UPDATE of 500M rows will lock the entire table and overwhelm your transaction log.
• Throttling: Add sleep between batches to avoid overwhelming the database. Monitor replication lag and pause if it exceeds thresholds.
• Idempotency: Backfill scripts must be safe to run multiple times. Use WHERE new_column IS NULL to skip already-backfilled rows.
• Progress tracking: Log progress so you can resume if the backfill is interrupted. Store the last processed ID.

Multi-Region Migration Coordination

If your database is replicated across regions, schema migrations become even more complex. You need to ensure that the migration is applied to the primary first, then replicated to all secondaries before deploying application code that depends on the new schema. In a blue-green database setup, the migration and application deployment must be carefully choreographed to avoid windows where the application expects a schema that doesn't exist yet on some replicas.

Lessons from the Trenches

Database migrations are where theory meets the unforgiving reality of production data. The companies that do this well share common traits: they have a migration review process (every migration PR is reviewed by a DBA or platform engineer), they use automated safety checks (strong_migrations, squawk for Postgres), they practice migrations on staging databases with production-scale data, and they treat migration failures as incidents with full postmortems. The platform team's role is to make safe migrations easy and dangerous migrations impossible. Provide tooling, guardrails, documentation, and expert review. The cost of getting this wrong is too high to leave it to chance.