SRE Concepts

Posted by Amol Dighe on June 12, 2025

Lets explore these SRE concepts:

1. Service Level Objectives (SLOs)

A Service Level Objective, or SLO, is a precise, measurable target for the reliability of a service. It is a key tool for defining what “good” looks like from a user’s perspective. An SLO is built on two core components:

  • Service Level Indicator (SLI): A quantitative measure of a service’s behavior. Common SLIs include:

    • Latency: The time it takes to serve a request. (e.g., 99% of requests served in under 200ms)

    • Availability: The percentage of time a service is operational and serving requests. (e.g., 99.9% uptime)

    • Throughput: The number of requests a service can handle per second.

    • Error Rate: The percentage of requests that result in an error.

  • Objective: The specific target you set for that SLI over a defined period.

Why they matter: SLOs shift the focus from internal metrics to what truly impacts the end user. They are the contract between the service provider and the customer (whether internal or external) that sets clear expectations for reliability.

2. Error Budgets

The error budget is a direct byproduct of your SLO. It represents the maximum amount of “unreliability” that a service can tolerate over a given period without violating its SLO.

  • The Math: If your SLO for a service’s availability is 99.9%, your error budget is the remaining 0.1% of time the service can be unavailable. For a 30-day month, that’s roughly 43 minutes of acceptable downtime.

Why they matter: The error budget is the central mechanism for balancing innovation and reliability. It provides a clear, quantitative threshold for risk.

  • When the budget is “in the green” (you have time left): The team can take more risks, like deploying a new feature, knowing that a brief failure won’t violate the SLO.

  • When the budget is “depleted” (you’ve used up your downtime): The team must stop all non-essential feature development and focus solely on improving reliability and fixing the underlying issues that caused the downtime. This “stop and fix” rule prevents the team from digging a deeper reliability hole.

3. Distributed Tracing

As applications become more complex and move from monoliths to microservices, it becomes nearly impossible to track a single request as it travels through a dozen different services. Distributed tracing is the solution to this problem.

  • What it is: A method for observing and profiling requests as they flow through a distributed system. It creates a complete timeline of a single request, from the moment it enters the system to the final response.

  • Key Concepts:

    • Span: A single unit of work within a trace, representing an operation like a database query, an API call, or a function execution. Each span has a start time, end time, and metadata.

    • Trace: A collection of spans that represents a complete end-to-end journey of a request. Spans are connected in a parent-child relationship to show the flow.

    • Context Propagation: The mechanism that passes unique trace and span IDs from one service to the next, allowing them to be connected into a single, cohesive trace.

Why it matters: Distributed tracing is essential for:

  • Root Cause Analysis: Quickly pinpointing which service or component failed or caused a slowdown.

  • Performance Optimization: Identifying bottlenecks and latency issues within a specific service or in the communication between services.

  • Understanding System Behavior: Providing a visual map of how different services interact with each other.

4. Architectural Fault Tolerance

Fault tolerance is the design philosophy of building a system that can continue to operate correctly, and with minimal impact, even when one or more of its components fail. It is about anticipating failure and designing a system to be resilient from the ground up.

  • Key Principles & Patterns:

    • Redundancy: Having backup or duplicate components ready to take over if a primary component fails. This can be in the form of a hot-standby (active-passive) or multiple active components (active-active).

    • Failover: The automatic process of switching to a redundant system or component when a failure is detected. This should be as fast and seamless as possible to minimize downtime.

    • Circuit Breakers: A pattern that prevents a failing service from cascading its failure to other services. If a service is consistently failing, the circuit breaker “trips” and all subsequent requests fail fast instead of waiting and overloading the failing service.

    • Load Balancing: Distributing incoming requests across multiple instances of a service to prevent a single point of failure and handle high traffic loads.

    • Rate Limiting: A mechanism to control the rate of requests a service receives, protecting it from being overwhelmed and failing.

By combining these concepts, SRE teams can move from a reactive, crisis-driven model to a proactive, data-informed approach to managing reliability. SLOs set the targets, error budgets provide the framework for risk management, distributed tracing offers the visibility to debug and optimize, and architectural fault tolerance ensures the system is built to withstand inevitable failures.