Observability Engineering. Achieving Production Excellence. 2nd Edition Charity Majors, Austin Parker, Liz Fong-Jones

Dostępność produktu

Produkt został zweryfikowany pod kątem ułatwień dostępu więcej przeczytasz w deklaracji dostępności.

Spis treści książki

• Foreword
• Preface
  • Why We Wrote This Book
  • Whats Different in the Second Edition
  • How to Read This Book
  • Conventions Used in This Book
  • Using Code Examples
  • OReilly Online Learning
  • How to Contact Us
  • Acknowledgments for the First Edition
  • Acknowledgments for the Second Edition
• I. Introduction to Observability
• 1. What Is Observability?
  • The Origins of Observability
  • Applying Observability to Software Systems
    • Properties of Software Dependability
    • Observability is a Property of Dependable Software
  • How Observable is Your Software?
  • Two Competing Models for Telemetry: Three Pillars Versus Unified Data
    • The Three Pillars Model
    • The Unified Storage Model
  • Observability is the Validation of Developer Intent
  • The Agentic Incursion Has Just Begun
  • Guardrails Are Having a Moment
  • Conclusion
• 2. How Code Crosses Over: Validating Developer Intent in Production
  • What Makes Software Good?
    • Good Software Serves Its Purpose
    • Good Software Delivers Efficiently Over Time
  • The Maintenance Horizon: Disposable Code Versus Durable Code
    • Durable Code Is a Model for Mutating and Maintaining Code in Place
    • Disposable Code Is a Model for Avoiding Maintenance Costs
    • The Maintenance Horizon Is Not Always Knowable Up Front
  • Production Quality Code Is a Function of Dependability
    • Identifying the Critical Path
    • The Closer You Get to Persisting Data, the More Cautious You Should Be
  • Development to Production: Tools for Crossing Over
    • Practice 0: Give Yourself the Gift of Rich Data and Precision Tooling
      • Alternative Options
    • Practice 1: Build a Feedback Loop Between Developers and Production
    • Practice 2: Test Your Code Before You Deploy It
    • Practice 3: Instrument Your Code and Validate in Production
    • Practice 4: Decouple Deploys from Releases Using Feature Flags
      • Feature Flags for Deploys
      • Feature Flags for Releases
      • Advanced Uses
    • Practice 5: Invest in Progressive Delivery, Canaries, Automated Rollbacks
    • Bonus Practices: Traffic Splitters, Capture/Replay, Strangler Figs
  • Observability Is the Feedback Loop of Feedback Loops
  • The Work of Development Is Not Done Until Its Working in Production
  • Conclusion
• 3. The Origins of Observability in Software
  • An Introduction from Charity
  • The Dominant Model of Observability Is Just Monitoring, Rebranded
  • We Lost the Fight To Define Observability
  • Production Has Become Too Complex for Us to Debug Via Intuition
  • The Facebook Experience That Showed Us Whats Possible
  • The Control Theory Definition Made It All Click
  • The Modern Observability Landscape Is Confusing and Continues to Expand
  • Costs Are Driving the Need to Change, and AI Is Enabling That Change
  • Distilling the Lessons That Matter
  • Conclusion
• II. Instrumentation Fundamentals
• 4. Getting Started with Instrumentation
  • Instrumentation Basics
  • The Vocabulary of Telemetry: Logs, Metrics, and Traces
  • OpenTelemetry: The Universal Language for Observability
  • Automatic Instrumentation Versus Custom Instrumentation
  • Ownership of Instrumentation
  • Building a Custom Instrumentation Strategy
  • Cost Considerations, Volume Management, and Processing
    • Processing and Pipelines
    • Cost Considerations and Volume
    • Essential Concepts for Sampling
      • Before
      • After
  • Conclusion
• 5. Structured Events Are the Building Blocks of Observability
  • What Is a Structured Event?
  • The Limitations of Metrics
    • What Is a Metric?
    • Metrics Are Typically Aggregated
  • The Inner Workings of Logs
    • Turning Traditional Logs Into Structured Logs
    • Is a Structured Log the Same as a Structured Event?
    • Tracking a Single Operation
  • The Inner Workings of Distributed Traces
    • A Brief Introduction to Distributed Tracing
    • The Components of Tracing
  • Turning Logs Into Distributed Traces
    • Traces Are Collections of Spans
    • Trace Context
  • Dynamically Generating the Right System Views
  • Conclusion
• 6. Making Structured Events Arbitrarily Wide
  • Service and Code Context
    • Service Metadata
    • Build Information
    • Feature Flags
    • Versions of Important Things
  • Request and Execution Flow
    • HTTP Information
    • Route Information
    • Timings
    • Async Request Summaries
    • Errors
  • User and Business Context
    • User and Customer Information
    • Rate Limits
    • Caching
    • Localization Information
  • Operational Information
    • Uptime
    • Metrics
    • A Convention To Filter Out Everything Else
  • Attributes Important to Your Specific Application
  • Conclusion
• 7. Instrumenting Your Code with OpenTelemetry
  • What It Means to Use OpenTelemetry
  • Effective Instrumentation With OpenTelemetry
    • Trace-First Telemetry
    • Traces For Common Architectural Patterns
      • Tracing streaming architectures
      • Asynchronous work and jobs
      • Serverless
      • Extended Berkeley Packet Filters
    • Metrics, Spans, Logs, EventsOh My?
  • Using AI Agents To Instrument Your Code
    • What Is An Agent, Anyway?
    • Instrumentation with Agents
    • Useful Strategies for Agents
  • Conclusion
• III. Analysis Workflows
• 8. Getting Started with Observability Analysis
  • Debugging from Known Conditions
  • Debugging from First Principles
    • Using the Core Analysis Loop
    • Automating the Brute-Force Portion of the Core Analysis Loop
  • Automating Analysis with Generative AI
    • Agentic AI Personas
    • Using Agentic AI for Observability In Practice
  • Conclusion
• 9. Observability-Driven Development
  • Test-Driven Development
  • Observability in the Development Cycle
  • Determining Where to Debug
  • Debugging in the Time of Microservices
  • How Instrumentation Drives Modern Observability
  • Shifting Observability Left
  • Using Observability to Speed Up Software Delivery
  • Observability-Driven Development with AI
  • Conclusion
• 10. The Role of AI Agents for Observability
  • What Is an AI Agent for Observability?
  • The Pitfalls of Querying Without Context
  • Proven Use Cases for Observability Agents
    • Incident Response
    • Explaining Errors and Patterns in Telemetry Data
    • Improving Instrumentation Quality
  • Observability-Adjacent Use Cases for Agents
  • The Production Problem: The Mental Model Thats Disappearing
  • The Need for Context
  • Conclusion
• 11. Using Service Level Objectives for Reliability
  • Threshold-Based Alerting Creates Alert Fatigue
  • Threshold Alerting Is Only for Known-Unknowns
  • User Experience Is a North Star
  • Reliable Alerting with Service Level Objectives
  • Case Study: Changing Culture Toward SLO-Based Alerts
  • Accelerating SLO Adoption with Generative AI
    • From Targets to Implementation: Drafting Service Level Indicator
    • Encoding Best Practices in Your Prompts
  • Conclusion
• IV. Observability Technical Deep Dives
• 12. Acting on and Debugging SLO-Based Alerts
  • Alerting Before Your Error Budget Is Empty
  • Framing Time as a Sliding Window
  • Forecasting to Create a Predictive Burn Alert
    • Threshold-Crossing Alerts
    • Relative Burn Alerts
    • Predictive Burn Alerts
    • The Lookahead Window
      • Extrapolating the future from current burn rate
      • Short-term burn alerts
      • Context-aware burn alerts
    • The Baseline Window
    • Acting on SLO Burn Alerts
  • Using Structured Event Data for SLOs Versus Time-Series Data
  • Conclusion
• 13. Efficient Data Storage with Retriever
  • The Functional Requirements for Observability
    • Time-Series Databases Are Inadequate for Unified Observability
    • Other Possible Datastores
    • Data Storage Strategies
  • Case Study: The Implementation of Retriever
    • Partitioning Data by Time
    • Storing Data by Column Within Segments
    • Performing Query Workloads
    • Queries on Parts of Fields and Aggregated Fields
    • Querying for Traces
    • Joins
    • Querying Data in Real Time
    • Making It Affordable with Tiering
    • Making It Fast with Parallelism
    • Dealing with High Cardinality
    • Scaling and Durability Strategies
  • Notes on Building Your Own Efficient Datastore
  • Conclusion
• 14. Efficient Data Storage with ClickHouse
  • ClickHouse Core Concepts
    • MergeTree Fundamentals
    • Query Execution
      • Structured Logging
      • Balancing Structure and Flexibility with Map
      • Primary Key Design
      • Caches
  • Collecting Data
    • The OpenTelemetry Collector and ClickHouse Exporter
    • Alternate Ingestion Approaches
  • Querying Data
    • Simple Queries
    • Complex Joins
  • Scaling Vertically
    • Profiling Your Queries
    • Practical Data Modelling and Optimization
      • Within a table
        • Skip indexes
        • Column materialization
        • Projections
      • Across tables
        • Materialized views (incremental)
        • Merge table
    • Data Lifecycle Management
  • Scaling Horizontally
    • Replication
    • Sharding
    • Single-Region Observability Cluster
    • Multiregion Observability Cluster
    • SharedMergeTree
  • Visualizing Your Data
  • Using ClickHouse for Observability Workloads
  • Conclusion
• 15. Cheap and Accurate Enough Sampling
  • Sampling to Refine Your Data Collection
  • Using Different Approaches to Sampling
    • Constant-Probability Sampling
    • Sampling on Recent Traffic Volume
    • Sampling Based on Event Content (Keys)
    • Combining per Key and Historical Methods
    • Choosing Dynamic Sampling Options
    • When to Make a Sampling Decision for Traces
  • Translating Sampling Strategies into Code
    • The Base Case
    • Fixed-Rate Sampling
    • Recording the Sample Rate
    • Consistent Sampling
    • Target-Rate Sampling
    • Having More Than One Static Sample Rate
    • Sampling by Key and Target Rate
    • Sampling with Dynamic Rates on Arbitrarily Many Keys
    • Putting It All Together: Head and Tail per Key Target-Rate Sampling
  • Conclusion
• 16. Telemetry Management with Pipelines
  • An Introduction to Telemetry Pipelines
    • The Telemetry Pipeline Solution
    • Why This Matters Now
  • Core Functions of a Modern Telemetry Pipeline
    • Collect
    • Normalize and Secure
    • Enrich
    • Reduce
    • Route
    • Data Resilience
    • Pipeline Control and Observability
    • What You Should Remember About Core Functions
  • Pipeline Architecture in Practice
    • Core Components
    • Deployment Patterns
    • Scaling and Performance in Practice
    • Build-Versus-Buy Considerations
    • What You Should Remember About Pipeline Architecture
  • Adoption and Migration
    • A Phased Path to Adoption
      • Phase 1: Greenfield
      • Phase 2: Redirecting Existing Agents
      • Phase 3: Replacing Existing Agents
    • What You Should Remember About Adoption and Migration
  • Use Cases: Collect and Reduce, Combined
  • Business Case
    • Cost Control
    • Vendor Neutrality
    • Telemetry as a Strategic Asset
  • The Role of Telemetry Pipelines
  • Conclusion
• 17. Ontologies as a Shared Language for Humans and AI
  • Ontologies and Their Role in Observability
  • Design the Ontology
    • Define the Core Entities (the Nouns)
    • Define the Invariants (the Rules)
    • Visualize the Semantic Grammar of the Domain
    • Glue the Schema Through Metadata
    • Schematize Intent with the ActionPlan
  • Validate the Contract: Continuous Integration
    • Layer Three Gates for Defense
    • Establish the Team Workflow
  • Create Signal Parity in Production: Continuous Deployment
    • Understand the Hierarchy of Signals
    • Create Shared Instrumentation: The Universal Payload
  • Implement the AI Sandwich Architecture
  • Close the Loop: Production Driving Tests
    • Putting Ontologies into Practice
      • Scenario A: The Deterministic Comparison
      • Scenario B: The AI Proposal
  • Conclusion
• V. Observability Use Cases
• 18. Observability for CI/CD Pipelines
  • Why Reliable and Fast CI/CD Matters
  • Build Observability Has the Best ROI of All Observability Applications
  • CI/CD Through the Lens of Observability
    • The Ontology of Continuous Integration and Deployment
    • Instrumentation Basics
    • Defining Service Level Indicators and Achieving Predictability
    • From Jobs, to Directed Acyclic Graphs and Traces
  • Making Improvements and Measuring Them
    • Understanding Performance: Treating Continuous Integration Like Production
    • Predictability: Real-World Trade-Offs
    • (The Lack of) Incrementality: The Bane of Continuous Integrations Existence
    • Keeping Your Build Performance Tight and Your Developers Happy
  • Case Study: The Importance of Quick Build Times at Honeycomb
    • History of Improving Build Times at Honeycomb
      • Build steps and data transfer
      • Docker bake
      • Parallel builds
      • Removing bottlenecks
      • Caching network fetches
      • Results
    • Applying These Lessons to Your Build System
  • Conclusion
• 19. Observability for Mobile and Frontend
  • Status Quo for Mobile and Frontend
  • Instrumentation Limitations
  • Overcoming Domain Challenges
    • Recognizing the Problem
    • Instrumentation Difficulties
      • Application architecture
      • Life in a chaotic, remote sandbox
  • Applying Local Storage and Real-Time Control to Mobile Observability
  • What to Observe and Why
  • Opportunities for Improvement
    • Making Observability User-Focused
    • Quantifying User Experience
    • Adapting Existing Approaches
  • Applying User-Focused Observability
    • Iterative Analysis Process
    • Example: Food Delivery App
  • Mobile and Frontend Applications Deserve Observability
  • Conclusion
• 20. Performance Engineering with Observability
  • The Case for Performance Engineering
  • Building a Performance Engineering Practice
  • Optimizing Cost Without Modifying Application Code
    • Infrastructure Purchasing Models
    • Fleet-Wide Optimization
    • Cost Optimizing Kubernetes
    • Cost Optimizing Serverless
    • Observing the Costs
  • How Application Observability Reduces Cost
    • Using CPU Profiling Tools
      • Setting up and interpreting profiling data
      • Scaling out profile collection and analysis
    • Using the Correct Observability Signals, Together
      • When to use each signal type
      • Blending performance engineering and observability
  • Conclusion
• 21. Observability for Large Language Models
  • Why Observability Matters for LLMs
  • Using Evaluations for LLM Reliability
  • Designing Telemetry for LLM Applications
  • Analyzing Telemetry for AI Applications
  • Feeding Observability Data Into AI Application Development
  • Using Evaluations and Observability Together
  • Conclusion
• 22. An Intercom Case Study in Modern Engineering
  • Increasing Resolution Rate
  • Speeding Up Without Losing Efficiency
  • And Thus, Time to First Token Was Born
  • Speeding Fin Up
  • Finance Enters the Chat
  • Empathy
  • Conclusion
• VI. Observability Governance
• 23. Organizational Learning Speed is Now Your Biggest Constraint: An Open Letter to CTOs
  • An Open Letter to CTOs
  • The Sociotechnical Debts That Will Hold You Back
    • When Developers Live In a World of Tests, Not Reality
    • When Telemetry Gets Treated Like Infrastructure, Not Product
    • When Developer Tools Were Never Designed as Products
  • These Debts Will Sabotage Your Adoption of AI
  • Turning the Ship Around
    • Measuring Value
    • Build Good Feedback Loops
    • Change Actions That Drive this Strategy
      • Match tools to architecture and engineer workflows
      • Run observability as a platform, not infrastructure
      • Treat product observability as an investment, not a cost center
      • Make it real
  • A Letter From a CTO: How Intercom Engineering Optimizes for Learning
  • Conclusion
• 24. Systems Thinking for Software Delivery
  • Sociotechnical Systems
    • You Must Optimize Both the Social and Technical At the Same Time
    • Changing Information Flows
  • How Feedback Loops Drive Change
    • Amplifying Feedback Loops Accelerate Change
    • Balancing Feedback Loops Create Stability and Equilibrium
    • Small Shifts Can Trigger Massive Changes
    • The Difference Between a Virtuous Cycle and a Death Spiral is the Ability to Self-Correct
  • Feedback Loops in Software Delivery Systems
    • Observability in Amplifying Loops
      • The confidence loop
      • The learning accumulation loop
      • The system quality loop
    • Observability in Balancing Loops
      • Incident involvement breadth
      • Observability cost and usage regulation
    • Partial Feedback Creates Systemic Distortions
  • Leverage Points in Sociotechnical Systems
    • Leverage Points and the Limits of Observability
    • Push in the Right Direction
  • Conclusion
• 25. The Observability Landscape Through a Systems Lens
  • The Landscape Feels Noisy Because the Labels Are Noisy
  • The Loops Most Organizations Run Today (and What Is Missing)
    • Development Feedback Loops
    • Operational Feedback Loops
    • The Missing Loop
  • The Feedback Loop for Value Creation
    • Shipping Is Your Heartbeat
    • How to Build the Loop and Close the Gap
    • Why Closing the Gap Is Rare: The Economics of Cognitive Load
  • Two Observability Models, Two Feedback Loops
    • Three-Pillars Model (Built for Operational Outcomes)
    • Unified Storage Model (Built for Developer Learning)
  • AI Changes the Game and Opens New Interaction Models
  • Both Feedback Loops Matter: Lead with the Right One
  • Align on the Outcomes Youre Building Toward
  • Conclusion
• 26. The Business Case for Observability
  • Identifying Your Priorities
    • Complementary Roles of the Two Feedback Loops
    • Read the Fire Codes
  • The Business Case for Operational Loops
    • Mapping Observability Models to Operational Outcomes
    • The Operational Mandate
  • The Business Case for Developer Learning Loops
    • Estimating the Value of Developer Learning
    • The Developer Mandate Has Two Halves
      • External measures: Product differentiators and customer experience
      • Internal measures: Velocity as a competitive advantage
  • Strategic Investment or Cost Center Optimization?
  • Observability as Organizational Investment
  • Conclusion
• 27. Diagnosing Your Observability Investment
  • Dont Pay Observability Prices for Monitoring Outcomes
    • The Firefighting Trap
    • An Investment Posture Mismatch
    • Activities Versus Learning
  • How to Know If Your Investment Is Working (or Not)
  • Advice for Observability Investments
  • Conclusion
• 28. The Organizational Shift
  • Recognizing Legacy Masquerading as Modern Observability
    • The Ownership Test
    • The Two-or-Three People Test
    • The Mystery Test
    • The Arbitrary Question Test
    • The Deployment Confidence Test
    • After the Tests
  • Understanding the Resistance
    • The Organizational Immune Response
    • Why Legacy Vendors Fight
    • Why Existing Teams Resist
    • Why Leadership Is Skeptical
    • Why You Cannot Fight It Alone
  • Building the Coalition
    • Find the People Whose Pain Is Immediate
    • Honor the Heroes Who Held It Together
    • What Makes a Good Ally
    • Building the Case Together
  • Securing the Mandate
    • Sponsorship Versus Authority
    • Finding the Right Executive
    • The Pitch That Works
    • The AI Forcing Function
  • The Roadmap
    • Start Small: One Domain, Full Depth
    • Pave the Path: Make the New Way Easier
    • Platform Engineering Principles
    • Demonstrate Wins: Rerun the Tests
    • Expand Deliberately
  • Building the Team
    • What the Observability Team Owns
    • Capabilities of the Observability Team
    • Bringing the Existing Team Along
    • Breaking the Vendor Identity Trap
    • What Not to Promise
  • Conclusion
• 29. Build Versus Buy (Versus Open Source)
  • Which Do You Hate More: Your Money or Your Time?
  • Surfacing Our Biases
  • Observability Is Not Like Other Software
  • A Simple Framework for Evaluating Build Versus Buy
    • The Four Quadrants: Build, Buy, Should Not Exist, or Decide
      • Build (upper right)
      • Buy (lower left)
      • Should not exist (upper left)
      • Decide (lower right)
    • Mapping Examples to Quadrants
      • Example 1: Custom SLO/error budget tracking
      • Example 2: Custom instrumentation library wrapper
      • Example 3: Cost attribution and chargeback
    • The Generalized Pattern
  • What About Open Source?
  • A Framework That Includes Open Source
    • The Four Quadrants: Avoid, Create, Adopt, Acquire
      • Create (upper right)
      • Avoid (upper left)
      • Adopt (lower left)
      • Acquire (lower right)
      • Commission (middle right)
    • Theres Always Vibe Coding
  • The Real Costs of Building Your Own
    • The Hidden Costs of Free Software
    • The Benefits of Building Your Own
    • The Risks of Building Your Own
  • The Real Costs of Buying Software
    • The Hidden Financial Costs of Commercial Software
    • The Hidden Nonfinancial Costs of Commercial Software
    • The Benefits of Buying Commercial Software
    • The Risks of Buying Commercial Software
  • Build Versus Buy Is Not a Binary Choice
    • This Is Where Your Observability Team Comes In
    • Get the Best of Both Worlds
  • Important Caveats
  • Conclusion
• 30. The Art and Science of Vendor Partnerships
  • Does Buying Software Even Count as Engineering?
    • The Difference Between Vendor Engineering and Buying Software
    • Training Grounds for Executive Skills
  • What to Know Before You Start
    • Find Local Guidance
    • Understand Your Role
  • Mapping Your Stakeholders
    • Engineering
    • Management
    • Finance
    • IT/Ops, and Security
    • Procurement
    • Communicating up the Ladder
  • Trust and Credibility with Internal Stakeholders
    • Credibility Versus Trust
    • Building Trust with Stakeholders
    • Breaking Trust with Stakeholders
  • Trust and Reciprocity with External Stakeholders
    • Building Vendor Relationships
    • The Vendor Trust Coefficient
    • The Reciprocity Principle
    • How to Influence Another Companys Roadmap
    • SLAs Encode the VendorCustomer Relationship
  • Early-Stage Considerations
    • Define the Scope of Work
    • Identify Your Executive Sponsor
    • Define Success Criteria Before You Start
    • What to Do When Your Coworkers Perceive You as a Threat
    • The OpenTelemetry Decision
  • Design and Run a Meaningful Proof of Concept
    • What Does a Good Proof of Concept Look Like?
    • Structure Your Technical Evaluation
    • Validate Pricing at Scale (If Cost Is a Concern)
    • Build Champions and Advertise Your Wins
    • Maintain Momentum Through Procurement
  • Migration and Follow-Through
    • Handing Off to Another Team
    • Most Migrations Have Three Steps
    • You Arent Done Until You Decommission
    • Plan for the Next Transition
  • Conclusion
• 31. Instrumentation for Observability Teams
  • What Observability Teams Need To Know
    • Semantic Conventions
    • Telemetry Schemas in Practice
  • Tooling, Migrations, and Paved Paths
    • Tools and Frameworks
    • Paved Paths and Special Hats
  • Instrumentation in Highly Secure Environments
    • Use Compliance Tiers to Your Advantage
    • A Minimal Set of Capabilities for Observability Teams
    • Valuable Capabilities In a Regulated Environment
    • This Is Hard Work, But Work Worth Doing
  • Conclusion
• 32. Where Do We Go From Here?
  • Developers Must Return to Production
  • The Changing Cost Model of Software
  • The Most Important Parts of Our System Have Never Been Specified
  • The Great Unbundling
  • The Human Cost of Compressed Change
  • The Claim: AI Is Unprecedented
  • This Moment Has Many Precedents
  • The Sysadmins Went Through This
  • From Software Code to Software Systems
  • Values Backed By Durability
  • The Tools to Build Durable Systems Will Come
  • Conclusion
• Index

pokaż cały spis treści

ukryj recenzje