Maintainable

Machine learning systems can degrade even when the underlying code has not changed. Diana Pfeil of Sunbeam Consulting joins Robby Russell on Maintainable to explain how changing data, model behavior, and non-deterministic outputs create a different kind of maintenance challenge.

Rather than asking whether a system is simply correct or incorrect, teams need reliable ways to measure confidence in its behavior.

Diana introduces evals as a way to test AI-generated outputs that may be phrased differently each time. She and Robby discuss using LLMs to judge other LLM outputs, reviewing production traces, sampling unusual cases, protecting sensitive data, and keeping humans involved when automated checks cannot provide enough confidence.

They also explore how prompts should be versioned and tested against representative examples.

The conversation turns to the operational costs that come with AI features. Model providers can deprecate dependencies quickly, prompts may behave differently after an upgrade, and teams must continue monitoring systems that might once have been considered finished.

Diana encourages organizations to ask what can now be automated, how accurate the result needs to be, and whether the benefit justifies the additional maintenance work.

Diana also makes the case for starting with the simplest model or deterministic process that can solve the problem. Teams can add complexity once the baseline proves insufficient, but designing around imagined future requirements often creates the wrong system.

Her advice for engineers trying to introduce AI internally is equally direct: build a small prototype that solves a real problem, then let the result make the case.

Episode Highlights

[00:00:50] Maintaining Probabilistic Software: Diana explains why maintaining AI systems involves the code, changing data, model behavior, and confidence in the output.

[00:02:35] What Are Evals?: Robby asks how teams test AI-generated results when the correct response may be worded differently each time.

[00:04:38] Using an LLM as a Judge: Diana describes using one model to evaluate another and why the judge must be calibrated against human decisions.

[00:06:39] Monitoring AI in Production: Diana introduces human review, traces, observability, and production sampling.

[00:08:44] Recognizing Input Drift: A meeting-notes example shows how changing inputs can degrade an otherwise unchanged system.

[00:12:06] Maintaining Models and Prompts: Diana outlines the code, model, prompt, and data-pipeline changes teams may need to make.

[00:14:24] Versioning and Testing Prompts: Robby asks how prompt experimentation fits into source control and repeatable testing.

[00:18:32] Building Confidence in a Black Box: Diana explains how evals and production reviews help teams avoid regressions.

[00:21:16] Diana’s Machine Learning Background: Diana shares her path from recommendation systems at Amazon to startup leadership and consulting.

[00:22:26] How Sunbeam Consulting Helps Teams: Diana describes advising leaders on AI strategy and helping teams build machine learning products.

[00:24:28] Finding Useful Automation Opportunities: Diana explains how teams can identify previously unstructured work that may now be practical to automate.

[00:27:40] The Cost of Automated Decisions: Robby and Diana compare human error with the oversight and infrastructure required by AI systems.

[00:30:14] AI Is Not Free to Maintain: Diana discusses model deprecations, vendor dependencies, and the ongoing support required after launch.

[00:36:45] Keeping Up With Rapidly Changing Tools: Diana explains why teams need room to experiment without constantly disrupting established workflows.

[00:38:49] Why Simpler Models Often Win: Diana makes the case for starting with a baseline before introducing more sophisticated approaches.

[00:45:18] Selling an AI Idea Without the Buzzwords: Diana recommends building a useful prototype and allowing the result to make the case.

[00:46:30] The Inner Game of Tennis: Diana recommends W. Timothy Gallwey’s book about learning, judgment, and performance.

Resources Mentioned

Sunbeam Consulting

Diana Pfeil on LinkedIn

Pydantic

Amazon Bedrock Guardrails

Claude Code

Cursor

OpenAI Codex

The Inner Game of Tennis by W. Timothy Gallwey

Thanks to Our Sponsors!

Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

Mailtrap is a modern email delivery platform built for developers. Native SDKs, a secure Email API and SMTP, and a free tier with 4,000 emails a month. When you need help, you'll reach real people on 24/7 support, not an AI chatbot. Try Mailtrap for free!

Subscribe to Maintainable on:
Apple Podcasts
Spotify
Or search "Maintainable" wherever you stream your podcasts.
Keep up to date with the Maintainable Podcast by joining the newsletter.

What does it take to keep a product healthy after more than 15 years of continuous evolution?

In this episode, Robby Russell talks with Chris Coyier, co-founder of CodePen, about the long game of maintaining software. Chris shares how CodePen has evolved over time, the trade-offs involved in migrating parts of the platform from Rails to Go, and the challenges of balancing maintenance work with the desire to build what's next.

They also explore the human side of maintainability, the role of technical debt in shaping priorities, and why small teams often have to make very intentional decisions about where to invest their limited time and attention.

Whether you're maintaining a side project, stewarding a legacy application, or helping a team navigate change, this conversation offers practical insights into building software that lasts.

Key Topics

Defining what "well-maintained software" really means

Why maintainability is often more of a people problem than a code problem

The origin story of CodePen

Supporting a product that has evolved over 15 years

Balancing maintenance work with product evolution

Gradually migrating from Rails to Go

Using GraphQL across multiple implementations

Technical debt and its many interpretations

Team size, communication overhead, and organizational design

Simplifying software by embracing browser capabilities

Links & Resources

ChrisCoyier.net

Chris Coyier on Bluesky

CodePen

ShopTalk Show

CSS-Tricks

Book Recommendation

Understanding Comics: The Invisible Art (Goodreads) by Scott McCloud

Thanks to Our Sponsors!

Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks. It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications. Keep your coding cool and error-free, one line at a time! Use the code maintainable to get a 10% discount for your first year. Check them out!

Subscribe to Maintainable on:
Apple Podcasts
Spotify
Or search "Maintainable" wherever you stream your podcasts.
Keep up to date with the Maintainable Podcast by joining the newsletter.

Sally Lait joins Robby Russell on Maintainable to explore software maintainability through a different lens… not just code quality, but how teams work together over time.

Sally is a fractional technology leader and advisor with more than two decades in the industry. You can follow her on LinkedIn or Mastodon.

They start with a familiar question: what makes software well maintained? Structure and standards matter, but Sally shifts the focus to signals around the edges… documentation, onboarding speed, knowledge sharing, and especially how confident people feel making changes.

That confidence becomes the thread throughout the conversation.

Teams with high confidence move faster and adapt more easily. Teams with low confidence hesitate, avoid parts of the system, and struggle to make progress… regardless of what the code looks like.

Robby and Sally also dig into why maintenance work often struggles to get traction. It rarely speaks for itself. Leaders need to connect it to outcomes the business already cares about… risk, hiring, delivery speed, and long-term sustainability.

Sally references a LeadDev panel she moderated on why maintenance still feels “stuck in 2015”: Why Software Maintenance Is Stuck in 2015.

They also discuss modernizing legacy systems and moving away from long-standing in-house software… work that is rarely just technical. It requires trust, clear communication, and navigating the emotional attachment teams have to what they’ve built.

The episode closes with advice for engineers joining older codebases: stay curious, build relationships early, and use onboarding gaps as opportunities to improve things for the next person.

Episode Highlights

[00:01:02] What Makes Software Maintainable: Technical quality matters, but cultural signals often tell the deeper story.

[00:05:45] Why Progress Still Feels Slow: Even with improvements, teams can feel stuck due to perception gaps.

[00:07:30] Communicating Small Wins: Lack of visibility into incremental progress impacts morale and confidence.

[00:12:40] Influencing Without Manipulating: Maintenance work needs to be framed in business terms.

[00:16:00] Technical Debt as a Hiring Problem: Outdated systems affect recruiting and retention.

[00:20:22] Modernizing a Siloed System: Unlocking legacy data required both technical and organizational change.

[00:26:55] Building Trust for Change: Surprise proposals fail… alignment takes time.

[00:32:39] Letting Go of “Our Baby”: Replacing systems involves emotional and cultural dynamics.

[00:46:25] Joining an Older Codebase: Practical advice for onboarding and building confidence quickly.

Resources Mentioned

Sally Lait

Sally Lait on LinkedIn

Sally Lait on Mastodon

Why Software Maintenance Is Stuck in 2015 (LeadDev Panel)

Lara Hogan

The Murderbot Diaries by Martha Wells

Death of the Author by Nnedi Okorafor

Sally’s Reading & Reviews Site

Thanks to Our Sponsors!

Your test coverage says 90%, but that might be misleading. Undercover CI looks at your Ruby pull requests and shows you which parts of your changes weren't tested- not just overall coverage, but what changed and what got missed, down to the method level. Visit undercover-ci.com and use code MAINTAINABLE for 15% off your first billing cycle. Free for public repos. Private repos with unlimited users also available.

Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks. It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications. Keep your coding cool and error-free, one line at a time! Use the code maintainable to get a 10% discount for your first year. Check them out!

Subscribe to Maintainable on:
Apple Podcasts
Spotify
Or search "Maintainable" wherever you stream your podcasts.
Keep up to date with the Maintainable Podcast by joining the newsletter.

Software maintenance is often framed as a technical problem. Refactoring code, fixing bugs, or upgrading dependencies. In this conversation, Robby Russell talks with Rein Henrichs about a different lens, one centered on understanding.

Rein is a Principal Software Engineer at Procore, where he works within a large, long-lived system used across the construction industry. Rather than focusing on tooling, Rein emphasizes that well-maintained software is software that makes sense to the people maintaining it.

To explain this, Rein introduces the idea of the line of representation, drawing on the work of Richard Cook. Engineers do not interact directly with systems. They rely on representations such as logs, dashboards, and code. These are approximations, not reality, echoing ideas from Plato’s Allegory of the Cave.

When those representations break down, teams lose shared understanding, what Rein describes as “common ground.” This often shows up as weak signals. Subtle indicators that something is not quite right. They are easy to ignore, but over time they lead to confusion and slower decision-making.

Incidents make this especially visible. Rein explains how teams build alignment under pressure, highlighting that the role of an incident commander is coordination, not control. Clear communication matters as much as technical correctness.

The conversation also explores how large systems behave in practice. They rarely fail completely. Instead, they degrade in multiple ways at once. While SLOs can help teams respond to customer-facing issues, they do not capture internal clarity or alignment.

Rein references W. Edwards Deming to highlight a common trap. Not everything that matters can be measured. High-performing teams often rely on judgment, experience, and shared context.

Toward the end, Rein connects these ideas to The Field Guide to Understanding Human Error by Sidney Dekker, challenging the idea that incidents are simply caused by mistakes. Instead, they emerge from the same behaviors that usually lead to success, just under different conditions.

For teams working in complex systems, the takeaway is straightforward. Maintaining software depends on maintaining understanding.

Links & Resources

Procore

Rein Henrichs on LinkedIn

Concepts & References

How Complex Systems Fail – Richard Cook

The Field Guide to Understanding Human Error – Sidney Dekker

W. Edwards Deming

Gerald Weinberg – Secrets of Consulting

Referenced in this Conversation

Kent Beck: You’re Ignoring Optionality and Paying for It

Charity Majors: Deploys Are Just the Beginning

Heidi Helfand: The Art and Wisdom of Changing Teams

Thanks to Our Sponsor!

Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks.

It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications.

Keep your coding cool and error-free, one line at a time! 

Use the code maintainable to get a 10% discount for your first year. Check them out!

Subscribe to Maintainable on:
Apple Podcasts
Spotify
Or search "Maintainable" wherever you stream your podcasts.
Keep up to date with the Maintainable Podcast by joining the newsletter.

Software doesn’t become hard to maintain only because the code is messy. It often becomes hard to maintain because the reasoning behind it disappears.

In this episode of Maintainable, Robby Russell talks with Russ Olsen about trade-offs, legacy systems, and why maintainability depends on context more than dogma. Russ brings decades of experience across very different kinds of systems, each with its own definition of what “maintainable” actually means.

A central theme is that software must be understandable to the people maintaining it. Teams tend to document implementation details well, but often fail to capture system-level intent and the trade-offs behind major decisions. Russ makes the case for preserving that thinking, including the alternatives that were rejected, so future maintainers don’t have to rediscover it the hard way.

The conversation also touches on Russ’s book Eloquent Ruby, Second Edition. Rather than teaching syntax, the book focuses on how Ruby is actually used in practice and why common patterns exist. That leads into a discussion about where elegance improves maintainability, and where it turns into unnecessary cleverness.

From there, the episode shifts into the realities of working in legacy systems. Russ explains how teams develop pessimism over time, often accepting flawed assumptions about how their systems behave. In some cases, major issues turn out to be far simpler than expected. The challenge is that teams stop looking.

Robby and Russ also discuss the value of fresh perspective. New engineers or outside contributors can surface assumptions that longtime maintainers overlook. Russ suggests finding “pinch points” in a system as a practical way to understand behavior without needing to fully untangle everything at once.

Later, the conversation explores developer quality of life. Long build and deploy cycles create daily friction that teams often underestimate. These slow feedback loops quietly degrade productivity and morale over time.

The episode also tackles rewrites. Russ warns that teams frequently underestimate how much knowledge is embedded in existing systems. Code that looks questionable may reflect constraints no one documented. In practice, most successful rewrites happen incrementally, not all at once.

The conversation wraps with a reminder that software development is fundamentally a social process. Russ argues that engineers undervalue storytelling, even though it’s one of the most effective ways to connect technical work to real human outcomes.

Episode Highlights

[00:00:40] Defining maintainability: Why context matters more than a universal standard

[00:02:01] Beyond code comments: Documenting system intent and trade-offs

[00:08:14] Who Eloquent Ruby is for: Understanding how Ruby is used in practice

[00:16:21] Elegance vs. cleverness: Where maintainability starts to erode

[00:23:18] Legacy pessimism: Why teams stop questioning assumptions

[00:29:25] Pinch points: A practical way to understand complex systems

[00:32:05] Developer experience: The hidden cost of slow feedback loops

[00:38:26] Rewrites: Why they fail and what teams overlook

[00:44:00] Storytelling: Connecting technical work to real-world impact

Resources Mentioned

Russ Olsen on LinkedIn

Eloquent Ruby, Second Edition

Getting Clojure

Zen and the Art of Motorcycle Maintenance

A History of Western Philosophy

Thanks to Our Sponsor!

Turn hours of debugging into just minutes! AppSignal is a performance monitoring and error-tracking tool designed for Ruby, Elixir, Python, Node.js, Javascript, and other frameworks.

It offers six powerful features with one simple interface, providing developers with real-time insights into the performance and health of web applications.

Keep your coding cool and error-free, one line at a time! 

Use the code maintainable to get a 10% discount for your first year. Check them out!

Subscribe to Maintainable on:
Apple Podcasts
Spotify
Or search "Maintainable" wherever you stream your podcasts.
Keep up to date with the Maintainable Podcast by joining the newsletter.