AWS re:Invent 2025 - Closing the gap between speed and safety: shipping faster at scale (AIM259)

Closing the Gap Between Speed and Safety: Shipping Faster at Scale

Overview

• Presentation by Edith Harbaugh (CEO and Co-founder of LaunchDarkly), Jeffrey Goldberg (Product Manager at Click), and Nate Gray (Digital Engineering Operations at Ally Financial)
• Discussed how organizations can ship software faster while maintaining reliability and safety

Bridging the Speed vs. Reliability Divide

• Historically, there was a perceived tradeoff between shipping software quickly (speed) and ensuring reliability/safety
• Presenters argued this is a false dichotomy - shipping more frequently with smaller changes can actually increase reliability
• Key is decoupling deployment from release, and using feature flags to control rollouts

Improving Release Cadence

• Ally Financial went from monthly releases + hotfixes to 68 releases per month across 15 applications
• Enabled by feature flags, API-first development, and phased rollouts
• Improved developer happiness by eliminating weekend/overnight deployments

Leveraging Feature Flags

• Feature flags allow teams to decouple deployment from release
• Developers can deploy code "dark" and toggle features on/off in production
• Enables gradual rollouts, A/B testing, and quick rollbacks if issues arise

Addressing Reliability Concerns

• Regulated industries (e.g. banking) have strict compliance requirements
• Leveraging feature flags allows targeted rollouts and testing in production without full system impact
• Ability to quickly isolate and debug issues by enabling debug logging for specific tenants/customers

Integrating AI and Machine Learning

• Treat AI models like feature flags - can be gradually rolled out and A/B tested
• Enables experimentation to find optimal AI configurations for different customer segments

Lessons Learned

• Empower developers with autonomy and control over releases
• Automate as much as possible, but maintain human oversight for critical processes
• Establish clear guardrails and governance around feature flag usage
• Leverage caching, redundancy, and other techniques to ensure reliability of the feature flagging system itself

Business Impact

• Faster software delivery with reduced risk and improved customer experience
• Increased developer productivity and happiness by eliminating weekend/overnight deployments
• Ability to rapidly experiment and iterate on new features, including AI-powered capabilities
• Compliance and reliability benefits in highly regulated industries like banking

Key Takeaways

• Feature flags are a critical enabler for shipping software faster at scale
• Decoupling deployment from release, and giving developers more control, are key to improving velocity without sacrificing safety
• Reliability can be maintained through techniques like phased rollouts, debug logging, and caching
• Integrating AI/ML models with feature flags allows for experimentation and optimization
• Empowering developers, automating where possible, and establishing clear governance are crucial for success