This article is based on reporting originally published by Computerworld UK, sourced from a presentation at ChefConf in Chicago.
Capital One’s Journey to a Cloud-Native Chef Architecture
US bank Capital One is a well-known proponent of DevOps practices at enterprise scale. The bank runs Chef Automate across more than 15,000 nodes spanning development, QA, and production environments — all hosted on AWS. To reach that level of reliability, the bank has had to iterate on its Chef architecture multiple times, learning hard lessons about what works at genuine enterprise scale in a heavily regulated industry.
Three Iterations: From Data Centres to AWS
Speaking at ChefConf in Chicago, John Casanova, Master Software Engineer at Capital One, outlined the evolution of the bank’s Chef platform. When he joined the organisation approximately four years before the talk, his first major task was migrating Chef out of three on-premises data centres and onto AWS — a move completed in 2015.
That first cloud-based architecture quickly revealed its own limitations. The team spent roughly a year redesigning the stack, completing the work and releasing the new version the year prior to the conference. By the time of Casanova’s presentation, Capital One was effectively on its third major iteration of the Chef platform.
“My first task was to get Chef out of three data centres and on to AWS. We did that in 2015 and that architecture had limitations, so we redesigned it over a year and finished that last year and released this year.”
John Casanova, Master Software Engineer, Capital One
Compliance as a First-Class Architectural Concern
As a federally regulated US bank, Capital One operates under strict compliance requirements that directly shape how its software delivery pipelines are designed. Compliance is not treated as an afterthought — it is baked into the architecture itself.
One of the more distinctive practices the bank has adopted is what it calls rehydration: a process of regularly relaunching infrastructure stacks as part of OS patching and security update cycles. Rehydration happens on a roughly two-month cadence and is a fundamental driver behind how the team designs and rebuilds its Chef stacks.
Automated Compliance Enforcement with Cloud Custodian
Capital One uses an open-source tool called Cloud Custodian to automatically enforce compliance policies across its AWS environment. Cloud Custodian is a rules engine that can monitor cloud resources and take automated action — including shutting down or quarantining resources — when they fall out of compliance with defined policies.
“Cloud Custodian will shut you down if you are out of compliance, so there is a bit of pressure. Rehydration is a big factor in how we approach building our stacks as we have to relaunch them every two months.”
John Casanova, Master Software Engineer, Capital One
The combination of automated compliance enforcement and mandatory rehydration cycles means that Capital One’s infrastructure must be designed to be fully ephemeral and rebuildable on a regular basis. This is a meaningful architectural constraint: any stack that cannot survive a relaunch every two months is, by definition, not production-ready at Capital One.
Key Takeaways for DevOps Teams
- Treat compliance as architecture, not audit. Capital One’s experience shows that regulated industries benefit from building compliance enforcement directly into the delivery pipeline rather than relying on periodic manual reviews.
- Plan for immutability from day one. The rehydration practice forces the team to treat infrastructure as truly disposable, which aligns with modern infrastructure-as-code and immutable infrastructure principles.
- Expect to iterate. Even a well-resourced team at a sophisticated organisation went through three major architectural revisions. Treat your first cloud-native architecture as a learning exercise, not a final destination.
- Automate compliance gates. Tools like Cloud Custodian remove the ambiguity around enforcement — non-compliant resources are shut down automatically, creating strong incentives for teams to maintain compliant configurations.
- Scale requires dedicated platform engineering. Managing 15,000+ Chef nodes across multiple environments is not a side project. Capital One’s investment in a dedicated platform engineering function reflects the operational overhead that comes with running configuration management at true enterprise scale.
Further Reading
Capital One has open-sourced Cloud Custodian and it is now a Cloud Native Computing Foundation (CNCF) project. Teams looking to implement similar automated compliance enforcement on AWS, Azure, or GCP can explore the project directly. For Chef Automate documentation and enterprise deployment guidance, refer to the official Chef documentation.
Original reporting: Computerworld UK. Presentation: ChefConf, Chicago, 2018.