This article is based on a personal account originally published on builtin.com. All technical experiences and conclusions described below belong to the original author.
The Unexpected Parallel Between Ice Skating and DevOps
Growing up in Canada, the author spent a lot of time around ice rinks. The first time he skated, he fell repeatedly — bruised, frustrated, and in tears. But he went back the following week, and the week after that. Eventually, he could skate fast and pull off a few tricks. That arc of painful failure followed by hard-won competence turned out to be a surprisingly accurate metaphor for learning DevOps.
Like an ice rink, DevOps is an open environment. You can skate in any direction, play different games, and assume different roles. You can also fall — in ways that feel uniquely humbling compared to other software engineering disciplines. This is one developer’s account of those falls, and what he learned from each one.
Starting With the Wrong Mindset
When the author first approached DevOps, he carried two assumptions that would cost him time. First, he assumed DevOps was mostly “glue” — a few bash scripts connecting existing services. Second, he was fiercely independent and resistant to pivoting once he had committed to a stack. Both attitudes are common among developers new to infrastructure work, and both tend to backfire.
The project that exposed these blind spots was a personal proof-of-concept: a PostgreSQL database exposing custom REST endpoints and WebSockets. Simple enough on paper — but the specific requirements quickly made off-the-shelf solutions inadequate.
Requirement 1: A Custom PostgreSQL Extension (pg_cron)
The author wanted to use pg_cron, a PostgreSQL extension built by Citus Data that schedules cron jobs directly on the database server and exposes those jobs as a queryable database table. It also enables server-side events driven by SQL queries, co-located with the database process itself.
The immediate problem: pg_cron is not an officially supported extension on AWS Relational Database Service (RDS). After Citus Data was acquired by Microsoft, the company shifted its focus to Azure and no longer maintains a formal build and test pipeline for pg_cron across arbitrary OS, architecture, and PostgreSQL version combinations. That means installing pg_cron outside of a supported environment requires compiling it from source.
Requirement 2: Root (or SSH) Access to the Database Host
Installing a custom-compiled PostgreSQL extension requires copying build artifacts to the server and having sufficient permissions to register the extension with PostgreSQL. AWS RDS does not support SSH access into managed instances and does not grant the elevated permissions needed to install extensions outside its supported list. In practice, this meant the author needed root access to the underlying host — something RDS fundamentally does not provide.
This is not necessarily a disadvantage. Root access over the long term provides genuine insight into database performance tuning, availability configuration, and the kind of low-level control that managed services deliberately abstract away. For a personal data architecture built around PostgreSQL, the tradeoff was worthwhile. For a production SaaS product where scalability and managed support matter most, AWS RDS remains a sensible default.
Requirement 3: Server-Side Events via PostgreSQL LISTEN/NOTIFY
The project also required pushing server-side events to clients. PostgreSQL’s CREATE TRIGGER combined with its LISTEN/NOTIFY mechanism can implement a lightweight pub/sub system inside the database. It is also possible to run custom stored procedures using PL/pgSQL or PL/Python in this context.
This capability introduces security considerations. Stored procedures that can fire on data changes need careful permissioning — ideally scoped to specific tables, rows, or columns — to ensure they cannot access data outside their intended domain. These requirements further reinforced that a self-managed PostgreSQL instance was the right approach for this project.
First Attempt: AWS Elastic Beanstalk
Rather than solving the database problem first, the author decided to get a backend process running quickly using AWS Elastic Beanstalk. The appeal was clear: minimal operational overhead, a user-friendly CLI, and access to underlying AWS resources without having to manage every layer manually.
After a week of experimentation, he moved on. The core tension is a classic one in managed platforms: Elastic Beanstalk trades objective safety (understanding exactly what is happening underneath when something breaks) for subjective safety (a guardrailed, abstracted interface). When his requirements drifted outside the platform’s intended use cases, he found himself debugging opaque failures with limited visibility. He had built himself into a corner.
“I realized I didn’t want a sandbox. I want an ice rink.”
The Right Tool: Docker and Docker Compose
The Elastic Beanstalk experience clarified what the author actually needed: accessible and descriptive system logs, an open-source and locally reproducible deployment framework, and explicit control over dependencies. That description points directly to Docker.
A few Docker features were particularly relevant to his situation:
- Multi-stage builds: You can define build, test, and production stages in a single Dockerfile. Intermediate build images are cached and discarded, keeping the final image lean. This matters when some dependencies must be compiled from source.
- Docker Compose for single-host deployments: When you need to run multiple containers on one host and want to compartmentalize resources per process (e.g., block storage for a database, static file storage for a web server), Docker Compose provides a clean, declarative way to define that topology locally.
Deploying to AWS: ECS, EBS, and the Complexity Cliff
Once a Docker Compose stack works locally, the next question is how to deploy it to AWS. The natural answer is AWS Elastic Container Service (ECS). The author found the initial experience frustrating for several interconnected reasons:
- Persistent volumes: Docker container filesystems are ephemeral by design. To persist data (such as a PostgreSQL data directory), you must attach an AWS Elastic Block Store (EBS) volume to the container.
- Fargate does not support EBS: AWS Fargate, the serverless container orchestration layer within ECS, does not support EBS volume attachment. Persisting stateful data requires using EC2-backed task definitions instead, which reintroduces infrastructure management overhead.
- EBS volumes are coupled to EC2 instances: Reproducibly provisioning an EC2 instance with an EBS volume attached at startup typically requires tooling such as Packer to define a custom Amazon Machine Image (AMI), adding another layer to the stack.
At this point the author seriously considered giving up. The complexity felt disproportionate to the goal. But he kept going — and found a path forward.
The Solution: CloudFormation as the Source of Truth
An AWS blog post describing EC2 task definition support for Docker volume drivers — which bind a Docker volume to a persistent state store — pointed toward AWS CloudFormation as the connective tissue. CloudFormation is AWS’s native infrastructure-as-code service, and it has a steeper learning curve than ECS alone. However, it offers significantly more documentation, community resources, and tutorials, and it serves as the canonical description of any AWS deployment.
Reading Docker on AWS by Justin Menga helped the author understand how well ECS and CloudFormation complement each other, and how much depth remains to explore. The conclusion he reached: writing your own CloudFormation templates, despite the upfront investment, gives you objective safety — you know precisely what AWS is building, and you retain the flexibility to make it do what you actually need.
The Final Stack
After roughly two months of iteration, the author settled on a stack that met his requirements:
- Docker Compose for local definition and development
- AWS Elastic Container Service (ECS) with EC2 task definitions (not Fargate) for container orchestration
- AWS Elastic Block Store (EBS) volumes mounted to containers for persistent storage
- A custom Amazon Machine Image (AMI) to reproducibly provision the EC2 host
- AWS CloudFormation templates as the single source of truth for all infrastructure
What the Skating Metaphor Actually Teaches Us
Looking back on the experience, the author draws three lessons that generalize beyond his specific project:
1. Environments Vary — Adapt Your Approach
A frozen neighborhood pond and the Rockefeller Center skating rink are both ice, but they demand different skills and different respect. Similarly, deploying to a local development machine, a cloud-based staging environment, and production are three distinct contexts. The tools, constraints, and failure modes differ enough that you cannot carry assumptions from one directly into another.
2. Competence Requires Deliberate Practice Over Time
You cannot learn to play hockey without first logging hours on the ice learning to skate. Likewise, DevOps fluency does not come from reading documentation alone. It comes from building things, breaking them, debugging unfamiliar systems, and steadily expanding your mental model of how infrastructure fits together.
3. Understanding First Principles Changes Your Options
Being able to deploy a stack from first principles — knowing why each component exists, what it replaces, and what happens when it fails — expands your decision-making leverage significantly. Managed services are not wrong choices; they are context-dependent ones. When you understand what they are abstracting, you can make that choice deliberately rather than by default.
Key Takeaways for DevOps Learners
- Managed database services like AWS RDS are excellent for scalable production workloads but impose hard limits on customization. Know those limits before you commit.
- Abstracted platforms (Elastic Beanstalk, Fargate) trade control for convenience. When your requirements drift outside a platform’s design envelope, that tradeoff reverses — you get less control and less convenience.
- Docker multi-stage builds are worth learning early. They keep images small, make dependency compilation reproducible, and separate build-time from runtime concerns cleanly.
- CloudFormation has a steep learning curve but pays off as a durable, explicit description of your infrastructure. Templates are reviewable, version-controllable, and portable across accounts and regions.
- Persistence matters more than speed in learning. Two months of falling down and getting back up produced a durable understanding that a single weekend tutorial never could.