Every engineering team eventually encounters the dilemma where a high-impact feature is fully built and passes all pipeline tests, but business, marketing, or operational constraints prevent it from going live to all users immediately. In traditional workflows, this forces teams to either stall delivery on brittle feature branches or brace for high-risk, off-hours deployments, simply because code deployment and feature release are treated as the same event. By integrating feature flags into your continuous delivery pipelines, you decouple technical deployments from user-facing releases, enabling dark code to ship safely to production while remaining dormant until you are ready to toggle it on. For teams looking to build robust delivery pipelines through practical, hands-on learning platforms like DevOpsSchool, mastering this separation transforms high-stakes release days into routine, predictable, and fully controlled events.
Understand the Building Blocks of Feature Flags
Before diving into code, let us break down the architectural components and decision criteria governing modern feature flags.
+-------------------------------------------------------------------------+
| Incoming User Request |
| { user_id: "usr_9481", role: "beta_tester", ip: "192.168.1.1" } |
+------------------------------------+------------------------------------+
|
v
+-------------------------------------------------------------------------+
| Evaluation Engine (Flag SDK) |
| |
| Rule 1: Is Master Kill Switch active? -------------> [ YES ] -> FALSE |
| Rule 2: Is Environment == "production"? ----------> [ PASS ] |
| Rule 3: User role in ["beta_tester", "admin"]? ----> [ MATCH ] -> TRUE |
| Rule 4: Percentage Rollout Hash(user_id) < 20% ---> [ SKIP ] |
+------------------------------------+------------------------------------+
|
v
[ Execute New Code Path ] or [ Execute Legacy Fallback ]
Code language: PHP (php)Core Terminology and Types
Boolean Flags
The simplest toggle type, returning either true or false. Boolean flags guard new UI elements, toggle third-party integrations, or switch backend execution paths.
Configuration-Based Flags (Multivariate Flags)
These return strings, numbers, or structured JSON payloads instead of binary booleans. For example, a configuration flag can pass rate-limiting thresholds, API endpoint routes, or dynamic theme settings to an application without redeploying code.
User-Specific and Role-Based Flags
Decisions based on contextual attributes extracted from incoming requests (such as email domain, account ID, geographic region, or assigned roles). This enables internal employee dogfooding, VIP account previews, or selective beta programs.
Environment-Specific Configuration
A flag definition evaluates differently based on the runtime tier (development, staging, or production). This allows a feature to stay permanently enabled in staging while remaining completely locked down in production.
Percentage-Based Rollouts
Traffic segmentation based on deterministic hashing. A system hashes a persistent identifier (like a user_id or session_token), maps it to a normalized range between 1 and 100, and enables the feature for requests falling below a specific percentage threshold.
Emergency Kill Switches
Hard-wired operational overrides designed to instantly disable degraded or failing features without rolling back container images or restarting services.
The Feature Flag Lifecycle
Every flag represents temporary control technical debt that must be managed deliberately across five discrete phases:
[ 1. Define & Instrument ]
│ (Declare flag key, establish defaults, add conditional logic)
▼
[ 2. Pipeline Verification ]
│ (Automated CI validation of both true and false paths)
▼
[ 3. Targeted Rollout ]
│ (Enable in Staging -> Internal Beta -> 5% Canary -> 100% Prod)
▼
[ 4. Operational Permanence ]
│ (Flag runs at 100% in production for a stabilization window)
▼
[ 5. Retirement & Cleanup ]
(Remove conditional checks, delete flag key from management hub)
Code language: JavaScript (javascript)Failing to complete Phase 5 is the primary reason teams accumulate dead code and conditional complexity.
Choose a Practical Setup for Your First Lab
To build practical mastery, we will use a lightweight, production-representative architecture that you can run locally or in a cloud environment.
Component Breakdown
| Layer | Lab Component | Production-Grade Alternative | Purpose |
| Application Layer | Node.js (Express REST API) | Microservices (Go, Java, Python, .NET) | Executes business logic and evaluates flags |
| Flag Management | JSON Config / Dynamic Endpoint | LaunchDarkly, Unleash, Flagsmith | Manages rules, audits, environments, and rollouts |
| CI/CD Automation | GitHub Actions (or Jenkins) | GitHub Actions, GitLab CI, Jenkins | Runs automated tests, builds images, coordinates deploys |
| Runtime Runtime | Docker Engine | Kubernetes (EKS, GKE, AKS) | Runs application workloads |
- LaunchDarkly: Enterprise SaaS platform featuring real-time streaming updates via server-sent events (SSE), granular targeting rules, and audit controls.
- Unleash: Popular open-source and enterprise feature management platform. Excellent for self-hosted compliance environments.
- Flagsmith: Open-source platform offering hosted or self-hosted flag evaluation, multivariate values, and segment analytics.
- Static/Dynamic JSON Config: Perfect for local labs and initial automated CI/CD integration exercises to understand the underlying mechanics before introducing SaaS dependencies.
Lab Assumptions and Prerequisites
- Node.js 18+ LTS installed locally
- Git installed and configured
- A free GitHub account for running automated pipeline workflows
- Basic familiarity with command-line operations and JSON syntax
Exercise One: Create and Control Your First Feature Flag
Objective
Build a lightweight REST service, instrument a Boolean feature flag using clean separation patterns, and observe real-time application behavior shifts without restarting or redeploying the application.
┌────────────────────────────────────────────────────────┐
│ App Configuration │
│ flags.json: { "new_pricing_engine": false } │
└───────────────────────────┬────────────────────────────┘
│
▼
┌────────────────────────────────────────────────────────┐
│ Client Request │
│ GET /api/v1/calculate-cart │
└───────────────────────────┬────────────────────────────┘
│
▼
[ Flag Evaluator ]
/ \
(flag is false) (flag is true)
/ \
▼ ▼
Legacy Pricing Path New Promotional Engine
(Standard Math) (Dynamic Discounts)
Code language: JavaScript (javascript)Step 1: Initialize the Project Workspace
Create a new directory and initialize your project files:
Bash
mkdir feature-flag-lab
cd feature-flag-lab
npm init -y
npm install express dotenv
Step 2: Create the Local Flag Configuration
Create a file named flags.json. This simulates our feature management store:
JSON
{
"new_pricing_engine": false,
"beta_tier_discount": false
}
Code language: JSON / JSON with Comments (json)Step 3: Implement Clean Feature Flag Evaluation Logic
Create featureFlags.js. A common anti-pattern is scattering raw environment variable lookups or direct configuration file reads throughout your business logic.
Instead, encapsulate flag reads inside an evaluation service that handles fallbacks safely:
JavaScript
// featureFlags.js
const fs = require('fs');
const path = require('path');
const FLAGS_PATH = path.join(__dirname, 'flags.json');
class FeatureFlagService {
constructor() {
this.flags = {};
this.loadFlags();
// Auto-reload configuration changes without restarting the process
fs.watchFile(FLAGS_PATH, (curr, prev) => {
console.log('[FLAGS] Configuration file changed on disk. Reloading...');
this.loadFlags();
});
}
loadFlags() {
try {
const rawData = fs.readFileSync(FLAGS_PATH, 'utf-8');
this.flags = JSON.parse(rawData);
console.log('[FLAGS] Active state:', JSON.stringify(this.flags));
} catch (err) {
console.error('[FLAGS] Error loading flags file. Retaining existing state.', err.message);
// Fallback: If uninitialized, assign empty safe defaults
if (!this.flags) this.flags = {};
}
}
isEnabled(flagKey, context = {}, defaultValue = false) {
if (Object.prototype.hasOwnProperty.call(this.flags, flagKey)) {
const flagVal = this.flags[flagKey];
// Handle standard boolean
if (typeof flagVal === 'boolean') {
return flagVal;
}
// Dynamic evaluations can be extended here
return Boolean(flagVal);
}
// Defensive coding: Return default if flag does not exist
return defaultValue;
}
}
module.exports = new FeatureFlagService();
Code language: JavaScript (javascript)Step 4: Build the Application Endpoints
Create server.js:
JavaScript
// server.js
const express = require('express');
const featureFlags = require('./featureFlags');
const app = express();
app.use(express.json());
const PORT = process.env.PORT || 3000;
// Business logic endpoints
app.post('/api/v1/calculate-cart', (req, res) => {
const { items, baseTotal } = req.body;
if (baseTotal === undefined || isNaN(baseTotal)) {
return res.status(400).json({ error: "Invalid 'baseTotal' supplied." });
}
// Feature Flag Branching Point
const useNewEngine = featureFlags.isEnabled('new_pricing_engine', {}, false);
if (useNewEngine) {
// New Implementation: Dynamic 15% promotional calculation
const discount = baseTotal * 0.15;
const finalTotal = baseTotal - discount;
return res.json({
engine: "v2-promotional-dynamic",
baseTotal: Number(baseTotal),
discountApplied: discount,
finalTotal: finalTotal,
message: "Special promotion applied successfully!"
});
}
// Legacy Path: Simple pass-through calculation
return res.json({
engine: "v1-legacy-standard",
baseTotal: Number(baseTotal),
discountApplied: 0,
finalTotal: Number(baseTotal),
message: "Standard pricing applied."
});
});
app.listen(PORT, () => {
console.log(`[APP] Service running on http://localhost:${PORT}`);
});
Code language: JavaScript (javascript)Step 5: Verify the Execution States
- Open your terminal and start the server:Bash
node server.js - In a second terminal window, send a test request:Bash
curl -X POST http://localhost:3000/api/v1/calculate-cart \ -H "Content-Type: application/json" \ -d '{"items": ["book", "pen"], "baseTotal": 100}'Expected Output:JSON{"engine":"v1-legacy-standard","baseTotal":100,"discountApplied":0,"finalTotal":100,"message":"Standard pricing applied."} - Open
flags.jsonin your editor and change"new_pricing_engine": falsetotrue. Save the file. Observe your server terminal: it logs that the configuration was dynamically reloaded. - Re-run the identical
curlcommand:Bashcurl -X POST http://localhost:3000/api/v1/calculate-cart \ -H "Content-Type: application/json" \ -d '{"items": ["book", "pen"], "baseTotal": 100}'Expected Output:JSON{"engine":"v2-promotional-dynamic","baseTotal":100,"discountApplied":15,"finalTotal":85,"message":"Special promotion applied successfully!"}
You have changed production business behavior in real time without executing a build, modifying server parameters, or restarting the process.
Exercise Two: Connect Feature Flags to a CI/CD Pipeline
Objective
Integrate feature flag assertions into an automated delivery pipeline using GitHub Actions, ensuring that continuous delivery gates validate code regardless of whether a flag is enabled or disabled.
Git Push / Pull Request
│
▼
[ GitHub Actions Runner ]
│
├─── Step 1: Checkout Code & Setup Environment
├─── Step 2: Install Project Dependencies
├─── Step 3: Run Unit Tests with Flag = FALSE (Legacy Pass)
├─── Step 4: Run Unit Tests with Flag = TRUE (New Feature Pass)
├─── Step 5: Validate Flag Configuration Schema
│
▼
[ Deployment Gate Passed: Safe to Ship to Production ]
Code language: PHP (php)Step 1: Add Unit Tests Covering Both Execution Paths
A major risk with feature flags is code rot: teams test the new feature with the flag turned on, but fail to notice that the legacy path broke for users who still have the flag turned off.
Install a lightweight testing tool:
Bash
npm install --save-dev mocha chai
Create a test file named test/pricing.test.js:
JavaScript
// test/pricing.test.js
const { expect } = require('chai');
const featureFlags = require('../featureFlags');
describe('Pricing Engine Feature Flag Verification', () => {
it('should correctly calculate legacy total when flag is disabled', () => {
// Force flag state for isolated unit testing
featureFlags.flags['new_pricing_engine'] = false;
const baseTotal = 200;
const isNewEngine = featureFlags.isEnabled('new_pricing_engine', {}, false);
expect(isNewEngine).to.be.false;
// Simulate legacy calculation
const finalTotal = baseTotal;
expect(finalTotal).to.equal(200);
});
it('should correctly calculate promotional discount when flag is enabled', () => {
// Force flag state for isolated unit testing
featureFlags.flags['new_pricing_engine'] = true;
const baseTotal = 200;
const isNewEngine = featureFlags.isEnabled('new_pricing_engine', {}, false);
expect(isNewEngine).to.be.true;
// Simulate v2 calculation
const discount = baseTotal * 0.15;
const finalTotal = baseTotal - discount;
expect(finalTotal).to.equal(170);
});
it('should return safe fallback default if flag is completely missing', () => {
const isMissingFlag = featureFlags.isEnabled('non_existent_key', {}, false);
expect(isMissingFlag).to.be.false;
});
});
Code language: PHP (php)Update your package.json test script:
JSON
"scripts": {
"test": "mocha test/**/*.test.js"
}
Code language: JavaScript (javascript)Verify tests run locally:
Bash
npm test
Step 2: Build the GitHub Actions CI/CD Pipeline
Create .github/workflows/pipeline.yml:
YAML
name: CI/CD Feature Flag Delivery Pipeline
on:
push:
branches: [ main ]
pull_request:
branches: [ main ]
jobs:
build-and-test:
runs-on: ubuntu-latest
steps:
- name: Check out repository
uses: actions/checkout@v4
- name: Setup Node.js runtime
uses: actions/setup-node@v4
with:
node-version: 18
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Validate Feature Flag JSON Schema
run: |
echo "Validating flags.json formatting..."
node -e "
try {
const flags = JSON.parse(require('fs').readFileSync('flags.json'));
if (typeof flags !== 'object' || flags === null) process.exit(1);
console.log('Valid JSON structure verified.');
} catch (e) {
console.error('Invalid JSON syntax in flags.json:', e.message);
process.exit(1);
}
"
- name: Execute Dual-Path Test Suite
run: npm test
- name: Simulation of Continuous Deployment
if: github.ref == 'refs/heads/main'
run: |
echo "============================================================"
echo "Deploying artifact to production environment..."
echo "Active flags deployment status: Inactive by default."
echo "Production is running latest code with safe default toggles."
echo "============================================================"
Code language: PHP (php)Commit this code to your Git repository and push to GitHub:
Bash
git init
git add .
git commit -m "feat: implement dual path tests and CI/CD workflow"
Code language: JavaScript (javascript)The pipeline validates that both the legacy code path and the new code path operate properly in parallel. This guarantees that deploying the artifact does not depend on whether a business feature is turned on or off.
Exercise Three: Test Features Across Multiple Environments
Objective
Configure isolated flag evaluation profiles across development, staging, and production tiers, preventing configuration leakage and enforcing least-privilege security controls.
+----------------------------------------------------------------------------+
| Environment Isolation |
+----------------------------------------------------------------------------+
| Development (DEV) | Staging (QA / UAT) | Production (PROD) |
| new_feature: TRUE | new_feature: TRUE | new_feature: FALSE |
| * Rapid iteration | * End-to-end integration | * Locked down to public |
| * Dev-local access | * Performance regression | * Strict change governance|
+----------------------------------------------------------------------------+
Code language: PHP (php)Step 1: Establish Environment Separation
Configuration leakage occurs when a flag toggled for testing in a development or staging tier inadvertently impacts production users. In enterprise platforms (such as LaunchDarkly or Unleash), this is handled via environment-scoped API tokens.
In our modular service, we implement this via environment configuration binding:
Create a folder named config/ containing three files:
config/flags.development.json:
JSON
{
"new_pricing_engine": true,
"experimental_checkout": true
}
Code language: JSON / JSON with Comments (json)config/flags.staging.json:
JSON
{
"new_pricing_engine": true,
"experimental_checkout": false
}
Code language: JSON / JSON with Comments (json)config/flags.production.json:
JSON
{
"new_pricing_engine": false,
"experimental_checkout": false
}
Code language: JSON / JSON with Comments (json)Step 2: Refactor Flag Service for Environment Resolution
Update featureFlags.js to read dynamically based on NODE_ENV:
JavaScript
// featureFlags.js (Updated for Environments)
const fs = require('fs');
const path = require('path');
class MultiEnvironmentFlagService {
constructor() {
this.env = process.env.NODE_ENV || 'development';
this.flags = {};
this.init();
}
init() {
const configFileName = `flags.${this.env}.json`;
const resolvedPath = path.join(__dirname, 'config', configFileName);
try {
if (fs.existsSync(resolvedPath)) {
const raw = fs.readFileSync(resolvedPath, 'utf-8');
this.flags = JSON.parse(raw);
console.log(`[FLAGS] Loaded configuration profile: ${configFileName}`);
} else {
console.warn(`[FLAGS] Config ${configFileName} not found. Defaulting to safe empty state.`);
this.flags = {};
}
} catch (err) {
console.error(`[FLAGS] Failed to parse environment flags:`, err.message);
this.flags = {};
}
}
isEnabled(flagKey, defaultValue = false) {
if (Object.prototype.hasOwnProperty.call(this.flags, flagKey)) {
return Boolean(this.flags[flagKey]);
}
return defaultValue;
}
}
module.exports = new MultiEnvironmentFlagService();
Code language: JavaScript (javascript)Step 3: Verify Environment Behavior
Run the application under different environment contexts:
Bash
# Test Development Context
NODE_ENV=development node -e "const f = require('./featureFlags'); console.log('DEV Engine:', f.isEnabled('new_pricing_engine'));"
# Expected: DEV Engine: true
# Test Production Context
NODE_ENV=production node -e "const f = require('./featureFlags'); console.log('PROD Engine:', f.isEnabled('new_pricing_engine'));"
# Expected: PROD Engine: false
Code language: PHP (php)Security Considerations for Multi-Environment Toggles
- Strict Access Permissions: Staging flags should be modifiable by QA engineers and developers. Production flags must require elevated privileges, multi-factor authentication (MFA), and audit logging.
- Never Put Secrets in Flags: Do not store database credentials, private API keys, or encryption secrets inside feature flag configurations. Flags should control operational flow, not store sensitive data.
- Audit History: Every production flag update should record who changed the toggle, the associated change ticket, the exact timestamp, and the previous state.
Exercise Four: Implement a Gradual Feature Rollout
Objective
Implement deterministic, percentage-based rollout logic that gradually exposes a feature to user traffic while ensuring individual users receive consistent experiences across requests.
User ID: "user_101" ──► Hash Function (MD5/CRC32) ──► Modulo 100 ──► Bucket Value: 14
│
▼
Rollout Percentage Target: [ 10% ] ───────────────────────────────► Output: FALSE (14 >= 10)
Rollout Percentage Target: [ 25% ] ───────────────────────────────► Output: TRUE (14 < 25)
Code language: JavaScript (javascript)Why Deterministic Hashing Matters
A naive rollout implementation uses a random generator like Math.random() < 0.2. Never do this in production. If a customer visits your application, clicks an item, and refreshes the page, a purely random check could flip them back and forth between two different interfaces, creating confusion and breaking multi-step flows.
Instead, map a consistent user identifier (such as user_id or account_id) to a stable integer between 0 and 99 using a hashing algorithm.
Step 1: Implement the Rollout Evaluator
Create rolloutEngine.js:
JavaScript
// rolloutEngine.js
const crypto = require('crypto');
class RolloutEngine {
/**
* Deterministically evaluates if a user is within a target percentage rollout
* @param {string} userId - Unique identifier for the user
* @param {string} flagKey - The feature flag identifier
* @param {number} rolloutPercentage - Target percentage (0 to 100)
* @returns {boolean}
*/
isFeatureEnabledForUser(userId, flagKey, rolloutPercentage) {
if (rolloutPercentage <= 0) return false;
if (rolloutPercentage >= 100) return true;
if (!userId) return false; // Fail-closed if no user context is provided
// Salt the hash with the flag key so rollouts across different features aren't identical
const hashInput = `${flagKey}:${userId}`;
const hash = crypto.createHash('sha256').update(hashInput).digest('hex');
// Convert the first 8 hex characters into an integer and scale to 0 - 99
const numericValue = parseInt(hash.substring(0, 8), 16);
const bucket = numericValue % 100;
return bucket < rolloutPercentage;
}
}
module.exports = new RolloutEngine();
Code language: JavaScript (javascript)Step 2: Validate User Consistency
Create a verification script named verifyRollout.js:
JavaScript
// verifyRollout.js
const rollout = require('./rolloutEngine');
const FLAG = 'checkout_redesign_v2';
const USERS = ['usr_alice', 'usr_bob', 'usr_charlie', 'usr_david', 'usr_eva'];
console.log('--- Phase 1: 20% Initial Rollout ---');
USERS.forEach(user => {
const enabled = rollout.isFeatureEnabledForUser(user, FLAG, 20);
console.log(`User: ${user.padEnd(12)} -> Enabled: ${enabled}`);
});
console.log('\n--- Phase 2: Expanding to 60% Rollout ---');
USERS.forEach(user => {
const enabled = rollout.isFeatureEnabledForUser(user, FLAG, 60);
console.log(`User: ${user.padEnd(12)} -> Enabled: ${enabled}`);
});
Code language: JavaScript (javascript)Run the script:
Bash
node verifyRollout.js
Code language: CSS (css)Expected Observations
- At 20%, only users whose deterministic bucket falls below 20 receive the feature.
- When expanding from 20% to 60%, any user who had the feature enabled during Phase 1 retains access in Phase 2. They never experience unpredictable feature toggling between sessions.
Exercise Five: Build a Kill Switch for Emergency Control
Objective
Simulate a production defect in a newly released feature and engage an emergency kill switch to restore system stability without running an infrastructure rollback.
[ Normal Flow ]
Client Request ──► Feature Flag Engine ──► New Payment Gateway (Crashing with 500s)
[ Kill Switch Engaged via API / Dashboard ]
Admin Action ────► Set 'enable_new_payment_gateway': FALSE (Propagated in ms)
[ Mitigated Flow ]
Client Request ──► Feature Flag Engine ──► Legacy Payment Gateway (100% Operational)
Code language: PHP (php)Step 1: Simulate a Flawed Feature
Create paymentService.js:
JavaScript
// paymentService.js
const express = require('express');
const app = express();
app.use(express.json());
// In-memory configuration state (simulating real-time control store)
let operationalFlags = {
enable_new_payment_gateway: true // Newly launched feature
};
app.post('/api/checkout', (req, res) => {
const { accountId, amount } = req.body;
try {
if (operationalFlags.enable_new_payment_gateway) {
// Unforeseen edge case: Throws unhandled exception on large amounts
if (amount > 500) {
throw new Error('CRITICAL: Memory leak and connection timeout in NewPaymentProvider SDK');
}
return res.json({ status: 'SUCCESS', provider: 'NextGen-Gateway-v2', amount });
}
// Battle-tested fallback path
return res.json({ status: 'SUCCESS', provider: 'Legacy-Gateway-v1', amount });
} catch (err) {
console.error(`[ERROR 500] Alert triggered: ${err.message}`);
return res.status(500).json({ error: 'Payment processing failed. Engineering alerted.' });
}
});
// Administrative Kill Switch Endpoint
app.post('/admin/flags/kill-switch', (req, res) => {
const { flagKey, authorizationKey } = req.body;
if (authorizationKey !== 'super-secret-ops-token') {
return res.status(403).json({ error: 'Unauthorized override attempt.' });
}
if (operationalFlags[flagKey] !== undefined) {
operationalFlags[flagKey] = false;
console.warn(`[KILL SWITCH] Flag '${flagKey}' has been forcefully deactivated!`);
return res.json({ message: `Kill switch executed for ${flagKey}`, state: false });
}
return res.status(404).json({ error: 'Unknown flag key.' });
});
app.listen(4000, () => console.log('[PAYMENT APP] Running on port 4000'));
Code language: JavaScript (javascript)Step 2: Walk Through the Incident Scenario
- Start the service:Bash
node paymentService.js - Send a valid transaction:Bash
curl -X POST http://localhost:4000/api/checkout \ -H "Content-Type: application/json" \ -d '{"accountId": "acc_101", "amount": 250}'Response:{"status":"SUCCESS","provider":"NextGen-Gateway-v2","amount":250} - Trigger the defect with an order above $500:Bash
curl -X POST http://localhost:4000/api/checkout \ -H "Content-Type: application/json" \ -d '{"accountId": "acc_102", "amount": 750}'Response: HTTP 500 Error. The application encounters an unhandled exception. - Engage the Kill Switch immediately:Bash
curl -X POST http://localhost:4000/admin/flags/kill-switch \ -H "Content-Type: application/json" \ -d '{"flagKey": "enable_new_payment_gateway", "authorizationKey": "super-secret-ops-token"}' - Re-run the problematic transaction:Bash
curl -X POST http://localhost:4000/api/checkout \ -H "Content-Type: application/json" \ -d '{"accountId": "acc_102", "amount": 750}'Response:{"status":"SUCCESS","provider":"Legacy-Gateway-v1","amount":750}
The system returns to normal operation within seconds.
Operational Insight: A kill switch is a first-response mitigation tool to stop customer impact immediately. It does not replace root-cause analysis, bug fixes, post-mortems, or formal patch deployments.
Exercise Six: Add Automated Tests for Flagged Features
Objective
Create a test matrix covering both states of a feature flag, missing configurations, and unexpected flag service outages.
┌──────────────────────────────────────────────────────────────┐
│ Test Execution Matrix │
├──────────────────────┬─────────────────┬─────────────────────┤
│ Test Scenario │ Flag Input │ Expected Behavior │
├──────────────────────┼─────────────────┼─────────────────────┤
│ Baseline Verification│ flag = false │ Legacy flow behaves │
│ Feature Verification │ flag = true │ New flow behaves │
│ Outage Resilience │ SDK returns Err │ Safe default active │
│ Corrupted Config │ flag = null/bad │ Fails closed │
└──────────────────────┴─────────────────┴─────────────────────┘
Code language: PHP (php)Implementation: Comprehensive Flag Test Suite
Create test/resilience.test.js:
JavaScript
// test/resilience.test.js
const { expect } = require('chai');
// Mock Flag Provider simulating network dependencies
class RemoteFlagProviderMock {
constructor(status = 'ONLINE', state = {}) {
this.status = status;
this.state = state;
}
async getBooleanFlag(key, defaultVal) {
if (this.status === 'TIMEOUT') {
throw new Error('ETIMEDOUT: Feature flag management server unreachable');
}
if (this.status === 'CORRUPT') {
return null;
}
return this.state[key] !== undefined ? this.state[key] : defaultVal;
}
}
// Business service wrapping the provider with defensive fallbacks
class NotificationService {
constructor(flagProvider) {
this.flagProvider = flagProvider;
}
async sendWelcome(userId) {
let useRichHtml = false;
try {
// Defensive fallback pattern: Always supply false on unexpected failures
const result = await this.flagProvider.getBooleanFlag('rich_html_welcome', false);
useRichHtml = typeof result === 'boolean' ? result : false;
} catch (err) {
console.warn(`[WARN] Flag service offline: ${err.message}. Defaulting to safe fallback.`);
useRichHtml = false;
}
return useRichHtml ? `<h1>Welcome, ${userId}!</h1>` : `Welcome, ${userId}!`;
}
}
describe('Notification Feature Flag Automation Suite', () => {
it('renders standard plain text when flag is explicitly FALSE', async () => {
const provider = new RemoteFlagProviderMock('ONLINE', { rich_html_welcome: false });
const service = new NotificationService(provider);
const output = await service.sendWelcome('usr_test');
expect(output).to.equal('Welcome, usr_test!');
});
it('renders HTML content when flag is explicitly TRUE', async () => {
const provider = new RemoteFlagProviderMock('ONLINE', { rich_html_welcome: true });
const service = new NotificationService(provider);
const output = await service.sendWelcome('usr_test');
expect(output).to.equal('<h1>Welcome, usr_test!</h1>');
});
it('handles remote flag service timeouts gracefully by falling back to safety', async () => {
const provider = new RemoteFlagProviderMock('TIMEOUT');
const service = new NotificationService(provider);
// The service must NOT throw an unhandled rejection; it falls back safely
const output = await service.sendWelcome('usr_test');
expect(output).to.equal('Welcome, usr_test!');
});
it('handles invalid non-boolean configuration values gracefully', async () => {
const provider = new RemoteFlagProviderMock('CORRUPT');
const service = new NotificationService(provider);
const output = await service.sendWelcome('usr_test');
expect(output).to.equal('Welcome, usr_test!');
});
});
Code language: JavaScript (javascript)Execute this automated suite in your local terminal:
Bash
npx mocha test/resilience.test.js
Exercise Seven: Combine Feature Flags With Canary Deployment
Objective
Learn how infrastructure-level canary releases and application-level feature flags work together to create a multi-layered release strategy.
INCOMING PRODUCTION TRAFFIC (100%)
│
┌──────────────────┴──────────────────┐
▼ ▼
CANARY PODS / INSTANCES (10%) STABLE PODS / INSTANCES (90%)
Runs: Build v2.4.0 Runs: Build v2.3.0
│ │
[ Flag Evaluation ] [ Baseline Code ]
Flag = false (90% traffic) │
Flag = true (10% traffic) │
│ │
Actual Impact: 1% Total Users ▼
(10% canary * 10% flag rollout) Standard Production
Canary Deployments vs. Feature Flags
Engineers often ask whether they should use canary deployments or feature flags. The answer is that they address different layers of risk:
| Comparison Attribute | Infrastructure Canary Deployment | Application-Level Feature Flag |
| Layer of Operation | Routing / Infrastructure (Ingress, Envoy, K8s) | Application Code Runtime (SDK, Logic) |
| Granularity | Coarse: routes traffic to whole container/node | Fine: routes specific methods, buttons, algorithms |
| Rollback Mechanism | Infrastructure redeployment or routing change | Toggle flag state via API or configuration change |
| Time to Mitigate | Minutes (pod restarts, DNS or ingress drain) | Milliseconds to seconds (configuration update) |
| Audience Targeting | Traffic weight, simple headers, IP subnets | Context-rich (User ID, Role, Org, Tenant) |
When to Combine Them
Using both techniques together provides the highest degree of release safety:
- Deploy Canary Infrastructure (10% Traffic): Direct 10% of total cluster traffic to the new container build
v2.4.0. The new feature flag is set tofalseby default. - Validate Baseline Metrics: Confirm the new build introduces no memory leaks, container crash-loops, or connection pool spikes.
- Turn Flag ON for Internal Employees: Expose the feature to employees hitting the canary build to confirm real-world performance.
- Gradually Expand the Flag: Expand the flag rollout to 20%, then 50%, and finally 100% within the canary environment.
- Complete Fleet Promotion: Promote
v2.4.0to the remaining 90% of the fleet. The feature remains safely manageable throughout the process.
Troubleshooting: What Can Go Wrong With Feature Flags?
Even well-designed systems run into edge cases. Here are the most common issues teams encounter with feature flags, along with their root causes and solutions.
┌───────────────────────────────┬──────────────────────────────────┬─────────────────────────────────┐
│ Observed Problem │ Probable Root Cause │ Concrete Remediation │
├───────────────────────────────┼──────────────────────────────────┼─────────────────────────────────┤
│ Flag toggled ON, but feature │ 1. In-memory cache not expiring. │ 1. Flush/reduce cache TTL. │
│ does not appear. │ 2. Flag key name typo. │ 2. Check logs for key typos. │
│ │ 3. Target rule mismatch. │ 3. Validate user attributes. │
├───────────────────────────────┼──────────────────────────────────┼─────────────────────────────────┤
│ Inconsistent behavior between │ Flag state updated in Staging │ Maintain separate, isolated │
│ environments. │ but read by Production. │ config keys and stores per env. │
├───────────────────────────────┼──────────────────────────────────┼─────────────────────────────────┤
│ Flag evaluation server │ Network failure, DNS issue, or │ Ensure the SDK has local │
│ becomes unreachable. │ rate limits reached. │ defaults and fails closed. │
├───────────────────────────────┼──────────────────────────────────┼─────────────────────────────────┤
│ Users see state flip between │ Non-deterministic evaluation │ Use consistent hashing on a │
│ requests or page reloads. │ (e.g., using Math.random()). │ unique key (User ID, Tenant). │
├───────────────────────────────┼──────────────────────────────────┼─────────────────────────────────┤
│ Pipeline tests fail on one │ Pipeline tests only the active │ Parameterize CI test runner to │
│ state, succeed on another. │ flag state, ignoring the other. │ execute both TRUE and FALSE. │
└───────────────────────────────┴──────────────────────────────────┴─────────────────────────────────┘
Code language: JavaScript (javascript)Deep Dive: Resolving Flag Service Outages
When an external flag management service goes down, your application should never crash. The client SDK should handle this using three layers of resilience:
- Local In-Memory Cache: The SDK keeps the last known good flag state in memory.
- Local Fallback File: If the application restarts while the flag service is down, it reads defaults from a locally bundled configuration file.
- Hardcoded In-Code Default: If the configuration file is missing or corrupted, the lookup function returns the in-code default value (e.g.,
false).
Managing Feature Flags Beyond the Initial Release
The most common long-term challenge with feature flags is not initial implementation—it is managing accumulated complexity over time. Left unmaintained, temporary flags turn codebases into tangled webs of legacy conditions.
+--------------------------------------------------------------------------+
| Flag Technical Debt Accumulation |
| |
| Year 1: 10 Flags ──────► Easy to track and reason about |
| Year 2: 75 Flags ──────► Overlapping conditions, unclear ownership |
| Year 3: 300 Flags ──────► Dead code branches, unexpected interactions |
+--------------------------------------------------------------------------+
Flag Governance Best Practices
- Establish Clear Naming Conventions: Include the team, feature name, and creation date in the flag key.
- Bad:
test_flag_1,new_button - Good:
checkout_team_stripe_integration_q3_2026
- Bad:
- Assign Explicit Ownership: Every flag must have an assigned owner (developer or team) responsible for its lifecycle.
- Enforce Expiration Dates (TTL): Set an intended retirement date when the flag is created. If a temporary flag remains active past its expiration, flag it for cleanup.
- Audit Access Regularly: Restrict who can toggle production flags. Require approvals for changes in production environments.
Temporary Flag Retirement Checklist
Use this checklist once a feature reaches 100% rollout and stabilizes in production:
Markdown
- 1. The feature has run at 100% in production for at least 14 days without incident.
- 2. Create a technical debt cleanup ticket linked to the original feature story.
- 3. Remove conditional `if (flag)` logic from the application code.
- 4. Remove dead code branches and outdated fallback logic.
- 5. Update unit and integration tests to verify only the new, permanent behavior.
- 6. Run the CI/CD pipeline and confirm all tests pass.
- 7. Deploy the cleaned-up code to production.
- 8. Delete the flag key from your feature flag management platform.
Code language: JavaScript (javascript)Final Practice Project: Build a Feature-Flag-Driven Delivery Workflow
To consolidate everything you have learned, build an end-to-end delivery pipeline that brings these concepts together into a working project.
┌────────────────────────────────────────────────────────────────────────┐
│ Final Project Workflow │
│ │
│ 1. Git Repository │
│ ├── src/ (App code with 2 distinct feature flags) │
│ ├── test/ (Test matrix for enabled/disabled/resilience paths) │
│ ├── config/ (Dev, Staging, and Production flag profiles) │
│ └── .github/workflows/deploy.yml (CI/CD pipeline) │
│ │
│ 2. Pipeline Execution │
│ ├── Validates code formatting and configuration syntax │
│ ├── Runs tests against both flag states │
│ └── Deploys to Staging with feature ON, Production with feature OFF │
│ │
│ 3. Operational Verification │
│ ├── Verify deterministic rollout (0% -> 25% -> 100%) │
│ └── Test emergency kill switch and confirm instant mitigation │
└────────────────────────────────────────────────────────────────────────┘
Code language: JavaScript (javascript)Project Requirements
- Application Code:
- Build a lightweight REST API or web application using your language of choice (Node.js, Go, Python, Java).
- Implement two flags:
- Flag A: A Boolean flag controlling a visible UI element or API response format.
- Flag B: A multivariate flag controlling a configuration value (e.g., promotional discount rate or rate limit).
- Automated CI/CD Pipeline:
- Create a pipeline using GitHub Actions, GitLab CI, or Jenkins.
- Add a validation stage that verifies your flag configuration files use valid JSON/YAML.
- Add a test stage that runs automated tests against both flag states (
trueandfalse).
- Multi-Environment Configuration:
- Configure separate flag profiles for Staging and Production.
- Set Flag A to
truein Staging andfalsein Production.
- Gradual Rollout and Kill Switch:
- Implement deterministic, percentage-based targeting for Flag A.
- Include an administrative endpoint or script that acts as an emergency kill switch.
Portfolio and Interview Takeaways
When sharing this project in a portfolio or technical interview, highlight the operational considerations:
- Explain how your pipeline tests both execution branches to prevent regressions in fallback code.
- Walk through your deterministic hashing logic to demonstrate how users receive consistent experiences during gradual rollouts.
- Discuss your flag retirement strategy and how you prevent technical debt from accumulating over time.
Frequently Asked Questions
What are feature flags in CI/CD?
Feature flags (or feature toggles) are conditional software mechanisms that enable or disable specific features at runtime without requiring new code deployments. In CI/CD workflows, they decouple technical deployments from business releases, allowing teams to merge and deploy code continuously while keeping unreleased features dormant in production.
How are feature flags different from environment variables?
Environment variables are set at application startup, requiring a process restart or container redeployment to change values. Feature flags are evaluated dynamically at runtime, allowing updates to take effect immediately without restarts. Additionally, modern feature flag platforms support advanced targeting rules (like user segments and percentage rollouts) that static environment variables cannot provide.
Can feature flags be used with Jenkins and GitHub Actions?
Yes. Both Jenkins and GitHub Actions can validate flag configurations, run automated tests across enabled and disabled states, and automate flag changes via REST APIs during pipeline execution.
Do feature flags require a separate deployment?
No. One of the primary advantages of feature flags is that changing a flag value does not require a code deployment, container rebuild, or service restart. The application reads the updated flag state dynamically at runtime.
How do feature flags support canary releases?
While canary deployments route a percentage of infrastructure traffic to a new software version, feature flags control access to specific functionality within that version. Combining them allows teams to deploy a canary build to 10% of their infrastructure, and then gradually enable a new feature for only a fraction of users on that canary build.
How should feature flags be tested?
Automated test suites should test both the enabled and disabled states of every active flag. Additionally, teams should write resilience tests that verify the application falls back safely if the feature flag service becomes unreachable.
What happens when a feature flag service becomes unavailable?
A well-architected feature flag SDK uses defensive fallbacks: it serves values from a local in-memory cache, falls back to a bundled configuration file, and ultimately defaults to a safe in-code value (typically false). The application continues running without interruption.
When should old feature flags be removed?
Temporary release flags should be removed as soon as a feature reaches 100% rollout and runs stably in production for a designated evaluation window (typically 7 to 30 days). Teams should treat flag cleanup as part of the feature delivery process to avoid accumulating technical debt.
Are feature flags suitable for production applications?
Yes. Feature flags are standard practice at modern software companies and continuous delivery teams. When implemented with defensive defaults, clear lifecycle management, and proper access controls, they significantly improve release safety and operational control.
Conclusion: Make Feature Releases Safer Through Practice
Separating code deployment from feature release gives engineering teams full control over when, how, and to whom new functionality is exposed, shifting software delivery away from risky, all-or-nothing releases toward safe, predictable, and continuous value delivery. By validating both flag states in automated CI/CD pipelines, employing deterministic hashing for gradual rollouts, enforcing graceful offline fallbacks, and routinely retiring obsolete toggles, you maintain a fast delivery cadence without accumulating technical debt. Platforms like DevOpsSchool provide the structured, hands-on environment needed to practice and operationalize these patterns across modern CI/CD architectures. Start small by introducing a single boolean flag in your pipeline tests today, and unlock the speed and confidence of modern continuous delivery.