<?xml version="1.0" encoding="UTF-8"?><rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>failure &#8211; Best DevOps</title>
	<atom:link href="https://www.bestdevops.com/tag/failure/feed/" rel="self" type="application/rss+xml" />
	<link>https://www.bestdevops.com</link>
	<description>Lets Learn, Do it &#38; Share! Thats a Best DevOps!!!</description>
	<lastBuildDate>Mon, 12 Oct 2026 13:45:42 +0000</lastBuildDate>
	<language>en-US</language>
	<sy:updatePeriod>
	hourly	</sy:updatePeriod>
	<sy:updateFrequency>
	1	</sy:updateFrequency>
	<generator>https://wordpress.org/?v=7.1.3</generator>
	<item>
		<title>Is AWS Too Big to Fail? Cloud Concentration Risk Worries the Tech Community</title>
		<link>https://www.bestdevops.com/possible-amazon-cloud-failure-worries-tech-community/</link>
					<comments>https://www.bestdevops.com/possible-amazon-cloud-failure-worries-tech-community/#respond</comments>
		
		<dc:creator><![CDATA[anil]]></dc:creator>
		<pubDate>Sat, 05 Aug 2017 06:18:00 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps News]]></category>
		<category><![CDATA[DevOps Updates]]></category>
		<category><![CDATA[Amazon’s cloud service]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[Cloud computing]]></category>
		<category><![CDATA[Cloud failure]]></category>
		<category><![CDATA[failure]]></category>
		<category><![CDATA[Tech community]]></category>
		<guid isPermaLink="false">http://www.bestdevops.com/?p=2334</guid>

					<description><![CDATA[As Amazon Web Services powers a growing share of global internet infrastructure—from Netflix to major banks—engineers and regulators are raising serious questions about the systemic risk of over-reliance on a single cloud provider.]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph"><em>This article is based on reporting originally published by Financial News (fnlondon.com) in August 2017, with community discussion sourced from Hacker News.</em></p>



<h2 class="wp-block-heading">A Single Point of Failure for the Internet?</h2>



<p class="wp-block-paragraph">Amazon Web Services (AWS) has become so deeply embedded in global internet infrastructure that its potential failure is no longer just a business continuity problem—it is beginning to look like a systemic risk. That concern, long discussed quietly in engineering circles, moved into broader public view in mid-2017 when Financial News revealed the scale of AWS adoption in the financial sector and the tech community responded with alarm.</p>



<p class="wp-block-paragraph">Discussions on <a href="https://news.ycombinator.com">Hacker News</a>—the community forum run by Y Combinator, the startup accelerator behind companies such as Dropbox and Airbnb—surfaced a range of concerns from working engineers about what a serious, prolonged AWS outage would actually mean in practice.</p>



<h2 class="wp-block-heading">How Dominant Is AWS?</h2>



<p class="wp-block-paragraph">AWS relaunched as a commercial cloud platform in 2006 and has since grown into the dominant infrastructure provider across multiple industries. Its customer list, as reported by Financial News, includes:</p>



<ul class="wp-block-list">
<li>Major banks: Citigroup, JPMorgan, and HSBC</li>
<li>Stock exchange operator Nasdaq</li>
<li>Media platform Netflix</li>
<li>The UK&#8217;s Financial Conduct Authority (FCA)</li>
<li>100% of the Fortune 50 companies (as claimed by Amazon at the time)</li>
</ul>



<p class="wp-block-paragraph">This level of concentration—where a single vendor underpins critical systems across finance, media, government, and e-commerce simultaneously—is what prompted the &#8220;too big to fail&#8221; framing borrowed from the 2008 banking crisis.</p>



<h2 class="wp-block-heading">What Engineers Are Saying</h2>



<p class="wp-block-paragraph">Hacker News contributors were direct about the downstream impact of an extended AWS outage. User <strong>Mgkimsal</strong> wrote:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>&#8220;How many government and/or university systems would be unable to function because of direct or indirect dependencies on AWS-related services? [Amazon] going down for, let&#8217;s say four days, would cause big havoc on so many projects and systems I know of. I&#8217;m pretty sure most people have no clue how much of their data and systems functionality is reliant on AWS-related services.&#8221;</p></blockquote>



<p class="wp-block-paragraph">User <strong>Fenwick67</strong> expanded the picture beyond consumer inconvenience:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>&#8220;If AWS went down hard, it would be more than just &#8216;I can&#8217;t watch Netflix&#8217;. Some people wouldn&#8217;t be able to do their computer work, send or receive emails, others might not receive their paychecks or be able to pay bills.&#8221;</p></blockquote>



<p class="wp-block-paragraph">User <strong>Dalbasal</strong> framed the concern more broadly:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>&#8220;It seems reasonable to start worrying about the fragility potentially introduced by these massive internet infrastructure companies.&#8221;</p></blockquote>



<p class="wp-block-paragraph">One contributor, <strong>Peteretep</strong>, raised the national security dimension, noting that concentration of critical infrastructure in a small number of physical data centers could make those facilities high-value targets—a concern that goes beyond typical business risk discussions.</p>



<h2 class="wp-block-heading">The February 2017 S3 Outage: A Real-World Warning</h2>



<p class="wp-block-paragraph">The debate is not hypothetical. In February 2017, a human error during routine maintenance of AWS&#8217;s S3 object storage service in the US-East-1 region triggered a significant outage that lasted several hours. Among the affected services were Business Insider, Quora, Medium, and notably AWS&#8217;s own service health dashboard—making it temporarily impossible to even check the status of the outage through official channels.</p>



<p class="wp-block-paragraph">This incident illustrated two important points for DevOps teams:</p>



<ol class="wp-block-list">
<li><strong>Indirect dependencies are invisible until they fail.</strong> Many services affected by the S3 outage were not direct AWS customers—they relied on third-party tools, libraries, or SaaS products that themselves depended on S3.</li>
<li><strong>Single-region architectures amplify risk.</strong> Organizations that had not configured cross-region replication or multi-cloud failover were fully exposed.</li>
</ol>



<h2 class="wp-block-heading">Amazon&#8217;s Position</h2>



<p class="wp-block-paragraph">Gavin Jackson, then head of EMEA at AWS, told Financial News that the platform operates data centers across multiple countries and regions, and that customers can—and should—replicate data across centers to protect against localized failures. He stated:</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>&#8220;I can pretty confidently say that I don&#8217;t think that there&#8217;s any other company on the planet that has as many people thinking about security every day.&#8221;</p></blockquote>



<p class="wp-block-paragraph">This is technically accurate—AWS does offer the tools to build highly available, multi-region architectures. The problem critics raise is not that those tools don&#8217;t exist, but that many organizations have not implemented them, either due to cost, complexity, or simply not understanding the dependency chain running beneath their systems.</p>



<h2 class="wp-block-heading">Regulatory Concern: The Financial Stability Board Weighs In</h2>



<p class="wp-block-paragraph">The concern is not limited to the engineering community. The Financial Stability Board (FSB), in a June 2017 report titled <em>Financial Stability Implications from FinTech</em>, noted that third-party service providers are &#8220;quickly becoming more prominent and critical, especially in areas of cloud computing and data services.&#8221; The FSB called on local regulators to evaluate whether tighter rules governing concentration risk in cloud infrastructure were needed.</p>



<p class="wp-block-paragraph">This regulatory angle is significant. Financial institutions already operate under strict operational resilience requirements, and supervisors in the UK (FCA, PRA) and EU (EBA, DORA framework, introduced later) have since moved toward explicit rules governing cloud provider concentration and exit strategies for regulated firms.</p>



<h2 class="wp-block-heading">Practical Takeaways for DevOps and Platform Teams</h2>



<p class="wp-block-paragraph">Whether or not AWS is &#8220;too big to fail&#8221; in a regulatory sense, the underlying engineering problem is real and actionable. Teams relying heavily on any single cloud provider should consider the following:</p>



<ul class="wp-block-list">
<li><strong>Map your dependency graph.</strong> Identify not just your direct AWS usage but the third-party services your application depends on that are themselves AWS-hosted.</li>
<li><strong>Design for multi-AZ as a minimum.</strong> Single Availability Zone deployments are inappropriate for production workloads with meaningful SLAs.</li>
<li><strong>Evaluate multi-region for critical paths.</strong> Cross-region replication of data and active-active or active-passive failover for critical services raises resilience significantly, at added cost and complexity.</li>
<li><strong>Test your failover.</strong> Documented failover procedures that have never been tested in production are not reliable failover procedures.</li>
<li><strong>Consider multi-cloud selectively.</strong> Full multi-cloud is operationally expensive, but abstracting storage or DNS layers can reduce lock-in for specific critical components.</li>
<li><strong>Monitor for indirect dependencies.</strong> Services like npm, Docker Hub, GitHub, and many SaaS tools run on AWS. An outage can affect your CI/CD pipeline even if your application itself is elsewhere.</li>
</ul>



<h2 class="wp-block-heading">Conclusion</h2>



<p class="wp-block-paragraph">The concentration of critical internet infrastructure in a small number of hyperscale cloud providers is a genuine systemic risk—not a theoretical one. The February 2017 S3 outage demonstrated that cascading failures can affect organizations with no direct relationship to the root cause. For DevOps engineers and platform architects, the lesson is that availability architecture must account for provider-level failure, not just instance or zone-level failure. For regulators, the question of how to govern this concentration remains open and increasingly urgent.</p>



<p class="wp-block-paragraph"><em>Original reporting by Financial News (fnlondon.com), August 2017. Community discussion via Hacker News.</em></p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.bestdevops.com/possible-amazon-cloud-failure-worries-tech-community/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
		<item>
		<title>Is AWS Too Big to Fail? Cloud Concentration Risk in Financial Services</title>
		<link>https://www.bestdevops.com/is-amazons-cloud-service-too-big-to-fail/</link>
					<comments>https://www.bestdevops.com/is-amazons-cloud-service-too-big-to-fail/#respond</comments>
		
		<dc:creator><![CDATA[anil]]></dc:creator>
		<pubDate>Thu, 03 Aug 2017 06:25:52 +0000</pubDate>
				<category><![CDATA[Cloud Computing]]></category>
		<category><![CDATA[DevOps]]></category>
		<category><![CDATA[Amazon Web Services]]></category>
		<category><![CDATA[Amazon’s]]></category>
		<category><![CDATA[Cloud]]></category>
		<category><![CDATA[cloud service]]></category>
		<category><![CDATA[failure]]></category>
		<guid isPermaLink="false">http://www.bestdevops.com/?p=2315</guid>

					<description><![CDATA[As major banks, fintechs, and even regulators migrate to AWS and Azure, financial watchdogs are asking a familiar question: has the financial sector concentrated too much of its critical infrastructure in too few cloud providers?]]></description>
										<content:encoded><![CDATA[
<p class="wp-block-paragraph"><em>This article is based on a 2017 report originally published by <strong>fnlondon.com</strong>, drawing on interviews with AWS and Microsoft executives, fintech founders, and security researchers. Some figures and market conditions have evolved since publication.</em></p>



<h2 class="wp-block-heading">From Snowballs to Snowmobiles: AWS&#8217;s Scale Problem</h2>



<p class="wp-block-paragraph">When large enterprises — particularly banks — first move to Amazon Web Services, they face an immediate physical challenge: how do you transfer petabytes of legacy data into the cloud? AWS solved this with its <strong>Snowball</strong> appliance: a ruggedized, encrypted storage device shipped to a client site, filled with data on-premises, then physically transported to an AWS data centre for upload.</p>



<p class="wp-block-paragraph">As customer data volumes grew, even the Snowball Edge (with higher capacity than the standard Snowball) was no longer sufficient for some enterprises. AWS&#8217;s response was characteristically blunt: mount a purpose-built, data-centre-grade container on a tractor-trailer. They called it a <strong>Snowmobile</strong>. The customer plugs it into their data centre, the data is encrypted and ingested, the truck drives to an AWS facility, and the data is uploaded and re-encrypted. It is a vivid illustration of how seriously AWS treats the mechanics of large-scale data migration.</p>



<h2 class="wp-block-heading">AWS and Azure Dominate Financial Services</h2>



<p class="wp-block-paragraph">Since AWS relaunched its commercial cloud platform in 2006, it has become the de facto standard for enterprise cloud infrastructure. By 2017, AWS revenues had reached approximately $14 billion annually, up from roughly $10 billion the prior year, and the platform counted Nasdaq and all Fortune 50 companies among its customers.</p>



<p class="wp-block-paragraph">Financial services adoption lagged other industries — banks are famously conservative — but the pace accelerated dramatically in the mid-2010s. Major institutions including Citigroup, JPMorgan, and HSBC moved workloads onto AWS. Regulators themselves followed: the UK&#8217;s Financial Conduct Authority (FCA) and the US Financial Industry Regulatory Authority (FINRA) became cloud customers.</p>



<p class="wp-block-paragraph">Microsoft Azure pursued a parallel strategy. By mid-2017, Microsoft claimed that 85% of systemically important banks had signed on to Azure, ranging from established names like UBS to new entrants like ClearBank.</p>



<p class="wp-block-paragraph">A survey of more than 40 London fintechs at the time found that the vast majority ran on AWS, with most of the remainder on Azure, and only three using both.</p>



<h2 class="wp-block-heading">The Systemic Risk Question</h2>



<p class="wp-block-paragraph">The concentration of financial infrastructure in two cloud providers triggered a familiar debate. In June 2017, the Financial Stability Board (FSB) raised concerns about the systemic risk created by banks&#8217; dependence on third-party providers, specifically calling out cloud computing and data services. The FSB urged local regulators to assess whether tighter rules were warranted, and acknowledged that it lacked sufficient data to draw firm conclusions — one proposed measure was requiring banks to disclose which third-party providers they rely on.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>&#8220;The moment that there is a security flaw or an availability flaw within the Amazon fabric, for us that is a catastrophic failure.&#8221;</p><cite>Gavin Jackson, Head of EMEA, Amazon Web Services</cite></blockquote>



<p class="wp-block-paragraph">The analogy to the 2008 financial crisis is not merely rhetorical. Systemically important financial institutions (SIFIs) were considered too big to fail because their collapse would cascade through the global economy. The question regulators began asking in 2017 was whether AWS or Azure had achieved a comparable degree of systemic importance — not through financial leverage, but through operational dependency.</p>



<h2 class="wp-block-heading">Why Multi-Cloud Is Harder Than It Sounds</h2>



<p class="wp-block-paragraph">The obvious mitigation for concentration risk is diversification: require financial institutions to spread workloads across multiple providers. In practice, this is more complicated than it appears.</p>



<p class="wp-block-paragraph">Modern cloud platforms are not simply storage and compute rentals. AWS and Azure each offer deeply integrated ecosystems of services — databases, analytics pipelines, machine learning frameworks, security tooling, identity management, and more. A financial institution that builds its core platform on AWS-native services (such as RDS, Redshift, or SageMaker) cannot simply replicate those workloads on Azure without significant re-engineering. The two platforms&#8217; APIs, data models, and operational tooling are fundamentally different.</p>



<p class="wp-block-paragraph">Andrew White, co-founder of FundApps, put it plainly: running on both AWS and Azure would &#8220;pretty much require near duplication of all operational infrastructure and staff and would be a nightmare for any company, large or small.&#8221; Charlie Henderson, co-founder of Feedstock, noted the same problem from the vendor side: when a bank insists on Azure and his product is built on AWS, &#8220;you have to rebuild it, which is very costly.&#8221;</p>



<p class="wp-block-paragraph">Gavin Jackson of AWS pushed back on the framing of &#8220;lock-in&#8221; altogether, arguing that AWS services are built on open standards and open-source software, making it technically possible for customers to leave. Microsoft&#8217;s Richard Peers acknowledged that switching difficulty is a &#8220;real issue&#8221; while arguing that the risk calculus must be proportionate — a small fintech representing 0.01% of UK payment volumes does not carry the same systemic weight as a clearinghouse or payment rail.</p>



<p class="wp-block-paragraph">One company, Credit Benchmark, took a different path entirely, selecting CenturyLink as its cloud provider rather than AWS or Azure. Its CTO described the trade-off plainly: the ready-made managed services on the major platforms make life easier in the short term, but deepen dependency over time.</p>



<h2 class="wp-block-heading">The Shared Responsibility Model and Its Gaps</h2>



<p class="wp-block-paragraph">Cloud providers operate on a <strong>shared responsibility model</strong>: the provider secures the underlying infrastructure, while the customer is responsible for securing everything they build on top of it — their data, their access controls, their application configurations, and their network rules.</p>



<blockquote class="wp-block-quote is-layout-flow wp-block-quote-is-layout-flow"><p>&#8220;If you are architecting your applications for a single data centre in a single availability zone in a single region, then you could argue there&#8217;s an accident waiting to happen.&#8221;</p><cite>Gavin Jackson, Head of EMEA, Amazon Web Services</cite></blockquote>



<p class="wp-block-paragraph">Javvad Malik, security advocate at AlienVault, identified a persistent and dangerous misunderstanding in the financial sector: the belief that moving to AWS transfers security responsibility to Amazon. &#8220;&#8216;We put it on Amazon so it must be secure,&#8221; he said, is a mindset that is &#8220;wrongly embedded&#8221; among bank management. It ignores firewalls, operating system hardening, and database configuration — all of which remain the customer&#8217;s domain.</p>



<p class="wp-block-paragraph">Malik also flagged a structural skills gap: cloud security expertise is not uniformly distributed across financial institutions&#8217; IT teams. Where that knowledge gap exists, risk follows. A further complication is outsourcing — some banks delegate management of their AWS accounts to third parties, adding another link in the chain and another potential point of failure.</p>



<p class="wp-block-paragraph">Ankur Modi, CEO of cybersecurity startup StatusToday, noted that even the most robust infrastructure is not immune to outages. In February 2017, a misconfigured command during routine maintenance of AWS S3 in the US-EAST-1 region caused a significant outage that disrupted a wide range of services — including Business Insider, Quora, Medium, and AWS&#8217;s own status dashboard — for several hours. The incident was caused by human error, not a security breach, but it illustrated that availability risk is real regardless of the provider&#8217;s security record.</p>



<h2 class="wp-block-heading">AWS&#8217;s Argument: Resilience by Design</h2>



<p class="wp-block-paragraph">AWS counters concentration risk arguments with its global infrastructure architecture. The platform operates across multiple geographic regions, each divided into multiple isolated <strong>Availability Zones</strong> — physically separate data centres within a region, connected by high-bandwidth, low-latency links. Customers who architect their applications to span multiple AZs and multiple regions gain a degree of resilience that most on-premises deployments cannot match.</p>



<p class="wp-block-paragraph">Jackson argued that AWS does not access customer data — it is always encrypted and controlled by the customer, who determines the region and replication strategy. He declined to disclose AWS&#8217;s security investment figures but pointed to over eleven years of operation without a major security breach of the underlying infrastructure as evidence of the platform&#8217;s trustworthiness.</p>



<p class="wp-block-paragraph">The broader argument from both AWS and Microsoft is that, for most financial institutions, running on a major cloud provider is more secure than maintaining aging on-premises infrastructure with limited security budgets and staff. The pendulum, as Jackson described it, had shifted: banks were no longer asking &#8220;How do I think about security in the cloud?&#8221; but rather &#8220;I have to get into the cloud, because of security.&#8221;</p>



<h2 class="wp-block-heading">Key Takeaways for DevOps and Platform Teams</h2>



<ul class="wp-block-list">
<li><strong>Shared responsibility is not optional.</strong> Cloud providers secure their infrastructure; you secure everything you build on it. Misconfigurations, weak IAM policies, and unpatched OS images are your problem, not AWS&#8217;s.</li>
<li><strong>Multi-AZ and multi-region architecture matters.</strong> Deploying to a single availability zone in a single region exposes you to outages like the 2017 S3 incident. Design for failure from the start.</li>
<li><strong>Multi-cloud is architecturally expensive.</strong> True workload portability between AWS and Azure requires avoiding provider-native managed services and accepting significant operational overhead. Evaluate this trade-off honestly before committing.</li>
<li><strong>Vendor lock-in is real, even with open-source tooling.</strong> Using open-source software does not eliminate lock-in if your operational tooling, IAM, networking, and data services are tightly coupled to one provider&#8217;s abstractions.</li>
<li><strong>Concentration risk is a governance issue, not just a technical one.</strong> For regulated financial institutions, documenting third-party dependencies — including cloud providers — is increasingly a regulatory expectation, not merely a best practice.</li>
<li><strong>Skills gaps are an underestimated risk vector.</strong> Migrating to the cloud without investing in cloud-native security expertise creates exposure regardless of the provider&#8217;s own security posture.</li>
</ul>



<h2 class="wp-block-heading">Where the Debate Stands</h2>



<p class="wp-block-paragraph">The concentration risk debate in financial services cloud has not been resolved — it has intensified. Regulators in the UK, EU, and US have continued to develop frameworks for operational resilience that explicitly address cloud dependency, including requirements for exit planning and third-party risk management. The FSB&#8217;s concerns about systemic risk from cloud concentration remain active policy questions.</p>



<p class="wp-block-paragraph">For DevOps and platform engineers working in or adjacent to financial services, this debate has direct operational implications. Architecture decisions that seem purely technical — which managed database to use, how to structure IAM, whether to use a provider&#8217;s native container orchestration — carry regulatory and systemic risk dimensions that teams need to understand and document.</p>



<p class="wp-block-paragraph">As Jackson noted in 2017, the cloud market — even dominated by two players — was still early. That remains true. But the infrastructural and regulatory stakes have only grown since.</p>
]]></content:encoded>
					
					<wfw:commentRss>https://www.bestdevops.com/is-amazons-cloud-service-too-big-to-fail/feed/</wfw:commentRss>
			<slash:comments>0</slash:comments>
		
		
			</item>
	</channel>
</rss>
