Back in May 2018, Twitter announced that they were collaborating with Google Cloud to migrate their services to the cloud. Today, after two years, they have successfully migrated and have also started reaping the benefits of this move.

For the past 14 years, Twitter has been developing its data transformation pipelines to handle the load of its massive user base. The first deployments for those pipelines were initially running in Twitter’s data centers. For example, Twitter’s Hadoop file systems hosted more than 300PB of data across tens of thousands of servers.

This system had some limitations despite having consistently sustained massive scale. With increasing user base, it became challenging to configure and extend new features to some parts of the system, which led to failures.

In the next section, we take a look at how Twitter’s engineering team, in collaboration with Google Cloud, have successfully migrated their platform.

How The Migration Took Place

For the first step, Twitter’s team left a few pipelines such as the data aggregation legacy Scalding pipelines, unchanged. These pipelines were made to run at their own data centers. But the batch layer’s output was switched to two separate storage locations in Google Cloud.

W3Schools
The output aggregations from the Scalding pipelines were first transcoded from Hadoop sequence files to Avro on-prem, staged in four-hour batches to Cloud Storage, and then loaded into BigQuery, which is Google’s serverless and highly scalable data warehouse, to support ad-hoc and batch queries.

This data from BigQuery is then read by a simple pipeline deployed on Dataflow, and some light transformations are applied. Finally, results from the Dataflow pipeline are written into Bigtable. This is a Cloud Bigtable for low-latency, fully managed NoSQL database that serves as a backend for online dashboards and consumer APIs.

Also Read COVID-19 Misinformation Tweets Being Removed By Twitter

With the successful installation of the first iteration, the team began to redesign the rest of the data analytics pipeline using Google Cloud technologies.

After evaluating all possible options, the team chose Apache Beam because of its deep integration with other Google Cloud products, such as Bigtable, BigQuery, and Pub/Sub, Google Cloud’s fully managed, real-time messaging service.

A BigQuery slot can be defined as a unit of computational capacity required to execute SQL queries.

The Twitter team re-implemented the batch layer as follows:

Data is first staged from on-prem HDFS to Cloud Storage
A batch Dataflow job then regularly loads the data from Cloud Storage and processes the aggregations, and
The results are then written to BigQuery for ad-hoc analysis and Bigtable for the serving system.
For instance, the results showed that processing 800+ queries(~1 TB of data each) took a median execution time of 30 seconds.

Migration final picture via GCP
The above picture illustrates the final architecture after the second step of migration.

For job orchestration, the Twitter team built a custom command line tool that processes the configuration files to call the Dataflow API and submit jobs.

What Do The Numbers Say

via Twitter
The migration for modernization of advertising data platforms started back in 2017, and today, Twitter’s strategies have come to fruition, as can be seen in their annual earnings report.

The revenue for Twitter can be divided mainly into two categories:

Ads
Data licensing and other services.
According to the quarterly earnings report for the year 2019, Twitter has declared decent profits with steady progress.

“We reached a new milestone in Q4 with quarterly revenue in excess of $1 billion, reflecting steady progress on revenue product and solid performance across most major geographies, with particular strength in US advertising,” said Ned Segal, Twitter’s CFO.

Also Read How Google’s BigQuery ML Is Empowering Data Analysts
The 2019 revenue was $3.46 billion, which is an increase of 14% year-over-year.

Advertising revenue totalled $885 million, an increase of 12% year-over-year
Total ad engagements increased by 29% year-over-year
The motivation behind Twitter’s migration to GCP also involves other factors like the democratization of data analysis. For Twitter’s engineering team, visualization, and machine learning in a secure way is a top priority, and this is where Google’s tools such as BigQuery and Data Studio came in handy.

Although Google’s tools were used for simple pipelines, Twitter, however, had to build their infrastructure called Airflow. In the area of data governance, BigQuery services for authentication, authorization, and auditing did well but, for metadata management and privacy compliance, in house systems had to be designed.

A 1992 Wake-Up Call That Still Resonates

In 1992, while leading development for a shipping company, a conversation with the Head of Sales reframed everything. The request was straightforward: a breakdown of revenues, costs, and key customer metrics that existing reports couldn’t answer. The problem was that producing a new report took two to three weeks. The sales leader’s response was blunt: “That’s not good enough. I need this for a board meeting tomorrow. We have new competitors disrupting our pricing.”

That moment crystallised a principle that still drives data architecture decisions today: business users need insights at the speed of the business, not at the speed of IT queues. IT should be the enabler of insight delivery, not the gatekeeper.

Decades later, the underlying need hasn’t changed — but the tools available to address it have transformed dramatically. The convergence of AI/ML platforms and Kubernetes-based infrastructure is making on-demand intelligence genuinely achievable at enterprise scale.

From Self-Service BI to Self-Driving Intelligence

The journey toward democratised data access has progressed through several distinct phases:

Phase 1 — The Push Model

Traditional BI was IT-driven: analysts defined reports, scheduled runs, and distributed results. Business users received answers to questions that IT had anticipated in advance. The 1992 scenario above is a textbook example of this model’s limitations.

Phase 2 — The Self-Service Pull Model

Tools from vendors like Business Objects, Cognos, and Tableau transformed the equation by letting business users formulate their own queries against predefined datasets. This pull-and-push hybrid gave analysts genuine autonomy — within the bounds of pre-modelled schemas. The growth of these vendors demonstrates the real market value of reducing IT dependency for routine analytical questions.

The limitation, however, is structural: predefined schemas constrain the questions that can be answered. Users can slice and filter data that was anticipated, but cannot easily explore unknown patterns or discover entirely new relationships.

Phase 3 — AI/ML-Augmented Intelligence

Machine Learning moves the boundary further. Instead of answering predefined questions, ML models can surface patterns that were never explicitly formulated. Practical examples already in production include:

• Automated credit decisioning: Loan applications assessed and approved or rejected without human review, using models trained on historical approval data and risk signals.
• Intelligent customer support: Chatbots that learn from prior interactions to answer novel questions without escalating to a human agent.
• Predictive operations: Supply chain and logistics systems that trigger restocking or rerouting decisions autonomously based on real-time signals.

The next evolution — already underway — is a self-driving model where intelligence is embedded at every layer. Rather than users pulling reports or machines surfacing recommendations, decisions are made autonomously to shape machine-to-machine, machine-to-human, and human-to-human interactions in real time.

Why Kubernetes Is the Right Foundation for AI/ML at Scale

The biggest obstacle to capitalising on AI/ML potential is not the algorithms — it is the infrastructure required to run them reliably at scale. Platforms like Apache Hadoop and Apache Spark are powerful, but operating them demands specialist skills that most organisations struggle to hire and retain.

Kubernetes addresses this by providing a container orchestration layer that abstracts much of the operational complexity.

How Kubernetes Works (A Practical Summary)

To understand why Kubernetes matters for data and AI workloads, it helps to trace the abstraction stack:

1. Physical server: Raw compute with a fixed operating system and application stack.
2. Virtual machine (e.g., VMware): One physical host is divided into multiple isolated VMs, each with its own OS. This improved utilisation but not portability.
3. Container (e.g., Docker): The entire application stack — runtime, dependencies, configuration — is packaged into a portable image that runs consistently across environments.
4. Kubernetes: Orchestrates fleets of containers across many nodes, handling scheduling, scaling, self-healing, networking, and secrets management automatically.

For AI/ML pipelines, this means a data engineering team can define a training job, a feature engineering pipeline, and a model-serving endpoint as containerised workloads. Kubernetes scales them up during high-demand periods and scales them down when idle — without manual intervention.

Multi-Cloud and Hybrid Environments

One of Kubernetes’ most strategically important properties is cloud portability. Enterprises rarely commit to a single cloud provider. AWS, Microsoft Azure, and Google Cloud Platform each offer managed Kubernetes services (EKS, AKS, and GKE respectively). While there are implementation differences between these managed offerings, the core Kubernetes API provides a largely consistent surface for defining workloads.

This makes Kubernetes the de facto standard framework for:

• Multi-cloud deployments: Running workloads across two or more public cloud providers without complete re-architecture.
• Hybrid environments: Spanning on-premises data centres and public cloud infrastructure under a unified control plane.

Software vendors are increasingly shipping their AI/ML products as Kubernetes-native applications — Helm charts, operators, and custom resource definitions — precisely because it is the lowest common denominator that works across these diverse environments.

Practical Tips for Getting Started

Tip 1: Start Your AI/ML Journey Now

If your organisation has not yet explored AI or ML use cases, the time cost of waiting is compounding. One of the most common blockers is a perceived need for deep in-house data science expertise. A growing number of software vendors now offer pre-built models, AutoML tooling, and model marketplaces that lower the barrier to entry significantly. Starting with a narrow, well-scoped use case — automated document classification, demand forecasting, or anomaly detection in logs — builds organisational muscle before tackling more complex problems.

Tip 2: Treat Data Preparation as a First-Class Workload

A persistent failure mode in AI/ML initiatives is under-investing in data preparation. Industry experience consistently shows that shaping, cleaning, deduplicating, and consolidating raw data into model-ready features consumes the majority of total project effort — commonly cited at around 80 percent. Deploying a sophisticated ML platform on top of untrustworthy data produces unreliable models.

Key actions:

• Invest in dedicated data engineering capacity, not just data science.
• Evaluate data preparation and integration tooling alongside modelling platforms.
• Avoid assuming that AI platforms can automatically correct data quality problems at source.

Tip 3: Engage Your Vendors on Kubernetes Standards

Kubernetes adoption is broad, but standardisation across managed cloud offerings remains incomplete. Differences in networking, storage, and authentication between EKS, AKS, and GKE can create real portability friction for complex workloads.

As a practitioner, you have more leverage than you may realise. Raise Kubernetes compatibility as an explicit requirement in vendor evaluations. Ask software vendors about their Kubernetes roadmap, which distributions they certify against, and how they handle upgrades across cloud providers. Collective customer pressure has historically been more effective at driving cross-vendor standards than top-down mandates from the hyperscalers themselves.

The Bigger Picture

The convergence of AI/ML and Kubernetes-based infrastructure is not a passing trend — it reflects a fundamental shift in how software is built, deployed, and operated. The organisations that build this capability now will be in the best position to embed intelligence into their products and operations as the tooling continues to mature.

The lesson from 1992 remains valid: the speed at which a business can act on information determines competitive outcomes. Kubernetes and AI together make it possible to close that loop faster than ever before — but only if the underlying data infrastructure is sound and the operational model is designed for scale from the start.

Why Big Data Still Matters

According to an Accenture study cited in the original source, 79% of enterprise executives believe that companies not embracing big data will lose their competitive edge, and 83% say their organizations have pursued big data projects to stay ahead. Despite this widespread awareness, actual success rates remain low: a Capgemini survey found that only 27% of executives described their big data initiatives as successful.

The gap between ambition and execution is a persistent problem. Most businesses know they should be leveraging big data, but stumble when it comes to implementation. With data volumes continuing to grow rapidly, the ability to process data quickly and derive meaningful insight from it is becoming a core operational requirement rather than a differentiator.

What Is Big Data?

“Big data” refers to extremely large datasets that are too voluminous, varied, or fast-moving for traditional data processing tools to handle effectively. The value of big data lies not in the raw volume, but in the ability to analyze it for patterns, trends, and correlations that inform better business decisions.

Big data is commonly characterized by three dimensions:

• Volume: Datasets ranging from terabytes to petabytes and beyond.
• Velocity: Data arriving in real time or near-real time from sources such as sensors, logs, and user interactions.
• Variety: Structured, semi-structured, and unstructured data from databases, documents, images, social media, and more.

The increasing diversity of data sources is what makes modern big data analytics particularly powerful — and particularly challenging to implement well.

Engines vs. Frameworks: Understanding the Terminology

Two terms that come up constantly in big data discussions are processing engines and frameworks. While the distinction is not always rigid, a useful working definition is:

• Processing engine: The core component responsible for executing operations on data — sorting, filtering, aggregating, or transforming it.
• Framework: A broader set of components, libraries, and conventions that work together to manage the full data processing lifecycle, including ingestion, storage, processing, and output.

Both share the same fundamental goal: operate over large amounts of data to surface patterns, broaden understanding, and expose complex interactions that would be invisible in smaller samples.

Why Cloud Infrastructure Is Central to Big Data

On-premises hardware can handle big data workloads, but the capital cost and operational overhead of building out that infrastructure at scale is prohibitive for most organizations. Cloud environments address this by providing:

• Elastic scalability: Compute and storage resources can expand to meet peak workloads and contract when demand drops, reducing waste.
• Managed services: Cloud providers offer managed versions of popular big data tools, reducing the operational burden of cluster management and patching.
• Cost model: Pay-as-you-go pricing allows organizations to run large batch jobs without permanently provisioning the underlying infrastructure.
• Integration: Modern cloud platforms offer tight integration between data storage, processing, and business intelligence (BI) tooling, streamlining the analytics pipeline.

The combination of cloud scalability with big data frameworks is what has made large-scale analytics accessible to organizations outside the technology sector.

Key Big Data Processing Frameworks

Apache Hadoop

Hadoop is a Java-based open-source framework under the Apache Software Foundation umbrella. It is designed to store and process very large datasets across clusters of commodity hardware using a distributed computing model. Its core components include:

• HDFS (Hadoop Distributed File System): Distributes data across cluster nodes with built-in replication for fault tolerance.
• MapReduce: The original Hadoop processing model, inspired by Google’s MapReduce paper, which breaks large jobs into parallel map and reduce tasks.
• YARN (Yet Another Resource Negotiator): Manages cluster resources and job scheduling.

Hadoop can be deployed on-premises in an organization’s own data center or run on cloud infrastructure. Its open-source nature and large ecosystem of compatible tools have driven widespread industry adoption.

Apache Spark

Apache Spark is a fast, general-purpose distributed processing engine that addresses some of Hadoop MapReduce’s limitations, particularly for iterative workloads. Key characteristics include:

• In-memory processing: Spark can cache intermediate data in memory across iterations, making it significantly faster than disk-based MapReduce for many workloads.
• Unified API: Spark supports batch processing, streaming (Spark Streaming / Structured Streaming), SQL queries (Spark SQL), machine learning (MLlib), and graph processing (GraphX) within a single framework.
• Language support: APIs are available in Scala, Java, Python, and R.

Apache Storm is another open-source option in this space, focused specifically on real-time stream processing rather than batch workloads.

Cloudera Data Platform

Cloudera provides a commercial distribution and managed platform built on open-source technologies including Apache Hadoop, Apache Spark, Apache Hive, Apache Kafka, and others. It is designed to simplify the discovery, storage, processing, modeling, and serving of large amounts of data, adding enterprise features such as security, governance, and support on top of the upstream open-source projects.

Hadoop and Apache CloudStack: An Open-Source Combination

Apache CloudStack is an open-source cloud computing platform used to deploy and manage large networks of virtual machines. Like Hadoop, it is implemented in Java and is part of the Apache Software Foundation. For organizations that prefer to build their own private cloud infrastructure rather than rely on public cloud providers, CloudStack offers a fully open-source path.

Running Hadoop on a CloudStack-managed private cloud gives organizations the operational flexibility of cloud-style resource management — elastic scaling, self-service provisioning, and multi-tenancy — while keeping data and compute on infrastructure they control. This pairing is particularly relevant for industries with strict data residency or regulatory requirements that make public cloud options complicated.

Turning Big Data Ambition into Execution

The statistics make clear that awareness of big data’s value is nearly universal among enterprise leaders. What separates successful organizations is execution: choosing the right framework for the workload, building the infrastructure to support it reliably, and ensuring the output connects to decisions that actually improve the business.

A practical starting point for most teams is to:

1. Define the business question the data analysis should answer before selecting a tool.
2. Start with a manageable proof-of-concept dataset rather than attempting to process everything at once.
3. Evaluate managed cloud services (such as Amazon EMR, Google Dataproc, or Azure HDInsight) before committing to self-managed cluster infrastructure.
4. Invest in data quality and pipeline reliability — bad input data produces unreliable insights regardless of how powerful the processing engine is.
5. Build feedback loops so that analytical outputs visibly influence decisions, creating organizational buy-in for further investment.

Big data processing is not inherently complex, but it does require deliberate architectural choices and a clear connection between the data being collected and the outcomes the organization is trying to improve.

The Production Gap in Big Data Analytics

Many organizations have successfully run Spark and Hadoop pilot projects, only to hit a wall when trying to move those prototypes into production. A custom-configured environment built by a single data scientist can take days or weeks to recreate, and may behave unreliably once shared across a larger team. This gap between proof-of-concept and operational deployment has become one of the most common obstacles in enterprise big data programs.

The root cause is often configuration complexity. Spark clusters typically require careful tuning of versions, dependencies, data access paths, and tooling — details that are difficult to document, reproduce, and hand off reliably. When a wider user pool starts sharing an environment built for one, things break.

Applying DevOps Thinking to Big Data Infrastructure

To address this reproducibility problem, some teams have started applying DevOps principles — specifically containerization and microservices architecture — to their Spark and Hadoop deployments. The same techniques that allow application developers to ship consistent, portable software environments are now being used to package and deliver big data tooling.

The core idea is straightforward: if a Spark cluster and its associated tools (notebooks, query engines, data connectors) can be defined as container images, those environments become reproducible, versionable, and deployable on demand — without manual intervention from a platform engineer each time.

A Real-World Example: The Advisory Board Company

Ramesh Thyagarajan, managing director at The Advisory Board Company — a Washington, D.C.-based firm providing analytics and consulting services to healthcare organizations — described the challenge his team faced directly: data science teams and business stakeholders did not want to wait days or weeks for a new Spark cluster to be provisioned with the right tools, versions, configurations, and data.

His team adopted BlueData Software’s EPIC platform, a container-based architecture designed for big data deployments. The platform runs Spark SQL, the Spark analytics engine, and Apache Zeppelin developer notebooks. Users — data scientists and business analysts alike — can spin up new Spark clusters on demand without needing to understand the underlying configuration complexity.

To handle data consistency across on-demand clusters, the Advisory Board team built an internal framework to ingest data into HDFS (Hadoop Distributed File System) from a single curated source. This centralized data layer is critical: with more than 400 users, allowing each person to manage their own cluster and data copy would have been operationally unmanageable.

“There is no way we could have supported 400-plus users, each creating their own cluster.”

— Ramesh Thyagarajan, Managing Director, The Advisory Board Company

The State of Containers in Big Data (circa Early 2017)

According to Tom Phelan, co-founder and chief architect at BlueData and a veteran of the virtualization industry, Spark clusters had historically been deployed on bare metal servers — a pattern that produces static, difficult-to-change environments with slow, manual setup processes. This was the dominant model at the time of Spark Summit East 2017 in Boston, where Phelan presented on the topic.

Container-based implementations were possible using custom scripting, but managing a growing pipeline of Spark components by hand quickly became complex and error-prone. Phelan noted that modern Spark workloads are often part of orchestrated, multi-component pipelines — which makes ad hoc containerization harder to sustain at scale.

“Today you have to navigate a river of container managers,” Phelan told conference attendees — pointing to the fragmentation in the container orchestration ecosystem as an additional challenge for teams trying to standardize their big data infrastructure.

Preconfigured Images and Elastic Scaling

A practical step toward making containers viable for data science teams is packaging common tools as preconfigured Docker images. BlueData’s platform, as described in early 2017 release notes, supported tools such as JupyterHub, RStudio Server, and Apache Zeppelin as ready-to-use images — eliminating per-environment configuration work for end users.

This approach directly enables elastic scaling: because environments are defined as images rather than as manually configured systems, administrators can provision and tear down compute resources as workloads demand — a critical capability in both cloud and on-premises deployments. For Spark and Hadoop to see broader enterprise adoption, this kind of dynamic resource management is increasingly expected by the organizations running them.

Key Takeaways for DevOps and Data Engineering Teams

• Reproducibility is the core problem. Spark and Hadoop environments that cannot be reliably recreated will always create friction between data scientists and operations teams.
• Containers solve the configuration drift problem. Packaging cluster environments as Docker images makes them portable, versionable, and deployable without manual setup.
• Centralized data layers matter. On-demand cluster provisioning only scales if users are working from a shared, curated data source — not isolated copies of data per cluster.
• Orchestration complexity is real. Hand-rolled scripting works for simple pipelines but breaks down quickly. Purpose-built platforms or mature orchestration tools are needed for multi-component Spark workloads.
• The goal is self-service analytics. Shielding data scientists and business analysts from infrastructure complexity — while maintaining operational control — is the practical outcome teams should design toward.

The convergence of container technology with big data infrastructure reflects a broader trend: the operational practices that made application delivery faster and more reliable are now being applied to the data platform layer. For teams still running Spark on bare metal with hand-configured environments, the case for containerization is increasingly hard to ignore.