This article is based on reporting originally published by TechRepublic, drawing on an interview with John ‘JG’ Chirapurath, then General Manager for Azure Data and AI at Microsoft.
Background: From a Unified Code Base to a Broader Value Proposition
For a period, the Azure SQL Database service and the on-premises SQL Server product ran on separate code bases. Microsoft eventually unified them so that applications remained compatible across both environments. That unification revealed a deeper customer need: enterprises did not just want to lift and shift their databases to the cloud — they wanted a simpler way to manage the growing complexity of operating multiple databases across hybrid environments.
“What we saw was that even in the era of the cloud there was so much interest in SQL Server that there was an opportunity to expand the value proposition of what SQL Server is,” said Chirapurath. That insight drove two significant milestones: the introduction of PolyBase in SQL Server 2016, which lets developers query heterogeneous data sources using familiar T-SQL, and the release of SQL Server 2017 for Linux. According to Chirapurath, SQL Server 2017 was downloaded more than 20 million times within 18 months of release, a figure he attributed partly to how easy container support made deployment.
Simplifying Kubernetes for Big Data Workloads
Once SQL Server 2017 ran reliably inside Linux containers, customers began asking for Kubernetes integration — but they also wanted that integration to be manageable. Kubernetes adoption was growing rapidly in enterprise data centres as a container orchestration framework, but its extensibility had also made it increasingly complex.
“One of the reasons why Kubernetes is becoming complicated is that there are lots of extensions that people are adding, so it has gone from what was a general-purpose framework to a ‘one size fits all’ idea. Kubernetes is lots of different things,” Chirapurath observed.
SQL Server 2019 addresses this by giving cluster administrators both an admin portal and a command-line interface to handle all cluster management tasks from a single surface. A full SQL Server Availability Group can be deployed into Kubernetes with a single command, with the cluster ready in approximately half an hour.
What SQL Server 2019 Big Data Clusters Actually Are
SQL Server 2019 Big Data Clusters are a tightly integrated stack that runs on Kubernetes and combines several components:
- SQL Server engine (master instance) — handles relational queries in T-SQL.
- HDFS storage pool — an Hadoop-compatible distributed file system embedded inside the SQL Server cluster, running on a set of Linux containers.
- Apache Spark — the open-source analytics engine, integrated directly with the SQL query processor so that Spark queries can be embedded inside SQL syntax statements.
- PolyBase extensions — SQL Server 2019 extends PolyBase beyond earlier versions to support external data sources including MongoDB and Oracle, in addition to HDFS.
- SQL Server Machine Learning Services — R, Python, Java, and machine-learning scoring run natively in the master cluster instance.
- Integrated security tier — a unified security layer that covers SQL Server, Spark, and HDFS, avoiding the credential and permission fragmentation common when these systems are integrated separately.
“We’ve essentially taken the open-source Spark engine and integrated it tightly with the SQL query processor, the SQL Server engine,” Chirapurath explained. “Now you can essentially embed Spark queries inside SQL Server SQL syntax statements, and you can directly query data lakes.”
Data Virtualisation: Querying Without Moving Data
The central architectural concept in SQL Server 2019 Big Data Clusters is data virtualisation: creating a logical layer that lets analysts and developers query both relational and non-relational data sources with standard SQL, without physically moving data between systems.
Before this capability existed, combining SQL Server data with data lake content typically required a significant ETL pipeline. For example, merging SQL Server data with Spark data in Azure Data Warehouse required Azure Data Factory to ingest and transform data first — a time-consuming and resource-intensive process that became impractical at petabyte scale.
With the data virtualisation layer in SQL Server 2019, queries can join results from a relational SQL Server table and a Spark-queried HDFS data lake in a single result set. Neither dataset needs to be moved or pre-transformed.
“Customers are loath to move data out of SQL Server to do a whole data engineering process of cleansing it to achieve an AI task or an ML task. What this allows you to do is just work with the data inside SQL Server, and marry it in the results set for the data you query using Spark from your data lake.”
— John ‘JG’ Chirapurath, General Manager for Azure Data and AI, Microsoft
Bridging Two Machine-Learning Ecosystems
Chirapurath described two distinct camps of enterprise machine-learning practitioners that SQL Server 2019 is designed to serve simultaneously:
- Big-data engineers who collect data in Hadoop-compatible storage and use Spark — including Spark ML — to analyse it at scale.
- Developer-centric data scientists who work primarily in R, Python, or Scala to build models against more structured datasets.
SQL Server 2019 bridges these two approaches within a single platform. A machine-learning pipeline can pull unstructured data through the Spark integration, pull relational data through the SQL query processor, and use R or Python to build models — all inside the SQL Server environment. Keeping the compute close to the data reduces the latency that would otherwise result from moving large datasets to an external model-training environment.
Data engineers, data scientists, DBAs, and developers can manage databases, run queries, and interact with both toolsets using Azure Data Studio, Microsoft’s cross-platform database management tool.
Why the Structured/Unstructured Split Happened — and Why It No Longer Makes Sense
Chirapurath argues that the sharp division between relational and non-relational data systems is an artefact of historical economics rather than a fundamental technical requirement. Relational databases were designed in an era when storage was expensive; schema-on-write and strict data curation were practical necessities because organisations could not afford to keep everything.
“Relational database systems were born at a time when storage came at a high cost. Because of technology innovations, [the cost of] storage has almost gone to zero,” he noted. As a reference point, he observed that a 20 MB hard drive for an early IBM PC cost $3,000–$4,000 at the time of its launch.
The collapse in storage costs created a new default: save everything. Enterprises now hold petabytes of data in data lakes, but the bottleneck has shifted from storage capacity to the ability to extract value from that stored data. SQL Server 2019 Big Data Clusters are Microsoft’s answer to that challenge — removing the need to choose between a relational system and a big-data ecosystem by making both work together.
Key Takeaways for DevOps and Data Engineering Teams
- SQL Server 2019 Big Data Clusters deploy on Kubernetes, making them manageable using existing container orchestration workflows.
- A full Availability Group cluster can be deployed with a single command in approximately 30 minutes.
- PolyBase in SQL Server 2019 supports external sources including HDFS, MongoDB, and Oracle — queryable via T-SQL without data movement.
- Apache Spark is embedded in the cluster, enabling Spark ML pipelines alongside R and Python-based machine-learning workflows.
- An integrated security tier covers SQL Server, Spark, and HDFS, reducing the operational overhead of multi-system security management.
- Azure Data Studio provides a unified management and query interface for the full stack.
- The data virtualisation layer eliminates the ETL overhead of joining relational and data-lake content in many common analytics scenarios.