Background: From a Unified Code Base to a Broader Value Proposition

For a period, the Azure SQL Database service and the on-premises SQL Server product ran on separate code bases. Microsoft eventually unified them so that applications remained compatible across both environments. That unification revealed a deeper customer need: enterprises did not just want to lift and shift their databases to the cloud — they wanted a simpler way to manage the growing complexity of operating multiple databases across hybrid environments.

“What we saw was that even in the era of the cloud there was so much interest in SQL Server that there was an opportunity to expand the value proposition of what SQL Server is,” said Chirapurath. That insight drove two significant milestones: the introduction of PolyBase in SQL Server 2016, which lets developers query heterogeneous data sources using familiar T-SQL, and the release of SQL Server 2017 for Linux. According to Chirapurath, SQL Server 2017 was downloaded more than 20 million times within 18 months of release, a figure he attributed partly to how easy container support made deployment.

Simplifying Kubernetes for Big Data Workloads

Once SQL Server 2017 ran reliably inside Linux containers, customers began asking for Kubernetes integration — but they also wanted that integration to be manageable. Kubernetes adoption was growing rapidly in enterprise data centres as a container orchestration framework, but its extensibility had also made it increasingly complex.

“One of the reasons why Kubernetes is becoming complicated is that there are lots of extensions that people are adding, so it has gone from what was a general-purpose framework to a ‘one size fits all’ idea. Kubernetes is lots of different things,” Chirapurath observed.

SQL Server 2019 addresses this by giving cluster administrators both an admin portal and a command-line interface to handle all cluster management tasks from a single surface. A full SQL Server Availability Group can be deployed into Kubernetes with a single command, with the cluster ready in approximately half an hour.

What SQL Server 2019 Big Data Clusters Actually Are

SQL Server 2019 Big Data Clusters are a tightly integrated stack that runs on Kubernetes and combines several components:

• SQL Server engine (master instance) — handles relational queries in T-SQL.
• HDFS storage pool — an Hadoop-compatible distributed file system embedded inside the SQL Server cluster, running on a set of Linux containers.
• Apache Spark — the open-source analytics engine, integrated directly with the SQL query processor so that Spark queries can be embedded inside SQL syntax statements.
• PolyBase extensions — SQL Server 2019 extends PolyBase beyond earlier versions to support external data sources including MongoDB and Oracle, in addition to HDFS.
• SQL Server Machine Learning Services — R, Python, Java, and machine-learning scoring run natively in the master cluster instance.
• Integrated security tier — a unified security layer that covers SQL Server, Spark, and HDFS, avoiding the credential and permission fragmentation common when these systems are integrated separately.

“We’ve essentially taken the open-source Spark engine and integrated it tightly with the SQL query processor, the SQL Server engine,” Chirapurath explained. “Now you can essentially embed Spark queries inside SQL Server SQL syntax statements, and you can directly query data lakes.”

Data Virtualisation: Querying Without Moving Data

The central architectural concept in SQL Server 2019 Big Data Clusters is data virtualisation: creating a logical layer that lets analysts and developers query both relational and non-relational data sources with standard SQL, without physically moving data between systems.

Before this capability existed, combining SQL Server data with data lake content typically required a significant ETL pipeline. For example, merging SQL Server data with Spark data in Azure Data Warehouse required Azure Data Factory to ingest and transform data first — a time-consuming and resource-intensive process that became impractical at petabyte scale.

With the data virtualisation layer in SQL Server 2019, queries can join results from a relational SQL Server table and a Spark-queried HDFS data lake in a single result set. Neither dataset needs to be moved or pre-transformed.

“Customers are loath to move data out of SQL Server to do a whole data engineering process of cleansing it to achieve an AI task or an ML task. What this allows you to do is just work with the data inside SQL Server, and marry it in the results set for the data you query using Spark from your data lake.”

— John ‘JG’ Chirapurath, General Manager for Azure Data and AI, Microsoft

Bridging Two Machine-Learning Ecosystems

Chirapurath described two distinct camps of enterprise machine-learning practitioners that SQL Server 2019 is designed to serve simultaneously:

1. Big-data engineers who collect data in Hadoop-compatible storage and use Spark — including Spark ML — to analyse it at scale.
2. Developer-centric data scientists who work primarily in R, Python, or Scala to build models against more structured datasets.

SQL Server 2019 bridges these two approaches within a single platform. A machine-learning pipeline can pull unstructured data through the Spark integration, pull relational data through the SQL query processor, and use R or Python to build models — all inside the SQL Server environment. Keeping the compute close to the data reduces the latency that would otherwise result from moving large datasets to an external model-training environment.

Data engineers, data scientists, DBAs, and developers can manage databases, run queries, and interact with both toolsets using Azure Data Studio, Microsoft’s cross-platform database management tool.

Why the Structured/Unstructured Split Happened — and Why It No Longer Makes Sense

Chirapurath argues that the sharp division between relational and non-relational data systems is an artefact of historical economics rather than a fundamental technical requirement. Relational databases were designed in an era when storage was expensive; schema-on-write and strict data curation were practical necessities because organisations could not afford to keep everything.

“Relational database systems were born at a time when storage came at a high cost. Because of technology innovations, [the cost of] storage has almost gone to zero,” he noted. As a reference point, he observed that a 20 MB hard drive for an early IBM PC cost $3,000–$4,000 at the time of its launch.

The collapse in storage costs created a new default: save everything. Enterprises now hold petabytes of data in data lakes, but the bottleneck has shifted from storage capacity to the ability to extract value from that stored data. SQL Server 2019 Big Data Clusters are Microsoft’s answer to that challenge — removing the need to choose between a relational system and a big-data ecosystem by making both work together.

Key Takeaways for DevOps and Data Engineering Teams

• SQL Server 2019 Big Data Clusters deploy on Kubernetes, making them manageable using existing container orchestration workflows.
• A full Availability Group cluster can be deployed with a single command in approximately 30 minutes.
• PolyBase in SQL Server 2019 supports external sources including HDFS, MongoDB, and Oracle — queryable via T-SQL without data movement.
• Apache Spark is embedded in the cluster, enabling Spark ML pipelines alongside R and Python-based machine-learning workflows.
• An integrated security tier covers SQL Server, Spark, and HDFS, reducing the operational overhead of multi-system security management.
• Azure Data Studio provides a unified management and query interface for the full stack.
• The data virtualisation layer eliminates the ETL overhead of joining relational and data-lake content in many common analytics scenarios.

Aurora Serverless Adds PostgreSQL Support

Amazon Web Services announced the general availability of PostgreSQL compatibility on its Aurora Serverless platform. For teams already using Aurora with MySQL and looking for a fully managed, auto-scaling PostgreSQL option, this represents a significant milestone. But before you migrate or spin up a new cluster, it pays to understand exactly what you are getting — and what you are paying for.

What Is Amazon Aurora?

Aurora is AWS’s managed relational database service built for cloud-scale workloads. Its architecture decouples compute from storage: the storage layer is always replicated across multiple Availability Zones (AZs) for durability, while the compute layer can be configured separately. Aurora also supports low-latency global reads through cross-region replicas.

AWS deliberately describes its Aurora offerings as MySQL-compatible and PostgreSQL-compatible rather than as MySQL or PostgreSQL. This is an important distinction — Aurora’s internal implementation differs from the upstream open-source databases, which can affect the behaviour of certain features, extensions, and edge-case queries.

Two Pricing Models: Provisioned vs. Serverless

Provisioned Aurora

The traditional Aurora model is instance-based. AWS recommends running a primary instance plus one or more read replicas. You pay for three components separately:

• Compute instances — pricing starts at approximately $0.088 per hour for a db.t3.medium instance (the minimum supported for Aurora PostgreSQL at the time of this announcement). A db.t3.medium provides 2 vCPUs, 4 GB RAM, and up to 5 Gbps network performance.
• Storage — approximately $0.11 per GB per month.
• I/O requests — approximately $0.22 per million requests.

With provisioned Aurora, you have predictable costs and full control over instance sizing. It suits steady, predictable workloads well.

Aurora Serverless

Aurora Serverless removes the need to select or manage instances. Instead, capacity is expressed in Aurora Capacity Units (ACUs). Each ACU represents a combination of compute and memory, analogous to a database instance but abstracted away from you. You set a minimum and maximum ACU range, and Aurora scales automatically within those bounds.

Pricing for the serverless model at launch was approximately $0.06 per ACU per hour, with a recommended minimum of two ACUs. Storage and I/O are billed at the same rates as provisioned Aurora.

Because the ACU abstraction layers over raw instance types, the exact mapping of ACUs to vCPUs and RAM is not publicly documented in precise detail. This makes it harder to do a direct like-for-like cost comparison with provisioned instances.

Cost Trade-offs: When Does Serverless Make Sense?

Serverless Aurora is generally more expensive than provisioned Aurora for steady workloads. The value proposition is auto-scaling: when your database is idle or lightly loaded, it scales down, and you stop paying for idle compute. For workloads that are highly variable, bursty, or intermittent — such as development databases, staging environments, or applications with unpredictable traffic patterns — serverless can reduce costs significantly compared to running provisioned instances around the clock.

Availability, Failover, and Limitations

One of the most important architectural differences to understand is how Aurora Serverless handles availability.

• Single AZ compute — A serverless DB cluster runs its compute in a single Availability Zone, unlike provisioned Aurora which can span multiple AZs simultaneously for compute.
• Failover time is undefined — If the AZ hosting your serverless cluster becomes unavailable, Aurora will attempt to recreate it in another AZ. AWS’s own documentation at the time stated: “The Aurora Serverless failover time is currently undefined because it depends on demand and capacity availability.” This is a meaningful caveat for production workloads with strict RTO requirements.
• Data durability is preserved — Because the storage layer always spans multiple AZs, your data is protected even if the compute layer fails. The risk is unavailability, not data loss.
• Warm pool for scaling — AWS maintains a pool of pre-warmed database capacity to accelerate scaling events, which reduces the latency of adding capacity during traffic spikes.

Is Aurora Serverless Right for Your PostgreSQL Workload?

Aurora Serverless is a strong fit for specific use cases, but it is not a universal replacement for provisioned Aurora. Consider the following decision criteria:

• Use serverless if your database has variable or unpredictable traffic, is used intermittently (overnight jobs, scheduled tasks), or is a non-production environment where you want to minimise idle costs.
• Use provisioned if you have a steady, high-throughput production workload, require guaranteed multi-AZ compute redundancy with fast, defined failover times, or need to tightly control and predict your monthly database costs.

Both options carry the premium of a fully managed service: AWS handles patching, backups, monitoring integration, and underlying infrastructure. You are paying for operational simplicity in addition to raw compute and storage.

The Bigger Picture

Aurora Serverless PostgreSQL represents AWS bringing its most cloud-native database deployment model to the most widely used open-source relational database. At launch, the serverless model carried trade-offs — higher per-ACU cost, undefined failover times, and single-AZ compute — that made it unsuitable for the most demanding production workloads. These limitations are consistent with first-generation serverless database products across cloud providers.

The trajectory is clear: serverless database abstractions will continue to mature, narrow the feature gap with provisioned options, and become cost-competitive for a broader range of workloads. For teams evaluating Aurora today, it is worth checking the current AWS documentation for the latest ACU pricing, supported PostgreSQL versions, and any updates to the failover SLA, as these specifics evolve with each Aurora release.

Always consult the official Aurora Serverless documentation and the Aurora pricing page for current figures before making architectural decisions.