ksqlDB Now Available as a Fully-Managed Service on Confluent Cloud

Original reporting by InfoQ. Post date: April 27, 2020.

What Is ksqlDB?

ksqlDB (previously marketed as KSQL) is the streaming SQL engine built on top of Apache Kafka. It lets developers write continuous SQL queries against Kafka topics without writing Java or Scala code, making real-time stream processing accessible to a much broader engineering audience. Common use cases include:

• Stream-to-stream joins — correlate events from two or more Kafka topics in real time.
• Aggregations — compute rolling counts, sums, or averages over time windows.
• Transformations and filtering — reshape, enrich, or drop events as they flow through a pipeline.
• Data enrichment — look up reference data stored in ksqlDB tables and join it with live streams.

Queries run on the ksqlDB Server engine, which handles all reads and writes to and from the underlying Kafka cluster. Developers interact with the engine either through a SQL query editor in the Confluent Control Center UI or directly via the ksqlDB REST API.

General Availability on Confluent Cloud

Confluent announced that ksqlDB moved from preview to production-ready status on Confluent Cloud, available to all customers on usage-based billing plans. The managed service removes the operational overhead of deploying and scaling ksqlDB Server clusters yourself. As Derek Nelson, senior product manager at Confluent, explained at the time of the announcement:

Confluent Cloud ksqlDB eliminates the innate complexities of operating infrastructure in production. We take care of running production-grade infrastructure at scale while you focus on solving business problems with ksqlDB. Transformations, stream-to-stream joins, aggregations, and data enrichment can all be accomplished using ksqlDB with just a few lines of SQL.

Derek Nelson, Senior Product Manager, Confluent

Dan Rosanova, senior group product manager at Confluent, also highlighted ksqlDB’s broader role in the platform when speaking to InfoQ:

ksqlDB is really a new kind of database that has been purpose-built for stream processing applications. We see ksqlDB as an integral service rounding out our cloud offering as a complete streaming platform, beyond Kafka, enabling end-to-end stream processing workloads. It complements our Connect service as well, which enables streaming connectivity to and from many services and platforms. These services all compose together to provide users an event streaming SaaS that they can use to build and run sophisticated and complete modern applications.

Dan Rosanova, Senior Group Product Manager, Confluent

Key Capabilities of the Managed ksqlDB Service

Running ksqlDB as a managed service on Confluent Cloud offers several advantages over self-hosting:

• No infrastructure management — Confluent provisions, scales, and monitors the ksqlDB cluster on your behalf.
• Integrated UI — The Confluent Control Center provides a graphical interface to create streams and tables, browse schemas, and submit continuous queries without leaving the browser.
• REST API access — All operations available in the UI are also available programmatically via the ksqlDB REST API, enabling CI/CD integration.
• Schema Registry integration — Streams and tables are schema-aware, enabling safer evolution of event formats over time.
• Kafka Connect complement — Confluent Cloud Connect can source or sink data to external systems, with ksqlDB handling in-flight transformations in between.

Multi-Cloud Support

Confluent Cloud — and by extension the managed ksqlDB service — is available on all three major public cloud providers:

• Amazon Web Services (AWS) — available since 2017.
• Google Cloud Platform (GCP) — available via the GCP Marketplace.
• Microsoft Azure — available via the Azure Marketplace.

This multi-cloud availability means teams can run ksqlDB workloads in the same region and cloud provider as the rest of their data infrastructure, minimising egress costs and latency.

A Practical Example: Streaming Data Pipeline

A common pattern is using Apache Kafka as the backbone of a data pipeline and ksqlDB to transform events before landing them in a downstream system (a data warehouse, search index, or operational database). Instead of writing a custom Kafka Streams application in Java, a developer can express the same logic in SQL:

-- Create a stream from a raw Kafka topic
CREATE STREAM orders_raw (
  order_id VARCHAR,
  customer_id VARCHAR,
  amount DOUBLE,
  currency VARCHAR
) WITH (
  KAFKA_TOPIC='orders',
  VALUE_FORMAT='JSON'
);

-- Filter and enrich: only USD orders above $100
CREATE STREAM high_value_usd_orders AS
  SELECT
    order_id,
    customer_id,
    amount,
    UCASE(currency) AS currency
  FROM orders_raw
  WHERE currency = 'USD' AND amount > 100
EMIT CHANGES;Code language: PHP (php)

The resulting high_value_usd_orders stream is itself a Kafka topic, ready to be consumed by downstream services or sinked to an external system via Kafka Connect.

Who Benefits Most?

As Bart Kersteter, a systems engineer at Confluent, noted at the time of the announcement, ksqlDB is particularly valuable for teams that need real-time stream processing but do not have deep JVM expertise:

KSQL is great for writing streaming data apps, especially for non-Java programmers. It is now fully supported and available in Confluent Cloud.

Bart Kersteter, Systems Engineer, Confluent

Data engineers already comfortable with SQL for batch processing can apply the same mental model to streaming workloads, significantly reducing the learning curve compared to the Kafka Streams DSL or Apache Flink.

Getting Started

To try the managed ksqlDB service on Confluent Cloud:

1. Sign up for a Confluent Cloud account at confluent.io/confluent-cloud.
2. Create a Kafka cluster in your preferred cloud region (AWS, GCP, or Azure).
3. Add a ksqlDB application from the Confluent Cloud console.
4. Use the built-in query editor to create your first stream and run a continuous SQL query.
5. Explore the official Confluent Cloud ksqlDB documentation for deeper configuration options.

Note: Pricing details and free-tier offers mentioned at the time of this announcement (April 2020) may have changed. Check the Confluent Cloud pricing page for current information.