This article is based on an announcement originally covered by InfoQ and an AWS blog post by Danilo Poccia, Chief Evangelist (EMEA) at Amazon Web Services.
Overview
Amazon has extended AWS Glue with support for streaming ETL jobs, allowing teams to build continuous data ingestion pipelines that process and prepare streaming data on the fly. The feature makes processed data available for analysis within seconds of ingestion, without requiring customers to manage any underlying compute infrastructure.
What Is AWS Glue?
AWS Glue is a fully managed, serverless extract, transform, and load (ETL) service designed to help teams prepare and load data for analytics workloads. Core capabilities include:
- Glue Data Catalog: A central metadata repository that provides a unified view of data across sources, usable by services such as Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum.
- Automatic code generation: AWS Glue generates Scala or Python ETL code that developers can inspect and customize.
- Serverless execution: No compute resources to provision, patch, or scale — AWS manages the underlying infrastructure.
What Streaming ETL Adds
Before this feature, AWS Glue jobs were primarily designed for batch workloads. The streaming ETL extension allows Glue jobs — built on Apache Spark — to run continuously and consume data from:
- Amazon Kinesis Data Streams
- Apache Kafka, including the fully managed Amazon Managed Streaming for Apache Kafka (Amazon MSK)
As Danilo Poccia described in the AWS blog post announcing the feature:
Managing continuous ingestion pipelines and processing data on-the-fly is quite complex, because it’s an always-on system that needs to be managed, patched, scaled, and generally taken care of. Today, we are making this easier and more cost-effective to implement by extending AWS Glue jobs, based on Apache Spark, to run continuously and consume data from streaming platforms such as Amazon Kinesis Data Streams and Apache Kafka (including the fully-managed Amazon MSK).
Danilo Poccia, Chief Evangelist (EMEA), Amazon Web Services
Key Capabilities
Spark Structured Streaming
Streaming ETL jobs in AWS Glue have access to the full capabilities of Spark Structured Streaming. This enables teams to apply a wide range of transformations to in-flight data, including:
- Aggregating records (e.g., computing rolling counts or sums)
- Partitioning data by time window or key
- Reformatting or normalizing fields
- Joining streaming data with static or slowly-changing reference datasets to enrich or cleanse records before writing them to a target
Supported Data Targets
Processed streaming data can be written to:
- Amazon S3 — for data lake storage
- Amazon Redshift — for data warehouse analytics
- Other supported data stores reachable through Glue connections
Automatic Code Generation
AWS Glue automatically generates Scala or Python boilerplate for streaming ETL jobs. Engineers can then customize the generated code using familiar tools and development workflows, lowering the barrier to entry for teams new to Spark Structured Streaming.
Unified Data Catalog
Data ingested via streaming ETL jobs is registered in the AWS Glue Data Catalog, making it immediately queryable and reportable through services such as Amazon Athena, Amazon EMR, and Amazon Redshift Spectrum — without requiring additional catalog configuration.
Common Use Cases
Streaming ETL in AWS Glue is well-suited to scenarios where data arrives continuously and needs to be immediately usable for analytics or downstream systems:
- IoT event streams: Ingest and normalize device telemetry in real time before storing it in a data lake.
- Clickstream analytics: Process user interaction events from web or mobile applications, enriching records with session or user data before loading them into Redshift.
- Network and security logs: Continuously parse and transform log data for near-real-time monitoring dashboards or alerting pipelines.
Serverless Operations and Pricing
Because AWS Glue is serverless, streaming ETL jobs require no cluster provisioning or capacity planning. AWS automatically scales the underlying Spark infrastructure as data volumes change. Poccia summarized the operational benefit:
Managing a serverless ETL pipeline with Glue makes it easier and more cost-effective to set up and manage streaming ingestion processes, reducing implementation efforts so you can focus on the business outcomes of analytics.
Danilo Poccia, Chief Evangelist (EMEA), Amazon Web Services
The streaming ETL feature is available in the same AWS regions as AWS Glue. For current pricing, refer to the AWS Glue pricing page on the AWS website, as rates may have changed since this announcement.
Further Reading
For a deeper walkthrough of the feature, including example job configurations, refer to the original AWS blog post by Danilo Poccia and the AWS Glue developer documentation on streaming ETL.