Source: itbrief.com.au. Originally published April 2019.
The Data Engineering Bottleneck Slowing AI Adoption
Data integration and big data management solutions provider Attunity announced an analytics platform designed to automate streaming data pipelines for AI, machine learning, and data science initiatives. The core problem it addresses is a familiar one: data engineering teams cannot keep pace with the demand for real-time datasets that machine learning applications require.
Data integration and migration are often manual, complex, and time-consuming processes. By the time a dataset is assembled and delivered to data scientists, it is frequently outdated — undermining the value of the analysis before it even begins. Attunity’s approach targets this lag directly by combining efficient change data capture (CDC) at scale with automated data transformations running on Apache Spark.
Attunity for Databricks Unified Analytics Platform
The integration — Attunity for Databricks Unified Analytics Platform — provides continuous change data capture, delivery, and automated data refinement to produce analytics-ready datasets inside Databricks. Databricks’ Unified Analytics Platform unifies data processing and machine learning in a single environment, with Apache Spark as the high-performance processing engine underneath.
By pairing Attunity’s CDC pipeline automation with Databricks, organizations gain a path from raw transactional source data to clean, ML-ready datasets without writing custom integration code. This matters in practice because manual pipeline development is a significant time sink for data engineers — time that could otherwise be spent on higher-value work.
“Our ability to reach into virtually all enterprise systems and deliver real-time, analytics-ready and transactionally consistent data without coding is a powerful enabler for data scientists, allowing them to generate the business insights that accelerate corporate decision making and growth.”
Itamar Ankorion, Chief Marketing Officer, Attunity
Key Capabilities
Accelerated Time to Value
Using Apache Spark as the processing engine, Attunity automates data delivery and creates analytics-ready datasets in Databricks. Data engineers can build reusable, automated pipelines that reduce reliance on manual coding and expensive custom development resources, shortening the delivery cycle from source systems to data scientists.
Reduced Data Preparation Overhead
Attunity’s pipeline automation reduces the data preparation burden on data scientists. Rather than spending significant time wrangling raw data, data scientists can focus on analysis and model development. More agile pipeline generation means that changes to source data schemas or business requirements can be accommodated without rebuilding pipelines from scratch.
Real-Time Data with CDC at Scale
Attunity’s change data capture (CDC) technology provides the low-latency, high-throughput characteristics needed for large-scale machine learning datasets in the cloud. CDC captures only the changes made to source data — inserts, updates, and deletes — and streams those changes continuously, rather than running expensive bulk extractions on a schedule.
Multi-Cloud Support
Attunity supports Databricks deployments on both Microsoft Azure and Amazon Web Services (AWS), as well as a broad range of data lake, data warehouse, and streaming services available on those platforms. This gives organizations flexibility to adopt the platform within their existing cloud strategy without vendor lock-in to a single cloud provider.
Transactionally Consistent, ACID-Compliant Data
Attunity provides full support for the Databricks Unified Analytics Platform’s ACID capabilities. Transactions are delivered and applied in the order they were committed on the source system, ensuring that data scientists and ML models work from a transactionally consistent view of the data — a critical requirement for models where data ordering and integrity affect accuracy.
Why This Matters for DataOps
The Attunity-Databricks integration reflects a broader shift toward DataOps practices — applying DevOps-style automation, collaboration, and continuous delivery principles to data pipelines. Automating the movement and transformation of data reduces the manual coordination between data engineering and data science teams, shortens feedback loops, and makes it easier to keep ML models trained on current data.
For enterprises with data spread across siloed storage systems — on-premises databases, cloud data warehouses, streaming platforms — the ability to unify those sources into a single analytics environment without hand-coded ETL is a meaningful operational improvement. The combination of CDC for real-time ingestion and Spark for transformation automation addresses the two most time-consuming stages of the modern data pipeline.
Note: Attunity was acquired by Qlik in May 2019, shortly after this announcement. Organizations evaluating this technology should review current product branding and availability under Qlik.