Originally reported via it.toolbox.com (2019). This article covers the key capabilities and performance improvements introduced in the updated Cloud Storage Connector for Hadoop (GCS Connector).
What Is the Google Cloud Storage Connector for Hadoop?
The Cloud Storage Connector for Hadoop — often referred to as the GCS Connector — is an open-source Java client library that runs inside Hadoop JVMs. It enables Hadoop-based workloads, including Apache Spark jobs and Hadoop Compatible File System (HCFS) CLI operations, to read from and write to Google Cloud Storage (GCS) directly, in place of or alongside the Hadoop Distributed File System (HDFS).
Google Cloud Platform announced an updated version of the connector that focuses on throughput efficiency for columnar file formats such as Apache Parquet and ORC, stronger isolation for Cloud Storage directory modifications, and a series of broad performance improvements including lower latency, increased parallelization, and smarter configuration defaults.
Why Replace HDFS with Cloud Storage?
Storing big data workloads in Google Cloud Storage rather than HDFS offers several practical advantages for teams running Hadoop or Spark at scale:
- Cost reduction: Cloud Storage eliminates the need for a long-running HDFS cluster with three on-disk replicas, which can be significantly more expensive to operate continuously.
- Storage–compute separation: Decoupling storage from compute lets teams scale each layer independently based on actual demand rather than keeping them locked together.
- Persistent storage after cluster shutdown: Unlike HDFS, which is tied to cluster lifetime, Cloud Storage buckets persist after Hadoop clusters are terminated, preventing data loss and reducing costs from always-on clusters.
- Shared buckets across clusters: Multiple ephemeral Hadoop clusters can read from and write to the same Cloud Storage buckets simultaneously, simplifying multi-cluster architectures.
- No storage administration overhead: Tasks like HDFS upgrades, balancing, and maintaining high availability are eliminated — GCS handles durability and availability natively.
Compatibility and Deployment
The GCS Connector is fully open source and officially supported by Google Cloud Platform. It comes pre-configured in Cloud Dataproc, GCP’s managed Hadoop and Spark service. However, it can also be installed and used with other Hadoop distributions, including Hortonworks, Cloudera, and MapR, making it a viable path for migrating on-premises HDFS data to the cloud or bursting workloads onto GCP infrastructure.
Performance Improvements in the Updated GCS Connector
The updated release includes a range of targeted performance optimizations. Key improvements include:
Directory Rename Parallelization
In addition to using Cloud Storage batch requests, the connector now executes those batches in parallel. According to Google, this reduces the rename time for a directory containing 32,000 files from approximately 15 minutes down to 1 minute 30 seconds — a roughly 10× improvement.
Latency Optimizations
The connector reduces the number of Cloud Storage API requests required to complete high-level Hadoop file system operations, cutting round-trip overhead and lowering overall operation latency.
Concurrent Glob Algorithm Execution
Both regular and flat glob algorithms now run concurrently, allowing the connector to automatically yield the best-performing result for a given file tree structure — whether it is deep and hierarchical or wide and shallow.
Smarter Implicit Directory Repair
Implicit directory repair is now triggered during delete and rename operations rather than during list and glob operations. This change reduces the latency of expensive list and glob calls and removes the requirement for write permissions on read-only requests, improving security posture for read workloads.
Cloud Storage Read Consistency
The connector enforces requests against a consistent version of a Cloud Storage object, preventing scenarios where different parts of a read operation inadvertently access different object versions. This improves both data correctness and read performance.
Columnar Format Throughput: Parquet and ORC
One of the headline improvements in this release is increased throughput efficiency for columnar file formats — particularly Apache Parquet and ORC (Optimized Row Columnar). These formats are widely used in data lake and analytics workloads on Hive, Spark SQL, and Presto, making this improvement directly relevant to teams running analytical pipelines on GCP.
Summary
The updated Google Cloud Storage Connector for Hadoop is a meaningful step toward making Cloud Storage a production-grade replacement for HDFS in big data environments. With improvements to directory operation speed, read latency, glob performance, and columnar format throughput, the connector lowers both the operational complexity and the performance gap that has historically made HDFS migration difficult. Teams running Hadoop or Spark — whether on Cloud Dataproc or on-premises distributions — should evaluate this release as part of their cloud migration or hybrid architecture planning.
The GCS Connector source code is available on GitHub under the Google Cloud Platform organization. Refer to the official GCP documentation for installation guides and current release notes.