Originally reported by Forbes (October 2019). This article expands on the announced updates to Google Cloud AI Platform’s training and inference capabilities.
What Is Google Cloud AI Platform?
Google Cloud AI Platform is an end-to-end machine learning platform-as-a-service (ML PaaS) designed for data scientists, ML developers, and AI engineers. It provides managed services covering the full ML lifecycle — from data preparation and model training through to model serving and monitoring — giving teams a unified environment to build and deploy production-grade machine learning models on Google Cloud.
In late 2019, Google announced a set of updates that made training and deploying ML models on the platform significantly more flexible and powerful. The key areas of improvement were model development (training) and model deployment (inference).
Model Training Improvements
Custom Container Support (GA)
One of the most significant announcements was the general availability of custom container support for training on Cloud AI Platform. Previously, users were constrained to the ML frameworks and library versions that Google pre-installed on the platform. Custom container support removes that restriction entirely.
With this feature, teams can:
- Package any ML framework, algorithm, or custom library into a Docker image and run it on AI Platform — regardless of whether Google natively supports it.
- Pin exact versions of Python, CUDA, framework dependencies, and internal tooling inside the container, ensuring reproducibility across local and cloud environments.
- Build and test container images locally before pushing them to the cloud, enabling faster iteration cycles.
- Integrate AI Platform training jobs into existing CI/CD pipelines, treating model training as a first-class automated build step.
For DevOps teams managing ML infrastructure, this effectively means AI Platform training can be treated like any other containerised workload — version-controlled, tested, and deployed through standard tooling.
Flexible Machine Types and Scale Tiers
To simplify hardware selection for training jobs, Google introduced scale tiers — predefined cluster specifications based on classes of Google Compute Engine (GCE) VMs. Each scale tier is mapped to workload characteristics, so teams can select an appropriate tier without manually specifying individual VM parameters.
For teams with more specific requirements, a custom tier option allows explicit configuration of the machine types for each role in a distributed training cluster:
- Master node — coordinates the training job.
- Worker nodes — execute the bulk of computation.
- Parameter servers — manage shared model parameters during distributed training.
This distributed training architecture allows large datasets and complex models to be trained at scale by parallelising computation across multiple nodes. Both custom containers and the expanded machine type options for training were made generally available with this release.
Model Deployment and Inference Improvements
What Is Inference?
In ML operations, inference refers to running a fully-trained model against new input data to generate predictions. The Cloud AI Platform Prediction service hosts trained models and handles the infrastructure required to serve predictions at scale — including provisioning, autoscaling, and load balancing — without requiring manual cluster management.
Expanded Machine Type Selection for Prediction
Previously, the Online Prediction service offered limited compute options, supporting only one or four vCPU machine configurations. With the updated release, customers can select from a broader range of GCE machine types when hosting prediction endpoints, and can attach accelerators to reduce inference latency:
- NVIDIA T4 GPUs — suited for accelerating deep learning inference workloads.
- TPUs (Tensor Processing Units) — Google’s custom ML accelerators, designed for high-throughput TensorFlow workloads.
The service continues to handle provisioning, scaling, and serving as a managed offering, meaning engineering teams do not need to manage the underlying Kubernetes infrastructure directly.
Prediction Logging to BigQuery
A notable operational addition is the ability to log prediction requests and responses directly to BigQuery. This is a practical capability for ML teams responsible for model quality in production, enabling:
- Skew detection — identifying when the distribution of live input data diverges from the training data distribution.
- Outlier analysis — flagging unusual inputs that the model may handle poorly.
- Retraining decisions — using logged data to determine when a model’s accuracy has degraded enough to warrant retraining.
This closes an important feedback loop between model serving and model improvement, supporting more mature MLOps practices.
GKE-Backed Prediction Infrastructure
The Cloud AI Platform Prediction service is powered by Google Kubernetes Engine (GKE), which provides the elastic scalability required for production prediction workloads. Teams benefit from GKE’s autoscaling and reliability characteristics without needing to manage cluster configuration themselves.
Significance for DevOps and MLOps Teams
These updates collectively represent a meaningful step towards treating ML workloads with the same operational rigour applied to standard software deployments. Key takeaways for DevOps and platform engineering teams include:
- Custom containers bring ML training into the same Docker-based workflow already used for application services, enabling consistent CI/CD integration.
- Flexible compute for both training and inference allows right-sizing of resources rather than accepting platform defaults.
- GPU and TPU support for prediction addresses latency-sensitive inference use cases without requiring self-managed accelerator infrastructure.
- BigQuery prediction logging provides the observability data needed to detect model drift and support data-driven retraining decisions.
Following the rebranding of its ML PaaS to AI Platform at Google Cloud Next, these updates reflect Google’s continued investment in making the platform competitive for production ML deployments across organisations of varying scale and maturity.