Amazon Web Services (AWS) announced nine new capabilities for the Amazon SageMaker machine learning (ML) service during the ML keynote at the 2020 re:Invent conference, which was held virtually.

“Today, we are announcing a set of tools for Amazon SageMaker that makes it much easier for developers to build end-to-end machine learning pipelines to prepare, build, train, explain, inspect, monitor, debug, and run custom machine learning models with greater visibility, explainability, and automation at scale,” says AWS Amazon machine learning vice president Swami Sivasubramanian.

The new capabilities are:

Data Wrangler – for developers to prepare data for machine learning.

Data Wrangler contains over 300 built-in data transformers that can help customers normalise, transform, and combine features without having to write any code, while managing all of the processing infrastructure under the hood.

Feature Store – a purpose-built data store for storing, updating, retrieving, and sharing features.

Feature Store provides a purpose-built feature store where developers can access and share features that make it easier to name, organise, find, and share sets of features among teams of developers and data scientists. Residing in Amazon SageMaker Studio, close to where machine learning models are run, it provides single-digit millisecond latency for inference.

Pipelines – a purpose-built continuous integration and continuous delivery service

Pipelines logs each step in SageMaker Experiments every time a workflow is run.

This helps developers visualise and compare machine learning model iterations, training parameters, and outcomes. Workflows can be shared and re-used between teams.

Clarify – providing greater visibility into training data to limit bias

Clarify integrates with Data Wrangler where it runs a set of algorithms on features to identify bias during data preparation with visualisations that include a description of the sources and severity of possible bias. Clarify also integrates with Experiments to make it easier to check trained models for statistical bias, details how each feature inputted into the model is affecting predictions, and integrates with Model Monitor to alert developers if the importance of model features shifts and causes model behaviour to change.

Deep profiling for Debugger – monitors machine learning training performance

Deep profiling enables the profiling and monitoring of system resource utilisation in Studio. This expands Debugger’s scope to monitor the utilisation of system resources, send out alerts on problems during training in Studio or via AWS CloudWatch, and correlate usage to different phases in the training job or a specific point in time during training. Debugger can also trigger actions based on alerts, works across frameworks (PyTorch, Apache MXNet, and TensorFlow) and collects necessary system and training metrics automatically without requiring any code changes in training scripts.

Distributed Training – can train large models up to two times faster

Distributed Training with SageMaker’s Data Parallelism engine scales training jobs from one GPU to hundreds or thousands by automatically splitting data across multiple GPUs, improving training time by up to 40%. Data Parallelism’s engine manages GPUs for optimal synchronisation.

Distributed Training with SageMaker’s Model Parallelism engine can split large, complex models with billions of parameters across multiple GPUs by automatically profiling and identifying the best way to partition models. They do this by using graph partitioning algorithms to optimally balance computation and minimise communication between GPUs, resulting in minimal code changes and fewer errors caused by GPU memory constraints.

Edge Manager – machine learning model monitoring and management for edge devices

Edge Manager optimises models to run faster on target devices and provides model management for edge devices, so customers can prepare, run, monitor, and update deployed machine learning models across fleets of devices at the edge.

Customers can cryptographically sign their models, upload prediction data from their devices to SageMaker for monitoring and analysis, and view a dashboard that tracks and visually reports on the operation of the deployed models within the SageMaker console.

Almost two years after unveiling its Inferentia high-performance machine learning inference chips, Amazon has almost completed a migration of the bulk of its Alexa text-to-speech ML inference workloads to Inf1 instances on its AWS EC2 platform. The dramatic move to infrastructure powered by the newer chips was made to gain significant cost savings and performance gains over the GPU-based instances on EC2 that were formerly being used, according to the company.

By migrating the workloads to Inf1 instances, the Alexa text-to-speech workloads experience 25% lower end-to-end latency, while cutting operating costs by 30% compared to the previously used GPU-based instances, Amazon announced on Thursday (Nov. 12). By reducing the latency, Alexa engineers are also able to use more complex algorithms to improve the Alexa experience for users.

The Amazon Inferentia chip design was unveiled at the AWS re:Invent conference in November 2018, while Inf1 inferences followed a year later with an announcement at re:Invent in 2019.

The Inferentia chips allow Inf1 instances to provide up to 30% higher throughput and up to 45% lower cost per inference compared to GPU-based G4 instances, which previously were the most economical AWS cloud instances for ML inference, according to Amazon.

“While training rightfully receives a lot of attention, inference actually accounts for the majority of complexity and the cost – for every dollar spent on training, up to nine are spent on inference – of running machine learning in production, which can limit much broader usage and stall innovation,” an AWS spokesperson told EnterpriseAI. “EC2 instances – virtual servers running in the AWS cloud – offer a broad range of instance types powered by a variety of processors to meet specific workload requirements. AWS offers EC2 instances based on the fastest processors from Intel, cost optimized instances with AMD processors, the most powerful GPU instances from Nvidia, as well as instances based on AWS Inferentia chips, a high performance machine learning inference chip designed by AWS.”

While the lower latency benefits Alexa engineers in their text-to-speech work, customers across a diverse set of industries are seeing similar benefits as they turn to machine learning to address common use cases such as personalized shopping recommendations, fraud detection in financial transactions, increasing customer engagement with chat bots and more, the AWS spokesperson said. “Many of these customers are evolving their use of machine learning from running experiments to scaling up production machine learning workloads where performance and efficiency really matter. Customers want high performance for their machine learning applications in order to deliver the best possible end user experience.”

That’s where the Inf1 instances provide answers, the spokesperson added. “With Inf1 instances, customers can run large scale machine learning inference to perform tasks like image recognition, speech recognition, natural language processing, personalization, and fraud detection at the lowest cost in the cloud.”

Inferencing is the science and practice of making predictions using a trained ML model. The Inferentia chips were developed by Israel-based Annapurna Labs, which was acquired by Amazon in 2015. Each Inferentia chip provides hundreds of TOPS (tera operations per second) of inference throughput. Hundreds of TOPS can be included in each chip and the chips can be banded together to harness thousands of TOPS.

The Inferentia chips are designed for larger workloads that consume entire GPUs or require lower latency. The chips provide hundreds of teraflops each and thousands of teraflops per Amazon EC2 instance for multiple frameworks, including TensorFlow, Apache MXNet, and PyTorch. They also are built for multiple data types, including INT-8 and mixed precision FP-16 and bfloat16. The Inferentia chips complement AWS’s Amazon Elastic Inference, which allows customers to avoid running workloads on whole Amazon EC2 P2 or P3 instances where there is relatively low utilization. Instead, developers can run on a smaller, general-purpose Amazon EC2 instance and use Elastic Inference to provision the right amount of GPU performance, resulting in up to 75% cost savings.