Source: TechCrunch (original reporting, April 2020)
Background: A Joint PyTorch Push from AWS and Facebook
AWS and Facebook announced two new open-source projects built around PyTorch, the widely adopted machine learning framework originally developed by Facebook AI Research. The announcement signalled a formal collaboration between two organisations that each operate PyTorch workloads at a very large scale, and it produced two distinct tools aimed at different parts of the ML engineering lifecycle: TorchServe for model serving and TorchElastic for distributed training resilience.
TorchServe: A Model Server Built Specifically for PyTorch
TorchServe is a model-serving framework designed to make it straightforward to deploy PyTorch models to production. Before its release, practitioners who wanted to serve PyTorch models had to rely on generic solutions such as TensorFlow Serving or the AWS Multi Model Server — both of which were not tailored to the idioms and workflows that PyTorch developers expect.
According to Bratin Saha, AWS VP and GM for Machine Learning Services, the primary motivation was community demand. Adapting an existing model server would have meant compromising on PyTorch-specific optimisations and on the API design that PyTorch developers are accustomed to. TorchServe addresses this by exposing an API surface aligned with how PyTorch models are structured and exported.
Saha also confirmed that the open-source release started from the internal model server that AWS uses within its own infrastructure — specifically work that originated inside Amazon SageMaker — and was then iterated on with Facebook’s engineering team and the wider community before being published.
Key Capabilities of TorchServe
- Multi-model serving: Serve multiple PyTorch models simultaneously from a single server process.
- REST and gRPC APIs: Standard inference endpoints that align with production service expectations.
- Model versioning and management: Register, update and retire model versions without restarting the server.
- Logging and metrics: Built-in hooks for operational observability.
- Custom handlers: Pre- and post-processing logic can be attached per model, keeping business logic close to the model artefact.
TorchElastic: Fault-Tolerant Distributed Training on Kubernetes
TorchElastic addresses a different problem: making distributed PyTorch training jobs resilient to node failures and to changes in cluster size at runtime. This is particularly relevant when training on cost-optimised infrastructure such as AWS EC2 Spot Instances, which can be interrupted with minimal notice, or when using AWS Elastic Kubernetes Service (EKS) clusters that autoscale.
Traditional distributed ML training frameworks typically assume a fixed worker count for the entire duration of a job. If one worker fails, the entire job fails. TorchElastic breaks this assumption by allowing the training process to continue — or gracefully restart from a checkpoint — when the number of available workers changes.
Why This Matters for Cost and Reliability
- Spot Instance compatibility: Spot or preemptible instances are significantly cheaper than on-demand instances, but they can be reclaimed by the cloud provider. TorchElastic enables training jobs to tolerate this without total loss of progress.
- Dynamic scaling: Worker counts can grow or shrink during training, making it easier to respond to changing cluster availability.
- Kubernetes-native: TorchElastic integrates with Kubernetes, which is the dominant container orchestration layer in enterprise ML infrastructure.
- Developer control: Unlike the fully managed equivalent inside SageMaker, TorchElastic gives teams direct access to the underlying mechanics, which is useful when running on-premises or on third-party clouds.
Why AWS and Facebook Collaborated
Bill Jia, Facebook’s VP of AI Infrastructure, explained the rationale simply: a large proportion of researchers and enterprise teams who use PyTorch also run their workloads on AWS. By investing jointly in shared infrastructure tooling, both companies benefit — and more importantly, the broader PyTorch community gains production-grade tooling that neither organisation would have had as strong an incentive to open-source alone.
For AWS, this announcement also represents a continuation of its engagement with the open-source ML ecosystem. The company has been a significant contributor to Apache MXNet and has contributed to projects including Jupyter, TensorFlow, and libraries such as NumPy. Adding TorchServe to that list reflects the practical reality that PyTorch has become the framework of choice for a large share of ML research and is increasingly adopted in production settings.
Where to Find the Projects
Both TorchServe and TorchElastic are open source and hosted under the PyTorch organisation. You can find TorchServe at the pytorch/serve GitHub repository and TorchElastic at pytorch/elastic. Official documentation is available through the PyTorch website.
Summary
| Project | Purpose | Key Use Case |
|---|---|---|
| TorchServe | Model serving framework for PyTorch | Deploying trained PyTorch models as REST/gRPC endpoints |
| TorchElastic | Fault-tolerant distributed training | Training on Kubernetes with Spot Instances or dynamic cluster sizes |