How Autonomous Vehicles Generate 20–100 TB of Data Per Day — and Why Cloud Infrastructure Matters

This article is based on a Microsoft and Frost & Sullivan whitepaper on autonomous vehicle development, originally reported by techau.com.au in March 2020.

Microsoft’s Long History in Automotive Software

Microsoft’s involvement in the automotive industry stretches back decades. Windows Embedded Automotive — first launched in 1998 — powered early connected-car platforms including Ford’s original SYNC, Toyota’s Entune, and Fiat’s Blue & Me. However, active development on Windows Embedded Automotive ceased around 2010, with end-of-support following shortly after.

With that platform era concluded, Microsoft shifted its automotive ambitions toward cloud and DevOps infrastructure. Rather than owning the in-vehicle operating system, Microsoft now positions Azure as the backbone for AV development toolchains, data storage, AI model training, and simulation pipelines.

The Data Problem: 20–100 TB Per Vehicle Per Day

The central engineering challenge for autonomous vehicle development is data scale. According to the Microsoft and Frost & Sullivan whitepaper, AVs operating at higher levels of autonomy are expected to generate between 20 and 100 terabytes of data per day during on-road development and testing.

This data comes from the sensor stack required for autonomous perception: LiDAR, radar, cameras, ultrasonic sensors, GPS, and inertial measurement units all running simultaneously. Every object detection event, lane-keep maneuver, and edge-case driving scenario must be logged, labeled, and fed back into AI training pipelines.

At that scale, on-premises data centers face serious limitations in cost, throughput, and elasticity. The whitepaper argues that cloud infrastructure becomes a practical necessity — not merely a preference — as AV programs scale from prototype to fleet-level trials.

The Four Technology Pillars of Autonomous Vehicles

Autonomous vehicle technology can be broken into four functional layers, each with distinct engineering and infrastructure requirements:

• Perception: Collection and processing of sensor data — object recognition, localization, and environmental mapping. Perception systems generate the bulk of raw data that must be stored and analyzed.
• Planning: AI models that predict and plan safe paths, avoid obstacles, and manage motion at intersections, merges, and complex scenarios. Requires large-scale model training infrastructure.
• Actuation: Low-level control of braking, steering, throttle, and the human-machine interface (HMI). Actuation systems must meet strict real-time and safety certification requirements.
• Software: The connective tissue across all three layers — managing data ingestion, model training, simulation, validation (including software-in-the-loop and hardware-in-the-loop testing), and continuous deployment.

Software is identified as the central competency that determines how fast an AV program can iterate. This is where cloud-based DevOps tooling becomes directly relevant to engineering teams.

Key Engineering Challenges in AV Development

The whitepaper outlines several recurring challenges that AV engineering teams face:

• End-to-end workflow management: AV toolchains span data acquisition, labeling, training, simulation, and validation. Keeping these stages integrated with feedback loops and audit trails for automotive safety certifications (such as ISO 26262) is a non-trivial systems problem.
• Data curation at scale: Not all 20–100 TB of daily sensor data is equally valuable. Teams need pipelines to filter, tag, and selectively store edge cases and high-value driving scenarios without incurring unsustainable storage costs.
• Compute demand for training: Training deep learning models on large labeled datasets — especially for perception tasks like object detection and semantic segmentation — requires significant GPU or specialized AI accelerator capacity that fluctuates based on project phase.
• Simulation and validation: Real-world testing must be complemented by software-in-the-loop (SIL) and hardware-in-the-loop (HIL) simulations to achieve the safety validation mileage required for regulatory approval without proportionally increasing road-test costs.

Emerging Business Models Enabled by Autonomy

The Frost & Sullivan analysis also addresses how autonomous driving is reshaping automotive business models — context that is important for DevOps and platform engineering teams building the services layer on top of AV hardware:

• Mobility-as-a-Service (MaaS): The industry is projected to shift from vehicle ownership to usership models. Robotaxi and shared autonomy platforms require backend infrastructure for fleet management, routing, and payment — all software-intensive problems.
• Vehicle services differentiation: As sensor hardware standardizes across manufacturers, software services and in-vehicle experiences become the competitive differentiator. OEM brand value increasingly depends on software update cadence and service quality.
• Unlocked in-vehicle experiences: With drivers no longer required to focus on the road, new productivity and entertainment applications become viable in-cabin products — creating demand for connected application platforms.
• Usage-based insurance and predictive maintenance: Connected vehicle data enables peripheral services that monetize the telemetry stream, requiring secure data pipelines and analytics platforms.
• Autonomous logistics and long-haul delivery: Vehicles that can operate continuously without driver rest requirements represent a structural efficiency gain for freight. Autonomous last-mile delivery pilots are already underway at several logistics providers.

How Azure Maps to the AV Development Stack

Microsoft’s whitepaper positions Azure as an integrated platform for AV development rather than just a storage backend. The following table, drawn from the whitepaper’s product mapping, illustrates how Microsoft’s services correspond to AV engineering functions:

This mapping is not exhaustive and reflects Microsoft’s own positioning. Organizations building AV toolchains will typically evaluate these services alongside competing offerings from AWS (such as AWS RoboMaker and SageMaker) and Google Cloud (Vertex AI, Cloud Storage), depending on existing vendor relationships and specific workload requirements.

Why Cloud-Native DevOps Is Central to AV Programs

For DevOps engineers working in or adjacent to the automotive space, the key takeaway from this analysis is structural: AV development is fundamentally a software engineering and data engineering problem at scale. The vehicle is, in many respects, a data collection and compute platform that happens to move.

That framing has direct implications for how AV programs should be organized:

• CI/CD pipelines must extend from model training and simulation through to over-the-air (OTA) software deployment on the vehicle.
• Data versioning and lineage tracking are as important as code versioning — knowing which dataset version trained which model version is essential for safety audit trails.
• Infrastructure must be elastic enough to absorb spikes in compute demand during major training runs while remaining cost-controlled during quieter development phases.
• Security and access control must satisfy both automotive safety standards and cloud security best practices, given that vehicles are safety-critical systems connected to public networks.

The Broader Strategic Picture

Microsoft’s positioning in the AV space reflects a deliberate choice to compete as an infrastructure and toolchain provider rather than as a vehicle software platform owner. This mirrors how major cloud vendors have approached other capital-intensive industries — offering the picks-and-shovels rather than mining the ore directly.

For legacy automakers under pressure from both technology-native AV startups and platform companies, the appeal of a proven, scalable cloud toolchain is clear. The alternative — building and maintaining proprietary data infrastructure capable of handling petabyte-scale AV data — is a significant capital and engineering investment that does not directly contribute to vehicle differentiation.

Whether Azure, AWS, or Google Cloud emerges as the dominant AV cloud platform will depend on which ecosystem best addresses the unique constraints of automotive engineering: functional safety certification support, low-latency edge integration, and long-term platform stability commitments that match automotive product lifecycles.