SwiftStack Solutions | AI, Machine Learning

The Platform for Intelligent Data

The only proven storage and data management solution that can accelerate deep learning at ultra-scale.

SwiftStack AI Architecture

White Paper

The SwiftStack AI Architecture is essential for deep learning at ultra-scale. This customer-proven architecture stack satisfies the storage performance needed by GPU compute complexes and modern AI frameworks accessing and processing hundreds of petabytes of data.

Download White Paper

Infrastructure challenges are the primary inhibitor for broader adoption of AI/ML workflows. SwiftStack’s multi-cloud data management solution is the first of its kind in the industry and effectively handles storage I/O challenges faced by edge-to-core-to-cloud, large-scale AI/ML data pipelines.

Download Report

Amita Potnis

Research Director at IDC’s Infrastructure Systems Platforms and
Technologies Group

Featured Use Case:

Autonomous Vehicle Development

Working closely with NVIDIA, SwiftStack powers the data factories and Deep Neural Network (DNN) training of leaders in autonomous vehicle development.

Workflow

Survey cars generate terabytes of data per day, which is quickly transferred to SwiftStack for curation in the Data Factory.
Curated data from SwiftStack feeds thousands of GPUs for training, validation, and replay testing.
Real-world and simulated data is version-controlled and retained immutably for safety and traceability of DNN improvement over time.

Training a single complex Deep Neural Network (DNN) will require at least three million images, and refining it to be production-ready will take countless more. Vehicles aimed at SAE autonomy level 2+ will need at least 10 DNNs, and level 5 will require 20 or more. Refer to the table below to see basic storage capacity requirements for this training.

1 survey car, driving 8 hours per day 250 days per year	5 cameras 30 frames per second 2,000 hours	2 megapixels ( 2 MB/image ) 1 billion images	33% useful images after data factory cleansing
2,000 hours per year	1 billion images per year	2 PB raw data per year	300 million training images 660 TB training data

Architecture

The SwiftStack AI Architecture is an engineered and customer-proven edge-to-core-to-cloud storage and data management solution.
Core architectural capabilities like ultra-scale performance, distributed ﬂash-based cache, data immutability, standards-based APIs, and workﬂow integration make deep learning at scale possible.
SwiftStack ingests, retains, and provides data to thousands of GPUs working in parallel—”feeding the beast” to train and retrain the deep neural networks necessary for AV.

Helping the AI Team

Leveraging AI successfully is a company-wide effort. We believe SwiftStack’s technology and experience can help many of the team members involved.

Executives & Business Leaders know that data is valuable and want to use it for business insight or competitive advantage, but they are rightfully cautious about investing heavily into unproven technologies or strategies to do so.
How can SwiftStack help? SwiftStack is trusted to manage hundreds of petabytes in the world’s biggest companies—like Verizon, Blizzard, eBay, NVIDIA, Cisco, and AstraZeneca. In AI and Autonomous Vehicles in particular, NVIDIA chose SwiftStack as a core storage platform. And when it’s needed, our award-winning support ranks a perfect 5 out of 5 according to Gartner Peer Insights. If you are well down the path toward AI or new to it but already seeing some value from Big Data (e.g., Hadoop), Operational Intelligence (e.g., Splunk), or Business Intelligence, we would love to share some of our experience in Artificial Intelligence. We believe we offer a unique combination of performance, flexibility, and TCO that may be helpful to your business.

Executives & Business Leaders know that data is valuable and want to use it for business insight or competitive advantage, but they are rightfully cautious about investing heavily into unproven technologies or strategies to do so.

How can SwiftStack help? SwiftStack is trusted to manage hundreds of petabytes in the world’s biggest companies—like Verizon, Blizzard, eBay, NVIDIA, Cisco, and AstraZeneca. In AI and Autonomous Vehicles in particular, NVIDIA chose SwiftStack as a core storage platform. And when it’s needed, our award-winning support ranks a perfect 5 out of 5 according to Gartner Peer Insights. If you are well down the path toward AI or new to it but already seeing some value from Big Data (e.g., Hadoop), Operational Intelligence (e.g., Splunk), or Business Intelligence, we would love to share some of our experience in Artificial Intelligence. We believe we offer a unique combination of performance, flexibility, and TCO that may be helpful to your business.

Data Scientists & Engineers face the challenge of capturing, creating, and/or using piles of raw data to generate something of value—maybe some kind of business insight or threat detection, an intelligent prediction algorithm, or even complex DNNs for self-driving cars, but progress slows when tools and infrastructure can’t keep up.
How can SwiftStack help? Only SwiftStack’s shared-nothing architecture was designed from its inception to scale linearly to infinite levels of throughput, concurrency, and capacity, which means only SwiftStack can ingest, retain, and feed training data fast enough to fully utilize thousands of parallel GPUs working to produce today’s most intelligent DNNs and ML algorithms. If you are using GPUs or pressing for an investment in them, you also need storage that can feed data fast enough to utilize them efficiently and can retain your training data immutably for replay and regression testing down the road. Starting with flash or the public cloud is reasonable, but both will prove to be overly expensive at scale and will eat into your budget for additional development tools and compute horsepower. We believe we offer a unique combination of performance, flexibility, and TCO that will be a perfect fit for your AI development.

Data Scientists & Engineers face the challenge of capturing, creating, and/or using piles of raw data to generate something of value—maybe some kind of business insight or threat detection, an intelligent prediction algorithm, or even complex DNNs for self-driving cars, but progress slows when tools and infrastructure can’t keep up.

How can SwiftStack help? Only SwiftStack’s shared-nothing architecture was designed from its inception to scale linearly to infinite levels of throughput, concurrency, and capacity, which means only SwiftStack can ingest, retain, and feed training data fast enough to fully utilize thousands of parallel GPUs working to produce today’s most intelligent DNNs and ML algorithms. If you are using GPUs or pressing for an investment in them, you also need storage that can feed data fast enough to utilize them efficiently and can retain your training data immutably for replay and regression testing down the road. Starting with flash or the public cloud is reasonable, but both will prove to be overly expensive at scale and will eat into your budget for additional development tools and compute horsepower. We believe we offer a unique combination of performance, flexibility, and TCO that will be a perfect fit for your AI development.

IT & Storage Architects & Admins face the challenge of providing the right tools to enable Data Science teams. They have to balance the cost, complexity, and flexibility of on-prem and cloud investments in storage, networking, and compute—both CPU and GPU, and they are rightfully careful to minimize risk in adopting new technologies.
How can SwiftStack help? Compared to incumbent technologies and alternatives, only SwiftStack offers maximum performance with an optimal TCO. SwiftStack is trusted to manage hundreds of petabytes in the world’s biggest companies—like Verizon, Blizzard, eBay, NVIDIA, Cisco, and AstraZeneca. And when it’s needed, our award-winning support ranks a perfect 5 out of 5 according to Gartner Peer Insights. Only SwiftStack’s shared-nothing architecture was designed from its inception to scale linearly to infinite levels of throughput, concurrency, and capacity, which means only SwiftStack can ingest, retain, and feed training data fast enough to fully utilize thousands of parallel GPUs working to produce today’s most intelligent DNNs and ML algorithms. Starting with flash or the public cloud is reasonable, but both will prove to be overly expensive and cumbersome to manage at scale, which will eat into your budget for additional development tools and compute horsepower. We believe we offer a unique combination of performance, flexibility, and TCO that will enable your data science teams today and scale with you tomorrow.

IT & Storage Architects & Admins face the challenge of providing the right tools to enable Data Science teams. They have to balance the cost, complexity, and flexibility of on-prem and cloud investments in storage, networking, and compute—both CPU and GPU, and they are rightfully careful to minimize risk in adopting new technologies.

How can SwiftStack help? Compared to incumbent technologies and alternatives, only SwiftStack offers maximum performance with an optimal TCO. SwiftStack is trusted to manage hundreds of petabytes in the world’s biggest companies—like Verizon, Blizzard, eBay, NVIDIA, Cisco, and AstraZeneca. And when it’s needed, our award-winning support ranks a perfect 5 out of 5 according to Gartner Peer Insights. Only SwiftStack’s shared-nothing architecture was designed from its inception to scale linearly to infinite levels of throughput, concurrency, and capacity, which means only SwiftStack can ingest, retain, and feed training data fast enough to fully utilize thousands of parallel GPUs working to produce today’s most intelligent DNNs and ML algorithms. Starting with flash or the public cloud is reasonable, but both will prove to be overly expensive and cumbersome to manage at scale, which will eat into your budget for additional development tools and compute horsepower. We believe we offer a unique combination of performance, flexibility, and TCO that will enable your data science teams today and scale with you tomorrow.

Solving Real-World Challenges

AI Data Challenges

Speed of Initial Data Ingest

A lot of data from the survey cars has to be moved and stored quickly.

Cost, Performance, and Scale in DNN Development

The “working set” for training data measures into the 10s of PB—which makes public-cloud egress costs prohibitive.

Thousands of GPUs need to be fed subsets of that data simultaneously to be leveraged effectively—which exceeds what alternative storage architectures can deliver.

Long-Term Data Retention, Traceability, and Immutability

Data captured by survey cars must be retained, organized, labeled, and accessible indefinitely to regression-test DNNs as they are developed, which directly impacts safety and confidence in the algorithms.

Infrastructure Complexity

Silos of storage are growing independently on edge devices, in core data centers using traditional flash and NAS systems, and in clouds as well, which makes it difficult for data scientists and GPUs to access it and IT teams to manage it.

SwiftStack Provides

Ultra-Scale Performance

Only SwiftStack’s shared-nothing distributed architecture was designed from its inception to scale linearly to infinite levels of throughput, concurrency, and capacity, which means only SwiftStack can ingest, retain, and feed training data fast enough to fully utilize thousands of parallel GPUs working to produce today’s most intelligent DNNs and ML algorithms.

Elasticity from Edge-to-Core-to-Cloud

Only SwiftStack’s 1space makes it possible for applications to ingest and access data anywhere—on edge devices, in core data centers, and/or in public cloud infrastructure. Policy-based data movement and the ability to merge data namespaces enables applications to move and scale beyond one cloud or one set of on-premises infrastructure, so they can be deployed with ultimate flexibility, and training isn’t slowed by relocating data.

Data Immutability

The unique combination of Versioning, Access Control, and SwiftStack’s Delete Protection means that data can be retained and referenced indefinitely as it was originally written—enabling traceability, accountability, confidence, and safety throughout the life of a DNN.

Optimal TCO

When compared to the cost of frequently accessed public cloud storage or PBs of on-premises flash, the TCO of SwiftStack software deployed on industry-standard hardware offers compelling savings—approximately ⅓ the cost of public cloud storage and 90% less than flash—which can be quickly invested in advancing top-level AI initiatives!

Real-World Confidence

In addition to dozens of large-scale customers each storing tens of petabytes on SwiftStack for other applications, notable AI deployments for autonomous vehicle development include (1) an industry-leading robo-taxi company seamlessly scaling DNN training and storage across Amazon, Google, and on-premises infrastructure, (2) NVIDIA’s internal at-scale reference architecture for AV development—with proven throughput over 100 GBytes/sec from SwiftStack to GPUs, and (3) a Big-3 automaker developing DNNs for their future vehicles. In every case, SwiftStack “just works”—it deploys easily on common servers, scales linearly and infinitely, supports the industry-standard S3 API, and seamlessly merges on-premises and public-cloud storage to enable GPUs in any location.

Choosing the Right Storage Strategy

Many AI strategies start with public cloud resources or a few on-premises GPUs and some flash storage, but—as you prepare to scale—it pays to consider what storage strategy will optimize performance, cost, and flexibility.

We have about a PB of data—more than a PB of data—that is coming in per month and about 15 PB that is curated. For any large-scale company, this is not a huge amount of data, but what makes it tricky and hard is that all this 15 PB is the active training data set, so all of the internal teams are accessing it all of the time every day; every model that you’re working on is going to be using that data.

Senior Engineer, AI Infrastructure

Watch Video Presentation

Reference Stack: Storage for Deep Learning Development

Whether you are focused on autonomous vehicles or some other application of deep learning, the challenges are familiar: You need to create or capture huge amounts of training data, label it, retain it indefinitely, and use it to feed hundreds or thousands of GPUs. SwiftStack’s reference stack for AI is an industry-proven storage architecture for deep learning.

Many of the leading AI frameworks and toolsets—including TensorFlow—can now leverage public- and private-cloud data directly with the S3 API.

DNN training is accelerated with NVIDIA’s industry-leading GPUs in NVIDIA-branded DGXs or OEM systems like the Cisco C480ML or Dell DSS 8440.

SwiftStack software deploys easily on industry-standard servers from partners like Cisco, Dell, HPE, and Supermicro.

NVIDIA Preferred Storage Solution Advisor Partner

SwiftStack is proud to be an esteemed storage solution advisor partner in the NVIDIA Partner Network.

SwiftStack for AI, ML and DL

StorageSwiss

Briefing Note

Learn More >

SwiftStack Announces World’s First Multi-Cloud AI/ML Data Management Solution

Press Release

Learn More >

SwiftStack adds a disk object-based AI reference architecture

TechTarget

News

Learn More >

SwiftStack’s Vision

SwiftStack’s assumption is that every organization is moving to a multi-cloud reality. We have a vision that data should be easily managed across clouds and made available to applications regardless of where they are executing. This aligns perfectly with the trends we see driving the advancement of AI workflows in the industry.

Container- and microservice-based applications are changing the landscape, as they are deploying transparently using on-premises GPU resources and/or public cloud GPU farms. Meanwhile, the scale of capacity and throughput required to fully utilize these GPU resources is growing well beyond what traditional architectures can provide. The answer can only be a cloud-native data-management and storage architecture designed to scale infinitely and span both edge and on-premises infrastructure and multiple public clouds.

Sincerely,