SwiftStack is staking its claim, working collaboratively with our customers and helping solve scale challenges with their distributed AI/ML pipelines.


Over the last several months, SwiftStack has been busy helping two large autonomous vehicle customers.These data pipelines are distributed across edge (vehicle sensors) to core (data center) to cloud/multi-cloud locations, and are challenged with ingest, labeling, training, inferencing, and retaining data at scale. And when we say “scale,” we mean it: one deployment is handling more than a petabyte of data per week, with four thousand GPU cores from NVIDIA DGX-1 servers fed with 100 GB/s of throughput from SwiftStack cluster.

SwiftStack is uniquely positioned to handle these challenges and provides differentiated solution for each stage of the AI/ML workflow. Here’s how:

We put GPU cycles to work

Due to its massive parallelism, SwiftStack is able to ingest huge volumes of data accumulated by radar, LIDAR (Light Detection and Ranging), and computer vision sensors. SwiftStack is also able to handle massive neural net training throughput needed by several hundred GPU cores in NVIDIA DGX-1 servers. The containerized frameworks (Tensorflow, MXnet, and others) are  orchestrated by Kubernetes.

We bring cloud and AI to your data

SwiftStack’s 1space connector and our multi-cloud data management tools enable cloud bursting to and from AWS and GCP, when more economical GPU cycles are available in the cloud. 1space also enables frameworks like AWS Sagemaker and Kubeflow to be used in the cloud, while data is secured on-prem. SwiftStack versioning maintains the exact data version paired with a model training version.

We don’t drown you in your data

SwiftStack’s sophisticated labeling middleware can enrich the data during ingest, enabling quick metadata search and supervised learning workflows. SwiftStack Client provides the ability to search this dataset using the labels.

We let you focus on data first and not worry about infrastructure

Because SwiftStack ensures seamless universal access to our global namespace using file or object APIs, and near-infinite scale, the data scientist, Chief data officer, or Chief analytics officer can focus on business outcomes and time to value.

We don’t break the bank

SwiftStack has subscription based pricing, which allows you to start small and grow to PB scale. SwiftStack provides best economics for petabyte scale data pipelines.

We democratize AI/ML workflows

Our success in autonomous vehicles is only the beginning. Industries like healthcare and genomics, video analytics, and industrial IoT are equally challenged. SwiftStack is working with other best-of-breed partners in the AI/ML ecosystem to make it easier for these users to adopt multi-cloud workflows and frameworks, and scale to petabytes of storage and hundreds of gigabytes of bandwidth to meet their unique requirements.

Valohai is a Deep Learning platform as a service, that automates machine orchestration, version control and pipeline coordination for data science teams.  Valohai integrates with SwiftStack with a choice of S3 or Swift protocols and users can scale models to hundreds of CPUs or GPUs on-prem or in cloud, at the click of a button.

Deep Learning demo –  showcases a pipeline execution using MNIST database. The demo showcases 1) data transformation with batch feature extraction 2) model training with tensorflow in a container environment 3) hyper parameter optimization using several GPU instances and finally inferencing, all using SwiftStack data store for storage. SwiftStack Client provides easy access to the on premise bucket, SwiftStack versioning feature preserves the matching snapshot to a model.

GPL Technologies is an elite NVIDIA and SwiftStack Value added Reseller (VAR). GPL provides customers with the flexibility, technology leadership, and breakthrough economics to build tailored solutions to match their use cases.



About Author

Shailesh Manjrekar

Shailesh Manjrekar

Shailesh has deep experience in infrastructure across storage and networking (EMC, NetApp, HGST, Brocade). As a thought leader on the how AI/ML/DL impacts infrastructure, Shailesh has worked on changes needed in the datacenter, the edge and the cloud. Shailesh serves as the Head of Product and Solutions Marketing for SwiftStack.

Comments are closed.