A few weeks ago, my colleague Jon Kelly posted a blog article titled “How I Used pico swiftstack to Test TensorFlow with S3 Storage”. Since pico swiftstack is a relatively new product, I would like to explain why having a containerized S3-compatible object storage option for testing and experimenting is helpful to anyone evaluating scale-out, software-driven storage.
SwiftStack is a highly-scalable distributed data storage and management platform, but before you use deploy it into production, you probably want to run it through some motions. Will it support all the functionality your application needs? How are the API responses structured, for you to parse? Can it spin in mid air with a torch between its teeth? Well, you must test that. But setting up the full platform to check application or workflow compatibility, even based on virtual machines, can add additional time and consume more physical resources than necessary. So, how can we make it easier?
Simple Option for Locally Testing S3 API Compatibility
One option we have had for a long time was for you to test against an instance of SwiftStack that we host. When requested, we can provision API endpoints for you or your developers to test against without having to set up any infrastructure. However, different developers have different needs, and having a local containerized instance, with no need to communicate over the Internet, is another option to test for object storage API (S3 and Swift) compatibility.
Containers have been around for a while, long before Docker came around and made them usable by a broad audience. Containers isolate applications and resources. They create a native confined space on a host, which looks and feels like a real server, and allow you to control their personality and the level of interaction they have with the real world. For example, you can create an Ubuntu container on a Red Hat server, choose if they get their own network stack or use the host’s, and much more.
So, we created a container-based SwiftStack and call it “pico swiftstack”. It is not meant to be run as a full storage system, just as an endpoint for developers to quickly instantiate the access layer of the SwiftStack platform locally and run a test suite against it.
Architecturally, pico swiftstack is running on Alpine Linux, which has been marked as the best OS to use for containers, and is a personal favorite of mine (just as a data point, moving pico swiftstack from an Ubuntu image to an Alpine image cut its size down by more than 50%). We use S6 process scheduler (via the excellent S6-overlay).
It runs all the SwiftStack object storage services along with SwiftStack Auth, and exposes both the S3 or Swift API for you to run any of the PUT, GET, POST, DELETE and other verbs. You can also connect to it with our SwiftStack Client, or any other S3 or Swift-compatible tool.
Our intention is to keep pico swiftstack on par with our SwiftStack platform release cycle, so that every release we make will have its own
swiftstack/picoswiftstack:<release> for it. So,
swiftstack/picoswiftstack:220.127.116.11 will use the same components as the full SwiftStack v18.104.22.168 product uses.
swiftstack/picoswiftstack:latest will also point to our latest release. So, if you want to compare or troubleshoot something from the past, you can run specific SwiftStack version, and if not – just run latest.
You can find pico swiftstack, along with how-to instructions, in our Test Drive page. We are continuously adding functionality to pico swiftstack, and if you have any feedback on its capabilities, please let us know at email@example.com.