A few weeks back, we announced SwiftStack 4.0 and one of the highlighted features was Metadata Search. For those of you who already have a SwiftStack cluster in production or are testing one out, you can use this feature now! This may be a bit confusing since the current version of SwiftStack is 3.12. Let me briefly explain…at SwiftStack we have a development and testing process that allows us to frequently release incremental pieces of functionality to best service our customers. So instead of holding back Metadata Search until all of the SwiftStack 4.0 features are done (within the next few months), we will release them independently as they are ready. So once all of the 4.0 features are complete, the version number will increment to 4.0. Hopefully that makes sense. Back to the topic of the day…
Since the first release of SwiftStack, you’ve been able to attach any amount of metadata to objects, but to utilize that metadata, your applications needed to track and manage it. While other applications can access those objects, they could not utilize the metadata unless they’d crawl through all of the objects to create their own index. This has resulted in more limited use of metadata than we’d like.
Our team evaluated multiple ways to add this capability to SwiftStack and since metadata is becoming prevalent across many different applications in enterprises, we decided to integrate with Elasticsearch and show how you can visualize your data with Kibana. Like SwiftStack, these solutions are powered by open source and backed by a commercial team who productized them. With the metadata being continuously indexed, applications and users with access to that data can now get significantly more value out of it by being able to quickly find what they need.
Here’s a quick explanation of how it works…when applications write objects into SwiftStack, any amount of metadata can be associated with the objects. That metadata is stored alongside the objects.
When configured, SwiftStack automatically sends metadata for each object to Elasticsearch, where it is indexed and becomes searchable. Every time metadata changes, objects or added, or objects are removed, SwiftStack updates Elasticsearch.
Applications or end users can then search for objects directly in Elasticsearch or use a visualization tool like Kibana. That will give them the knowledge to know exactly what objects to retrieve from the SwiftStack cluster.
This powerful piece of functionality allows you to store petabytes of information in a single namespace while allowing applications and users, that may even be globally distributed, to make best use of the valuable data.
To see a demonstration of how this works, check out this video that Joe Arnold and Trey Duskin put together.
At any point, if you’d like to learn more about how you can utilize metadata in your environment, please feel free to contact us.