SHARE   

Swift developers work for different companies and live all over the world. Every so often, we get together for a few days of design, coding, and fun. Last week IBM hosted our hackathon in Austin, Texas. We had 27 people from 9 different companies (and 5 countries) attend.

cookies

We’re contributors to a common project, but in-person events help us better understand and trust one another. When we understand each other, we’re more effective collaborating when we aren’t together in the same room, and we begin to look for ways to make each other successful. And as a bonus, we develop friendships that extend beyond the code.

What we covered this week

But let’s not forget why we are meeting in the first place: the code. This past week in Austin was very productive. There is a *lot* going on in Swift these days–more than any one person or company can do alone. This fact became very obvious at the Vancouver summit. We kicked off this hackathon talking about all the things that are going on and our need to focus.

topics_joined

There’s more stuff on the list than there are people at the hackathon, so we’ve got to prioritize. I’ve proposed thinking about things in one of three ways:

  • Stuff that we can get done in the near-term (by the Tokyo summit)
  • Stuff that we can make measurable progress on in the near-term
  • Stuff that we can talk about but won’t be done soon (discuss over beers)

So with that being said, what were the “big things” we covered this past week?

How to focus and get patches landed faster

Our community is growing, and since we’ve got so much going on, we took a good look at how we can improve our track record of reviewing and accepting proposed changes in Swift. It’s not something we can change overnight, but we’ll be looking into improving some of our tools (like gerrit code review), building new tools to help prioritize activity, and better communication from patch authors and reviewers about the state of things.

Encryption

We’re currently working on providing data-at-rest encryption for data stored in Swift. This gives deployers peace of mind and protects against data breaches if hard drives leave the data center. We’ve made good progress on this feature so far, but there are still quite a few big things to finish before we can call the feature done. This is something that we’re able to make measurable progress on before the Tokyo summit, but we likely won’t have complete by then.

Erasure codes “done”

Swift’s erasure code support has been out in “beta” form for a few months now. Many companies have been rigorously testing it, and we’ve found and fixed several issues. We’re currently working on closing the last known open issues for this, and I expect we’ll be able to recommend erasure codes for production use by the Tokyo summit.

Global cluster improvements

Many deployers use Swift’s global cluster support to provide geographic durability and locality of access. In a storage system, the singular challenge with geographically disperse data is around data movement. Many people in the Swift community are improving the code to reduce the amount of data that is sent between different regions, and these improvements make global clusters better for everyone.

What do do with full clusters?

We spent some time last week identifying and writing down some best practices for what to do if a Swift cluster fills up. We’ll be creating some better official documentation about this, and we’ve got some great ideas on features to help deployers out in these situations.

Performance improvements

We’re always interested in improving performance in Swift clusters, and there are a few topics related to this that came up in Austin. We’ve got a patch under review that should lower latency variability for read operations, and another that improves the speed of background consistency processes

matt_sharding

Container sharding

We’re working on a feature in Swift to transparently shard containers so that end-users don’t have to worry about how many objects are stored in just one container. Today, we recommend that users spread their data across many containers to ensure that they get the best performance from the system. We’re looking at ways to handling this internally so that application developers don’t have to worry about it.

The Golang object server work

In an effort to improve performance of some background processes, the Swift community has a branch where some parts of the system have been reimplemented in the Go programming language. This work is proceding well, and it can show some very impressive initial performance improvements. This is a long-term project, and I’m excited to see the dieas that are coming out of it to improve Swift.

people

Where do we go from here?

Looking forward to the next few months, we’re focusing on closing all known issues with erasure code support, improving performance in a few key areas (eg object write latency and expiring objects), and improving some operator tools.

Beyond that, I’m excited about the ongoing work towards at-rest encryption (we’ll be finishing it up as quickly as possible) and the container sharding work.

I love the Swift hackathons because we make measurable progress on the code and we deepen relationships with one another. To all of you contributing to Swift today: thank you for working on Swift. To those not yet contributing, join us!

About Author

John Dickinson

John Dickinson

John joined SwiftStack as Director of Technology coming from Rackspace where he worked on Rackspace Cloud Files and OpenStack Swift, the large-scale distributed object storage system which SwiftStack is built on. As the Project Technical Lead (PTL) for OpenStack Swift, John is also leading much of its community development efforts.