Custom Cassandra Comparators Using Protocol Buffers

At BackType we serve a lot of our analytics data from Cassandra. We love the write performance, flexible schema, horizontal scalability and range queries that it allows.

The out of the box ordering implementations (ASCII, UTF-8, Long, and UUID) get you a long way, but to do a complex column type, such as one made up of a timestamp and an auto-increment ID, you're on your own.

The bad way

The hacky solution to create a custom column would be to pack two ints into a LongType:
struct.pack('!II', int1, int2)

This is a really bad idea. We're just smashing the ints together and there's nothing to protect us against careless mistakes. Also, by packing two ints into the LongType we've achieved the custom sorting we sought but in an inflexible and non obvious way.

The good way

A proper solution should do a few things:

1. Work for structured data and throw an error when we make a mistake
2. Give us direct control over how the sorting is implemented
3. Be language and platform neutral

We decided to go with Google Protocol Buffers:

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data ? think XML, but smaller, faster, and simpler. You define how you want your data to be structured once, then you can use special generated source code to easily write and read your structured data to and from a variety of data streams and using a variety of languages ? Java, C++, or Python.

That they are small, fast, simple and can be used in Python and Java make them perfect for reading and writing our custom columns. We simply define a "message" that represents our structure and then serialize and deserialize it when generating and parsing column names.

Here's an example .proto file for our new column type:
message MyCustomColumn { � required uint64 timestamp = 1; � required uint64 some_id = 2; }

Next we generate the Python and Java libraries and write a quick custom comparator for Cassandra to use when inserting into our column family:
package backtype.cassandra; import backtype.cassandraext.Cassandraext; import org.apache.cassandra.db.marshal.AbstractType; import com.google.protobuf.InvalidProtocolBufferException; public class MyCustomComparator extends AbstractType { ��� @Override ��� public String getString(byte[] bytes) { ������� if (bytes.length == 0) return ""; ������� try { ����������� Cassandraext.MyCustomColumn c = Cassandraext.MyCustomColumn.parseFrom(bytes); ����������� return c.toString(); ������� } catch (com.google.protobuf.InvalidProtocolBufferException e) { ����������� throw new RuntimeException(e); ������� } ��� } ��� ��� public int compare(byte[] o1, byte[] o2) { ������� if (o1.length == 0) { ����������� return o2.length == 0 ? 0 : -1; ������� } ������� if (o2.length == 0) { ����������� return 1; ������� } ������� try { ����������� Cassandraext.MyCustomColumn c1 = Cassandraext.MyCustomColumn.parseFrom(o1); ����������� Cassandraext.MyCustomColumn c2 = Cassandraext.MyCustomColumn.parseFrom(o2); ����������� if (c1.getTimestamp() < c2.getTimestamp()) { ��������������� return -1; ����������� } else if (c1.getTimestamp() > c2.getTimestamp()) { ��������������� return 1; ����������� } else { ��������������� if (c1.getSomeId() < c2.getSomeId()) { ������������������� return -1; ��������������� } else if (c1.getSomeId() > c2.getSomeId()) { ������������������� return 1; ��������������� } else { ������������������� return 0; ��������������� } ����������� } ������� } catch (com.google.protobuf.InvalidProtocolBufferException e) { ����������� throw new RuntimeException("Could not parse protobuf object", e); ������� } ��� } }

Finally we add our new column to the schema and restart our nodes with the new jars for our comparator and protobuf in cluded in the lib directory:
<ColumnFamily Name="MyColumn" CompareWith="backtype.cassandra.MyColumnComparator"/>

What about Thrift?

We've used Thrift to do this before ? it works in the exact same way. Unfortunately, Cassandra is built against an older version of Thrift and if you want things to work you have to make sure you're using the exact same one.

Permalink | Leave a comment  »

Source: http://tech.backtype.com/custom-cassandra-comparators-using-protocol-b

computer repair kansas city area computer repair kansas city waldo computer repair kansas city computer repair kansas city wornall