Redis Streams is a new data type in the Redis 5.0, append-log like data structure. Detailed information can be found in the offical web link and antirez’s blog post as well.

Motivation

In our project, we use Redis Streams as lightweight streaming support service in our backend. The reason why we select Redis Streams is that:

• Lightweight. Comparing with Kafka, etc,
• Memory efficency.
• Works both on Cloud and Edge Computing.
• Reuse with our existed Redis experience.

Comparing with “AMQP”-like broker, such as RabbitMQ,etc, those kind of brokers has a important feature that brokers can route the message with some predefined rules to the right ‘channel’.

In append-log like streaming service, this needs to be done by the Consumer, that means even you only care about some subset on this ‘topic’, you need to accept that and make another filter with that.

You can create a new ‘topic’ in Kafka or ‘Stream’ in Redis Streams to differ them into another catalog. Most time this solution will be the right choice, but in our case, it has some limitations.

In our project, we maintain vendor-specific OID to represent data structure internally, such like this example in the wiki as example:

1.3.6.1.4.1.343

corresponds to the following path through the OID tree:

1 ISO
1.3 identified-organization,
1.3.6 dod,
1.3.6.1 internet,
1.3.6.1.4 private,
1.3.6.1.4.1 IANA enterprise numbers,
1.3.6.1.4.1.343 Intel Corporation
1.3.6.1.4.1.343.1 CPUs

You can define your own database on this tree with agreement by IANA. (just broadcast some leaf node will not be taken by other vendors).

So, if you only care about some subtrees than others, for example, only ‘CPUs’ subtree was interested on me, etc.

In the real world, you can find other examples, as IoT home-appliance world, sensors broad the temperature into backend services,

/building/1/floor/3/room/4  temp 20
/building/2/floor/5/room/2  temp 30

you might think we should choose kind of ‘RabbitMQ’ routing-capability as the message distribution broker, yes, you are right, but as the reasons mentioned in the beginning of this post, we decided to use Redis Streams in this time.

Routing capability

Topic exchange in RabbitMQ

Topic exchange is one of the most important feature in the RabbitMQ, it use such patterns to filter topic-message.

# (star) can substitute for exactly one word.
* (hash) can substitute for zero or more words.

take examples above:

/building/1/*        // filter all the sensors in building-1
*/floor/3/*          // filter all the sensors in floor-3

So, I don’t want to invent another semantic on this pattern, just copy this semantic on my solution.

Topic filter in Redis Streams

We expected:

• All the topic-message in Redis Steams should have the first key-value with the following pattern.

    'routing_key':'/building/1/floor/3/room/4' temp 20 ....

routing_key is the keyword of the key in the first key-value pair, all the message with filtered should have this first pair.

• Add ‘FILTER’ keyword in the RESP to request filter rule.

> XREAD FILTER '*/floor/3/*' STREAMS mystream 0
> XREAD FILTER '*/floor/3/*' BLOCK 0 STREAMS mystream $

Implementation

The source code was here.

Demo:

SQL-like filter engine?

If blocking read is not needed on your case, using redis gears is recommended solution. redis gears is a lambda function which you can run it in the redis, in your lambda function you can select more flexible match rules as your want.

How about if we want to filter the specified k-v in the Redis Steams?

In Kafka, it use SQL like kSql to filter/extract message in the streams, Should we consider use the similar idea in Redis Steams?

Redis Gears is the recommended way to do such jobs. We can make a dedicated gears recipes which can accept ‘sql’-like queries as a standard recipes. Actually I am working on this now.

127.0.0.1:30001> RG.PYEXECUTE "GB().run(rules="SELECT * FROM mysteam WHERE routing_key LIKE '*/floor/3/*'")"

Hope enjoying this post :-)