Horizontal scaling in/out techniques for redis cluster

7 min readJul 5, 2019

This story is in continuation of my previous story “Creating a Redis Cluster”. If you are still not comfortable creating a redis cluster, i suggest you go and read about it here before diving into this story.

Understanding scaling

Scalability refers to the idea of a system in which every application or piece of infrastructure can be expanded to handle increased load. There are two ways to scale a system as follows-

Vertical scaling

Addition of resources to an existing system to achieve desired level of performance is called vertical scaling. For example adding more RAM or compute power to a system. This method has limitations as resources in one single system can be increased only up to a certain limit. At some point, you won’t be able to scale it further vertically See image below

Horizontal scaling

Adding new nodes of comparatively lower performance together to form a cluster or removing nodes from that cluster is called horizontal scaling. It enables applications to avoid having a SPF (single point of failure) and in theory can be scaled out infinitely.

In this story i will be covering techniques to horizontally scale a redis cluster.

scale in (removing machines from the cluster) and scale out (linking together machines to collectively do the work of a much more advanced one) a redis cluster. You should not confuse scale out— Horizontal scaling technique with scale up — Vertical scaling technique

Now that you understand scaling well, let’s dive deep into how to scale out or scale in a redis cluster.

Adding new nodes to existing cluster or Scale out

While adding a new node to an existing cluster, a new node can be added as a master node or a slave node (In redis, slave nodes are generally called as replicas as they hold the replicated slots that their masters have).

Adding a new Master node

The first step to perform is adding an empty node.

This is as simple as to start a new node in port 6007 (we already used from 6001 to 6006 for our existing 6 nodes. Read this) with the same configuration used for the other nodes, except for the port number, so in order to conform with the setup we used for the previous nodes we create a redis.conf file, similar to the one used for the other nodes but using 6007 as port number then finally start the server with following command

../redis-server ./redis.conf

At this point the server should be running.

Now we can use redis-cli as usual in order to add the node to the existing cluster.

./redis-cli --cluster add-node 127.0.0.1:6007 127.0.0.1:6001

As you can see I used the add-node command specifying the address of the new node as first argument, and the address of a random existing node in the cluster as second argument. See image below as output of above command

A new master node has been added to the cluster and you check the same by executing following command-

./redis-cli — cluster-check 127.0.0.1:6001

Following is the output of executing above command.

If you notice in above image, a new master node is already connected to the cluster it is already able to redirect client queries correctly and is generally speaking part of the cluster. However It holds no data as it has no assigned hash slots(0 slots). Now it is possible to assign hash slots to this node using the resharding feature of redis-cli.

Let’s see how we can re-shard the cluster.

Resharding the cluster

Resharding basically means to move hash slots from a set of nodes to another set of nodes, and like cluster creation it is accomplished using the redis-cli utility. To start a resharding just type:

./redis-cli --cluster reshard 127.0.0.1:6001

You only need to specify a single node, redis-cli will find the other nodes automatically.

Currently redis-cli is only able to reshard with the administrator support, you can’t just say move 5% of slots from this node to the other one (but this is pretty trivial to implement). So it starts with questions. The first is how much a big resharding do you want to do:

How many slots do you want to move (from 1 to 16384)?

We can try to reshard 1000 hash slots, that should already contain a non trivial amount of keys

Then redis-cli needs to know what is the target of the resharding, that is, the node that will receive the hash slots. I’ll use the first master node, that is, 127.0.0.1:6001, but I need to specify the Node ID of the instance. This was already printed in a list by redis-cli, but I can always find the ID of a node with the following command if I need:

./redis-cli -p 6001 cluster nodes | grep myself

Now you’ll get asked from what nodes you want to take those keys. I’ll just type all in order to take a bit of hash slots from all the other master nodes. After this, resharding will complete. At the end of the resharding, you can test the health of the cluster with the following command:

./redis-cli --cluster check 127.0.0.1:6001

See below the output of above command after resharding. You can clearly see the node that had 0 slots before resharding now has 1000 nodes.

At this point we know how to add a new master node in redis cluster and how to rehshard it to assign it some slots in order to allow this newly added master node to store some keys

Adding a new replica node

Adding a new Replica can be performed in two ways. The obvious one is to use redis-cli again, but with the — cluster-slave option, like this:

./redis-cli --cluster add-node 127.0.0.1:6007 127.0.0.1:6001 --cluster-slave

Note that the command line here is exactly like the one we used to add a new master, so we are not specifying to which master we want to add the replica. In this case what happens is that redis-cli will add the new node as replica of a random master among the masters with less replicas.

However you can specify exactly what master you want to target with your new replica with the following command line:

./redis-cli --cluster add-node 127.0.0.1:6007 127.0.0.1:6001 --cluster-slave --cluster-master-id 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

This way we assign the new replica to a specific master.

A more manual way to add a replica to a specific master is to add the new node as an empty master, and then turn it into a replica using the CLUSTER REPLICATE command. This also works if the node was added as a slave but you want to move it as a replica of a different master.

For example in order to add a replica for the node 127.0.0.1:6006 that is currently serving hash slots in the range 11423–16383, that has a Node ID 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e, all I need to do is to connect with the new node (already added as empty master) and send the command:

./redis 127.0.0.1:6006> cluster replicate 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e

That’s it. Now we have a new replica for this set of hash slots, and all the other nodes in the cluster already know (after a few seconds needed to update their config). We can verify with the following command:

./redis-cli -p 7000 cluster nodes | grep slave | grep 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e
f093c80dde814da99c5cf72a7dd01590792b783b 127.0.0.1:6006 slave 3c3a0c74aae0b56170ccb03a76b60cfe7dc1912e 0 1385543617702 3 connected

Removing nodes from existing cluster or Scale In

To remove a slave node just use the del-node command of redis-cli:

./redis-cli --cluster del-node 127.0.0.1:6001 `<node-id>`

The first argument is just a random node in the cluster, the second argument is the ID of the node you want to remove.

You can remove a master node in the same way as well, however in order to remove a master node it must be empty. If the master is not empty you need to reshard data away from it to all the other master nodes before.

Reference — https://redis.io/topics/cluster-tutorial

What’s next?

So now that you know how to create a redis cluster and how to horizontally scale it in/out, you should know techniques to access this redis cluster programmatically from your application. Know how to connect to redis cluster using a java/python clients.