Redis Manual

IMPORTANT:
Some of the content here is a personal summary/abbreviation of contents on the Offical Redis Guide. Feel free to refer to the official site if you think some of the sections written here are not clear.

Redis Intro

Redis is an in-memory remote database that offers high performance, replication, and a unique data model to produce a platform for solving problems. By supporting five different types of data structures (Strings, Lists, Sets, Hashes, and Sorted sets), Redis accommodates a wide variety of problems that can be naturally mapped into what Redis offers.

To be specific, Redis is a very fast non-relational database that stores a mapping of keys to five different types of values. Redis supports in-memory persistent storage on disk, replication to scale read performance, and client-side sharding1 to scale write performance.

non-relational
- In Redis, there are no tables, and there’s no database-defined or -enforced way of relating data in Redis with other data in Redis. Generally speaking, many Redis users will choose to store data in Redis only when the performance or functionality of Redis is necessary, using other relational or non-relational data storage for data where slower performance is acceptable, or where data is too large to fit in memory economically.

Data Structures in Redis

Structure type	What it contains	Structure read/write ability
`STRING`	Strings, integers, or floating-point values	Operate on the whole string, parts, increment/ decrement the integers and floats
`LIST`	Linked list of strings	Push or pop items from both ends, trim based on offsets, read individual or multiple items, find or remove items by value
`SET`	Unordered collection of unique strings	Add, fetch, or remove individual items, check membership, intersect, union, difference, fetch random items
`HASH`	Unordered hash table of keys to values	Add, fetch, or remove individual items, fetch the whole hash
`ZSET` (sorted set)	Ordered mapping of string members to floating-point scores, ordered by score	Add, fetch, or remove individual values, fetch items based on score ranges or member value

In the following sections, we will use the redis cli to pratice using redis commands and data structures.

`String`s in Redis - Basics

The operations available to STRINGs start with what’s available in other key-value stores. We can GETvalues, SET values, and DEL values. After you have installed and tested Redis, within redis-cli you can try to SET, GET, and DEL values in Redis. The basic meanings are shown below in the table:

Command	What it does
`GET`	Fetches the value stored at the given key
`SET`	Sets the value stored at the given key
`DEL`	Deletes the key-value pair stored at the given key (works for all types)

Example demonstrating each of the commands:

127.0.0.1:6379> set hello value
OK
127.0.0.1:6379> get hello
"value"
127.0.0.1:6379> del hello
(integer) 1
127.0.0.1:6379> get hello
(nil)

Note:

You see outputs such as (integer) 1, which usually represents execution success/true. (nil) represents execution failure, and (integer) 0 also means failure but to the extent where the output means false, for example when a value (rather than the key) DNE.

`List`s in Redis - Basics

The operations that can be performed on LISTs are typical of what we find in LinkedList data structures in almost any programming language. We can push items to the front and the back of the LIST with LPUSH/RPUSH; we can pop items from the front and back of the list with LPOP/RPOP; we can fetch an item at a given position with LINDEX; and we can fetch a range of items with LRANGE.

Command	What it does
`RPUSH`	Pushes the value onto the right end of the list. If list DNE, creates a new list
`LRANGE`	Fetches a range of values from the list
`LINDEX`	Fetches an item at a given position in the list (begins at index `0`)
`LPOP`	Pops the value from the left end of the list and returns it

For example:

127.0.0.1:6379> rpush test-list item1
(integer) 1
127.0.0.1:6379> rpush test-list item2
(integer) 2
127.0.0.1:6379> rpush test-list item1
(integer) 3
127.0.0.1:6379> lrange test-list 0 -1
1) "item1"
2) "item2"
3) "item1"
127.0.0.1:6379> lindex test-list 1
"item2"
127.0.0.1:6379> lpop test-list
"item1"
127.0.0.1:6379> lrange test-list 0 -1
1) "item2"
2) "item1"

Note:

See that duplicates in values are allowed.

`Set`s in Redis - Basics

In Redis, SETs are similar to LISTs in that they’re a sequence of strings, but unlike LISTs, Redis SETs use a hash table to keep all values/strings unique.

Because Redis SETs are unordered, we can’t push and pop items from the ends like we did with LISTs. Instead, we add and remove items by value with the SADD and SREM commands. We can also find out whether an item is in the SET quickly with SISMEMBER, or fetch the entire set with SMEMBERS (this can be slow for large SETs, so be careful)

Comamnd	What it does
`SADD`	Adds the item to the set
`SMEMBERS`	Returns the entire set of items
`SISMEMBER`	Checks if an item is in the set
`SREM`	Removes the item from the set, if it exists

For example:

127.0.0.1:6379> sadd test-set item1
(integer) 1
127.0.0.1:6379> sadd test-set item2
(integer) 1
127.0.0.1:6379> sadd test-set item1
(integer) 0
127.0.0.1:6379> smembers test-set
1) "item2"
2) "item1"
127.0.0.1:6379> sismember test-set item1
(integer) 1
127.0.0.1:6379> sismember test-set item3
(integer) 0
127.0.0.1:6379> srem test-set item1
(integer) 1
127.0.0.1:6379> srem test-set item1
(integer) 0
127.0.0.1:6379> smembers test-set
1) "item2"

`Hash`es in Redis - Basics

Whereas LISTs and SETs in Redis hold sequences of items, Redis HASHes store a mapping of (distinct) keys to values. The values that can be stored in HASHes are the same as what can be stored as normal STRINGs: strings themselves, or if a value can be interpreted as a number, that value can be incremented or decremented.

In a lot of ways, we can think of HASHes in Redis as miniature versions of Redis itself.

Command	What it does
`HSET`	Creats a hashtable, and stores the key-value pair in tha hashtable
`HGET`	Fetches the value at the given hash key
`HGETALL`	Fetches the entire hashtable
`HDEL`	Removes a key-value pair from the hash, if it exists

For example:

127.0.0.1:6379> hset test-hasht key1 value1
(integer) 1
127.0.0.1:6379> hset test-hasht key2 value2
(integer) 1
127.0.0.1:6379> hset test-hasht key1 value2
(integer) 0
127.0.0.1:6379> hgetall test-hasht
1) "key1"
2) "value2"
3) "key2"
4) "value2"
127.0.0.1:6379> hget test-hasht key1
"value2"
127.0.0.1:6379> hdel test-hasht key2
(integer) 1
127.0.0.1:6379> hgetall test-hasht
1) "key1"
2) "value2"

`Sorted set`s/`Zset`s in Redis - Basics

Like Redis HASHes, ZSETs also hold pairs of key and value. The keys (called members) are unique, and the values (called scores) are limited to floating-point numbers. ZSETs have the unique property in Redis of being able to be accessed by member (like a HASH), but items can also be accessed by the sorted order and values of the scores.

Command	What it does
`ZADD`	Adds member/key with the given score to the ZSET. The order of input is `score` and then `member`.
`ZRANGE`	Fetches the items in the ZSET from their positions in sorted order
`ZRANGEBYSCORE`	Fetches items in the ZSET based on a range of scores
`ZREM`	Removes the item from the ZSET, if it exists

For example:

127.0.0.1:6379> zadd test-zset 100 member1
(integer) 1
127.0.0.1:6379> zadd test-zset 200 member2
(integer) 1
127.0.0.1:6379> zadd test-zset 200 member3
(integer) 1
127.0.0.1:6379> zrange test-zset 0 -1 withscores
1) "member1"
2) "100"
3) "member2"
4) "200"
5) "member3"
6) "200"
127.0.0.1:6379> zrange test-zset 0 -1
1) "member1"
2) "member2"
3) "member3"
127.0.0.1:6379> zrangebyscore test-zset 0 150 withscores
1) "member1"
2) "100"
127.0.0.1:6379> zrem test-zset member1
(integer) 1
127.0.0.1:6379> zrange test-zset 0 -1 withscores
1) "member2"
2) "200"
3) "member3"
4) "200"

Connecting a Python Standalone Container to Redis

The basic idea is as follows (assuming you already have a redis container):

create a bridge that allows the two containers to automatically resolve each other’s IP by container name
connect your redis container to that bridge
create a python container, connecting it to the same bridge you just created, and configuring your mount volume(s)
install Redis libraries in the python container
connect your to the redis container by connected = redis.Redis(host='<your-redis-container-name>', port=6379).
(alternatively, you could use docker-compose to achieve the same thing.)

Create a user-defined bridge with the name redis-py in Docker with the following command:
1
$ docker network create -d bridge redis-py
Connect your already exitsed redis container to that bridge. If you do not have a redis container yet, you need to set one up first. You connect the redis container (called test-redis in this example) to that bridge redis-py by running:
1
$ docker network connect redis-py test-redis
Note:
- Here you would also need to remember the port name your redis container uses. By default, it should be 6379. You can see the port once you run your redis container with docker exec -it test-redis redis-cli, and it will show, for example: 127.0.0.1:6379>.
Now you need to build another python container. You can first pull the image with docker pull python, and then start a container with:
1
2
3
4
5
6
7
$ docker run -it --name=test-py-red \
--network redis-py \
-e REDIS_HOST=test-redis \
-e REDIS_PORT=6379 \
-p 8080:80 \
-v python-vol:/data \
python bash
Note that some of the commands above are optional:
- -e REDIS_HOST=test-redis
  - this will configure your enviromental variable REDIS_HOST to be set to test-redis (your redis container name). However, this is not necessary.
- e REDIS_PORT=6379
  - this will configure your enviromental variable REDIS_PORT to be set to 6379 (your redis port). However, this is not necessary.
- v python-vol:/data (recommended to have a mount)
  - this will mount the directory /data in your python container to the external local volume python-vol
Now, since we specified python bash in the last line, it will start in bash inside the container. This allows you to install the redis library for python:
1
$ pip install --trusted-host pypi.python.org Flask Redis
Note:
- A problem with this is that if you remove the container, you will need to execute to re-install this library again. A solution could be to build your own Dockerfile and create your own image with:
  1
  2
  3
  4
  5
  FROM python:latest
  WORKDIR /app
  ADD . /app
  RUN pip install --trusted-host pypi.python.org Flask Redis
  EXPOSE 80
1
Finally, you can connect to your redis container names test-redis with:
1
$ python
which enters the python command line.
1
>>> import redis
which imports the redis library you just installed
1
>>> connected = redis.Redis(host='test-redis',port=6379)
which connects to the test-redis container at port 6379, so that now you can access redis contents and comamnds such as:
1
>>> connected.set('hello','world')

Antonomy of a Redis Web Application

Consider the case that you are managing a large online retail store, and you have millions of customers using it. Now, besides managing your website’s functionality, you would need several things to consider related to user-generated information:

login management
user browsing data
…
You may thing that you could simply use a database like SQL. However, the problem is that most relational databases are limited to inserting, updating, or deleting roughly 200–2,000 individual rows every second per database server. Though bulk inserts/updates/deletes can be performed faster, a customer will only be updating a small handful of rows for each web page view, so higher-speed bulk insertion doesn’t help here.

At present, due to the relatively large load through the day (assume averaging roughly 1,200 writes per second, close to 6,000 writes per second at peak), you needed to set up 10 relational database servers to deal with the load during peak hours. But now, you would like to try the new redis tool and see if all those data could be managed more easily.

Whenever we sign in to services on the internet, such as bank accounts or web mail, these services remember who we are using cookies. Cookies are small pieces of data that websites ask our web browsers to store and resend on every request to that service.

For login cookies, there are two common methods of storing login information in cookies: a signed cookie or a token cookie.

In general, cookies do not store passwords.

Signed cookies typically store the username when they last logged in, or maybe their user ID, and whatever else the service may find useful. Along with this user-specific information, the cookie also includes a signature that allows the server to verify that the information that the browser sent hasn’t been altered (like replacing the login name of one user with another).
Token cookies use a series of random bytes as the data in the cookie. The series of random bytes is generated when you logged in successfully, and then the server issues you that long random number token as a secret identifier. Then, on the server, the token is used as a key to look up the user who owns that token by querying a database of some kind. Over time, old tokens can be deleted to make room for new tokens (so that your loggin session expires).

Cookie type	Pros	Cons
Signed cookie	Everything needed to verify the cookie is in the cookie. Additional information can be included and signed easily	Correctly handling signatures is hard. It’s easy to forget to sign and/or verify data, allowing security vulnerabilities
Token cookie	Adding information is easy. Very small cookie, so mobile and slow clients can send requests faster	More information to store on the server. If using a relational database, cookie loading / storing can be expensive

To get started, we’ll use a HASH to store our mapping from login cookie tokens to the user that’s logged in.

To check the login, we need to fetch the user based on the token and return it, if it’s available. The following listing shows how we check login cookies.

# conn=redis.Redis(host='<yourhost>', port='<yourPort>')

def check_token(conn, token):
	return conn.hget('login:', token)  # Fetch and return the given user, if available.

Then, for a visit, we’ll update the login HASH for the user and record the current timestamp for the token in the ZSET of recent users. If the user was viewing an item, we also add the item to the user’s recently viewed ZSET and trim that ZSET if it grows past 25 items. The function that does all of this can be seen next.

def update_token(conn, token, user, item=None):
	timestamp = time.time()  #Get the timestamp.

	conn.hset('login:', token, user)  #Keep a mapping from the token to the logged-in user.
	conn.zadd('recent:', token, timestamp)  #Record when the token was last seen.

	if item:
		conn.zadd('viewed:' + token, item, timestamp)  #Record that the user viewed the item.
		conn.zremrangebyrank('viewed:' + token, 0, -26)  #Remove old items, keeping the most recent 25.

And you know what? That’s it. We’ve now recorded when a user with the given session last viewed an item and what item that user most recently looked at. On a server made in the last few years, you can record this information for at least 20,000 item views every second, which is more than three times what we needed to perform against the database. This can be made even faster, which we’ll talk about later. But even for this version, we’ve improved performance by 10–100 times over a typical relational database in this context.

Over time, memory use will grow, and we’ll want to clean out old data. As a way of limiting our data, we’ll only keep the most recent 10 million sessions.

For our cleanup, we’ll fetch the size of the ZSET in a loop. If the ZSET is too large, we’ll fetch the oldest items up to 100 at a time (because we’re using timestamps, this is just the first 100 items in the ZSET), remove them from the recent ZSET, delete the login tokens from the login HASH, and delete the relevant viewed ZSETs. If the ZSET isn’t too large, we’ll sleep for one second and try again later. The code for cleaning out old sessions is shown next.

QUIT = False
LIMIT = 10000000
def clean_sessions(conn):
	while not QUIT:
		size = conn.zcard('recent:')  # Find out how many tokens are known.

		if size <= LIMIT:
			time.sleep(1)  # We’re still under our limit; sleep and try again.
		else:
			end_index = min(size - LIMIT, 100)
			tokens = conn.zrange('recent:', 0, end_index-1)  # Fetch the token IDs that should be removed.
		
			session_keys = []
			for token in tokens:
				session_keys.append('viewed:' + token)  # Prepare the key names for the tokens to delete.
		
			conn.delete(*session_keys)
			conn.hdel('login:', *tokens)
			conn.zrem('recent:', *tokens)

Note:

The cleanup function written above may be run as a daemon process, to be run periodically via a cron job, or even to be run during every execution. As a general rule, if the function includes a while not QUIT: line, it’s supposed to be run as a daemon, though it could probably be modified to be run periodically, depending on its purpose.

You also notice that we called three functions with a syntax similar to conn.delete(*vtokens). Basically, we’re passing a sequence of arguments to the underlying function without previously unpacking the arguments. For further details on the semantics of how this works, you can visit the Python language tutorial website by visiting this short url: https://mng.bz/8I7W.

Storing Shopping Carts Info

Because we’ve had such good luck with session cookies and recently viewed items, we’ll push our shopping cart information into Redis. Since we’re already keeping user session cookies in Redis (along with recently viewed items), we can use the same cookie ID for referencing the shopping cart.

The shopping cart that we’ll use is simple: it’s a HASH that maps an item ID to the quantity of that item that the customer would like to purchase. We’ll have the web application handle validation for item count, so we only need to update counts in the cart as they change. If the user wants more than 0 items, we add the item(s) to the HASH (replacing an earlier count if it existed). If not, we remove the entry from the hash. Our add_to_cart() function can be seen in this listing.

def add_to_cart(conn, session, item, count):
	if count <= 0:
		conn.hrem('cart:' + session, item)  # Remove the item from the cart.

	else:
		conn.hset('cart:' + session, item, count)  # Add the item to the cart.

Now, we’ll update our session cleanup function to include deleting old shopping carts as clean_full_sessions() in the next listing.

def clean_full_sessions(conn):
	while not QUIT:
		size = conn.zcard('recent:')
		if size <= LIMIT:
			time.sleep(1)
		else:
			end_index = min(size - LIMIT, 100)
			sessions = conn.zrange('recent:', 0, end_index-1)

			session_keys = []
			for sess in sessions:
				session_keys.append('viewed:' + sess)
				session_keys.append('cart:' + sess)  # The required added line to delete the shopping cart for old sessions.


			conn.delete(*session_keys)
			conn.hdel('login:', *sessions)
			conn.zrem('recent:', *sessions)

We now have both sessions and the shopping cart stored in Redis, which helps to reduce request size, as well as allows the performing of statistical calculations on visitors to our site based on what items they looked at, what items ended up in their shopping carts, and what items they finally purchased.

Webpage Caching

Caching is different from a cookie. While they both act as some kind of storage, cache is actually a part of computer memory, whereas cookie is just stored and used by web browsers.

In genearl, when a processor wants something (e.g. values(a,b) for doing an operation (a+b)), it first search in cache if cache have these values then processor will fetch values form cache (fetching values from cache is really fast), but if cache don’t have these values then it will search in RAM (fetching values from ram is faster than hdd,ssd but slower than cache) if ram don’t have then it will search in hhd or ssd.

In the context of web services, this means that cache in general can deal with larger data pieces than cookies, so one common way of using it in web services is for storing static website contents. Since those static contenst don’t need to be dynamically generated upon every request, we can store most of the information in cache for faster browsing experience.

def cache_request(conn, request):
	
	if not can_cache(request):
		return callback(request)  # calls the request back to generate the content

	page_key = "cache:" + hash_request(request)  # hash the request for lookup and storing
	
	content = conn.get(pagekey)

	if content is None:  # reached here means content is cachable but not stored in cache

		content = callback(request) # content not cached. calls the request back to generate the content
		conn.setex(page_key, 300, content)  # cache the content and set expire in 300 seconds

	return content

Database row caching

For a daily deal with inventory counts being reduced and affecting whether someone can buy the item, the site cannot be entirely cached. However, an easy solution would be caching but also remmeber updating that specific database row. This means that you can choose to update the cached row for that specific item every few seconds, if there are many buyers. But if the data doesn’t change often, or when back-ordered items are acceptable, it may make sense to only update the cache every minute.

This can be done via using two ZSETs, one called delay: that tells you how often you should update your database row, and another called schedule: which tells you when the database row has been last updated in your cache.

def cache_rows(conn):  # this will be a daemon process
	while not QUIT:
		next = conn.zrange('schedule:',0,0, withscores=TRUE)  # this fetches the first/earliest item that we have updated, if there is

		now = time.time()
		if next is None or next[0][1] > now:  # if there are no items in schedule: 
											  # or time hasn't come yet for updating
			time.sleep(0.05)
		else:
			row_id = next[0][0]

			delay = conn.zscore('delay:', row_id)  # delay also stores information on whether
												   # we should remove the item in the scheduling
			
			if delay <= 0:
				conn.zrem('delay:', row_id)
				conn.zrem('schedule:', row_id)
				conn.delete('inv:' + row_id)  
			else:  # now we need to handle the cache updating

				d_row = Inventory.get(row_id)  # gets the database row

				conn.zadd('schedule:', row_id, now+delay)  # re-schedule the item for update
				
				conn.set('inv:`+row_id, json.dumps(row.to_dict()))  # updates the cache with the latest database row

def add_schedule_row_cache(conn, row_id, delay):
	conn.zadd('delay:', row_id, delay)  # adds the item and tells how often it should be updated

	conn.zadd('schedule:', row_id, time.time())  # adds the item with the time it was placed in cache

Simple Webpage Analytics

In the previous examples, we were able to cache webpage contents. However, we did not limit how much pages should be cached, as we did not specify the can_cache() function yet. But what would be a sensible way to decide which webpage should be cached and which one shouldn’t?

A simple idea is to only include (for example) 10,000 webpages that has items you are browing the most often. For the other webpages with items that you browse only occaasionally, you do not store them in cache. However, you also need to be able to include possible new trending items, so you cannot just simply discard all items below the 10,000 limit. One solution you could use is to rescale the view counts of the top 10,000 items you viewed to be half has much as they were before.

def rescale_viewed(conn):
	while not QUIT:
		conn.zremrangebyrank('viewed:', 20000, -1)  # Remove any item not in the top 20,000 viewed items.

		conn.zinterstore('viewed:', {'viewed:': .5})  # Rescale all counts to be 1/2 of what they were before.

		time.sleep(300)  # Do it again in 5 minutes.

where the line:

conn.zinterstore('viewed:', {'viewed:': .5}), the function ZINTERSTORE lets us combine one or more ZSETs and multiply every score in the input ZSETs by a given number. (Each input ZSET can be multiplied by a different number.)

Now, one sensible way to call this function is to place it in the place where you kept a reference to every item visited when you do login and shopping cart info storing. So you can place it in here:

def update_token(conn, token, user, item=None):
	timestamp = time.time()
	conn.hset('login:', token, user)
	conn.zadd('recent:', token, timestamp)
	if item is not None:
		conn.zadd('viewed:' + token, item, timestamp)
		conn.zremrangebyrank('viewed:' + token, 0, -26)

		conn.zincrby('viewed:', item, -1)  # the added line, move the item forward if viewed again

With the rescaling and the counting, we now have a constantly updated list of the most-frequently viewed items. Now all we need to do is to specify our can_cache() function to take into consideration our new method of deciding whether a page can be cached, and we’re done:

def can_cache(conn, request):
	item_id = extract_item_id(request)  # Get the item ID for the page, if any.

	if item_id is not None or is_dynamic(request):  # Check whether the page can be statically cached and whether this is an item page.

		return False

	rank = conn.zrank('viewed:', item_id)  # Get the rank of the item.

	return rank is not None and rank < 10000  # Return whether the item has a high enough view count to be cached.

`String`s in Redis

In this section, we’ll talk about the simplest structure available to Redis, the STRING. This builds on the section Strings in Redis - Basics, and here it covers the basic numeric increment and decrement operations, followed later by the bit and substring manipulation calls, and you’ll come to understand that even the simplest of structures has a few surprises that can make it useful in a variety of powerful ways.

Command	Example use and description
`INCR`	`INCR key-name` — Increments the value stored at the key by 1
`DECR`	`DECR key-name` — Decrements the value stored at the key by 1
`INCRBY`	`INCRBY key-name amount` — Increments the value stored at the key by the provided integer value
`DECRBY`	`DECRBY key-name amount` — Decrements the value stored at the key by the provided integer value
`INCRBYFLOAT`	`INCRBYFLOAT key-name amount` — Increments the value stored at the key by the provided float value (available in Redis 2.6 and later)

This means that, when setting a STRING value in Redis, if that value could be interpreted as a base-10 integer or a floating-point value, Redis will detect this and allow you to manipulate the value using the various INCR* and DECR* operations. So, if you try to increment or decrement a key that has a value that can’t be interpreted as an integer or float, you’ll receive an error.

Note:

If you try to increment or decrement a key that doesn’t exist or is an empty string, Redis will operate as though that key’s value were zero.

>>> import redis; conn=redis.Redis(host='cust-redis',port=6379)
>>> conn.get('new-key')  
>>> conn.incr('new-key')  # incrementing a non-existant key by 1
1
>>> conn.incr('new-key-2')  # incrementing a non-existant key by 
1
>>> conn.get('new-key-2')
b'1'
>>> conn.incr('new-key',10)  # python implemented to have the option of passing in a value
11
>>> conn.incrby('new-key',10)
21
>>> conn.decr('new-key',10)
11
>>> conn.decrby('new-key',10)
1
>>> conn.set('new-key',10)
True
>>> conn.get('new-key')
b'10'

Note:

You see those b in front of the result string because these data are byte literals instead of String objects. Therefore, to get rid of the b, you can add the method decode('utf8') to the end.

Redis additionally offers methods for reading and writing parts of byte string values (integer and float values can also be accessed as though they’re byte strings, though that use is somewhat uncommon).

Command	Example use and description
`APPEND`	`APPEND key-name value` — Concatenates the provided value to the string already stored at the given key
`GETRANGE`	`GETRANGE key-name start end` — Fetches the substring, including all characters from the start offset to the end offset, inclusive
`SETRANGE`	`SETRANGE key-name offset value` — Sets the substring starting at the provided offset to the given value
`GETBIT`	`GETBIT key-name offset` — Treats the byte string as a bit string, and returns the value of the bit in the string at the provided bit offset
`SETBIT`	`SETBIT key-name offset value` — Treats the byte string as a bit string, and sets the value of the bit in the string at the provided bit offset
`BITCOUNT`	`BITCOUNT key-name [start end]` — Counts the number of 1 bits in the string, optionally starting and finishing at the provided byte offsets
`BITOP`	`BITOP operation dest-key key-name [key-name …]` — Performs one of the bitwise operations, AND, OR, XOR, or NOT, on the strings provided, storing the result in the destination key

>>> conn.append('new-string-key', 'hello ')  # append the string ‘hello ’ to the previously nonexistent key 
6L  # When appending a value, Redis returns the length of the string so far.

>>> conn.append('new-string-key', 'world!')
12L

>>> conn.substr('new-string-key', 3, 7)  # Redis uses 0-indexing. It is inclusive of the endpoints by default.
'lo wo'

>>> conn.setrange('new-string-key', 0, 'H')  # Sets the 0th char to 'H'
12

>>> conn.setrange('new-string-key', 6, 'W')
12

>>> conn.get('new-string-key')
'Hello World!'

>>> conn.setrange('new-string-key', 11, ', how are you?')  # With setrange, we can replace anywhere inside the string
                                                           # and we can make the string longer as a result
25

>>> conn.get('new-string-key')
'Hello World, how are you?'

>>> conn.setbit('another-key', 2, 1)  # If we write to a bit beyond the size of the string, it’s filled with 0s.
0

>>> conn.setbit('another-key', 7, 1)
0

>>> conn.get('another-key')  # If you want to interpret the bits stored in Redis
                             # you have 0010 0001, which is '!' in ASCII
'!'

`List`s in Redis

In this section, we’ll talk about LISTs, which store an ordered sequence of STRING values. This builds on from the section Lists in Redis - Basics. We’ll cover some of the most commonly used LIST manipulation commands for pushing and popping items from LISTs.

Command	Example use and description
`RPUSH`	`RPUSH key-name value [value …]` — Pushes the value(s) onto the right end of the list
`LPUSH`	`LPUSH key-name value [value …]` — Pushes the value(s) onto the left end of the list
`RPOP`	`RPOP key-name` — Removes and returns the rightmost item from the list
`LPOP`	`LPOP key-name` — Removes and returns the leftmost item from the list
`LINDEX`	`LINDEX key-name offset` — Returns the item at the given offset/index
`LRANGE`	`LRANGE key-name start end` — Returns the items in the list at the offsets from start to end, inclusive
`LTRIM`	`LTRIM key-name start end` — Trims the list to only include items at indices between start and end, inclusive

>>> conn.rpush('list-key', 'last')
1L  # When we push items onto the list, it returns the length of the list after the push has completed.

>>> conn.lpush('list-key', 'first')
2L

>>> conn.rpush('list-key', 'new last')
3L

>>> conn.lrange('list-key', 0, -1)
['first', 'last', 'new last']

>>> conn.lpop('list-key')
'first'

>>> conn.lpop('list-key')
'last'

>>> conn.lrange('list-key', 0, -1)
['new last']

>>> conn.rpush('list-key', 'a', 'b', 'c')  # We can push multiple items at the same time.
4L

>>> conn.lrange('list-key', 0, -1)
['new last', 'a', 'b', 'c']

>>> conn.ltrim('list-key', 2, -1)  # Only keep elements from the 2nd position to the last
True

>>> conn.lrange('list-key', 0, -1)
['b', 'c']

Among the LIST commands we didn’t introduce before are a few commands that allow you to move items from one list to another, and even block while waiting for other clients to add items to LISTs.

Command	Example use and description
`BLPOP`	`BLPOP key-name [key-name …] timeout` — Pops the leftmost item from the first non-empty LIST, or waits the timeout in seconds for an item
`BRPOP`	`BRPOP key-name [key-name …] timeout` — Pops the rightmost item from the first non-empty LIST, or waits the timeout in seconds for an item
`RPOPLPUSH`	`RPOPLPUSH source-key dest-key` — Pops the rightmost item from the source and LPUSHes the item to the destination, also returning the item to the user
`BRPOPLPUSH`	`BRPOPLPUSH source-key dest-key timeout` — Pops the rightmost item from the source and LPUSHes the item to the destination, also returning the item to the user, and waiting up to the timeout if the source is empty

# Let’s add some items to a couple of lists to start.
>>> conn.rpush('list', 'item1')
1
>>> conn.rpush('list', 'item2')
2
>>> conn.rpush('list2', 'item3')
1

>>> conn.brpoplpush('list2', 'list', 1)  # move an item from one list to the other, also returning the item.
'item3'

>>> conn.brpoplpush('list2', 'list', 1)  # When a list is empty, the blocking pop will stall for the timeout, 
                                         # and return None (which isn’t displayed in the interactive console).
>>> conn.lrange('list', 0, -1)
['item3', 'item1', 'item2']

>>> conn.brpoplpush('list', 'list2', 1)
'item2'

>>> conn.blpop(['list', 'list2'], 1)
('list', 'item3')

>>> conn.blpop(['list', 'list2'], 1)
('list', 'item1')

>>> conn.blpop(['list', 'list2'], 1)
('list2', 'item2')

>>> conn.blpop(['list', 'list2'], 1) # block for 1 second until this operation is completed
(1.02s)  

>>>

`Set`s in Redis

In this section, we’ll discuss some of the most frequently used commands that operate on SETs. This builds on from the section Sets in Redis - Basics. You’ll learn about the standard operations for inserting, removing, and moving members between SETs, as well as commands to perform intersection, union, and differences on SETs.

Command	Example use and description
`SADD`	`SADD key-name item [item …]` — Adds the items to the set and returns the number of items added that weren’t already present
`SREM`	`SREM key-name item [item …]` — Removes the items and returns the number of items that were removed
`SISMEMBER`	`SISMEMBER key-name item` — Returns whether the item is in the SET
`SCARD`	`SCARD key-name` — Returns the number of items in the SET
`SMEMBERS`	`SMEMBERS key-name` — Returns all of the items in the SET as a Python set
`SRANDMEMBER`	`SRANDMEMBER key-name [count]` — Returns one or more random items from the SET. When count is positive, Redis will return count distinct randomly chosen items, and when count is negative, Redis will return count randomly chosen items that may not be distinct.
`SPOP`	`SPOP key-name` — Removes and returns a random item from the SET
`SMOVE`	`SMOVE source-key dest-key item` — If the item is in the source, removes the item from the source and adds it to the destination, returning if the item was moved

>>> conn.sadd('set-key', 'a', 'b', 'c')
3  # Adding items to the SET returns the number of items that weren’t already in the SET.

>>> conn.srem('set-key', 'c', 'd')
True

>>> conn.srem('set-key', 'c', 'd')
False  # Removing items from the SET returns whether an item was removed; 
       # note that the client is buggy in that respect — Redis itself returns the total number of items removed.

>>> conn.scard('set-key')  # We can get the number of items in the SET.
2

>>> conn.smembers('set-key')
set(['a', 'b'])

>>> conn.smove('set-key', 'set-key2', 'a')
True

>>> conn.smove('set-key', 'set-key2', 'c')
False

>>> conn.smembers('set-key2')
set(['a'])

`Hash`es in Redis

In this section, we’ll talk about the most commonly used commands that manipulate HASHes. This builds on fro the section Hashes in Redis - Basics. You’ll learn more about the operations for adding and removing key-value pairs to HASHes, as well as commands to fetch all of the HASH contents along with the ability to increment or decrement values.

Command	Example use and description
`HMGET`	`HMGET key-name key [key …]` — Fetches the values at the fields/keys in the HASH
`HMSET`	`HMSET key-name key value [key value …]` — Sets the values of the fields in the HASH
`HDEL`	`HDEL key-name key [key …]` — Deletes the key-value pairs in the HASH, returning the number of pairs that were found and deleted
`HLEN`	`HLEN key-name` — Returns the number of key-value pairs in the HASH

>>> conn.hmset('hash-key', {'k1':'v1', 'k2':'v2', 'k3':'v3'})
True  # We can add multiple items to the hash in one call.

>>> conn.hmget('hash-key', ['k2', 'k3'])
['v2', 'v3']

>>> conn.hlen('hash-key')
3  # The HLEN command is typically used for debugging very large HASHes.

>>> conn.hdel('hash-key', 'k1', 'k3')
True

Note:

The HMGET/HMSET commands are similar to their single-argument versions (HSET/HGET), only differing in that they take a list or dictionary for arguments instead of the single entries.

With the availability of HGETALL, it may not seem as though HKEYS and HVALUES would be that useful, but when you expect your values to be large, you can fetch the keys, and then get the values one by one to keep from blocking other requests.

Command	Example use and description
`HEXISTS`	`HEXISTS key-name key` — Returns whether the given key exists in the HASH
`HKEYS`	`HKEYS key-name` — Fetches the keys in the HASH
`HVALS`	`HVALS key-name` — Fetches the values in the HASH
`HGETALL`	`HGETALL key-name` — Fetches all key-value pairs from the HASH
`HINCRBY`	`HINCRBY key-name key increment` — Increments the value stored at the given key by the integer increment
`HINCRBYFLOAT`	`HINCRBYFLOAT key-name key increment` — Increments the value stored at the given key by the float increment

HINCRBY and HINCRBYFLOAT should remind you of the INCRBY and INCRBYFLOAT operations available on STRING keys, and they have the same semantics, applied to HASH values.

>>> conn.hmset('hash-key2', {'short':'hello', 'long':1000*'1'})
True

>>> conn.hkeys('hash-key2')
['long', 'short']

>>> conn.hexists('hash-key2', 'num')
False  # We can also check the existence of specific keys.

>>> conn.hincrby('hash-key2', 'num')  # Incrementing a previously nonexistent key in a hash behaves just like on strings; 
                                      # Redis operates as though the value had been 0.
1L

>>> conn.hexists('hash-key2', 'num')
True

`ZSet`s in Redis

In this section, we’ll talk about commands that operate on ZSETs. This builds on from the section ZSets in Redis - Basics. You’ll learn how to add and update items in ZSETs, as well as how to use the ZSET intersection and union commands.

Command	Example use and description
`ZADD`	`ZADD key-name score member [score member …]` — Adds members with the given scores to the ZSET
`ZREM`	`ZREM key-name member [member …]` — Removes the members from the ZSET, returning the number of members that were removed
`ZCARD`	`ZCARD key-name` — Returns the number of members in the ZSET
`ZINCRBY`	`ZINCRBY key-name increment member` — Increments the member in the ZSET
`ZCOUNT`	`ZCOUNT key-name min max` — Returns the number of members with scores between the provided minimum and maximum
`ZRANK`	`ZRANK key-name member` — Returns the position of the given member in the ZSET
`ZSCORE`	`ZSCORE key-name member` — Returns the score of the member in the ZSET
`ZRANGE`	`ZRANGE key-name start stop [WITHSCORES]` — Returns the members and optionally the scores for the members with ranks between start and stop

>>> conn.zadd('zset-key', 'a', 3, 'b', 2, 'c', 1)  # Adding members to ZSETs in Python has the arguments 
                                                   # reversed compared to standard Redis, which makes the order the same as HASHes.
3  

>>> conn.zcard('zset-key')
3  # Knowing how large a ZSET is can tell us in some cases if it’s necessary to trim our ZSET.

>>> conn.zincrby('zset-key', 'c', 3)
4.0

>>> conn.zscore('zset-key', 'b')
2.0  # Fetching scores of individual members can be useful if we’ve been keeping counters or toplists.

>>> conn.zrank('zset-key', 'c')
2  # By fetching the 0-indexed position of a member, we can then later use ZRANGE to fetch a range of the values easily.

>>> conn.zcount('zset-key', 0, 3)
2L  # Counting the number of items with a given range of scores can be quite useful for some tasks.

>>> conn.zrem('zset-key', 'b')
True

>>> conn.zrange('zset-key', 0, -1, withscores=True)
[('a', 3.0), ('c', 4.0)]

For debugging, we usually fetch the entire ZSET with this ZRANGE call, but real use cases will usually fetch items a relatively small group at a time.

Below shows several more ZSET commands in Redis that you’ll find useful.

Command	Example use and description
`ZREVRANK`	`ZREVRANK key-name member` — Returns the position of the member in the ZSET, with members ordered in reverse
`ZREVRANGE`	`ZREVRANGE key-name start stop [WITHSCORES]` — Fetches the given members from the ZSET by rank, with members in reverse order
`ZRANGEBYSCORE`	`ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]` — Fetches the members between min and max
`ZREVRANGEBYSCORE`	`ZREVRANGEBYSCORE key max min [WITHSCORES] [LIMIT offset count]` — Fetches the members in reverse order between min and max
`ZREMRANGEBYRANK`	`ZREMRANGEBYRANK key-name start stop` — Removes the items from the ZSET with ranks between start and stop
`ZREMRANGEBYSCORE`	`ZREMRANGEBYSCORE key-name min max` — Removes the items from the ZSET with scores between min and max
`ZINTERSTORE`	`ZINTERSTORE dest-key key-count key [key …] [WEIGHTS weight [weight …]]` — Performs a SET-like intersection of the provided ZSETs
`ZUNIONSTORE`	`ZUNIONSTORE dest-key key-count key [key …] [WEIGHTS weight [weight …]]` — Performs a SET-like union of the provided ZSETs

# We’ll start out by creating a couple of ZSETs.
>>> conn.zadd('zset-1', 'a', 1, 'b', 2, 'c', 3)
3
>>> conn.zadd('zset-2', 'b', 4, 'c', 1, 'd', 0)
3

>>> conn.zinterstore('zset-i', ['zset-1', 'zset-2'])  # When performing ZINTERSTORE or ZUNIONSTORE, our default aggregate is sum (which adds the value of the shared members)
2L

>>> conn.zrange('zset-i', 0, -1, withscores=True)
[('c', 4.0), ('b', 6.0)]

>>> conn.zunionstore('zset-u', ['zset-1', 'zset-2'], aggregate='min')  # here we are using min, which selects the minimum of the shared members
4L

>>> conn.zrange('zset-u', 0, -1, withscores=True)
[('d', 0.0), ('a', 1.0), ('c', 1.0), ('b', 2.0)]

>>> conn.sadd('set-1', 'a', 'd') 
2

>>> conn.zunionstore('zset-u2', ['zset-1', 'zset-2', 'set-1']) 
4L

>>> conn.zrange('zset-u2', 0, -1, withscores=True)
[('d', 1.0), ('a', 2.0), ('c', 4.0), ('b', 6.0)]

`Publish`/`Subscribe` in Redis

Generally, the concept of publish/subscribe, also known as pub/sub, is characterized by listeners subscribing to channels, with publishers sending binary string messages to channels.

Anyone listening to a given channel will receive all messages sent to that channel while they’re connected and listening. You can think of it like a radio station, where subscribers can listen to multiple radio stations at the same time
Any publishers can send messages to a given channel, like being a radio station.

Command	Example use and description
`SUBSCRIBE`	`SUBSCRIBE channel [channel …]` — Subscribes to the given channels
`UNSUBSCRIBE`	`UNSUBSCRIBE [channel [channel …]]` — Unsubscribes from the provided channels, or unsubscribes all channels if no channel is given
`PUBLISH`	`PUBLISH channel message` — Publishes a message to the given channel
`PSUBSCRIBE`	`PSUBSCRIBE pattern [pattern …]` — Subscribes to messages broadcast to channels that match the given pattern
`PUNSUBSCRIBE`	`PUNSUBSCRIBE [pattern [pattern …]]` — Unsubscribes from the provided patterns, or unsubscribes from all subscribed patterns if none are given

>>> def publisher(n):
	time.sleep(1)  # We sleep initially in the function to let the SUBSCRIBEr connect and start listening for messages.

	for i in xrange(n):
				conn.publish('channel', i)
				time.sleep(1)  # After publishing, we’ll pause for a moment so that we can see this happen over time.


>>> def run_pubsub():
	threading.Thread(target=publisher, args=(3,)).start()  # start the publisher thread to send three messages.

	pubsub = conn.pubsub()
	pubsub.subscribe(['channel'])  # We’ll set up the pubsub object and subscribe to a channel.

	count = 0
	for item in pubsub.listen():  # listen to subscription messages by iterating over the result of pubsub.listen().

		print item  # print every message that we receive.

		count += 1
		if count == 4:  # We’ll stop listening for new messages after the subscribe message and three real messages by unsubscribing.
			pubsub.unsubscribe()
			break

>>> run_pubsub()  # Actually run the functions to see them work.

{'pattern': None, 'type': 'subscribe', 'channel': 'channel', 'data': 1L}

# When subscribing, we receive a message on the channel.
# These are the structures that are produced as items when we iterate over pubsub.listen().
{'pattern': None, 'type': 'message', 'channel': 'channel', 'data': '0'}
{'pattern': None, 'type': 'message', 'channel': 'channel', 'data': '1'}
{'pattern': None, 'type': 'message', 'channel': 'channel', 'data': '2'}

# When we unsubscribe, we receive a message telling us which channels we have unsubscribed from 
# and also the number of channels we are still subscribed to.
{'pattern': None, 'type': 'unsubscribe', 'channel': 'channel', 'data': 0L}

Note:

But in the case of clients that have subscribed, if the client is disconnected and a message is sent before it can reconnect, the client will never see the message. When you’re relying on receiving messages over a channel, the semantics of PUBLISH/SUBSCRIBEin Redis may let you down.

However, there are different methods to handle reliable message delivery (you will see in a later section), which works in the face of network disconnections, and which won’t cause Redis memory to grow (even in older versions of Redis) unless you want it to.

Operations in Redis - Sorting

Sorting in Redis is similar to sorting in other languages: we want to take a sequence of items and order them according to some comparison between elements. SORT allows us to sort LISTs, SETs, and ZSETs according to data in the LIST/SET/ZSET data stored in STRING keys, or even data stored in HASHes. If you’re coming from a relational database background, you can think of SORT as like the order by clause in a SQL statement that can reference other rows and tables.

Command	Example use and description
`SORT`	`SORT source-key [BY pattern] [LIMIT offset count] [GET pattern [GET pattern …]] ...` — Sorts the input `LIST`, `SET`, or `ZSET` according to the options provided, and returns or stores the result

Some of the more basic options with SORT include the ability to order the results in descending order rather than the default ascending order, consider items as though they were numbers, compare as though items were binary strings, and etc.

# Start by adding some items to a LIST.
>>> conn.rpush('sort-input', 23, 15, 110, 7)
4

>>> conn.sort('sort-input')  # We can sort the items numerically.
['7', '15', '23', '110']

>>> conn.sort('sort-input', alpha=True)  # And we can sort the items alphabetically. In this case, sort each number character by character
['110', '15', '23', '7']  

# Adding some additional data for SORTing and fetching.
>>> conn.hset('d-7', 'field', 5)
1L
>>> conn.hset('d-15', 'field', 1)
1L
>>> conn.hset('d-23', 'field', 9)
1L
>>> conn.hset('d-110', 'field', 3)
1L

>>> conn.sort('sort-input', by='d-*->field')  # We can sort our data by fields of HASHes.
['15', '110', '7', '23']

>>> conn.sort('sort-input', by='d-*->field', get='d-*->field')
['1', '3', '5', '9']

Sorting can be used to sort LISTs, but it can also sort SETs, turning the result into a LIST.

Though SORT is the only command that can manipulate three types of data at the same time, basic Redis transactions can let you manipulate multiple data types with a series of commands without interruption.

Basic Redis Transactions

Though there are a few commands to copy or move items between keys, there isn’t a single command to move items between types (though you can copy from a SET to a ZSET with ZUNIONSTORE). For operations involving multiple keys (of the same or different types), Redis has five commands that help us operate on multiple keys without interruption: WATCH, MULTI, EXEC, UNWATCH, and DISCARD.

In Redis, a basic transaction involving MULTI and EXEC is meant to provide the opportunity for one client to execute multiple commands A, B, C, … without other clients being able to interrupt them. This isn’t the same as a relational database transaction, which can be executed partially, and then rolled back or committed. In Redis, every command passed as part of a basic MULTI/EXEC transaction is executed one after another until they’ve completed. After they’ve completed, other clients may execute their commands.

To perform a transaction in Redis, we first call MULTI, followed by any sequence of commands we intend to execute, followed by EXEC. When seeing MULTI, Redis will queue up commands from that same connection until it sees an EXEC, at which point Redis will execute the queued commands sequentially without interruption.

For example, in a redis-cli:

127.0.0.1:6379> multi
OK
127.0.0.1:6379> get hello
QUEUED
127.0.0.1:6379> set hello no
QUEUED
127.0.0.1:6379> exec
1) "worldanother-world"
2) OK
127.0.0.1:6379> get hello
"no"

Semantically, Python library handles this by the use of what’s called a pipeline. Calling the pipeline() method on a connection object will create a transaction, which when used correctly will automatically wrap a sequence of commands with MULTI and EXEC. Incidentally, the Python Redis client will also store the commands to send until we actually want to send them. This reduces the number of round trips between Redis and the client, which can improve the performance of a sequence of commands.

import redis
import threading
import time

def trans():
        pipeline = conn.pipeline()
        pipeline.incr("trans")
        time.sleep(0.1)
        pipeline.incr("trans",-1)
        print(pipeline.execute())
		# this in the end has the same effect of MULTI ... and then EXEC

if __name__ == "__main__":
        conn = redis.Redis(host='cust-redis',port=6379)
        for i in range(3):
                threading.Thread(target=trans).start()

Now, notice that by using a transaction, each thread is able to execute its entire sequence of commands without other threads interrupting it (as otherwise one of them might print 1 instead of 0), despite the delay between the two calls.

Basically, you can think of this as having the same effect as synchrnoized keyword in Java, so that this transaction is locked once one thread enters.

Key Expiration

When we say that a key has a time to live, or that it’ll expire at a given time, we mean that Redis will automatically delete the key when its expiration time has arrived.

Having keys that will expire after a certain amount of time can be useful to handle the cleanup of cached data. If you look through other chapters, you won’t see the use of key expiration in Redis often. This is mostly due to the types of structures that are used; few of the commands we use offer the ability to set the expiration time of a key automatically. And with containers (LISTs, SETs, HASHes, and ZSETs), we can only expire entire keys, not individual items (this is also why we use ZSETs with timestamps in a few places).

Command	Example use and description
`PERSIST`	`PERSIST key-name` — Removes the expiration from a key
`TTL`	`TTL key-name` — Returns the amount of time remaining before a key will expire
`EXPIRE`	`EXPIRE key-name seconds` — Sets the key to expire in the given number of seconds
`EXPIREAT`	`EXPIREAT key-name timestamp` — Sets the expiration time as the given Unix timestamp
`PTTL`	`PTTL key-name` — Returns the number of milliseconds before the key will expire (available in Redis 2.6 and later)
`PEXPIRE`	`PEXPIRE key-name milliseconds` — Sets the key to expire in the given number of milliseconds (available in Redis 2.6 and later)
`PEXPIREAT`	`PEXPIREAT key-name timestamp-milliseconds` — Sets the expiration time to be the given Unix timestamp specified in milliseconds (available in Redis 2.6 and later)

# We’re starting with a very simple STRING value.
>>> conn.set('key', 'value')
True
>>> conn.get('key')
'value'

>>> conn.expire('key', 2)
True
>>> time.sleep(2)
>>> conn.get('key')  # it’s already been deleted.

>>> conn.set('key', 'value2')
True
>>> conn.expire('key', 100); conn.ttl('key')
True
100

Persistence in Redis - Snapshot

In Redis, we can create a point-in-time copy of in-memory data by creating a snapshot. After creation, these snapshots can be backed up, copied to other servers to create a clone of the server, or left for a future restart.

On the configuration side of things, snapshots are written to the file referenced as dbfilename in the configuration, and stored in the path referenced as dir. Until the next snapshot is performed, data written to Redis since the last snapshot started (and completed) would be lost if there were a crash caused by Redis, the system, or the hardware.

As an example, say that we have Redis running with 10 gigabytes of data currently in memory. A previous snapshot had been started at 2:35 p.m. and had finished. Now a snapshot is started at 3:06 p.m., and 35 keys are updated before the snapshot completes at 3:08 p.m. If some part of the system were to crash and prevent Redis from completing its snapshot operation between 3:06 p.m. and 3:08 p.m., any data written between 2:35 p.m. and now would be lost. But if the system were to crash just after the snapshot had completed, then only the updates to those 35 keys would be lost (so that the snapshot only includes data at the point of initiation, at 3:06 p.m.).

There are five methods to initiate a snapshot, which are listed as follows:

Any Redis client can initiate a snapshot by calling the BGSAVE command. On platforms that support BGSAVE (basically all platforms except for Windows), Redis will fork, 1 and the child process will write the snapshot to disk while the parent process continues to respond to commands.
- When a process forks, the underlying operating system makes a copy of the process. On Unix and Unix-like systems, the copying process is optimized such that, initially, all memory is shared between the child and parent processes. When either the parent or child process writes to memory, that memory will stop being shared.
A Redis client can also initiate a snapshot by calling the SAVE command, which causes Redis to stop responding to any/all commands until the snapshot completes. This command isn’t commonly used, except in situations where we need our data on disk, and either we’re okay waiting for it to complete, or we don’t have enough memory for a BGSAVE.
If Redis is configured with save lines, such as save 60 10000, Redis will automatically trigger a BGSAVE operation if 10,000 writes have occurred after 60 seconds since the last successful save has started (using the configuration option described). When multiple save lines are present, any time one of the rules match, a BGSAVE is triggered.
When Redis receives a request to shut down by the SHUTDOWN command, or it receives a standard TERM signal, Redis will perform a SAVE, blocking clients from performing any further commands, and then shut down.
If a Redis server connects to another Redis server and issues the SYNC command to begin replication, the master Redis server will start a BGSAVE operation if one isn’t already executing or recently completed.

Note:

There are two ways to do the above configuration for redis. One is to use the config set <value> command. The other is to change the redis.conf file, which should be under /usr/local/etc/redis/redis.conf.

For docker users, if that file does not exist, you might need to manually create one and mount that directory.

However, as mentioned before, if a system crashes while you are doing a snapshot, data will be lost. Then the question becomes how to recover it. Suppose you are processing log files, and the system crashes before you are diong a dump. Then, in order to recover from it, you need to find a way to first know what has been lost, in other words, you need to somehow track your progress:

def process_logs(conn, path, callback):  # the callback function will be used to process the log line

   current_file, offset = conn.mget(  # mget is multiple get
      'progress:file', 'progress:position')  # Get the current progress.

   pipe = conn.pipeline()

   def update_progress():  # this is used later to update the line number

      pipe.mset({
         'progress:file': fname,
         'progress:position': offset
      })

      pipe.execute()

   for fname in sorted(os.listdir(path)):  # Iterate over the log files in sorted order.

      if fname < current_file:  # Skip over files that are before the current file.
         continue

      inp = open(os.path.join(path, fname), 'rb')

      if fname == current_file:
         inp.seek(int(offset, 10))  # If we’re continuing a file, skip over the parts that we’ve
								    # already processed.

      else:
         offset = 0

      current_file = None

      for lno, line in enumerate(inp):

         callback(pipe, line)  # Process/handle the log line.

         offset = int(offset) + len(line)  # Update our information about the offset into the file.

         if not (lno+1) % 1000:
            update_progress()  # updates the progress information for every 1000 actual updates
      
	  update_progress()  #update again when we are done with the file

      inp.close()

Deciding whether to use BGSAVE or SAVE

When the amount of data that we store in Redis tends to be under a few gigabytes, snapshotting with BGSAVE can be the right answer. Redis will fork, save to disk, and finish the snapshot faster than you can read this sentence. But as our Redis memory use grows over time, so does the time to perform a fork operation for the BGSAVE. In situations where Redis is using tens of gigabytes of memory, there isn’t a lot of free memory, or if we’re running on a virtual machine, letting a BGSAVE occur may cause the system to pause for extended periods of time.

To prevent forking from causing such issues, we may want to disable automatic saving entirely. When automatic saving is disabled, we then need to manually call BGSAVE (which has all of the same potential issues as before, only now we know when they will happen), or we can call SAVE. With SAVE, Redis does block until the save is completed, but because there’s no fork, there’s no fork delay. And because Redis doesn’t have to fight with itself for resources, the snapshot will finish faster.

For example, when trying to use BGSAVE with clients writing to Redis with data of about 50 gigabytes, forking would take 15 seconds or more, followed by 15–20 minutes for the snapshot to complete. But with SAVE, the snapshot would finish in 3–5 minutes.

Persistence in Redis - Append-only file

In basic terms, append-only log files keep a record of data changes that occur by writing each change to the end of the file. In doing this, anyone could recover the entire dataset by replaying the append-only log from the beginning to the end. Redis has functionality that does this as well, and it’s enabled by setting the configuration option appendonly yes.

You also have several options to configuring syncing and writing the append-only file to disk. When that sync is completed, we can be fairly certain that our data is on disk and we can read it later if the system otherwise fails.

`appendfsync` Option	How often syncing will occur
`always`	Every write command to Redis results in a write to disk. This slows Redis down substantially if used.
`everysec`	Once per second, explicitly syncs write commands to disk.
`no`	Lets the operating system control syncing to disk.

Warning:

You’ll want to be careful if you’re using SSDs with appendfsync always. Writing every change to disk as they happen, instead of letting the operating system group writes together as is the case with the other appendfsync options, has the potential to cause an extreme form of what is known as write amplification. By writing small amounts of data to the end of a file, you can reduce the lifetime of SSDs from years to just a few months in some cases.

As a reasonable compromise between keeping data safe and keeping our write performance high, we can also set appendfsync everysec. This configuration will sync the append-only log once every second. For most common uses, we’ll likely not find significant performance penalties for syncing to disk every second compared to not using any sort of persistence. By syncing to disk every second, if the system were to crash, we could lose at most one second of data that had been written or updated in Redis. Also, in the case where the disk is unable to keep up with the write volume that’s happening, Redis would gracefully slow down to accommodate the maximum write rate of the drive.

Append-only files are flexible, offering a variety of options to ensure that almost every level of paranoia can be addressed. But there’s a dark side to AOF persistence, and that is file size.

Configuring/Compacting your Append-only Files

If using AOF would make data loss to one second of work (for example, appendfsync everysecond), then why do we want to use Snapshots in general? Isn’t the choice clear that we would want to use AOF almost always? In reality, the choice is actually not so simple: because every write to Redis causes a log of the command to be written to disk, the append-only log file will continuously grow. Over time, a growing AOF could cause your disk to run out of space, but more commonly, upon restart, Redis will be executing every command in the AOF in order. When handling large AOFs, Redis can take a very long time to start up.

One solution is to use BGREWRITEAOF, which will rewrite the AOF to be as short as possible by removing redundant commands. BGREWRITEAOF works similarly to the snapshotting BGSAVE: performing a fork and subsequently rewriting the append-only log in the child. As such, all of the same limitations with snapshotting performance regarding fork time, memory use, and so on still stand when using append-only files. But even worse, because AOFs can grow to be many times the size of a dump (if left uncontrolled), when the AOF is rewritten, the OS needs to delete the AOF, which can cause the system to hang for multiple seconds while it’s deleting an AOF of tens of gigabytes.

There are also two configuration options that enable and customize the automatic BGREWRITEAOF execution:

auto-aof-rewrite-percentage
auto-aof-rewrite-min-size

For example, if we use auto-aof-rewritepercentage 100 and auto-aof-rewrite-min-size 64mb, when AOF is enabled, Redis will initiate a BGREWRITEAOF when the AOF is at least 100% larger than it was when Redis last finished rewriting the AOF, and when the AOF is at least 64 megabytes in size. As a point of configuration, if our AOF is rewriting too often, we can increase the 100 that represents 100% to something larger, though it will cause Redis to take longer to start up if it has been a while since a rewrite happened.

Replication

Replication is a method by which other servers receive a continuously updated copy of the data as the main server is being written, so that the replicas can service read queries. In the relational database world, it’s not uncommon for a single master database to send writes out to multiple slaves, with the slaves performing all of the read queries. Redis has adopted this method of replication as a way of helping to scale, and this section will discuss configuring replication in Redis, and how Redis operates during replication.

With a master/slave setup, instead of connecting to the master for reading data, clients will connect to one of the slaves to read their data (typically choosing them in a random fashion to try to balance the load).

Configuring Master and Slave Server

To configure replication on the master side of things, we only need to ensure that the path and filename listed under the dir and dbfilename configuration options shown below are to a path and file that are writable by the Redis process.
1
2
3
4
# inside the redis.conf file

dbfilename dump.rdb # also the default
dir ./
Though a variety of options control behavior of the slave itself, only one option is really necessary to enable slaving: slaveof. If we were to set slaveof host port in our configuration file, the Redis that’s started with that configuration will use the provided host and port as the master Redis server it should connect to. If we have an already running system, we can tell a Redis server to stop slaving, or even to slave to a new or different master. To connect to a new master, we can use the SLAVEOF host port command, or if we want to stop updating data from the master, we can use SLAVEOF no one.

`Master`/`Slave` startup process

These are the operations that occur on both the master and slave when a slave connects to a master.

Step	Master operations	Slave operations
1	(waiting for a command)	(Re-)connects to the `master`; issues the `SYNC` command
2	Starts `BGSAVE` operation; keeps a backlog of all write commands sent after `BGSAVE`	Serves old data (if any), or returns errors to commands (depending on configuration)
3	Finishes `BGSAVE`; starts sending the snapshot to the slave; continues holding a backlog of write commands	Discards all old data (if any); starts loading the dump as it’s received
4	Finishes sending the snapshot to the slave; starts sending the write command backlog to the slave	Finishes parsing the dump; starts responding to commands normally again
5	Finishes sending the backlog; starts live streaming of write commands as they happen	Finishes executing backlog of write commands from the master; continues executing commands as they happen

So, Redis manages to keep up with most loads during replication, except in cases where network bandwidth between the master and slave instances isn’t fast enough, or when the master doesn’t have enough memory to fork and keep a backlog of write commands. Though it isn’t necessary, it’s generally considered to be a good practice to have Redis masters only use about 50–65% of the memory in our system, leaving approximately 30–45% for spare memory during BGSAVE and command backlogs.

Note:

Just to make sure that we’re all on the same page (some users forget this the first time they try using slaves): when a slave initially connects to a master, any data that had been in memory will be lost, to be replaced by the data coming from the master.

But what happens when you have more than one slave? When another slave connects to an existing master, sometimes it can reuse an existing dump file (if there is). In general, only one of the two scenario will happen:

When additional slaves connect	Master operation
Before step 3 in table above	All slaves will receive the same dump and same backlogged write commands.
On or after step 3 in table above	While the master is finishing up the five steps for earlier slaves, a new sequence of steps 1-5 will start for the new slave(s).

`Master`/`Slave` Chain

Some developers have found that when they need to replicate to more than a handful of slaves, some networks are unable to keep up — especially when replication is being performed over the internet or between data centers. Because there’s nothing particularly special about being a master or a slave in Redis, slaves can have their own slaves, resulting in master/slave chaining.

In general, when read load significantly outweighs write load, and when the number of reads pushes well beyond what a single Redis server can handle, it’s common to keep adding slaves to help deal with the load. As load continues to increase, we can run into situations where the single master can’t write to all of its slaves fast enough, or is overloaded with slaves reconnecting and resyncing. To alleviate such issues, we may want to set up a layer of intermediate Redis master/slave nodes:

Picture from https://redislabs.com/ebook/part-2-core-concepts/chapter-4-keeping-data-safe-and-ensuring-performance/4-2-replication/4-2-3-masterslave-chains/

Verifying Disk Writes

Verifying that the data we wrote to the master made it to the slave is easy: we merely need to write a unique dummy value after our important data, and then check for it on the slave. But verifying that the data made it to disk is more difficult. If we wait at least one second, we know that our data made it to disk. But if we’re careful, we may be able to wait less time by checking the output of INFO for the value of aof_pending_bio_fsync, which will be 0 if all data that the server knows about has been written to disk.

Now, if we know this, we could write a program to automate the check for us:

def wait_for_sync(mconn, sconn):
   identifier = str(uuid.uuid4())

   time = time.time()
   mconn.zadd('sync:wait', identifier, time)  # Add a token to the master to verify slave data later

   while not sconn.INFO()['master_link_status'] != 'up':  # Wait for the slave to sync (if necessary).

      time.sleep(.001)

   while sconn.zscore('sync:wait', identifier) != time:  # Wait for the slave to receive the data change.

      time.sleep(.001)

   deadline = time.time() + 1.01

   while time.time() < deadline:  # Loops up to one second. 
			                      # After a second, the AOF will be completed anyway

      if sconn.INFO()['aof_pending_bio_fsync'] == 0:  # Check to see if the data is known to be on disk.
         break

      time.sleep(.001)

   mconn.zrem('sync:wait', identifier)
   mconn.zremrangebyscore('sync:wait', 0, time.time()-900)  # Clean up our status by 
															# cleaning out older entries that may have been left there

Here we see the use of the INFO command. In general, the INFO command can offer a wide range of information about the current status of a Redis server—memory used, the number of connected clients, the number of keys in each database, the number of commands executed since the last snapshot, and more. Generally speaking, INFOis a good source of information about the general state of our Redis servers, and many resources online can explain more.

Fixing `Snapshot`s and `AOF`s

We have covered how to replicate to handle redis data storage. However, we have not yet covered how to fix the Dumps or the AOFs themselves if there is a system failure.

When confronted with system failures, we have tools to help us recover when either snapshotting or append-only file logging had been enabled. Redis includes two command-line applications for testing the status of a snapshot and an append-only file. These commands are redis-check-aof and redis-check-dump. If we run either command without arguments (not in the redis-cli, but in command-line), we’ll see the basic help that’s provided:

$ redis-check-aof
Usage: redis-check-aof [--fix] <file.aof>
$ redis-check-dump
Usage: redis-check-dump <dump.rdb>
$

If we provide –fix as an argument to redis-check-aof, the command will fix the file. Its method to fix an append-only file is simple: it scans through the provided AOF, looking for an incomplete or incorrect command. Upon finding the first bad command, it trims the file to just before that command would’ve been executed. For most situations, this will discard the last partial write command. However, it is possible that all the correct commands after that first bad command will be lost due to this process.
Unfortunately, there’s no currently supported method of repairing a corrupted snapshot. Though there’s the potential to discover where the first error had occurred, because the snapshot itself is compressed, an error partway through the dump has the potential to make the remaining parts of the snapshot unreadable. It’s for these reasons that I’d generally recommend keeping multiple backups of important snapshots, and calculating the SHA1 or SHA256 hashes to verify content during restoration.

Now, after we’ve verified that our backups are what we had saved before, and we’ve corrected the last write to AOF as necessary, the last step is for us to replace a Redis server with that corrected file.

Replacing a Failed Master

Suppose machine A is running a copy of Redis that’s acting as the master, and machine B is running a copy of Redis that’s acting as the slave. Unfortunately, machine A has just lost network connectivity for some reason that we haven’t yet been able to diagnose. But we have machine C with Redis installed that we’d like to use as the new master.

Note:

If we cannot access machine A, then usually we cannot access and correct the AOF. However, if we could, then we could simply use the correct AOF file we just fixed in the last section to become the AOF file for Machine C. Then, we set machine B to be the slave of machine C, and we finish.

In this case, we can also tell machine B to produce a fresh snapshot with SAVE. We’ll then copy that snapshot over to machine C. After the snapshot has been copied into the proper path, we’ll start Redis on machine C. Finally, we’ll tell machine B to become a slave of machine C.

user@vpn-master ~:$ ssh root@machine-b.vpn
Last login: Wed Mar 28 15:21:06 2012 from ...
Connect to machine B on our VPN network.

root@machine-b ~:$ redis-cli  # Start up the commandline redis client to do a few simple operations.

redis 127.0.0.1:6379> SAVE
OK
redis 127.0.0.1:6379> QUIT  # Start a SAVE, and when it’s done, QUIT so that we can continue.

root@machine-b ~:$ scp +
> /var/local/redis/dump.rdb machine-c.vpn:/var/local/redis/  # Copy the snapshot over to the new master, machine C.
dump.rdb                  100%      525MB   8.1MB/s   01:05

root@machine-b ~:$ ssh machine-c.vpn
Last login: Tue Mar 27 12:42:31 2012 from ...

root@machine-c ~:$ sudo /etc/init.d/redis-server start
Starting Redis server...
Connect to the new master and start Redis.

root@machine-c ~:$ exit
root@machine-b ~:$ redis-cli
redis 127.0.0.1:6379> SLAVEOF machine-c.vpn 6379
OK

redis 127.0.0.1:6379> QUIT
root@machine-b ~:$ exit
user@vpn-master ~:$

As an alternative to creating a new master, we may want to turn the slave into a master and create a new slave. Either way, Redis will be able to pick up where it left off, and our only job from then on is to update our client configuration to read and write to the proper servers, and optionally update the on-disk server configuration if we need to restart Redis.

REDIS SENTINEL

A relatively recent addition to the collection of tools available with Redis is Redis Sentinel. By the final publishing of this manuscript, Redis Sentinel should be complete. Generally, Redis Sentinel pays attention to Redis masters and the slaves of the masters and automatically handles failover if the master goes down. For more details, please visit this link.

Redis Transactions

(This section builds on from the section Basic Redis Transactions)

Because of Redis’s delaying execution of commands until EXEC is called when using MULTI/EXEC, many clients (including the Python client that we’re using) will hold off on even sending commands until all of them are known. When all of the commands are known, the client will send MULTI, followed by the series of commands to be executed, and EXEC, all at the same time. The client will then wait until all of the replies from all of the commands are received. This method of sending multiple commands at once and waiting for all of the replies is generally referred to as pipelining, and has the ability to improve Redis’s performance when executing multiple commands by reducing the number of network round trips that a client needs to wait for.

However, there could be a potential problem of error: since MULTI holds the command back before executing, if some data has been changed (e.g. deleted) in the redis server, then there could be an error or data corruption when you execute your EXEC.

Consider the example of a Fake Game Company has seen major growth in their webbased RPG that’s played on YouTwitFace, a fictional social network. Because it pays attention to the needs and desires of its community, it has determined that the players need the ability to buy and sell items in a marketplace. It’s our job to design and build a marketplace that can scale to the needs of the community.

Posting Items in a Marketplace

Our requirements for the market are simple: a user can list an item for a given price, and when another user purchases the item, the seller receives the money. We’ll also say that the part of the market we’ll be worrying about only needs to be ordered by selling price. In a later chapter, we’ll cover some topics for handling other orders.

Our basic marketplace that includes an ItemA being sold by user 4 for 35 e-dollars. To include enough information to sell a given item in the market, we’ll concatenate the item ID for the item with the user ID of the seller and use that as a member of a market ZSET, with the score being the item’s selling price. By including all of this information together, we greatly simplify our data structures and what we need to look up, and get the benefit of being able to easily paginate through a presorted market.

Picture from https://redislabs.com/ebook/part-2-core-concepts/chapter-4-keeping-data-safe-and-ensuring-performance/4-4-redis-transactions/4-4-1-defining-users-and-their-inventory/

Now, we need to deal with the code part.

In the process of listing, we’ll use a Redis operation called WATCH, which we combine with MULTI and EXEC, and sometimes UNWATCH or DISCARD. When we’ve watched keys with WATCH, if at any time some other client replaces, updates, or deletes any keys that we’ve WATCHed before we have performed the EXEC operation, our operations against Redis will fail with an error message when we try to EXEC (at which point we can retry or abort the operation). By using WATCH, MULTI/EXEC, and UNWATCH/DISCARD, we can ensure that the data that we’re working with doesn’t change while we’re doing something important, which protects us from data corruption.

Basically, WATCH works by making the EXEC command conditional; in other words, Redis will only perform a transaction if the WATCHed keys were not modified (otherwise it returns nil no matter what commands you queued). If a WATCHed key was indeed modified, the transaction won’t be entered at all.

The WATCH command can be called numerous times (before the MULTI command), and one WATCH call can involve any number of keys. Watch calls simply monitor for changes starting from the point where WATCH was called until EXEC is called. Once EXEC is invoked, all the keys will be UNWATCHed, whether or not the transaction in question was aborted. Closing a client connection also triggers all keys to be UNWATCHed.

Note:

While UNWATCH is obvious as it sounds, DISCARD needs more explaination. In the same way that UNWATCH will let us reset our connection if sent after WATCH but before MULTI, DISCARD will also reset the connection if sent after MULTI but before EXEC. That is to say, if we’d WATCHed a key or keys, fetched some data, and then started a transaction with MULTI followed by a group of commands, we could cancel the WATCH and clear out any queued commands with DISCARD. We don’t use DISCARD here, primarily because we know whether we want to perform a MULTI/EXEC or UNWATCH, so a DISCARD is unnecessary for our purposes.

Therefore, all we need to do for listing/posting those items would be to add the item to the market ZSET, while WATCHing the seller’s inventory to make sure that the item is still available to be sold.

def list_item(conn, itemid, sellerid, price):
   inventory = "inventory:%s"%sellerid
   item = "%s.%s"%(itemid, sellerid)
   end = time.time() + 5
   pipe = conn.pipeline()

   while time.time() < end:
      try:
         conn.watch(inventory)  # Watch for changes to the user’s inventory.

         if not conn.sismember(inventory, itemid):  # Verify that the user still has the item to be listed.

            conn.unwatch()  # If the item isn’t in the user’s inventory, stop watching the inventory key and return.

            return None

         pipe.multi()
         pipe.zadd("market:", item, price)
         pipe.srem(inventory, itemid)  # Actually list the item.

         pipe.execute()  # If execute returns without a WatchError being raised, 
						 # then the transaction is complete and the inventory key is no longer watched.

         return True
      except redis.exceptions.WatchError:
         pass  # The user’s inventory was changed; retry the loop.

   return False

Generally, listing an item should occur without any significant issue, since only the user should be selling their own items (which is enforced farther up the application stack). But as mentioned before, if a user’s inventory were to change between the WATCH and EXEC, our attempt to list the item would fail, and we’d retry.

Purchasing an Item

To process the purchase of an item, we need to:

first WATCH the market and the user who’s buying the item.
then fetch the buyer’s total funds and the price of the item
verify that the buyer has enough money. If they don’t have enough money, we cancel the transaction.
if they do have enough money, we perform the transfer of money between the accounts, move the item into the buyer’s inventory, and remove the item from the market.
on WATCH error, we retry for up to 10 seconds in total.

def purchase_item(conn, buyerid, itemid, sellerid, lprice):
   buyer = "users:%s"%buyerid
   seller = "users:%s"%sellerid
   item = "%s.%s"%(itemid, sellerid)
   inventory = "inventory:%s"%buyerid
   end = time.time() + 10
   pipe = conn.pipeline()

   while time.time() < end:
      try:
         conn.watch("market:", buyer)  # Watch for changes to the market and to the buyer’s account information.

         price = conn.zscore("market:", item)
         funds = int(conn.hget(buyer, "funds"))
         if price != lprice or price > funds:  # Check for a sold/repriced item or insufficient funds.
            conn.unwatch()  

            return None

		 # Transfer funds from the buyer to the seller, and transfer the item to the buyer.
         pipe.multi()
         pipe.hincrby(seller, "funds", int(price))
         pipe.hincrby(buyer, "funds", int(-price))
         pipe.sadd(inventory, itemid)
         pipe.zrem("market:", item)
         pipe.execute()  

         return True
      except redis.exceptions.WatchError:
         pass  # Retry if the buyer’s account or the market changed.

   return False

Q: Why can’t the data be simply locked in Redis?

When accessing data for writing (SELECT FOR UPDATE in SQL), relational databases will place a lock on rows that are accessed until a transaction is completed with COMMIT or ROLLBACK. If any other client attempts to access data for writing on any of the same rows, that client will be blocked until the first transaction is completed. This form of locking works well in practice (essentially all relational databases implement it), though it can result in long wait times for clients waiting to acquire locks on a number of rows if the lock holder is slow.

Because there’s potential for long wait times, and because the design of Redis minimizes wait time for clients (except in the case of blocking LIST pops), Redis doesn’t lock data during WATCH. Instead, Redis will notify clients if someone else modified the data first, which is called optimistic locking (the actual locking that relational databases perform could be viewed as pessimistic). Optimistic locking also works well in practice because clients are never waiting on the first holder of the lock; instead they retry if some other client was faster.

Non-transactional Pipeline

In the python redis libraray, you can actually put arguments in the pipeline() function:

1	pipe = conn.pipeline(True) # this is the same as conn.pipeline()

And as we have seen before, this will wrap the commands with MULTI/EXEC, so that all commands will execute in order in a transactional way. However, we could also pass in the argument False:

1	pipe = conn.pipeline(False)

This will not wrap the commands with MULTI/EXEC, but it will have an effect of executing multiple commands at once in the redis server, and thereby reduce the time taken for roundtrips. You can also think of these as comamnds with a multiple arguments, like MGET, MSET, HMGET, HMSET, RPUSH/LPUSH, SADD, ZADD, and others (but the performance improvement is perhaps not as drastic as these commands).

However, this does means that it is appropriate only when we are sure the result of one command doesn’t affect the input to another, and we don’t need them all to execute transactionally. However, we are sure that it will reduce the amount of round trips between the client and the server (to one in the example below), so pipeline(False) method can further improve overall Redis performance.

For example:

def update_token(conn, token, user, item=None):
   timestamp = time.time()

   conn.hset('login:', token, user)

   conn.zadd('recent:', token, timestamp)

   if item:
      conn.zadd('viewed:' + token, item, timestamp)

      conn.zremrangebyrank('viewed:' + token, 0, -26)

      conn.zincrby('viewed:', item, -1)

could be replaced by:

def update_token_pipeline(conn, token, user, item=None):
   timestamp = time.time()
   pipe = conn.pipeline(False)

   pipe.hset('login:', token, user)  # these two commands don't have to be executed in transaction
   pipe.zadd('recent:', token, timestamp)
   if item:
      pipe.zadd('viewed:' + token, item, timestamp)
      pipe.zremrangebyrank('viewed:' + token, 0, -26)  # those technically do
      pipe.zincrby('viewed:', item, -1)

   pipe.execute()  # execute the above commands but there is no lock

Redis Performance and Benchmarking

Improving performance in Redis requires having an understanding of what to expect in terms of performance for the types of commands that we’re sending to Redis. To get a better idea of what to expect from Redis, we’ll quickly run a **benchmark that’s included within Redis: redis-benchmark:

root@a93d918c25f7:/data# redis-benchmark -c 1 -q

PING_INLINE: 21353.83 requests per second
PING_BULK: 19845.21 requests per second
SET: 16220.60 requests per second
GET: 19334.88 requests per second
INCR: 16592.00 requests per second
LPUSH: 18621.97 requests per second
RPUSH: 19033.12 requests per second
LPOP: 18556.32 requests per second
RPOP: 18231.54 requests per second
SADD: 19264.11 requests per second
HSET: 15893.20 requests per second
SPOP: 17445.92 requests per second
LPUSH (needed to benchmark LRANGE): 16342.54 requests per second
LRANGE_100 (first 100 elements): 13616.56 requests per second
LRANGE_300 (first 300 elements): 9668.37 requests per second
LRANGE_500 (first 450 elements): 8774.24 requests per second
LRANGE_600 (first 600 elements): 7390.44 requests per second
MSET (10 keys): 15544.85 requests per second

The output of redis-benchmark shows a group of commands that are typically used in Redis, as well as the number of commands of that type that can be run in a single second. A standard run of this benchmark without any options will try to push Redis to its limit using 50 clients, but it’s a lot easier to compare performance of a single benchmark client against one copy of our own client (by specifying the 1 option, as in the command show in the example above), rather than many.

However, be careful if you are using the python redis library. When looking at the output of redis-benchmark, we must be careful not to try to directly compare its output with how quickly our application (e.g. python) performs. This is because redis-benchmark doesn’t actually process the result of the commands that it performs, which means that the results of some responses that require substantial parsing overhead aren’t taken into account. Generally, compared to redis-benchmark running with a single client, we can expect the Python Redis client to perform at roughly 50–60% of what redis-benchmark will tell us for a single client and for nonpipelined commands, depending on the complexity of the command to call.

Logging in Redis

In the world of Linux and Unix, there are two common logging methods. The first is logging to a file, where over time we write individual log lines to a file, and every once in a while, we write to a new file. Many thousands of pieces of software have been written do this (including Redis itself). But this method can run into issues because we have many different services writing to a variety of log files, each with a different way of rolling them over, and no common way of easily taking all of the log files and doing
something useful with them.

Running on TCP and UDP port 514 of almost every Unix and Linux server available is a service called syslog, the second common logging method. Syslog accepts log messages from any program that sends it a message and routes those messages to various on-disk log files, handling rotation and deletion of old logs. With configuration, it can even forward messages to other servers for further processing. As a service, it’s far more convenient than logging to files directly, because all of the special log file rotation and deletion is already handled for us.

Recent Logs

To keep a recent list of logs, we’ll LPUSH log messages to a LIST and then trim that LIST to a fixed size. Later, if we want to read the log messages, we can perform a simple LRANGE to fetch the messages. We’ll take a few extra steps to support different named log message queues and to support the typical log severity levels, but you can remove either of those in your own code if you need to. The code for writing recent logs to Redis is shown below:

SEVERITY = {
   logging.DEBUG: 'debug',
   logging.INFO: 'info',
   logging.WARNING: 'warning',
   logging.ERROR: 'error',
   logging.CRITICAL: 'critical',
}

def log_recent(conn, name, message, severity=logging.INFO, pipe=None):
   severity = str(SEVERITY.get(severity)).lower()

   destination = 'recent:%s:%s'%(name, severity)  # Create the key that messages will be written to.

   message = time.asctime() + ' ' + message  # Add the current time so that we know when the message was sent.

   pipe = pipe or conn.pipeline()
   pipe.lpush(destination, message)  # Add the latest message to the beginning of the log list.

   pipe.ltrim(destination, 0, 99)  # Trim the log list to only include the most recent 100 messages.

   pipe.execute()

Logging Common operations

The problem with the above code snippet is that it’s not very good at telling you whether any important messages were lost in the noise. By recording information about how often a particular message appears, you could then look through the messages ordered by how often they happened to help you determine what’s important.

def log_common(conn, name, message, severity=logging.INFO, timeout=5):
   severity = str(SEVERITY.get(severity)).lower()  # Handle the logging level.

   destination = 'common:%s:%s'%(name, severity)  # Set up the destination key for keeping recent logs.

   start_key = destination + ':start'  # Keep a record of the start of the hour for this set of messages.

   pipe = conn.pipeline()
   end = time.time() + timeout
   while time.time() < end:
      try:
         pipe.watch(start_key)  # We’ll watch the start of the hour key for changes 
                                # that only happen at the beginning of the hour.

         now = datetime.utcnow().timetuple()

         hour_start = datetime(*now[:4]).isoformat()  # Find the current start hour.

         existing = pipe.get(start_key)
         pipe.multi()  # Set up the transaction.

         if existing and existing < hour_start:  # If the current list of common logs is for a previous hour…

            pipe.rename(destination, destination + ':last')
            pipe.rename(start_key, destination + ':pstart')  # …move the old common log information to the archive.

            pipe.set(start_key, hour_start)  # Update the start of the current hour for the common logs.

         pipe.zincrby(destination, message)  # Actually increment our common counter.

         log_recent(pipe, name, message, severity, pipe)  # Call the log_recent() function to record these, 
                                                          # and rely on its call to execute().

         return
   except redis.exceptions.WatchError:
         continue

Locks in Redis

In the context of Redis, we’ve been using WATCH as a replacement for a lock, and we call it optimistic locking, because rather than actually preventing others from modifying the data, we’re notified if someone else changes the data before we do it ourselves.

With distributed locking, we have the same sort of acquire, operate, release operations, but instead of having a lock that’s only known by threads within the same process, or processes on the same machine, we use a lock that different Redis clients on different machines can acquire and release. When and whether to use locks or WATCH will depend on a given application; some applications don’t need locks to operate correctly, some only require locks for parts, and some require locks at every step.

Problems with Watch

Consider the market example that we solved before with WATCH and UNWATCH: we WATCH the seller’s inventory to make sure the item is still available, add the item to the market ZSET, and remove it from the user’s inventory; to purchase of an item, we WATCH the market and the buyer’s HASH.

This would work correctly, however, when the scale of operation increases, the amount of retries that happen becomes huge, which could affect the latency:

-	Listed items	Bought items	Purchase retries	Average wait per purchase
1 lister, 1 buyer	145,000	27,000	80,000	14ms
5 listers, 1 buyer	331,000	<200	50,000	150ms
5 listers, 5 buyers	206,000	<600	161,000	498ms

To get past this limitation and actually start performing sales at scale, we must make sure that we only list or sell one item in the marketplace at any one time. We do this by using a lock.

Building a Simple Lock

The first part of making sure that no other code can run is to acquire the lock. The natural building block to use for acquiring a lock is the SETNX command, which will only set a value if the key doesn’t already exist. We’ll set the value to be a unique identifier to ensure that no other process can get the lock, and the unique identifier we’ll use is a 128-bit randomly generated UUID.

If we fail to acquire the lock initially, we’ll retry until we acquire the lock, or until a specified timeout has passed, whichever comes first, as shown here.

def acquire_lock(conn, lockname, acquire_timeout=10):
   identifier = str(uuid.uuid4())# A 128-bit random identifier.

   end = time.time() + acquire_timeout
   while time.time() < end:
      if conn.setnx('lock:' + lockname, identifier):  # Get the lock.

         return identifier

      time.sleep(.001)  # sleep and retries

   return False  # failed to get the lock within the amount of time

Now that we have the lock, we can perform our buying or selling without WATCH errors getting in our way. We’ll acquire the lock and, just like before, check the price of the item, make sure that the buyer has enough money, and if so, transfer the money and item. When completed, we release the lock.

def purchase_item_with_lock(conn, buyerid, itemid, sellerid):
   buyer = "users:%s"%buyerid
   seller = "users:%s"%sellerid
   item = "%s.%s"%(itemid, sellerid)
   inventory = "inventory:%s"%buyerid
   end = time.time() + 30

   lock_obtained = acquire_lock(conn, market)  # Get the lock.

   if !lock_obtained:
       return False
   else:
	   pipe = conn.pipeline(True)
	   try:
	      while time.time() < end:
	         try:
	            pipe.watch(buyer)
	
	            pipe.zscore("market:", item)
	            pipe.hget(buyer, 'funds')
	            price, funds = pipe.execute()
	            if price is None or price > funds:  # Check for a sold item or insufficient funds.
	               pipe.unwatch()
	               return None
	
	            pipe.hincrby(seller, int(price))
	            pipe.hincrby(buyerid, int(-price))
	            pipe.sadd(inventory, itemid)
	            pipe.zrem("market:", item)
	            pipe.execute()  # Transfer funds from the buyer to the seller, and transfer the item to the buyer.
	
	            return True
	         except redis.exceptions.WatchError:
	            pass
		     finally:
		        release_lock(conn, market, lock_obtained)

Between the time when we acquired the lock and when we’re trying to release it, someone may have done bad things to the lock. To release the lock, we need to WATCH the lock key, and then check to make sure that the value is still the same as what we set it to before we delete it. This also prevents us from releasing a lock multiple times. The release_lock() function is shown next.

def release_lock(conn, lockname, identifier):
   pipe = conn.pipeline(True)
   lockname = 'lock:' + lockname

   while True:
      try:
         pipe.watch(lockname)
         if pipe.get(lockname) == identifier:  # Check and verify that we still have the lock.

            pipe.multi()
            pipe.delete(lockname)
            pipe.execute()
            return True  # Release the lock.

         pipe.unwatch()
         break

      except redis.exceptions.WatchError:  # Someone else did something with the lock; retry.
										   # this is actually kind of rare
         pass

   return False

Note:

For simplicity, we performed lock on the entire market data, which also means that listing an item could be blocked. However, if we replace the market-level lock with one specific to the item to be bought or sold, we can reduce lock contention and increase performance.

Locks with timeouts

The above locks still had one important problem: it doesn’t handle cases where a lock holder crashes without releasing the lock, or when a lock holder fails and holds the lock forever. To handle the crash/failure cases, we add a timeout to the lock.

In order to give our lock a timeout, we’ll use EXPIRE to have Redis time it out automatically.

The natural place to put the EXPIRE is immediately after the lock is acquired, and we’ll do that. But if our client happens to crash (and the worst place for it to crash for us is between SETNX and EXPIRE), we still want the lock to eventually time out. To handle that situation, any time a client fails to get the lock, the client will check the expiration on the lock, and if it’s not set, set it. This will make sure that the lock will always have a timeout, and will eventually expire, letting other clients get a timed-out lock.

def acquire_lock_with_timeout(conn, lockname, acquire_timeout=10, lock_timeout=10):
   identifier = str(uuid.uuid4())

   lock_timeout = int(math.ceil(lock_timeout))  # Only pass integers to our EXPIRE calls.

   end = time.time() + acquire_timeout
   while time.time() < end:
      if conn.setnx(lockname, identifier):
         conn.expire(lockname, lock_timeout)  # Get the lock and set the expiration.

         return identifier

      elif not conn.ttl(lockname):  # Check and update the expiration time as necessary.
         conn.expire(lockname, lock_timeout)

      time.sleep(.001)

   return False

This new acquire_lock_with_timeout() handles timeouts. It ensures that locks expire as necessary, and that they won’t be stolen from clients that rightfully have them. Even better, we were smart with our release lock function earlier, which still works.

Note:

As of Redis 2.6.12, *the SET command added options to support a combination of SETNX and SETEX functionality, which makes our lock acquire function trivial. We still need the complicated release lock to be correct. In python, this looks like: set(name, value, ex=None, px=None, nx=False, xx=False, keepttl=False)

Building a Semaphore

Sometimes, we need to use a semaphore to hold the ocks, instead of having a timeout lock shown above for a single use.

When building a counting semaphore, we run into many of the same concerns we had with other types of locking. We must decide who got the lock, how to handle processes that crashed with the lock, and how to handle timeouts. If we don’t care about timeouts, or handling the case where semaphore holders can crash without releasing semaphores.

In almost every case where we want to deal with timeouts in Redis, we’ll generally look to one of two different methods. Either we’ll use EXPIRE like we did with our standard locks, or we’ll use ZSETs. In this case, we want to use ZSETs, because that allows us to keep information about multiple semaphore holders in a single structure.

Now, to build a semaphore, we will have: for each process that attempts to acquire the semaphore, we’ll generate a unique identifier. This identifier will be the member of a ZSET. For the score, we’ll use the timestamp for when the process attempted to acquire the semaphore. When a process wants to attempt to acquire a semaphore, it first generates an identifier, and then the process adds the identifier to the ZSET using the current timestamp as the score. After adding the identifier, the process then checks for its identifier’s rank. If the rank returned is lower than the total allowed count (Redis uses 0-indexing on rank), then the caller has acquired the semaphore. Otherwise, the caller doesn’t have the semaphore and must delete its identifier from the ZSET.

def acquire_semaphore(conn, semname, limit, timeout=10):
   identifier = str(uuid.uuid4())  # A 128-bit random identifier.

   now = time.time()

   pipeline = conn.pipeline(True)
   pipeline.zremrangebyscore(semname, '-inf', now - timeout)  # Time out old semaphore holders.

   pipeline.zadd(semname, identifier, now)  # Try to acquire the semaphore.

   pipeline.zrank(semname, identifier)
   if pipeline.execute()[-1] < limit:  # Check to see if we have it.

      return identifier  # got the semaphore

   conn.zrem(semname, identifier)  # We failed to get the semaphore; discard our identifier.

   return None

Releasing the semaphore is easy: we remove the identifier from the ZSET, as can be
seen in the next listing.

1 2	def release_semaphore(conn, semname, identifier): return conn.zrem(semname, identifier)

However, there is a problem in this design as well. Since it is relying on every process having access to the same system time in order to get the semaphore can cause problems if we have multiple hosts. This isn’t a huge problem for our specific use case, but if we had two systems A and B, where A ran even 10 milliseconds faster than B, then if A got the last semaphore, and B tried to get a semaphore within 10 milliseconds, B would actually “steal” A’s semaphore without A knowing it.

Any time we have a lock or a semaphore where such a slight difference in the system clock can drastically affect who can get the lock, the lock or semaphore is considered unfair. Unfair locks and semaphores can cause clients that should’ve gotten the lock or semaphore to never get it, and this is something that we’ll fix in the next section.

Fair Semaphore

In order to minimize problems with inconsistent system times, we’ll add a counter (a String) and a second ZSET. The counter creates a steadily increasing timer-like mechanism that ensures that whoever incremented the counter first should be the one to get the semaphore.

We then enforce our requirement that clients that want the semaphore who get the counter first also get the semaphore inside an “owner” ZSET with the counter-produced value as the score, checking our identifier’s rank in the new ZSET to determine which client got the semaphore.

Picture from https://redislabs.com/ebook/part-2-core-concepts/chapter-6-application-components-in-redis/6-3-counting-semaphores/6-3-2-fair-semaphores/

We continue to handle timeouts the same way as our basic semaphore, by removing entries from the system time ZSET. We propagate those timeouts to the new owner ZSET by the use of ZINTERSTORE and the WEIGHTSargument.

def acquire_fair_semaphore(conn, semname, limit, timeout=10):
   identifier = str(uuid.uuid4())  # A 128-bit random identifier.

   czset = semname + ':owner'
   ctr = semname + ':counter'

   now = time.time()
   pipeline = conn.pipeline(True)
   pipeline.zremrangebyscore(semname, '-inf', now - timeout)  # Time out old entries.
   pipeline.zinterstore(czset, {czset: 1, semname: 0})  # since they have the same UUID member name
											  		    # weights for czset is 1, semanme is 0

   pipeline.incr(ctr)
   counter = pipeline.execute()[-1]  # Get the counter.

   pipeline.zadd(semname, identifier, now)
   pipeline.zadd(czset, identifier, counter)

   pipeline.zrank(czset, identifier)  # Try to acquire the semaphore.
   if pipeline.execute()[-1] < limit:  # Check the rank to determine if we got the semaphore.

      return identifier  # We got the semaphore.

   pipeline.zrem(semname, identifier)
   pipeline.zrem(czset, identifier)  # We didn’t get the semaphore; clean out the bad data.

   pipeline.execute()
   return None

Releasing the semaphore is almost as easy as before, only now we remove our identifier from both the owner and timeout ZSETs, as can be seen in this next listing.

def release_fair_semaphore(conn, semname, identifier):
   pipeline = conn.pipeline(True)
   pipeline.zrem(semname, identifier)
   pipeline.zrem(semname + ':owner', identifier)
   return pipeline.execute()[0]

Redis Intro

Data Structures in Redis

Strings in Redis - Basics

Lists in Redis - Basics

Sets in Redis - Basics

Hashes in Redis - Basics

Sorted sets/Zsets in Redis - Basics

Connecting a Python Standalone Container to Redis

Antonomy of a Redis Web Application

Login and Cookie Caching

Storing Shopping Carts Info

Webpage Caching

Database row caching

Simple Webpage Analytics

Strings in Redis

Lists in Redis

Sets in Redis

Hashes in Redis

ZSets in Redis

Publish/Subscribe in Redis

Operations in Redis - Sorting

Basic Redis Transactions

Key Expiration

Persistence in Redis - Snapshot

Persistence in Redis - Append-only file

Configuring/Compacting your Append-only Files

Replication

Configuring Master and Slave Server

Master/Slave startup process

Master/Slave Chain

Verifying Disk Writes

Fixing Snapshots and AOFs

Replacing a Failed Master

Redis Transactions

Posting Items in a Marketplace

Purchasing an Item

Non-transactional Pipeline

Redis Performance and Benchmarking

Logging in Redis

Recent Logs

Logging Common operations

Locks in Redis

Problems with Watch

Building a Simple Lock

Locks with timeouts

Building a Semaphore

Fair Semaphore

`String`s in Redis - Basics

`List`s in Redis - Basics

`Set`s in Redis - Basics

`Hash`es in Redis - Basics

`Sorted set`s/`Zset`s in Redis - Basics

`String`s in Redis

`List`s in Redis

`Set`s in Redis

`Hash`es in Redis

`ZSet`s in Redis

`Publish`/`Subscribe` in Redis

`Master`/`Slave` startup process

`Master`/`Slave` Chain

Fixing `Snapshot`s and `AOF`s