Python redis get all keys

Get all keys in Redis database with python

scan_iter() is superior to keys() for large numbers of keys because it gives you an iterator you can use rather than trying to load all the keys into memory.

I had a 1B records in my redis and I could never get enough memory to return all the keys at once.

SCANNING KEYS ONE-BY-ONE

Here is a python snippet using scan_iter() to get all keys from the store matching a pattern and delete them one-by-one:

import redisr = redis.StrictRedis(host='localhost', port=6379, db=0)for key in r.scan_iter("user:*"): # delete the key r.delete(key)

SCANNING IN BATCHES

If you have a very large list of keys to scan — for example, larger than >100k keys — it will be more efficient to scan them in batches, like this:

import redisfrom itertools import izip_longestr = redis.StrictRedis(host='localhost', port=6379, db=0)# iterate a list in batches of size ndef batcher(iterable, n): args = [iter(iterable)] * n return izip_longest(*args)# in batches of 500 delete keys matching user:*for keybatch in batcher(r.scan_iter('user:*'),500): r.delete(*keybatch)

I benchmarked this script and found that using a batch size of 500 was 5 times faster than scanning keys one-by-one. I tested different batch sizes (3,50,500,1000,5000) and found that a batch size of 500 seems to be optimal.

Note that whether you use the scan_iter() or keys() method, the operation is not atomic and could fail part way through.

DEFINITELY AVOID USING XARGS ON THE COMMAND-LINE

I do not recommend this example I found repeated elsewhere. It will fail for unicode keys and is incredibly slow for even moderate numbers of keys:

redis-cli --raw keys "user:*"| xargs redis-cli del

In this example xargs creates a new redis-cli process for every key! that’s bad.

I benchmarked this approach to be 4 times slower than the first python example where it deleted every key one-by-one and 20 times slower than deleting in batches of 500.

Источник

Get All Keys in Redis Database With Python

1. Use keys() to Get All Keys in Redis Database
2. Use scan_iter() to Get All Keys in Redis Database
3. Use zip_longest to Get All Keys in Redis Database

The Redis design system will be familiar if you know JSON. It uses a key-value structure and a distributed and in-memory approach to achieve a resilient database.

Hashes, lists, sets, sorted sets, strings, JSON, and streams are one of the many data structures that Redis supports. This open-source database supports different languages, including Python, and if you are developing a backend system with it, some modules and packages help with that.

One of the many operations you would often do with databases is retrieving data, and in a database like Redis , the keys are important to achieve such an operation.

This article will discuss getting all the keys in a Redis database.

Use keys() to Get All Keys in Redis Database

To make use of redis , we need to have it installed; you can check through the Redis download page to find out how. For Linux and macOS users, it’s quite easier; however, for Windows users, you might have to make use of the Windows Subsystem for Linux (WSL2), and you can follow their instructions video guide.

With the assumption that you have your Redis database set up, we will install the redis package, which provides client access to the Redis database. To install it, we will use the pip command.

Collecting redis  Downloading redis-4.3.4-py3-none-any.whl (246 kB)  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 246.2/246.2 kB 794.4 kB/s eta 0:00:00 Collecting deprecated>=1.2.3  Downloading Deprecated-1.2.13-py2.py3-none-any.whl (9.6 kB) Collecting async-timeout>=4.0.2  Downloading async_timeout-4.0.2-py3-none-any.whl (5.8 kB) Collecting packaging>=20.4  Downloading packaging-21.3-py3-none-any.whl (40 kB)  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 1.9 MB/s eta 0:00:00 Collecting wrapt=1.10  Downloading wrapt-1.14.1-cp310-cp310-win_amd64.whl (35 kB) Collecting pyparsing!=3.0.5,>=2.0.2  Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 624.8 kB/s eta 0:00:00 Installing collected packages: wrapt, pyparsing, async-timeout, packaging, deprecated, redis Successfully installed async-timeout-4.0.2 deprecated-1.2.13 packaging-21.3 pyparsing-3.0.9 redis-4.3.4 wrapt-1.14.1 

So, the redis package uses wrapt , pyparsing , async-timeout , packaging , and deprecated modules to power its module.

To make use of the redis package, we need to import it.

Moving on to the main issue, we can use the keys() method provided by the redis module to access and get all the keys in.

The key() method returns a list of keys matching the pattern passed within its arguments from a given Redis database. There is a default pattern if no argument is not passed, which is * , which means all the keys.

To show the method at work, we have manually prepopulated the redis database with the alias Temp with some keys using the +Key button or the following code.

import redis  redisHost = 'localhost' redisPort = 6379 redisDecodeRes = True  r = redis.StrictRedis(  host=redisHost,  port=redisPort,  decode_responses=redisDecodeRes  )  r.set("ConnectionPool", "Ox1212af34w3141") 

The above code results in the key ConnectionPool being added to the Temp database with the corresponding value.

The set() method applies the key-value pair to the database, and the StrictRedis() method creates a Redis connection object that gives us access to the database.

To show the database (with the alias Temp ) and its keys via a GUI, we can use the RedisInsight application, as seen in the image.

A total of 11 keys have been added to the database manually to test the key() method.

Now, to the key() method, we will have to use the StrictRedis() method to create a Redis connection object to access the keys. The host , port , and decode_responses arguments are passed to define the parameters of the connection.

The host and port define the hostname and port number, and the decode_responses defines that the data passed is decoded to Python strings that we can easily work with. The keys() method then accesses all available keys since no argument isn’t passed.

import redis  redisHost = 'localhost' redisPort = 6379 redisDecodeRes = True  r = redis.StrictRedis(  host=redisHost,  port=redisPort,  decode_responses=redisDecodeRes,  db=0  )  print(r.keys()) 

['bar-two', 'information', 'bar-one', 'details', 'foo', 'jinku', 'bar', 'User-One', 'delft', 'bar-three', 'ConnectionPool'] 

We have a list of all the keys in the Temp database, which we can work with.

If we have a key pattern we need, we can pass it as an argument. Let’s list all the keys that start with bar .

['bar-two', 'bar-one', 'bar', 'bar-three'] 

Use scan_iter() to Get All Keys in Redis Database

With a large database, scan_iter() allows us to manage the data better within our Python application. Also, the key() method blocks the server and prevents other usage operations, and with scan_iter() , its batch-based operation allows other usage operations.

Though keys() might be faster, it is not great for multiple request-based systems.

Now, let’s see it in action.

import redis  redisHost = 'localhost' redisPort = 6379 redisDecodeRes = True  try:  r = redis.StrictRedis(  host=redisHost,  port=redisPort,  decode_responses=redisDecodeRes,  db=0  )  for key in r.scan_iter(match="bar*"):  print(key) except Exception as e:  print(e) 

bar-three bar-one bar-two bar 

Using the try/except helps deal with connection issues when we try to work with the database. After the connection using the StrictRedis() , we use the match argument to define the key pattern we are looking for and loop through the result to give the key.

Use zip_longest to Get All Keys in Redis Database

As we stated, for larger databases with lots of keys, the scan_iter() method is better, but we can improve it further by retrieving the keys in batches of a specified number to better manage the result.

To create the batch , we need the itertools module which provides different iterators (or methods) that can be used for different cases. Within the itertools module, we have the zip_longest method which returns a zip_longest object whose .next() method returns a tuple and aggregates elements from the iterable passed to it.

We can use the zip_longest() method to make a function that creates a batch of a specified number of keys depending on the arguments passed. For example, we create a batch of 2, which can be used for many cases.

import redis from itertools import zip_longest  redisHost = 'localhost' redisPort = 6379 redisDecodeRes = True  try:  r = redis.StrictRedis(  host=redisHost,  port=redisPort,  decode_responses=redisDecodeRes,  db=0  )   def batch(iterable, num):  initIter = [iter(iterable)] * num  return zip_longest(*initIter)   for keyBatch in batch(r.scan_iter('bar*'), 2):  print(keyBatch) except Exception as e:  print(e) 

('bar-three', 'bar-one') ('bar-two', 'bar') 

Olorunfemi is a lover of technology and computers. In addition, I write technology and coding content for developers and hobbyists. When not working, I learn to design, among other things.

Источник

How can I get all keys in redis in [2022]

Redis is a popular open-source,key-value type storage. Basically, redis saves the data in keys and value format. In this blog, we will learn how to get all keys in redis. so let’s get started.

Redis get the value of a key

If you are a beginner, then you can follow the below tutorial to install redis on your system based on the operating system.

Let’s insert the below 5 keys into the redis instance.

set my_key1 my_value1 set my_key2 my_value2 set my_key3 my_value3 set my_key4 my_value4 set my_key5 my_value5

To get a value of a single key user needs to use the GET command followed by the key_name.

The above query will return the value of my_key1

Get all keys in redis

Similarly to get all the keys from a redis instance user needs to use the pound Asterisk(*) sign with KEYS.

The above query returns all keys in the redis instance.

Get all keys in redis with the prefix

So far we have seen how to get all keys in redis.In this session, we will understand how to get keys using regex. Let’s insert two more keys into our redis instance.

set key1 value1 set key2 value2

Check all the keys in the redis instance

Now let’s assume you wish to get the keys which start from my_. You can use the below command to achieve the same.

Redis get all keys python

In this session, we will understand how to retrieve all the keys in redis using the python client. If you are a beginner you can follow my tutorial on redis python to install a python client to interact with redis.

Now users can refer to the below code to get all the keys in redis using python.

import redis r = redis.Redis(host='localhost', port=6379, db=0) r.keys()

Conclusion

I hope you have liked this small tutorial on redis get all keys. Please do let me know if you are facing any issues while following along.

Источник