How to Hash Files in Python
Hashing files allows us to generate a string/byte sequence that can help identify a file. This can then be used by comparing the hashes of two or more files to see if these files are the same as well as other applications.
Note: This post assumes you already know what a hash is, if you don’t, read up on hashes before reading this post. Sites like this or this can help you out.
What Does it Mean to Hash a File?
Hashing a file is when a file of arbitrary size is read and used in a function to compute a fixed-length value from it. This fixed-length value can help us get a sort of ‘id’ of a file which can then help us do particular tasks (examples follow).
Strong hashes with large amounts of bytes can help distinguish between many different files, but we do always need to remember the birthday problem. For example, the birthday problem reminds us that if we have a 160bit hash, there can only be 2^160 different hashes, meaning that as soon as we generate 2^160 + 1 hashes of different files, it is guaranteed that we find a duplicate hash. This duplicate hash means we get the same hash for different files which can cause issues in some applications.
What Can I Do With A File Hash?
File hashes are quite useful as they can represent (not substitute) a file, meaning that you do not have to store the whole file when trying to identify a file. When hashing files (or anything in general), you will get the same result hash result every time you hash a particular file. This makes hashing files useful for things like:
- Comparing files: Instead of comparing whole files, take hashes and compare hashes together. This is particularly efficient when comparing more than one file to another file.
- Easily identifying files without storing the whole file: If you are looking for a file, simply get the hash of your current file and hash other files as you look at them while looking for a match. This technique is used by some anti-virus software.
- Stopping filename clashes: Rename files where many files are located in one directory to their hash. This will mean two different files will not have the same name and will save on space when two files have the same name as they will overwrite (does not work if you need two of the same file).
- Object keys: Just like in the point «Stopping filename clashes», using hashes as object keys can help identify what objects match to which file.
- Detecting changes in a file: If you had a file hash before it was modified, you can re-hash the file and compare it against the original hash to see if it has been modified.
What Can I Not Do With A File Hash?
Even though you can compute a hash using a file, this does not mean you cannot get the original file back using this hash. Hashing is a one-way function (lossy) and is not an encryption scheme.
Supported Hashing Algorithms in Python
The hashlib Python module «implements a common interface to many different secure hash and message digest algorithms». To look at the hashing algorithms Python offers, execute:
import hashlib print(hashlib.algorithms_guaranteed) This will print a set of strings that are hash algorithms guaranteed to be supported by this module on all platforms. To use these, select a hashing algorithm from the set and then use it as shown below (this example uses sha256):
import hashlib h = hashlib.sha256() # Construct a hash object using our selected hashing algorithm h.update('My content'.encode('utf-8')) # Update the hash using a bytes object print(h.hexdigest()) # Print the hash value as a hex string print(h.digest()) # Print the hash value as a bytes object hashlib.algorithms_guaranteed vs hashlib.algorithms_available
To get all the algorithms available in your interpreter, you can execute hashlib.algorithms_available . Unlike the output from hashlib.algorithms_guaranteed , these hashes aren’t guaranteed to exist in interpreters on other machines. You can visit the docs to get more information on this.
To hash a file, read it in bit-by-bit and update the current hashing functions instance. When all bytes have been given to the hashing function in order, we can then get the hex digest.
import hashlib file = ".\myfile.txt" # Location of the file (can be set a different way) BLOCK_SIZE = 65536 # The size of each read from the file file_hash = hashlib.sha256() # Create the hash object, can use something other than `.sha256()` if you wish with open(file, 'rb') as f: # Open the file to read it's bytes fb = f.read(BLOCK_SIZE) # Read from the file. Take in the amount declared above while len(fb) > 0: # While there is still data being read from the file file_hash.update(fb) # Update the hash fb = f.read(BLOCK_SIZE) # Read the next block from the file print (file_hash.hexdigest()) # Get the hexadecimal digest of the hash This snippet will print the hash value of the file specified in the file generated using the SHA256 algorithm. The call .hexdigest() returns a string object containing only hexadecimal digits; you can use .digest() as shown before to get the bytes representation of the hash.
Why do I Need to Worry About the Buffer Size?
You would have noticed in the script above, the variable BLOCK_SIZE is set to 65536 . This is the number of bytes that is read into memory in a single read operation. This is used so larger files are not completely loaded into memory before computing the hash.
For example, if we did not do this and were hashing a video file that was 2Gb large, the whole 2Gb file would be loaded into memory (or at least tried to) and then hashed. This approach reads the file block-by-block so we don’t load the whole file into memory.
The buffer I have used is 64Kb but you can use any value you wish. Making this larger reads files faster, but in turn, loads more of the file into memory at once.
Owner of PyTutorials and creator of auto-py-to-exe. I enjoy making quick tutorials for people new to particular topics in Python and tools that help fix small things.
Create MD5 Hash of a file in Python
As a Python enthusiast, I’m always on the lookout for handy tools and techniques that can streamline my development process. One such technique that I find particularly useful is generating MD5 hashes of files. Whether you’re ensuring data integrity, verifying file integrity during transmission, or simply looking to add an extra layer of security, MD5 hashes can be invaluable. In this blog post, I’m excited to guide you through the process of creating an MD5 hash of a file in Python. So, let’s dive in and unlock the power of file hashing!
MD5 is (atleast when it was created) a standardized 1-way function that takes in data input of any form and maps it to a fixed-size output string, irrespective of the size of the input string.
Though it is used as a cryptographygraphic hash function, it has been found to suffer from a lot of vulnerabilities.
The hash function generates the same output hash for the same input string. This means that, you can use this string to validate files or text or anything when you pass it across the network or even otherwise. MD5 can act as a stamp or for checking if the data is valid or not.
| Input String | Output Hash |
|---|---|
| hi | 49f68a5c8493ec2c0bf489821c21fc3b |
| debugpointer | d16220bc73b8c7176a3971c7f73ac8aa |
| computer science is amazing! I love it. | f3c5a497380310d828cdfc1737e8e2a3 |
Check this out — If you are looking for MD5 hash of a String.
Create MD5 hash of a file in Python
MD5 hash can be created using the python’s default module hashlib .
Incorrect Way to create MD5 Hash of a file in Python
But, you have to note that you cannot create a hash of a file by just specifying the name of the file like this-
# this is NOT correct import hashlib print(hashlib.md5("filename.jpg".encode('UTF-8')).hexdigest()) 03e6eda992afdeda6b2acaed17722515 The above value is NOT the MD5 hash of the file. But, it is the MD5 hash of the string filename.jpg itself.
Correct Way to create MD5 Hash of a file in Python
You have to read the contents of the file to create MD5 hash of the file itself. It’s simple, we can just read the contents of the file and create the hash.
The process of creating an MD5 hash in python is very simple. First import hashlib, then encode your string that you want to hash i.e., converts the string into the byte equivalent using encode(), then pass it through the hashlib.md5() function. We print the hexdigest value of the hash m , which is the hexadecimal equivalent encoded string.
import hashlib file_name = 'filename.jpg' with open(file_name) as f: data = f.read() md5hash = hashlib.md5(data).hexdigest() MD5 Hash of Large Files in Python
In the above code, there is one problem. If the file is a 10 Gb file, let’s say a large log file or a dump of traffic or a Game like FIFA or others. If you want to compute MD5 hash of it, it would probably chew up your memory.
Here is a memory optimised way of computing MD5 hash, where we read chunks of 4096 bytes (can be customised as per your requirement, size of your system, size of your file etc.,). So, in this process we sequentially process the chunks and update the hash. So, in this process, let’s say there are 1000 such chunks of the file, the hash_md5 is updated 1000 times.
At the end we return the hexdigest value of the hash m , which is the hexadecimal equivalent encoded string.
import hashlib # A utility function that can be used in your code def compute_md5(file_name): hash_md5 = hashlib.md5() with open(file_name, "rb") as f: for chunk in iter(lambda: f.read(4096), b""): hash_md5.update(chunk) return hash_md5.hexdigest() Compare and Verify MD5 hash of a file using python
You need to verify the MD5 hash at the server or at some other point or logic in your code.
To verify the MD5 hash you will have to create the MD5 hash of the original file again.
Then compare the original MD5 value that the source has generated and MD5 that you generate.
import hashlib file_name = 'filename.jpg' original_md5 = '5d41402abc4b2a76b9719d911017c592' with open(file_name) as f: data = f.read() md5_returned = hashlib.md5(data).hexdigest() if original_md5 == md5_returned: print "MD5 verified." else: print "MD5 verification failed." The process of MD5 creation and verification is easy as we discussed above. Happy Coding!
NOTE : Please do not use this to hash passwords and store it in your databases, prefer SHA-256 or SHA-512 or other superior cryptographygraphic hash functions for the same.
I’m glad that you found the content useful. Happy Coding.
We’ve reached the end of our journey through the world of file hashing using MD5 in Python. I hope this exploration has empowered you with the knowledge and skills to incorporate this powerful technique into your own projects. The ability to generate MD5 hashes of files not only enhances data security but also provides a means to validate the integrity of files. As you continue your Python coding adventures, remember the importance of data integrity and the role that MD5 hashes can play in achieving it. Keep coding, keep exploring, and keep harnessing the power of Python!