Easiest way to persist a data structure to a file in python?
What’s the easiest way to progammatically get that into a file that I can load from python later? Can I somehow save it as python source (from within a python script, not manually!), then import it later? Or should I use JSON or something?
The easiest way would be JSON because it’s structuring data similar to Python dictionary. Luckily, python has a bundled JSON module. All you need to do is just import json .
7 Answers 7
import pickle d = < "abc" : [1, 2, 3], "qwerty" : [4,5,6] >afile = open(r'C:\d.pkl', 'wb') pickle.dump(d, afile) afile.close() #reload object from file file2 = open(r'C:\d.pkl', 'rb') new_d = pickle.load(file2) file2.close() #print dictionary object loaded from file print new_d The r» denotes a raw string, described here: docs.python.org/reference/lexical_analysis.html#string-literals. Basically, it means that backslashes in the string are included as literal backslashes, not character escapes (though a raw string can’t end in a backslash).
I’ve corrected the example—the file needs to be opened in binary mode. It still needs to be for Python 2, but it won’t fail as dramatically.
Make sure you read the Python documentation (including for the appropriate version) and don’t just rely on examples! 🙂 docs.python.org/3.0/library/pickle.html (Sorry for the comment spam!)
Technically pickling will work for text mode files, so long as you’re not using a binary pickle format (ie. protocol = 0) and you use it consistently (ie. also use text mode for reading back). Using binary is generally a better idea though, especially if you could be moving data between platforms.
Take your pick: Python Standard Library — Data Persistance. Which one is most appropriate can vary by what your specific needs are.
pickle is probably the simplest and most capable as far as «write an arbitrary object to a file and recover it» goes—it can automatically handle custom classes and circular references.
For the best pickling performance (speed and space), use cPickle at HIGHEST_PROTOCOL .
Try the shelve module which will give you persistent dictionary, for example:
import shelve d = < "abc" : [1, 2, 3], "qwerty" : [4,5,6] >shelf = shelve.open('shelf_file') for key in d: shelfPython save set to file = dPython save set to file shelf.close() . # reopen the shelf shelf = shelve.open('shelf_file') print(shelf) # => JSON has faults, but when it meets your needs, it is also:
- simple to use
- included in the standard library as the json module
- interface somewhat similar to pickle , which can handle more complex situations
- human-editable text for debugging, sharing, and version control
- valid Python code
- well-established on the web (if your program touches any of that domain)
JSON ain’t valid Python. It looks so, superficially, but use some bools and you’ll see the problem (JSON uses true and false, while Python uses True and False). Also: JSON arrays (dicts) only have string keys. So it doesn’t preserve the data structure correctly.
You also might want to take a look at Zope’s Object Database the more complex you get:-) Probably overkill for what you have, but it scales well and is not too hard to use.
Just to add to the previous suggestions, if you want the file format to be easily readable and modifiable, you can also use YAML. It works extremely well for nested dicts and lists, but scales for more complex data structures (i.e. ones involving custom objects) as well, and its big plus is that the format is readable.
If you want to save it in an easy to read JSON-like format, use repr to serialize the object and eval to deserialize it.
repr(object) -> string
Return the canonical string representation of the object. For most object types, eval(repr(object)) == object .
The main thing I don’t like about this solution is that you have an object in the structure where the eval(repr()) identity doesn’t hold, repr() will «succeed» but then eval() will barf.
pickle, YAML, JSON, etc. are all safer and work with more types than this method. IMO, eval() should be avoided whenever possible.
@Jason: Actually, pickle is not any safer than eval — malicious input can execute code just as easily, and here at least it is obvious that it is doing so, so I think downvoting this is a little unfair. There are other reasons to avoid eval() (eg. only handles objects with evalable repr()s and silently loses data if they don’t self-eval, as Miles pointed out), but security wise, it’s no worse than pickle.
How can I save a list of dictionaries to a file?
I have a list of dictionaries. Occasionally, I want to change and save one of these dictionaries so that the new message is utilized if the script is restarted. Right now, I make that change by modifying the script and rerunning it. I’d like to pull this out of the script and put the list of dictionaries into a configuration file of some kind. I’ve found answers on how to write a list to a file, but this assumes that it is a flat list. How can I do it with a list of dictionaries? My list looks like this:
5 Answers 5
provided that the object only contains objects that JSON can handle ( lists , tuples , strings , dicts , numbers , None , True and False ), you can dump it as json.dump:
import json with open('outputfile', 'w') as fout: json.dump(your_list_of_dict, fout) if you want each dictionary in one line:
import json output_file = open(dest_file, 'w', encoding='utf-8') for dic in dic_list: json.dump(dic, output_file) output_file.write("\n") @ElizaR dest_file is the location of the destination file. This can be something like /home/user/file on linux
Just for completeness I add also the json.dumps() method:
with open('outputfile_2', 'w') as file: file.write(json.dumps(logic_steps, indent=4)) Have a look here for the difference between json.dump() and json.dumps()
The way you will have to follow to write a dict to a file is kind different from the post you have mentioned.
First, you need serialize the object and than you persist it. These are fancy names for «write python objects to a file».
Python has 3 serialization modules included by default that you can use to achieve your objective. They are: pickle, shelve and json. Each one has its own characteristics and the one you have to use is the one which is more suitable to your project. You should check each module documentation to get more on it.
If your data will be only be accessed by python code, you can use shelve, here is an example:
import shelve my_dict = # file to be used shelf = shelve.open("filename.shlf") # serializing shelf["my_dict"] = my_dict shelf.close() # you must close the shelve file. To retrieve the data you can do:
import shelve shelf = shelve.open("filename.shlf") # the same filename that you used before, please my_dict = shelf["my_dict"] shelf.close() See that you can treat the shelve object almost the same way you do with a dict.
Writing a set to an output file in python
I usually use json for lists, but it doesn’t work for sets. Is there a similar function to write a set into an output file,f? Something like this, but for sets:
f=open('kos.txt','w') json.dump(list, f) f.close() 2 Answers 2
json is not a python-specific format. It knows about lists and dictionaries, but not sets or tuples.
But if you want to persist a pure python dataset you could use string conversion.
with open('kos.txt','w') as f: f.write(str()) # set of numbers & a tuple then read it back again using ast.literal_eval
import ast with open('kos.txt','r') as f: my_set = ast.literal_eval(f.read()) this also works for lists of sets, nested lists with sets inside. as long as the data can be evaluated literally and no sets are empty (a known limitation of literal_eval ). So basically serializing (almost) any python basic object structure with str can be parsed back with it.
For the empty set case there’s a kludge to apply since set() cannot be parsed back.
import ast with open('kos.txt','r') as f: ser = f.read() my_set = set() if ser == str(set()) else ast.literal_eval(ser) You could also have used the pickle module, but it creates binary data, so more «opaque», and there’s also a way to use json : How to JSON serialize sets?. But for your needs, I would stick to str/ast.literal_eval
Writing a Set To a File in Python
I was wondering if there were any pre-made modules in python to remove the opening Set( and closing ) strings from the string:
2 Answers 2
What you see is a set string representation. To get a list from a set , call list() on it:
>>> s = set(['item1', 'item2', 'item3']) >>> s set(['item2', 'item3', 'item1']) >>> list(s) ['item2', 'item3', 'item1'] Keep in mind that sets are unordered collections.
There’s a few different ways to accomplish this. The first two have already been mentioned in answers or comments, but I’d like to add some context as well as a third method.
1. Slice the String
As @Pynchia said in his comment, you can simply slice the result. This will work for any set and would be my preferred solution. Very simple:
>>> x = set(['item1','item2','item3']) >>> print str(x)[4:-1] ['item2', 'item3', 'item1'] 2. Convert to a List
As @alecxe said in his answer, you could simply convert the variable to a List and then convert to a String . However, this is mildly hackish because you’re creating an entirely new List object with whatever overhead that entails just to get its string representation. Probably fine even for pretty large lists, but eventually the overhead will be meaningful.
>>> x = set(['item1','item2','item3']) >>> print list(x) ['item2', 'item3', 'item1'] 3. Create a new class
This might be overkill for your specific situation, but you could create your own Set subclass with its own string method. This code is assuming Python 2.
class mySet(set): def __str__(self): return super(Set,self).__str__()[4:-1] >>> x = mySet(['item1','item2','item3']) >>> print x ['item2', 'item3', 'item1']