How to save data in Python

Save data in Python

Suppose we have spent a lot of calculations to get a result in Python and we want to save all the data in a separate file so that we can reuse it later.
This post suggests some methods to do that.

numpy.savez

numpy.savez is a method to save several arrays into a single file in uncompressed .npz format.

import numpy as np 

x_save = np.arange(10)

np.savez("data.npz", x=x_save) # save x_save with name "x" to file data.npz

data = np.load("data.npz") 

x_use = data["x"]

print(x_use) # get [0 1 2 3 4 5 6 7 8 9]

In the case that x_save is a dictionary, use allow_pickle=True in the np.load().

import numpy as np 

x_save = {"a": [1,2,3], "b": "hello from x_save"}

np.savez("data.npz", x=x_save) # save x_save with name "x" to file data.npz

data = np.load("data.npz",  allow_pickle=True) 

x_use = data["x"]

print(x_use) # get {'x': [1, 2, 3], 'y': 'hello from x_save'}

shelve

The second solution is to use the package Shelve.

A “shelf” object is a persistent, dictionary-like object. It can handle most of datatypes in Python.

The following demo shows us how to save the dictionary named save_dict.

import shelve
save_dict = {"a": 1, "b": 2} 
with shelve.open('mydata') as db:
    db["save_dict"] = save_dict 

Here db stands for database.

Now, the dictionary save_dict has been saved in the file mydata.db.
We can retrieve this dictionary with a new name use_dict as follows.

import shelve
with shelve.open('mydata') as db:
    use_dict = db["save_dict"]
print(use_dict)

The result we get should be {'a': 1, 'b': 2}. Done!

pickle

In practice, I couldn’t load the file saved by shelve in another Colab Notebook. So solution for this case is to use pickle instead of shelve.

import pickle
save_dict =  {"a":  1,  "b":  2}

with open("mydata.pkl", "wb") as f:
	pickle.dump(save_data, f)

with open("mydata.pkl", "rb") as f:
	output = pickle.load(f)