Sharing a complex object between processes?
I have a fairly complex Python object that I need to share between multiple processes. I launch these processes using multiprocessing.Process . When I share an object with multiprocessing.Queue and multiprocessing.Pipe in it, they are shared just fine. But when I try to share an object with other non-multiprocessing-module objects, it seems like Python forks these objects. Is that true? I tried using multiprocessing.Value. But I’m not sure what the type should be? My object class is called MyClass. But when I try multiprocess.Value(MyClass, instance) , it fails with: TypeError: this type has no size Any idea what’s going on?
6 Answers 6
After a lot research and testing, I found that «Manager» does this job at a non-complex object level.
The code below shows that object inst is shared between processes, which means property var of inst is changed outside when child process changes it.
from multiprocessing import Process, Manager from multiprocessing.managers import BaseManager class SimpleClass(object): def __init__(self): self.var = 0 def set(self, value): self.var = value def get(self): return self.var def change_obj_value(obj): obj.set(100) if __name__ == '__main__': BaseManager.register('SimpleClass', SimpleClass) manager = BaseManager() manager.start() inst = manager.SimpleClass() p = Process(target=change_obj_value, args=[inst]) p.start() p.join() print inst # print inst.get() # 100 Okay, above code is enough if you only need to share simple objects.
Why no complex? Because it may fail if your object is nested (object inside object):
from multiprocessing import Process, Manager from multiprocessing.managers import BaseManager class GetSetter(object): def __init__(self): self.var = None def set(self, value): self.var = value def get(self): return self.var class ChildClass(GetSetter): pass class ParentClass(GetSetter): def __init__(self): self.child = ChildClass() GetSetter.__init__(self) def getChild(self): return self.child def change_obj_value(obj): obj.set(100) obj.getChild().set(100) if __name__ == '__main__': BaseManager.register('ParentClass', ParentClass) manager = BaseManager() manager.start() inst2 = manager.ParentClass() p2 = Process(target=change_obj_value, args=[inst2]) p2.start() p2.join() print inst2 # print inst2.getChild() # print inst2.get() # 100 #good! print inst2.getChild().get() # None #bad! you need to register child class too but there's almost no way to do it #even if you did register child class, you may get PicklingError :) I think the main reason of this behavior is because Manager is just a candybar built on top of low-level communication tools like pipe/queue.
So, this approach is not well recommended for multiprocessing case. It’s always better if you can use low-level tools like lock/semaphore/pipe/queue or high-level tools like Redis queue or Redis publish/subscribe for complicated use case (only my recommendation lol).
I think this is a better answer due to actual code, 2 examples simple and complex. The docs linked in the chosen answer do not explain this that well. Was able to pass a Tensorboard writer using this strategy. I would clarify, this is good if custom object has parallel processing in mind such as the Tensorboard writer.
You can do this using Python’s multiprocessing «Manager» classes and a proxy class that you define. See Proxy Objects in the Python docs.
What you want to do is define a proxy class for your custom object, and then share the object using a «Remote Manager» — look at the examples in the same linked doc page in the «Using a remote manager» section where the docs show how to share a remote queue. You’re going to be doing the same thing, but your call to your_manager_instance.register() will include your custom proxy class in its argument list.
In this manner, you’re setting up a server to share the custom object with a custom proxy. Your clients need access to the server (again, see the excellent documentation examples of how to setup client/server access to a remote queue, but instead of sharing a Queue , you are sharing access to your specific class).
The code in this question helped supplement the doc page for me. It is an example with a custom class. stackoverflow.com/questions/11951750/…
In Python 3.6 the docs say:
Changed in version 3.6: Shared objects are capable of being nested. For example, a shared container object such as a shared list can contain other shared objects which will all be managed and synchronized by the SyncManager.
As long as instances are created through the SyncManager, you should be able to make the objects reference each other. Dynamic creation of one type of object in the methods of another type of object might still be impossible or very tricky though.
Edit: I stumbled upon this issue Multiprocessing managers and custom classes with python 3.6.5 and 3.6.7. Need to check python 3.7
Edit 2: Due to some other issues I can’t currently test this with python3.7. The workaround provided in https://stackoverflow.com/a/50878600/7541006 works fine for me
here’s a python package I made just for that (sharing complex objects between processes).
The idea is you create a proxy for your object and pass it to a process. Then you use the proxy like you have a reference to the original object. Although you can only use method calls, so accessing object variables is done threw setters and getters.
Say we have an object called ‘example’, creating proxy and proxy listener is easy:
from pipeproxy import proxy example = Example() exampleProxy, exampleProxyListener = proxy.createProxy(example) Now you send the proxy to another process.
p = Process(target=someMethod, args=(exampleProxy,)) p.start() Use it in the other process as you would use the original object (example):
def someMethod(exampleProxy): . exampleProxy.originalExampleMethod() . But you do have to listen to it in the main process:
exampleProxyListener.listen() Read more and find examples here:
I tried to use BaseManager and register my customized class to make it happy, and get the problem about nested class just as Tom had mentioned above.
I think the main reason is irrelevant to the nested class as said, yet the communication mechanism that python take in low level. The reason is python use some socket-alike communication mechanism to synchronize the modification of customized class within a server process in low level. I think it encapsulate some rpc methods, make it just transparent to the user as if they called the local methods of a nested class object.
So, when you want to modify, retrieve your self-defined objects or some third-party objects, you should define some interfaces within your processes to communicate to it rather than directly get or set values.
Yet when operating the multi-nested objects in the nested objects, one can ignore the issues mentioned above, just as what you do in your common routine because your nested objects in the registered class is not a proxy objects any longer, on which the operation will not go through the socket-alike communication routine again and is localized.
Here is the workable code I wrote to solve the problem.
from multiprocessing import Process, Manager, Lock from multiprocessing.managers import BaseManager import numpy as np class NestedObj(object): def __init__(self): self.val = 1 class CustomObj(object): def __init__(self, numpy_obj): self.numpy_obj = numpy_obj self.nested_obj = NestedObj() def set_value(self, p, q, v): self.numpy_obj[p, q] = v def get_obj(self): return self.numpy_obj def get_nested_obj(self): return self.nested_obj.val class CustomProcess(Process): def __init__(self, obj, p, q, v): super(CustomProcess, self).__init__() self.obj = obj self.index = p, q self.v = v def run(self): self.obj.set_value(*self.index, self.v) if __name__=="__main__": BaseManager.register('CustomObj', CustomObj) manager = BaseManager() manager.start() data = [[0 for x in range(10)] for y in range(10)] matrix = np.matrix(data) custom_obj = manager.CustomObj(matrix) print(custom_obj.get_obj()) process_list = [] for p in range(10): for q in range(10): proc = CustomProcess(custom_obj, p, q, 10*p+q) process_list.append(proc) for x in range(100): process_list[x].start() for x in range(100): process_list[x].join() print(custom_obj.get_obj()) print(custom_obj.get_nested_obj()) Converting python code to shared object
I would like to prepare a shared object (.so) from a python module. I came across Cython which would a) first convert a *.pyx module to a *.c code,b) and then this *.c code would be converted to a shared object (.so). All the examples of Cython then state how this .so can be imported into python. But, I am interested in reading this shared object from a C code. When I wrote a sample C code to read a .so, it throws a error saying that the methods that are actually present in .pyx are not present in the .so object. I would like to know a) whether is it possible to read the shared objects from Cython from a different language such as C b) And, if the above statement is True, what changes do I have to make in my code so that I would be able to read a shared object from C. Thanks, Python code (saved as square_number.pyx)
def square_me(int x): return x * x from distutils.core import setup from Cython.Build import cythonize setup( ext_modules=cythonize("square_number.pyx"), ) python setup.py build_ext --inplace This would create a square_number.so in the same folder. Now, I renamed this to libSquareNumber.so C code for reading the .so
#include int main(int argc,char *argv[])
When I am trying to compile and build an executable from the above command, I am getting an error Compilation of C code:
gcc -L/home/USRNAME/work/cython-codes/squaring/ -Wall -o test so_reader_in_c.c -lSquareNumber so_reader_in_c.c: In function ‘main’: so_reader_in_c.c:11:4: warning: implicit declaration of function ‘square_me’ [- Wimplicit-function-declaration] result=square_me(2); ^ /tmp/ccE5vIOH.o: In function `main': so_reader_in_c.c:(.text+0x1a): undefined reference to `square_me' collect2: error: ld returned 1 exit status