python 多进程共享数据的读与写详解大数据

1. 父进程向子进程传参

1.1python通常的数据结构可以传给子进程读，但子进程写无效：

from multiprocessing import Pool, Manager 
 
def chid_proc(test_dict, i): 
    print '{} process, before modify: {}'.format(i, test_dict) 
    test_dict[i] = i * i 
    print '{} process, after modify: {}'.format(i, test_dict) 
 
 
if __name__ == '__main__': 
    td = {'a':1} 
    pool = Pool(3) 
    for i in xrange(0, 3): 
        pool.apply_async(chid_proc, args=(td, i)) 
    pool.close() 
    pool.join() 
    print 'after pool:  {}'.format(td)

结果：

0 process, before modify: {'a': 1} 
0 process, after modify: {'a': 1, 0: 0} 
1 process, before modify: {'a': 1} 
2 process, before modify: {'a': 1} 
1 process, after modify: {'a': 1, 1: 1} 
2 process, after modify: {'a': 1, 2: 4} 
after pool:  {'a': 1}

1.2 使用Manager.dict实现进程间共享数据的修改

 
from multiprocessing import Pool, Manager 
 
def chid_proc(test_dict, i): 
    print '{} process, before modify: {}'.format(i, test_dict) 
    test_dict[i] = i * i 
    print '{} process, after modify: {}'.format(i, test_dict) 
 
 
 
if __name__ == '__main__': 
    #td = {'a':1} 
    td = Manager().dict() 
    td['a'] = 1 
    pool = Pool(3) 
    for i in xrange(0, 3): 
        pool.apply_async(chid_proc, args=(td, i)) 
    pool.close() 
    pool.join() 
    print 'after pool:  {}'.format(td)

结果：

0 process, before modify: {'a': 1} 
0 process, after  modify: {'a': 1, 0: 0} 
1 process, before modify: {'a': 1, 0: 0} 
1 process, after  modify: {'a': 1, 0: 0, 1: 1} 
2 process, before modify: {'a': 1, 0: 0, 1: 1} 
2 process, after  modify: {'a': 1, 0: 0, 2: 4, 1: 1} 
after pool:  {'a': 1, 0: 0, 2: 4, 1: 1}