[@sddddgZddlZddlZddlZddlZddlmZddlmZdd l m Z dd l m Z dd l m Z dd l m Z dd l mZddl mZddl mZddZejejedddddfDZedek rMddZxeD]Zejeeq0WnGdddeZfiddZddZGdddeZd d!Zd"d#ZGd$d%d%eZGd&d'd'eZ ie j!e j"fd(6e j#e j$fd)6Z%Gd*ddeZ&Gd+d,d,e'Z(Gd-ddeZ)d.d/Z*id0d1Z+dddd2d3d4Z,Gd5d6d6eZ-Gd7d8d8eZ.d2d9d:Z/Gd;d<d<e)Z0Gd=d>d>e)Z1Gd?d@d@e1Z2GdAdBdBe)Z3GdCdDdDe)Z4GdEdFdFe)Z5GdGdHdHe)Z6e+dIdJdKdLdMdNdOdPdQdRdSdTdUdVdWdXdYdZd[d\fZ7Gd]d^d^e7Z8e+d_dKdLdMdNdRd`dadbdcdddXdddedfdfZ9e+dgdNdMdRfZ:e+dhdidjdkdldmdndodpdqdrdsf Z;idtdj6dtdp6dtdr6dudl6dudm6e;_<Gdvdhdhe;Z=Gdwdde&Z>e>jdxej?e>jdyej?e>jdzej@e3e>jd{ejAe1e>jd|ejBe1e>jd}ejCe1e>jd~ejDe1e>jdejEe2e>jdejFe4e>jde jGe=e>jdee8e>jdeHe9e>jd8e.e6e>jd:e/e:e>jd6e-e5e>jdude0dde>jdtdddS) BaseManager SyncManager BaseProxyTokenN)time) format_exc) connection)context)pool)process) reduction)util) get_contextcCstj|j|jffS)N)arraytypecodetobytes)ar./usr/lib/python3.4/multiprocessing/managers.py reduce_array%srcCs(g|]}tti|qSr)typegetattr).0namerrr )s ritemskeysvaluescCstt|ffS)N)list)objrrrrebuild_as_list+sr!c@sLeZdZdZdZddZddZd d Zd d Zd S)rz4 Type to uniquely indentify a shared object typeidaddressidcCs!||||_|_|_dS)N)r"r#r$)selfr"r#r$rrr__init__:szToken.__init__cCs|j|j|jfS)N)r"r#r$)r%rrr __getstate__=szToken.__getstate__cCs|\|_|_|_dS)N)r"r#r$)r%staterrr __setstate__@szToken.__setstate__cCsd|j|j|jfS)Nz#Token(typeid=%r, address=%r, id=%r))r"r#r$)r%rrr__repr__CszToken.__repr__N)ztypeidzaddresszid) __name__ __module__ __qualname____doc__ __slots__r&r'r)r*rrrrr4s    cCsN|j||||f|j\}}|dkr;|St||dS)zL Send a message to manager using connection `c` and return response z#RETURNN)sendrecvconvert_to_error)cr$ methodnameargskwdskindresultrrrdispatchKs  r9cCs~|dkr|S|dkr>t|tks4tt|S|dkrpt|tksbttd|StdSdS)Nz#ERRORz #TRACEBACKz#UNSERIALIZABLEzUnserializable message: %s zUnrecognized message type)rstrAssertionError RemoteError ValueError)r7r8rrrr2Us    r2c@seZdZddZdS)r<cCs)ddddt|jdddS)N -Kr)r:r5)r%rrr__str__bszRemoteError.__str__N)r+r,r-rArrrrr<as r<cCsLg}x?t|D]1}t||}t|r|j|qqW|S)z4 Return a list of names of methods of `obj` )dirrcallableappend)r temprfuncrrr all_methodsis  rGcCsddt|DS)zP Return a list of names of methods of `obj` which do not start with '_' cSs&g|]}|ddkr|qS)r_r)rrrrrrxs z"public_methods..)rG)r rrrpublic_methodstsrIc @seZdZdZdddddddd d g Zd d Zd dZddZddZddZ ddZ ddZ ddZ ie d6e d6e d6Z ddZd d!Zd"d#Zd$d%Zd&d'Zd(d)Zd*d+Zd,d-Zd.d/Zd0S)1ServerzM Server class which runs in a process controlled by a manager object shutdowncreateaccept_connection get_methods debug_infonumber_of_objectsdummyincrefdecrefcCst|tst||_tj||_t|\}}|d|dd|_|jj |_ idffd6|_ i|_ t j |_dS)Nr#Zbacklog0) isinstancebytesr;registryr AuthenticationStringauthkeylistener_clientlistenerr# id_to_objid_to_refcount threadingRLockmutex)r%rXr#rZ serializerListenerClientrrrr&s  zServer.__init__cCstj|_|tj_zqtjd|j}d|_|j y*x#|jj sq|jj dqOWWnt t fk rYnXWdtjtjkrtjdtjt_tjt_ntjdXdS)z( Run the server forever targetTrNzresetting stdout, stderrr)r_Event stop_eventr current_process_manager_serverThreadaccepterdaemonstartis_setwaitKeyboardInterrupt SystemExitsysstdout __stdout__rdebug __stderr__stderrexit)r%rkrrr serve_forevers      zServer.serve_foreverc Csgx`y|jj}Wntk r-wYnXtjd|jd|f}d|_|jqWdS)Nrer5T)r\ZacceptOSErrorr_rjhandle_requestrlrm)r%r3trrrrks  zServer.accepterc 'Csd}}}yvtj||jtj||j|j}|\}}}}||jksttd|t||}Wn!tk rdt f} YnGXy||||}Wn!tk rdt f} Yn Xd|f} y|j | Wntk r} zdy|j dt fWntk rBYnXt j d| t j d|t j d| WYdd} ~ XnX|j dS)z) Handle a new connection Nz%r unrecognizedz #TRACEBACKz#RETURNzFailure to send message: %rz ... request was %rz ... exception was %r)r Zdeliver_challengerZZanswer_challenger1publicr;r Exceptionrr0rinfoclose) r%r3Zfuncnamer8requestignorer5r6rFmsgerrrr{s4     #zServer.handle_requestc*Cs+tjdtjj|j}|j}|j}x|jj s&y/d}}|}|\}}} } ||\}} } || krt d|t || fnt ||} y| | | }Wn1t k r}zd|f}WYdd}~XnpX| o| j|d}|rh|j|||\}}t||j|}d||ff}n d|f}Wnt k r|dkrdtf}n\y8|j|}|||||| | }d|f}Wn!t k rdtf}YnXYnXtk r:tjdtjjtjd Yn!t k rZdtf}YnXyPy||Wn;t k r}z|d t|fWYdd}~XnXWq7t k r"}zUtjd tjjtjd |tjd ||jtjdWYdd}~Xq7Xq7WdS)zQ Handle requests from the proxies in a particular process/thread z$starting server thread to service %rNz+method %r of %r object is not in exposed=%rz#ERRORz#PROXYz#RETURNz #TRACEBACKz$got EOF -- exiting thread serving %rrz#UNSERIALIZABLEzexception in thread serving %rz ... message was %rz ... exception was %rr)rrur_current_threadrr1r0r]rgrnAttributeErrorrrr~getrLrr#rfallback_mappingEOFErrorrrrxreprrr)r%connr1r0r]r4r ridentr5r6exposedZ gettypeidZfunctionZresrrr"ZridentZrexposedtokenZ fallback_funcr8rrr serve_clientsl              -  zServer.serve_clientcCs|S)Nr)r%rrr rrrfallback_getvalueszServer.fallback_getvaluecCs t|S)N)r:)r%rrr rrr fallback_str"szServer.fallback_strcCs t|S)N)r)r%rrr rrr fallback_repr%szServer.fallback_reprrAr*z #GETVALUEcCsdS)Nr)r%r3rrrrQ.sz Server.dummycCs|jjzg}t|jj}|jxY|D]Q}|dkr<|jd||j|t|j|dddfq<q<Wdj |SWd|jj XdS)zO Return some info --- useful to spot problems with refcounting rUz %s: refcount=%s %srNr@r>) raacquirerr]rsortrDr^r:joinrelease)r%r3r8rrrrrrO1s      -zServer.debug_infocCst|jdS)z* Number of shared objects r)lenr])r%r3rrrrPCszServer.number_of_objectscCsXzCytjd|jdWnddl}|jYnXWd|jjXdS)z' Shutdown this process z!manager received shutdown message#RETURNNr)rN)rrur0 traceback print_excrgset)r%r3rrrrrKIs  zServer.shutdownc Os\|jjz:|j|\}}}}|dkrat|dkrN| sTt|d} n|||} |dkrt| }n|dk rt|tkstt|t|}ndt | } t j d|| | t ||f|j | <| |jkr&d|j| Create a new shared object and return its id Nrrz%xz&%r callable returned object with id %r)rarrXrr;rIrdictrr$rrurr]r^rRtupler) r%r3r"r5r6rCrmethod_to_typeid proxytyper rrrrrLVs(     z Server.createcCst|j|jdS)zL Return the methods of the shared object indicated by token r)rr]r$)r%r3rrrrrN|szServer.get_methodscCs-|tj_|jd|j|dS)z= Spawn a new thread to serve this connection #RETURNN)rN)r_rrr0r)r%r3rrrrrMs zServer.accept_connectionc Cs7|jjz|j|d7 Connect manager object to the server process rZNrQ) r[rrrr9rrrr)r%rcrdrrrrconnectszBaseManager.connectc Csm|jjtjkst|dk rCt| rCtdntjdd\}}|j j dt |j d|j |j|j|j|||f|_djdd |jjD}t |jd ||j_|jj|j|j|_|jtj|j_tj|t |jd|j|j|j|j|jfd d |_dS) z@ Spawn a server process for this manager object Nzinitializer must be a callableZduplexFrer5:css|]}t|VqdS)N)r:)rirrr sz$BaseManager.start..r? exitpriorityr)rrrrr;rC TypeErrorr ZPiperZProcessr _run_serverrrrr_processrZ _identityr+rrmrr1rrFinalize_finalize_managerrrK)r% initializerinitargsreaderwriterrrrrrms( "   zBaseManager.startc Csl|dk r||n|j||||}|j|j|jtjd|j|jdS)z@ Create a server, report its address and run it Nzmanager serving at %r)_Serverr0r#rrrry) clsrXr#rZrbrrrserverrrrrs   zBaseManager._run_serverc Os|jjtjks!td|j|jd|j}z)t|dd|f||\}}Wd|j Xt ||j||fS)zP Create a new shared object; return the token and exposed tuple zserver not yet startedrZNrL) rrrrr;rrrr9rr)r%r"r5r6rr$rrrr_creates !) zBaseManager._createcCsA|jdk r=|jj||jjs=d|_q=ndS)zC Join the manager process (if it has been spawned) N)rris_alive)r%timeoutrrrrszBaseManager.joinc CsA|j|jd|j}zt|ddSWd|jXdS)zS Return some info about the servers shared objects and connections rZNrO)rrrr9r)r%rrrr _debug_info$szBaseManager._debug_infoc CsA|j|jd|j}zt|ddSWd|jXdS)z5 Return the number of shared objects rZNrP)rrrr9r)r%rrrr_number_of_objects.szBaseManager._number_of_objectscCsA|jjtjkr"|jn|jjtjks=t|S)N)rrrrrmrr;)r%rrr __enter__8s zBaseManager.__enter__cCs|jdS)N)rK)r%exc_typeexc_valexc_tbrrr__exit__>szBaseManager.__exit__cCs|jrtjdy8||d|}zt|ddWd|jXWntk reYnX|jdd|jrtjdt|drtjd |j|jdd |jrtjd qqqnt j |_ yt j |=Wntk rYnXdS) zQ Shutdown the manager process; will be registered as a finalizer z#sending shutdown message to managerrZNrKrg?zmanager still alive terminatez'trying to `terminate()` manager processg?z#manager still alive after terminate)rrrr9rr~rhasattrrrrrr_address_to_localKeyError)r r#rZr(rrrrrrAs.          zBaseManager._finalize_managercCs|jS)N)r)r%rrraszBaseManager.Tc s-d|jkr$|jj|_ndkr9tn|pNtdd}|pftdd}|rxgt|jD]P\}}t|tkst d|t|tkst d|qWn|||f|j<|r)fdd} | _ t || ndS)z9 Register a typeid with the manager type rN _exposed__method_to_typeid_z%r is not a stringc stjd|j||\}}||jd|d|jd|}|j|jd|j}t|dd|jf|S)Nz)requesting creation of a shared %r objectmanagerrZrrS) rrurrrrr#r9r$)r%r5r6rZexpproxyr)rr"rrrE~sz"BaseManager.register..temp) __dict__rcopy AutoProxyrrrrr:r;r+setattr) rr"rCrrr create_methodkeyrrEr)rr"rregistercs   ") zBaseManager.register)r+r,r-r.rrJrr&rrrm classmethodrrrrrrr staticmethodrpropertyr#rrrrrrs*    $     c@s(eZdZddZddZdS)ProcessLocalSetcCstj|dddS)NcSs |jS)N)clear)r rrrrsz*ProcessLocalSet.__init__..)rregister_after_fork)r%rrrr&szProcessLocalSet.__init__cCst|ffS)N)r)r%rrr __reduce__szProcessLocalSet.__reduce__N)r+r,r-r&rrrrrrs  rc@seZdZdZiZejZddddddZddZ fidd Z d d Z d d Z e ddZddZddZddZddZddZdS)rz. A base for proxies of shared objects NTc CsAtjjzPtjj|jd}|dkr\tjtf}|tj|jz0x%x)rr+rr"r$)r%rrrr*4szBaseProxy.__repr__c CsBy|jdSWn*tk r=t|dddSYnXdS)zV Return representation of the referent (or a fall-back if that fails) r*Nrz; '__str__()' failed>r)rr~r)r%rrrrA8s zBaseProxy.__str__)r+r,r-r.rrZForkAwareThreadLockrr&rrrrrrrrrr*rArrrrrs  %      cCsttjdd}|rB|j|jkrB|j|jdS|jddojttjdd }|||d||SdS)z Function used for unpickling proxy objects. If possible the shared object is returned, or otherwise a proxy for it. riNrrRTZ _inheritingF)rr rhr#r]r$pop)rFrrbr6rrRrrrrEs rc Cst|}y|||fSWntk r2YnXi}x%|D]}td||f|q@Wt|tf|}||_||||f<|S)zC Return an proxy type whose methods are given by `exposed` zLdef %s(self, *args, **kwds): return self._callmethod(%r, args, kwds))rrexecrrr)rr_cacheZdicmeth ProxyTyperrr MakeProxyTypeZs    rTc Cst|d}|dkr\||jd|}zt|dd|f}Wd|jXn|dkr|dk r|j}n|dkrtjj}ntd|j |}|||d|d|d|} d| _ | S) z* Return an auto-proxy for `token` rNrZrNz AutoProxy[%s]rrRT) r[r#r9rrr rhrZrr"r) rrbrrZrrRrrrrrrrrps     rc@s(eZdZddZddZdS) NamespacecKs|jj|dS)N)rupdate)r%r6rrrr&szNamespace.__init__cCsyt|jj}g}x=|D]5\}}|jds"|jd||fq"q"W|jdtjd|S)NrHz%s=%rz Namespace(%s)z, )rrr startswithrDrr:r)r%rrErrrrrr*s zNamespace.__repr__N)r+r,r-r&r*rrrrrs  rc@sReZdZdddZddZddZdd ZeeeZd S) ValueTcCs||_||_dS)N) _typecode_value)r%rrlockrrrr&s zValue.__init__cCs|jS)N)r)r%rrrrsz Value.getcCs ||_dS)N)r)r%rrrrrsz Value.setcCs dt|j|j|jfS)Nz %s(%r, %r))rr+rr)r%rrrr*szValue.__repr__N) r+r,r-r&rrr*rrrrrrrs    rcCstj||S)N)r)rZsequencerrrrArraysrc@sReZdZdZddZddZd d Zd d Zd dZdS) IteratorProxy__next__r0throwrcCs|S)Nr)r%rrr__iter__szIteratorProxy.__iter__cGs|jd|S)Nr)r)r%r5rrrrszIteratorProxy.__next__cGs|jd|S)Nr0)r)r%r5rrrr0szIteratorProxy.sendcGs|jd|S)Nr)r)r%r5rrrrszIteratorProxy.throwcGs|jd|S)Nr)r)r%r5rrrrszIteratorProxy.closeN)z__next__zsendzthrowzclose) r+r,r-rrrr0rrrrrrrs     rc@sLeZdZd ZddddZddZd d Zd d ZdS) AcquirerProxyrrTNcCs1|dkr|fn ||f}|jd|S)Nr)r)r%Zblockingrr5rrrrs!zAcquirerProxy.acquirecCs |jdS)Nr)r)r%rrrrszAcquirerProxy.releasecCs |jdS)Nr)r)r%rrrrszAcquirerProxy.__enter__cCs |jdS)Nr)r)r%rrrrrrrszAcquirerProxy.__exit__)zacquirezrelease)r+r,r-rrrrrrrrrr s   r c@sLeZdZdZdddZd d Zd d Zdd dZdS)ConditionProxyrrronotify notify_allNcCs|jd|fS)Nro)r)r%rrrrroszConditionProxy.waitcCs |jdS)Nr )r)r%rrrr szConditionProxy.notifycCs |jdS)Nr )r)r%rrrr szConditionProxy.notify_allcCs|}|r|S|dk r/t|}n d}d}xL|s|dk rp|t}|dkrpPqpn|j||}q>W|S)Nr)_timero)r%Z predicaterr8ZendtimeZwaittimerrrwait_fors        zConditionProxy.wait_for)zacquirezreleasezwaitznotifyz notify_all)r+r,r-rror r rrrrrr s   r c@sIeZdZdZddZddZd d Zd d d Zd S) EventProxyrnrrrocCs |jdS)Nrn)r)r%rrrrnszEventProxy.is_setcCs |jdS)Nr)r)r%rrrrszEventProxy.setcCs |jdS)Nr)r)r%rrrrszEventProxy.clearNcCs|jd|fS)Nro)r)r%rrrrroszEventProxy.wait)zis_setzsetzclearzwait)r+r,r-rrnrrrorrrrrs    rc@sseZdZdZdddZdd Zd d Zed d ZeddZ eddZ dS) BarrierProxy__getattribute__roabortresetNcCs|jd|fS)Nro)r)r%rrrrroszBarrierProxy.waitcCs |jdS)Nr)r)r%rrrrszBarrierProxy.abortcCs |jdS)Nr)r)r%rrrrszBarrierProxy.resetcCs|jddS)Nrparties)zparties)r)r%rrrrszBarrierProxy.partiescCs|jddS)Nr n_waiting)z n_waiting)r)r%rrrrszBarrierProxy.n_waitingcCs|jddS)Nrbroken)zbroken)r)r%rrrrszBarrierProxy.broken)z__getattribute__zwaitzabortzreset) r+r,r-rrorrrrrrrrrrrs   rc@s:eZdZd ZddZddZdd Zd S) NamespaceProxyr __setattr__ __delattr__cCsB|ddkr tj||Stj|d}|d|fS)NrrHrr)objectr)r%r callmethodrrr __getattr__szNamespaceProxy.__getattr__cCsH|ddkr#tj|||Stj|d}|d||fS)NrrHrr)rrr)r%rrrrrrrszNamespaceProxy.__setattr__cCsB|ddkr tj||Stj|d}|d|fS)NrrHrr)rrr)r%rrrrrr szNamespaceProxy.__delattr__N)z__getattribute__z __setattr__z __delattr__)r+r,r-rrrrrrrrrs   rc@s=eZdZdZddZddZeeeZdS) ValueProxyrrcCs |jdS)Nr)r)r%rrrrszValueProxy.getcCs|jd|fS)Nr)r)r%rrrrrszValueProxy.setN)zgetzset)r+r,r-rrrrrrrrrrs   r BaseListProxy__add__ __contains__ __delitem__ __getitem____len____mul__ __reversed____rmul__ __setitem__rDcountextendindexinsertrremovereverser__imul__c@s(eZdZddZddZdS) ListProxycCs|jd|f|S)Nr))r)r%rrrr__iadd__$szListProxy.__iadd__cCs|jd|f|S)Nr.)r)r%rrrrr.'szListProxy.__imul__N)r+r,r-r0r.rrrrr/#s  r/ DictProxyrrrZhas_keypopitem setdefaultr ArrayProxy PoolProxyZapplyZ apply_asyncrZimapZimap_unorderedrmapZ map_asyncstarmapZ starmap_asyncrZ AsyncResultIteratorc@s(eZdZddZddZdS)r5cCs|S)Nr)r%rrrrDszPoolProxy.__enter__cCs|jdS)N)r)r%rrrrrrrFszPoolProxy.__exit__N)r+r,r-rrrrrrr5Cs  c@seZdZdZdS)ra( Subclass of `BaseManager` which supports a number of shared object types. The types registered are those intended for the synchronization of threads, plus `dict`, `list` and `Namespace`. The `multiprocessing.Manager()` function creates started instances of this class. N)r+r,r-r.rrrrrMs QueueZ JoinableQueuerfLockr` SemaphoreBoundedSemaphore ConditionBarrierPoolrrrrF)I__all__rrr_rZqueuerr rrr r r r r rrrrZ view_typesrr!Z view_typerrr9r2r~r<rGrIrJrrcrdZ XmlListenerZ XmlClientr[rrrrrrrrrrrr r rrrrrr/r1r4Z BasePoolProxyrr5rr9rfr:r`r;r<r=r>r?rrrrr s         #