3 d)@s dZddlZddlZddlZddlZddlZddlmZddlm Z ddlm Z ddl m Z ddl mZddl Z ddlZddlmZddlZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZddlmZmZmZm Z m!Z!m"Z"ddl#m$Z$ddl%m&Z&y ddl'Z'Wne(k r.dZ'YnXyddl)m*Z*Wne(k rZdZ*YnXy ddl+Z+Wne(k rdZ+YnXddl,m-Z-ddl,m.Z.ddl,m/Z/ddl,m0Z0ddl,m1Z1ddl,m2Z2ddl3m4Z4m5Z5ddl3m6Z6m7Z7ddl3m8Z8m9Z9ddl3m:Z:m;Z;mm?Z?ddl>m@Z@dgeAe;jBZCeCdd e;D7ZCd!d"gZDeDd#gZEeFe:d$r~eCjGe:jHd%e;e<ZId&d'ZJd(d)ZKd*d+ZLGd,d-d-eMZNd2d/d0ZOd1d2ZPd3d3d4ZQd5d6ZRd7d8ZSd9d:ZTd;d<ZUejVjd=d4d5e8d6d?fd7gd@dAdBgdCedDdEZWedFeCedGddHd8d9gedIdHdJdKgdLdMZXedFeCdNdOZYdPdQZZeedRd:d;d<d=d>d?gdTdUZ[eedFd"d!d#gdVdWZ\dXdYZ]edZe;ed[e;d\d]Z^d^d_Z_d`daZ`dbdcZaedGddHd>gdddeZbedFeEdfdgZcedhdiZdedFeEdjdkZeeedFeEdldmZfeedFeDdndoZgdpdqZhedFe;edrddsdtdudvdwdxdydzd{d|d}d~g fddsdudwdydtdvdxdzd{d}d|d~g fgddZieedFeEddZjddZkeedFeDddZlddZmddZnddZoeedFeEejVjpddddZqGddde-ZrddZsedFeCddZtddZuddZvddZweEddZxexjydd ezd>DeedFexddZ{Gddde-Z|ddZ}ddZ~ddZddZGdddejj1Ze6deeedFd#d"d!dgddZeedGdHd@dgedFeEddZddZeedeEedeEddZeddZddZddZedddAdgddZeddBdCdDdEdFdGdHdIdJdKdLg dd΄ZeddЄZeeedFeDdd҄ZdZeedFeDddՄZdZdZdZdZeedFeDe:dkrXgnd%gedeeegeddddgddZdjejjejjejdZeddZdjejjejjejdZeddZddZddZeeedFeDddZddZeeddZddZedFeCedGddHdMdNgddZeeedFeDddZedFdd"d#gddZeedFeDddZddZddZddZddZddZdOddZeedFd"d#gdd Zee e+dkd dd d ZdPd dZedFd"d#gddZddZedFdd"ge e*dkddddZe e+dk ddddZddZddZddZdd Zd!d"ZeeedGdHddQdRgd#d$Zeeed%ddHddgedGdHdSgd&d'ZeedGdHdTged(d)d*d+gd,d-ZeeedFd!d#e/e.gd.d/ZeedGdHddUgd0d1ZdS(Vz Test the parallel module. N)format_exception)sqrt)sleep) PicklingError) TimeoutError)reload)parallel)dumpload)get_reusable_executor)np with_numpy)with_multiprocessing) parametrizeraisescheck_subprocess_callskipifSkipTestwarns)TerminatedWorkerError)Queue) parallel_sum)SequentialBackend)ThreadingBackend)MultiprocessingBackend)ParallelBackendBase) LokyBackend) SafeFunction)Paralleldelayed)register_parallel_backendparallel_backend)effective_n_jobs cpu_count)mpBACKENDSDEFAULT_BACKENDEXTERNAL_BACKENDS)JoblibException)WorkerInterruptcCsg|]}t|qS)r&).0Z backend_strr+r+;/tmp/pip-build-v9q4h5k9/joblib/joblib/test/test_parallel.py Gsr.multiprocessingloky threading get_contextspawncCst|dt|ddS)N_poolZ_workers)getattr)backendr+r+r- get_workersRsr7cCs||S)Nr+)xyr+r+r-divisionVsr:cCs|dS)Nr+)r8r+r+r-squareZsr<c@seZdZdZddZdS)MyExceptionWithFinickyInitz1An exception class with non trivial __init__ cCsdS)Nr+)selfabcdr+r+r-__init__asz#MyExceptionWithFinickyInit.__init__N)__name__ __module__ __qualname____doc__rCr+r+r+r-r=^sr=FcCs"|dkr|rtddddnt|S)Nr?r@rArB)r= ValueError)r8custom_exceptionr+r+r-exception_raiseresrKcCstjdtdS)Ng?)timerKeyboardInterrupt)r8r+r+r-interrupt_raiserls rNcCs|d||S)zO A module-level function so that it can be spawn with multiprocessing. r;r+)r8r9zr+r+r-fqsrPcCsttjdS)Nr)typerget_active_backendr+r+r+r-_active_backend_typexsrScCst||dddtdDS)N)n_jobsr6css|]}tt|VqdS)N)r r<)r,ir+r+r- ~sz parallel_func..)rrange) inner_n_jobsr6r+r+r- parallel_func|s rZcCstdkstdS)Nr)r$AssertionErrorr+r+r+r-test_cpu_countsr\cCstdkstdS)Nr)r#r[r+r+r+r-test_effective_n_jobssr]zbackend_n_jobs, expected_n_jobsrW)rTz positive-intz negative-intNone)idsc Cs@td|dtdd|ks tWdQRXtdddks.)rTr6racss|]}tt|VqdS)N)r r<)r,r8r+r+r-rVsz'test_simple_parallel..)rXrr[)r6rTrar+r+r-test_simple_parallelsrec Csz|jtjdddtd6}td|dddtdD}|d d d gksNtWdQRXd d |D}t|d ksvtdS)NnameZ!some_new_name_for_the_main_thread)targetrfvaluer;)rTr6css|]}tt|VqdS)N)r r<)r,r8r+r+r-rVsz6test_main_thread_renamed_no_warning..rWrrcSsg|]}dt|jkr|qS)zworker timeout)strmessage)r,wr+r+r-r.sz7test_main_thread_renamed_no_warning..)setattrr1current_threadrrrXr[len)r6 monkeypatchwarninforesultsr+r+r-#test_main_thread_renamed_no_warnings   rsc Cs`td}t||dWdQRX|rHt|dkrDd|djjdkSdSt|dksXtdSdS)N)r6rYrzbacked parallel loops cannotFT)rrZrorkargsr[)r6rYexpectedrecordsr+r+r-_assert_warning_nesteds  rwz%parent_backend,child_backend,expectedTcsltd|dfddtdDtd|dfddtdD}|dkr\t|shtn t|shtdS)Nr;)rTr6c3s |]}ttdddVqdS)rF)r6rYruN)r rw)r,_) child_backendr+r-rVsz0test_nested_parallel_warnings..rdc3s |]}ttddVqdS)r;)r6rYruN)r rw)r,rx)ryrur+r-rVsr1)rrXanyr[all)parent_backendryruresr+)ryrur-test_nested_parallel_warningss    r~cCs>dg}dd}tj||fd}|j|j|ds:tdS)NFc SsPtd"}tddddtdDWdQRXtt|t|dk|d<dS)Nr;)rTcss|]}ttdVqdS)g?N)r r)r,rxr+r+r-rVszPtest_background_thread_parallelism..background_thread..rir)rrrXprintro)is_run_parallelrvr+r+r-background_threads   z=test_background_thread_parallelism..background_thread)rgrtr)r1Threadstartjoinr[)r6rrtr+r+r-"test_background_thread_parallelisms rcCs"td|dddtdDdS)Nr;)rTr6css|]}ttdVqdS)g{Gz?N)r r<)r,rxr+r+r-rVsznested_loop..)rrX)r6r+r+r- nested_loops rryr|cs&td|dfddtdDdS)Nr;)rTr6c3s|]}ttVqdS)N)r r)r,rx)ryr+r-rV sz#test_nested_loop..)rrX)r|ryr+)ryr-test_nested_loops rcCstdS)N)rI)r6r+r+r-raise_exception srcCsLtt:tddd"}|ttdttdgWdQRXWdQRXdS)Nr;r0)rTr6)rrIrr rr)rr+r+r-)test_nested_loop_with_exception_with_lokys  rcs<tddtdddfddtdDjs8tdS) z1Input is mutable when using the threading backendrd)maxsizer;r1)rTr6c3s|]}tjdVqdS)rN)r put)r,rx)qr+r-rVsz1test_mutate_input_with_threads..N)rrrXfullr[r+r+)rr-test_mutate_input_with_threadss  rcCs6td}dd|Dt|ddd|Dks2tdS)z.Check the keyword argument processing of pmap. cSsg|]}t|ddqS)r)r9)rP)r,r8r+r+r-r.#sz(test_parallel_kwargs..)rTcss|]}tt|ddVqdS)r)r9N)r rP)r,r8r+r+r-rV$sz'test_parallel_kwargs..N)rXrr[)rTlstr+r+r-test_parallel_kwargss rc Cstd}dd|D}td|d^}|j}||dd|DksDt||dd|Dks^ttdk r|t|t|jks|tWdQRXtdk rt|jdkst||d d|Dksttdk rt|jdkstdS) NrcSsg|]}t|ddqS)r)r9)rP)r,r8r+r+r-r.*sz4test_parallel_as_context_manager..ri)rTr6css|]}tt|ddVqdS)r)r9N)r rP)r,r8r+r+r-rV3sz3test_parallel_as_context_manager..css|]}tt|ddVqdS)r)r9N)r rP)r,r8r+r+r-rV4scss|]}tt|ddVqdS)r)r9N)r rP)r,r8r+r+r-rV@s)rXr_backendr[r%r7)r6rrupZmanaged_backendr+r+r- test_parallel_as_context_manager's rc sLGdddtttdd&tddfddtd DWd QRXd S) ze Check that pmap captures the errors when it is passed an object that cannot be pickled. c@seZdZddZdS)z1test_parallel_pickling..UnpicklableObjectcSs tddS)NZ123) RuntimeError)r>r+r+r- __reduce__Ksz.UnpicklableObject.__reduce__N)rDrErFrr+r+r+r-UnpicklableObjectJsrzthe task to send)matchr;)rTc3s|]}ttVqdS)N)r id)r,rx)rr+r-rVOsz)test_parallel_pickling..rN)objectrrrrXr+r+)rr-test_parallel_picklingEsrcCs0ttd|ddddtdDdks,tdS)Nr;r)rTr6timeoutcss|]}ttdVqdS)gMbP?N)r r)r,r8r+r+r-rVVsz0test_parallel_timeout_success..)rorrXr[)r6r+r+r-test_parallel_timeout_successRsrc Cs8tt&td|ddddtdDWdQRXdS)Nr;g{Gz?)rTr6rcss|]}ttdVqdS)rN)r r)r,r8r+r+r-rV_sz-test_parallel_timeout_fail..r)rrrrX)r6r+r+r-test_parallel_timeout_failYs  rcACstdk rtt&td|dddtddDWdQRXtt td|ddddDWdQRXtd|d&}t|jdk stt|j}tt|ddtddDWdQRXt|jdk stt|j|k std dt d D|d d t d Dkstt|j}tt|d ddDWdQRXt|jdk sZtt|j|k sntddt d D|dd t d DkstWdQRXt|jdkstn,tt tdddddDWdQRXtt$tddddtdd DWdQRXtt $tddddd t dDWdQRXy"tdddd td!d"DWn6t k r}zt |t stWYdd}~Xn XtddS)#Nr;)rTr6cSsg|]\}}tt||qSr+)r r:)r,r8r9r+r+r-r.jsz&test_error_capture..rrcSsg|]}tt|qSr+)r rN)r,r8r+r+r-r.nscSsg|]\}}tt||qSr+)r r:)r,r8r9r+r+r-r.vscSsg|]}t|ddqS)r)r9)rP)r,r8r+r+r-r.srcss|]}tt|ddVqdS)r)r9N)r rP)r,r8r+r+r-rVsz%test_error_capture..cSsg|]}tt|qSr+)r rN)r,r8r+r+r-r.scSsg|]}t|ddqS)r)r9)rP)r,r8r+r+r-r.scss|]}tt|ddVqdS)r)r9N)r rP)r,r8r+r+r-rVs)rTcSsg|]}tt|qSr+)r rN)r,r8r+r+r-r.scSsg|]\}}tt||qSr+)r r:)r,r8r9r+r+r-r.s)rTracss|]}tt|ddVqdS)T)rJN)r rK)r,rUr+r+r-rVscss |]\}}tt||VqdS)N)r r:)r,r8r9r+r+r-rVsz'The excepted error has not been raised.)rr)rr)rr)rr)rr)rr)rr)rr)rr)rr)rr)r%rZeroDivisionErrorrzipr*r7rr[rXrMr= Exception isinstancer)rI)r6rZoriginal_workersexr+r+r-test_error_capturebsT         (     $rcCs|jd|dS)Nz Consumed %s)append)queueitemr+r+r-consumersrzbatch_size, expected_queuez Produced 0z Consumed 0z Produced 1z Consumed 1z Produced 2z Consumed 2z Produced 3z Consumed 3z Produced 4z Consumed 4z Produced 5z Consumed 5ricsTtfdd}td||dfdd|D|ks@ttdksPtdS) zC Test that with only one job, Parallel does act as a iterator. c3s*x$tdD]}jd||Vq WdS)Nz Produced %i)rXr)rU)rr+r-producersz'test_dispatch_one_job..producerr)rT batch_sizer6c3s|]}tt|VqdS)N)r r)r,r8)rr+r-rVsz(test_dispatch_one_job.. N)listrr[ro)r6rZexpected_queuerr+)rr-test_dispatch_one_jobs    rcstj}|jfdd}tddd|dfdd|Dt}|d d ksXt|d d jd }|dksvt|jd}||ksttdkstd S)zW Check that using pre_dispatch Parallel does indeed dispatch items lazily. c3s*x$tdD]}jd||Vq WdS)Nrz Produced %i)rXr)rU)rr+r-rsz/test_dispatch_multiprocessing..producerr;rrW)rTr pre_dispatchr6c3s|]}ttdVqdS)rzN)r r)r,rx)rr+r-rVsz0test_dispatch_multiprocessing..rz Produced 0Nriz Consumed anyz Produced 3r)r%Managerrrr[indexro)r6managerrZqueue_contentsZfirst_consumption_indexZproduced_3_indexr+)rr-test_dispatch_multiprocessings    rc CsFtdddd.}|ddtdD|jjdks8tWdQRXdS) Nr;autor1)rTrr6css|]}tt|VqdS)N)r r)r,rUr+r+r-rVsz/test_batching_auto_threading..ir)rrXrcompute_batch_sizer[)rr+r+r-test_batching_auto_threadingsrc CsFtdd|d.}|ddtdD|jjdks8tWdQRXdS)Nr;r)rTrr6css|]}tt|VqdS)N)r r)r,rUr+r+r-rVsz2test_batching_auto_subprocesses..ir)rrXrrr[)r6rr+r+r-test_batching_auto_subprocessessrc Cs8tt&tddddddtdDWdQRXdS) zCMake sure that exception raised during dispatch are indeed capturedr;r)rTrracss|]}tt|VqdS)N)r rK)r,rUr+r+r-rVsz*test_exception_dispatch..rN)rrIrrXr+r+r+r-test_exception_dispatchs  rcCs tddddtdDdS)Nr;)rTcss|]}tt|VqdS)N)r rK)r,jr+r+r-rV sz(nested_function_inner..r)rrX)rUr+r+r-nested_function_inner srcCs tddddtdDdS)Nr;)rTcss|]}tt|VqdS)N)r r)r,rr+r+r-rVsz(nested_function_outer..r)rrX)rUr+r+r-nested_function_outersrz'https://github.com/joblib/loky/pull/255)reasonc Cstt$}td|dddtdDWdQRXt|j|j|j}dj|}d|ksZt d |ksft d |ksrt t|jtkst dS) zEnsure errors for nested joblib cases gets propagated We rely on the Python 3 built-in __cause__ system that already report this kind of information to the user. r;)rTr6css|]}tt|VqdS)N)r r)r,rUr+r+r-rVsz1test_nested_exception_dispatch..rNrrrK) rrIrrXrrQrhtbrr[)r6excinfoZ report_linesreportr+r+r-test_nested_exception_dispatchs      rc@s$eZdZdZdddZd ddZdS) FakeParallelBackendz8Pretends to run concurrently while running sequentially.rNcKs|j||_||_|S)N)r#rTr)r>rTrZ backend_argsr+r+r- configure/s zFakeParallelBackend.configurecCs"|dkrttjd|d}|S)Nrr)maxr%r$)r>rTr+r+r-r#4sz$FakeParallelBackend.effective_n_jobs)rN)r)rDrErFrGrr#r+r+r+r-r,s rc Cs"tttddWdQRXdS)Nz unit-testing)r6)rrIrr+r+r+r-test_invalid_backend:s rc Cs:tt}td|djWdQRXdt|jks6tdS)Nr)rTr6z&n_jobs == 0 in Parallel has no meaning)rrIrZ_initialize_backendrjrhr[)r6rr+r+r-test_invalid_njobs?s rc Cs8z*tdtdtksttdtks(tWdtd=XdS)NZ test_backend)r!rr&r[r+r+r+r-test_register_parallel_backendFs   rc CsNttkstz$tdtdddttks0tWdtt_XttksJtdS)Nr1T)Z make_default)rSDefaultBackendr[r!r&rr'rr+r+r+r-test_overwrite_default_backendOs rc Cst|ddtj\}}|dks&ttddks6tt}|jdksJt|dkrvt|tksbtt|j tkstn|dkrt|t kstt|j t kstnX|dkrt|t kstt|j t kstn,|j drt|t kstt|j t kstWdQRXdS)NrW)rTr/r0r1Ztest_)r"rrRr[r#rrTrQrrrr startswithr)Z backend_nameactive_backend active_n_jobsrr+r+r-check_backend_context_manager[s$   rcCsg|] }d|qS)ztest_backend_%dr+)r,rUr+r+r-r.rscCsf|tkr|jt|tttks$tt|ttks:ttdddddtDttksbtdS)Nr;r1)rTr6css|]}|stt|VqdS)N)r r)r,r@r+r+r-rVsz/test_backend_context_manager..) r&setitemrrSrr[rr all_backends_for_context_manager)rpr6r+r+r-test_backend_context_managervs  rc@seZdZdZdddZdS)ParameterizedParallelBackendz9Pretends to run conncurrently while running sequentially.NcCs|dkrtd||_dS)Nzparam should not be None)rIparam)r>rr+r+r-rCsz%ParameterizedParallelBackend.__init__)N)rDrErFrGrCr+r+r+r-rsrc Cs|jtdtttksttddddttj\}}t |tksHt|j dksVt|dksbtt }|j dksvt|j |kst|ddtdD}WdQRX|dd tdDkstttkstdS) NZ param_backend*rW)rrTcss|]}tt|VqdS)N)r r)r,rUr+r+r-rVsz=test_parameterized_backend_context_manager..rdcSsg|] }t|qSr+)r)r,rUr+r+r-r.sz>test_parameterized_backend_context_manager..)rr&rrSrr[r"rrRrQrrrTrrX)rprrrrrr+r+r-*test_parameterized_backend_context_managers   rc Csttkstttddddttj\}}t|tks>t|jdksLt|dksXtt }|j dkslt|j |kszt|ddt dD}WdQRX|ddt dDkstttkstdS) N+)rrd)rTcss|]}tt|VqdS)N)r r)r,rUr+r+r-rVszFtest_directly_parameterized_backend_context_manager..cSsg|] }t|qSr+)r)r,rUr+r+r-r.szGtest_directly_parameterized_backend_context_manager..) rSrr[r"rrrRrQrrrTrrX)rrrrrr+r+r-3test_directly_parameterized_backend_context_managers   rcCstdtjS)Ng?)rosgetpidr+r+r+r-sleep_and_return_pidsrcCs<ttksttjdks ttddddtdDS)Nrr;)rTcss|]}ttVqdS)N)r r)r,rxr+r+r-rVsz"get_nested_pids..)rSrr[r_effective_n_jobsrXr+r+r+r-get_nested_pidssrcs eZdZdZfddZZS) MyBackendz:Backend to test backward compatibility with older backendscstt|jdS)Nr)superrget_nested_backend)r>) __class__r+r-rszMyBackend.get_nested_backend)rDrErFrGr __classcell__r+r+)rr-rsrZback_compat_backendc CsVt|DtddddtdD}x |D]}tt|dks,tq,WWdQRXdS)Nr;)rTcss|]}ttVqdS)N)r r)r,rxr+r+r-rVsz6test_nested_backend_context_manager..rr)r"rrXrosetr[)r6Z pid_groupsZ pid_groupr+r+r-#test_nested_backend_context_managers   rc shddtddfddtdDtd*tddfddtdDWdQRXdS) NcSs0tt|kstt|}tj|ks,tdS)N)rSr&r[r#rr)Zexpected_backend_typeZexpected_n_jobr+r+r-check_nested_backendsz?test_nested_backend_in_sequential..check_nested_backendr)rTc3s|]}tddVqdS)r0rN)r )r,rx)rr+r-rVsz4test_nested_backend_in_sequential..rc3s|]}tVqdS)N)r )r,rx)r6rrTr+r-rVs)rrXr")r6rTr+)r6rrTr-!test_nested_backend_in_sequentials  rc Cs*t|\}}|j|kstWdQRXdS)N)r" nesting_levelr[) inner_backendZexpected_levelr6rTr+r+r-check_nesting_levelsr outer_backendrc sdt|dtd|dfddtdDtdd"tfddtdDWdQRXdS) Nrr;)rTr6c3s|]}ttdVqdS)rN)r r)r,rx)rr+r-rVsz-test_backend_nesting_level..r)rTc3s|]}ttdVqdS)rN)r r)r,rx)rr+r-rVs)rrrXr")rrr+)rr-test_backend_nesting_level s   rc sxddlGfdddt}td|ddtd8\}}tddfd d td D|jd ksjtWdQRXdS) NrcseZdZdZjddZdS)z)test_retrieval_context..MyBackendrcss|jd7_dVdS)Nr)rU)r>r+r+r-retrieval_context!sz;test_retrieval_context..MyBackend.retrieval_contextN)rDrErFrUcontextmanagerrr+) contextlibr+r-rsrZ retrievalcSstddddt|DS)Nr;)rTcss|]}tt|VqdS)N)r r)r,rUr+r+r-rV)sz>test_retrieval_context..nested_call..)rrX)nr+r+r- nested_call(sz+test_retrieval_context..nested_callr;)rTc3s|]}t|VqdS)N)r )r,rU)rr+r-rV-sz)test_retrieval_context..rdr)rrr!r"rrXrUr[)rZbarxr+)rrr-test_retrieval_contexts  rcCstd}t|tj|dS)NZfoobar)r)reprpickledumps)er+r+r-test_joblib_exception5srcCsNtt}tt|ddWdQRXtt}tt|dWdQRXdS)Nrrr8)rr:rrrNr*)Z safe_divisionZsafe_interruptr+r+r-test_safe_function>s   rrgQ?c Cs"ttt|dWdQRXdS)N)r)rrIr)rr+r+r-test_invalid_batch_sizeHs rz)n_tasks, n_jobs, pre_dispatch, batch_sizer{rrr 2 * n_jobsrHcCsJ|||d}ddt|D}tf|ddt|D}||ksFtdS)N)rTrrcSsg|] }t|qSr+)r<)r,rUr+r+r-r.`sz0test_dispatch_race_condition..css|]}tt|VqdS)N)r r<)r,rUr+r+r-rVasz/test_dispatch_race_condition..)rXrr[)Zn_tasksrTrrparamsrurrr+r+r-test_dispatch_race_conditionNs rcCs8tj}tddd}|jjd}|j}||ks4tdS)Nr;r/)rTr6context)r%get_start_methodrZ _backend_argsgetr[)Zmp_start_methodrrZ start_methodr+r+r-test_default_mp_contextes   rcs^|dkrtjd}tjjd}|jddtjjtd|dfddt dDdS) Nr/r3rir;)rTr6c3s |]}ttjjVqdS)N)r r dotT)r,rU)r?r+r-rVszAtest_no_blas_crash_or_freeze_with_subprocesses..) r%r2r randomZ RandomStaterandnrrrrX)r6rngr+)r?r-.test_no_blas_crash_or_freeze_with_subprocessesns    r afrom joblib import Parallel, delayed def square(x): return x ** 2 backend = "{}" if backend == "spawn": from multiprocessing import get_context backend = get_context(backend) print(Parallel(n_jobs=2, backend=backend)( delayed(square)(i) for i in range(5))) cCsB|dkrtjdkrtjdtj|}ttjd|gddddS)Nr/forkzVRequire fork start method to use interactively defined functions with multiprocessing.z-crz\[0, 1, 4, 9, 16\])r stdout_regex) r%rpytestskip,UNPICKLABLE_CALLABLE_SCRIPT_TEMPLATE_NO_MAINformatrsys executable)r6coder+r+r-2test_parallel_with_interactively_defined_functionss    raimport sys # Make sure that joblib is importable in the subprocess launching this # script. This is needed in case we run the tests from the joblib root # folder without having installed joblib sys.path.insert(0, {joblib_root_folder!r}) from joblib import Parallel, delayed def run(f, x): return f(x) {define_func} if __name__ == "__main__": backend = "{backend}" if backend == "spawn": from multiprocessing import get_context backend = get_context(backend) callable_position = "{callable_position}" if callable_position == "delayed": print(Parallel(n_jobs=2, backend=backend)( delayed(square)(i) for i in range(5))) elif callable_position == "args": print(Parallel(n_jobs=2, backend=backend)( delayed(run)(square, i) for i in range(5))) else: print(Parallel(n_jobs=2, backend=backend)( delayed(run)(f=square, x=i) for i in range(5))) z!def square(x): return x ** 2 zcdef gen_square(): def square(x): return x ** 2 return square square = gen_square() zsquare = lambda x: x ** 2 define_funccallable_positionr rtkwargscCsv|d kr$|tkstjdkr$tjdtj|||tjj tjj t j d}|j d}|j |ttj|jgddd dS) Nr/r3win32zNot picklable with pickle)rr6rjoblib_root_folderzunpicklable_func_script.pyrz\[0, 1, 4, 9, 16\])rr )r/r3) SQUARE_MAINrplatformrr)UNPICKLABLE_CALLABLE_SCRIPT_TEMPLATE_MAINrrpathdirnamejoblib__file__rwriterrstrpath)r6rrtmpdirrZ code_filer+r+r-0test_parallel_with_unpicklable_functions_in_argss    r%aimport sys # Make sure that joblib is importable in the subprocess launching this # script. This is needed in case we run the tests from the joblib root # folder without having installed joblib sys.path.insert(0, {joblib_root_folder!r}) from joblib import Parallel, delayed from functools import partial class MyClass: '''Class defined in the __main__ namespace''' def __init__(self, value): self.value = value def square(x, ignored=None, ignored2=None): '''Function defined in the __main__ namespace''' return x.value ** 2 square2 = partial(square, ignored2='something') # Here, we do not need the `if __name__ == "__main__":` safeguard when # using the default `loky` backend (even on Windows). # The following baroque function call is meant to check that joblib # introspection rightfully uses cloudpickle instead of the (faster) pickle # module of the standard library when necessary. In particular cloudpickle is # necessary for functions and instances of classes interactively defined in the # __main__ module. print(Parallel(n_jobs=2)( delayed(square2)(MyClass(i), ignored=[dict(a=MyClass(1))]) for i in range(5) )) )rcCs.|jd}|jtttj|jgddddS)Nz(joblib_interactively_defined_function.pyz\[0, 1, 4, 9, 16\]rd)r r)rr"5INTERACTIVE_DEFINED_FUNCTION_AND_CLASS_SCRIPT_CONTENTrrrr#)r$scriptr+r+r-Btest_parallel_with_interactively_defined_functions_default_backends    r(auimport sys # Make sure that joblib is importable in the subprocess launching this # script. This is needed in case we run the tests from the joblib root # folder without having installed joblib sys.path.insert(0, {joblib_root_folder!r}) from joblib import Parallel, delayed, hash import multiprocessing as mp mp.util.log_to_stderr(5) class MyList(list): '''MyList is interactively defined by MyList.append is a built-in''' def __hash__(self): # XXX: workaround limitation in cloudpickle return hash(self).__hash__() l = MyList() print(Parallel(n_jobs=2)( delayed(l.append)(i) for i in range(3) )) cCs0|jd}|jtttj|jgdddddS)Nz)joblib_interactive_bound_method_script.pyz\[None, None, None\]Z LokyProcess)r Z stderr_regexr)rr"9INTERACTIVELY_DEFINED_SUBCLASS_WITH_METHOD_SCRIPT_CONTENTrrrr#)r$r'r+r+r-5test_parallel_with_interactively_defined_bound_method9s    r+cCs"tg}tdd|gkstdS)Nr;)rT)iterrr[)Zexhausted_iteratorr+r+r-%test_parallel_with_exhausted_iteratorCsr-cCs"t|tjstdt||jS)Nz#Expected np.memmap instance, got %r)rr memmap TypeErrorrQcopy)r?r+r+r- check_memmapHs  r1cCsdd}tdd|ddd|dD}x,t||t|D]\}}tjj||q.generate_arraysr;r)rT max_nbytesr6css|]}tt|VqdS)N)r r1)r,r?r+r+r-rV]sz.rcricss|]}tt|VqdS)N)r r1)r,r?r+r+r-rVes)rrror testingassert_array_equal)r6r4rrresultrur+r+r-)test_auto_memmap_on_arrays_from_generatorOs  r9cCs|S)Nr+)argr+r+r-identityjsr;cs|jdj}tj}tj|ddtj|ddg}t||t|ddt ddfdd d D\}t d tj svt d j |kst tjj||dS) Nz test.mmapuint8)r2r)Z mmap_moder;)rTc3s|]}ttVqdS)N)r r;)r,rx)r.r+r-rVvsz.test_memmap_with_big_offset..rr)r)rr#mmapZALLOCATIONGRANULARITYr zerosr3r r rrr.r[offsetr6r7)r$fnamesizeobjr8r+)r.r-test_memmap_with_big_offsetns   rDc Csntd"}tddddtdDWdQRXt|dks@t|d}t|jtsXtt|jdksjtdS)Nr)rcss|]}tt|VqdS)N)r r<)r,rUr+r+r-rV~szFtest_warning_about_timeout_not_supported_by_backend..2rzThe backend class 'SequentialBackend' does not support timeout. You have set 'timeout=1' in Parallel but the 'timeout' parameter will not be used.) rrrXror[rrk UserWarningrj)rqrlr+r+r-3test_warning_about_timeout_not_supported_by_backend|s &rGc Csbdgdgd}tt(tj}t||ddd|DWdQRXtj|}|dks^tdS)Nr?rrc)rTr6css|]}ttj|VqdS)N)r rLr)r,rUr+r+r-rVsz%test_abort_backend..)rr/rLrr[)rTr6ZdelaysZt_startdtr+r+r-test_abort_backends   rJc Cs|j}tdd||d<}|ddtjjdgdDttj|dksNtWdQRXx,tdD]}tj|srPt d qbWtd tdd|d }|d dtjjdgdDx,tdD]}tj|sPt d qWtd dS) Nr;r)rTr5r6Z temp_foldercss|]}tt|VqdS)N)r r1)r,r?r+r+r-rVsz(test_memmapping_leaks..rrrcg?z/temporary directory of Parallel was not removed)rTr5r6css|]}tt|VqdS)N)r r1)r,r?r+r+r-rVs) r#rr rrorlistdirr[rXr)r6r$rrxr+r+r-test_memmapping_leakss"        rLcCs<td|dddtdD}|ddtdDks8tdS)Nr;)rTr6css|]}tdd|VqdS)cSs|dS)Nr;r+)r8r+r+r-sz2test_lambda_expression...N)r )r,rUr+r+r-rVsz)test_lambda_expression..rcSsg|] }|dqS)r;r+)r,rUr+r+r-r.sz*test_lambda_expression..)rrXr[)r6rrr+r+r-test_lambda_expressions rNcsd}d}d|td||d}|fddt|D|jj|jjksLt|jj|jjks`t|fddt|D|jj|jjkst|jj|jjkstd S) zCTest that a parallel backend correctly resets its batch statistics.r;ig@r)rarTr6c3s|]}ttjVqdS)N)r rLr)r,rU) task_timer+r-rVsz6test_backend_batch_statistics_reset..c3s|]}ttjVqdS)N)r rLr)r,rU)rOr+r-rVsN)rrXrZ_effective_batch_sizeZ_DEFAULT_EFFECTIVE_BATCH_SIZEr[Z_smoothed_batch_durationZ _DEFAULT_SMOOTHED_BATCH_DURATION)r6rTZn_inputsrr+)rOr-#test_backend_batch_statistics_resetsrPc Csx~dD]v}tt|djtks"tt|dd}t|jtks@tt|dd}t|jtks^tt|dd}t|jtkstqWtdd dd }t|jtksttd dd0tdd }t|jtkst|jdkstWdQRXtd dd6td dd}t|jtkst|jd ks tWdQRXtd dd4tdd }t|jtksVt|jdksftWdQRXtd dd6td dd}t|jtkst|jd kstWdQRXdS)Nrr;)rTthreads)rTprefer processes sharedmem)rTrequirer0)rTr6rR)rRrW)rUr)rr;r)rQrrrr[rr"rT)rTrr+r+r-$test_backend_hinting_and_constraintss4        rVcCs@Gdddt}t|Btddd}t|j|ks:ttddd}t|j|ksXtWdQRXGdd d t}t|tddd}t|j|kst|j\}}|d kst|d ksttddd d }t|jtkst|j\}}d }|j|kst|d kstWdQRXt t t|ddWdQRXdS)Nc@s$eZdZdZdZddZddZdS)z[test_backend_hinting_and_constraints_with_custom_backends..MyCustomThreadingBackendTcSsdS)Nr+)r>r+r+r- apply_async szgtest_backend_hinting_and_constraints_with_custom_backends..MyCustomThreadingBackend.apply_asynccSs|S)Nr+)r>rTr+r+r-r#szltest_backend_hinting_and_constraints_with_custom_backends..MyCustomThreadingBackend.effective_n_jobsN)rDrErFsupports_sharedmem use_threadsrWr#r+r+r+r-MyCustomThreadingBackend srZr;rS)rTrRrT)rTrUc@s$eZdZdZdZddZddZdS)z\test_backend_hinting_and_constraints_with_custom_backends..MyCustomProcessingBackendFcSsdS)Nr+)r>r+r+r-rWszhtest_backend_hinting_and_constraints_with_custom_backends..MyCustomProcessingBackend.apply_asynccSs|S)Nr+)r>rTr+r+r-r#!szmtest_backend_hinting_and_constraints_with_custom_backends..MyCustomProcessingBackend.effective_n_jobsN)rDrErFrXrYrWr#r+r+r+r-MyCustomProcessingBackendsr[rr)rTrUrazUsing ThreadingBackend as joblib.Parallel backend instead of MyCustomProcessingBackend as the latter does not provide shared memory semantics.)r6rU) rr"rrQrr[Z readouterrrstriprrI)ZcapsysrZrr[outerrrur+r+r-9test_backend_hinting_and_constraints_with_custom_backendss*          r_c'CstttddWdQRXtttddWdQRXtttdddWdQRXtttdddWdQRXtttd ddWdQRXdS) Ninvalid)rR)rUrSrT)rRrUr0)r6rUr/)rrIrr+r+r+r-,test_invalid_backend_hinting_and_constraints:s     racCsFtj}td}ttjts"t|jttjt|ksBtdS)Nr1)rrr"rrr[ unregisterrQ)defaultZpbr+r+r-test_global_parallel_backendNs rdc Cs:dd}|td<tdttjts,tWdQRXdS)NcSs ttd<dS)Nfoo)rr&r+r+r+r- register_fooYsz,test_external_backends..register_foore)r(r"rrrrr[)rfr+r+r-test_external_backendsXs rgc s`tddL}t|jj|jjfg}dkr.|S|fddtdD}||dSQRXdS)zCPerform nested parallel calls and introspect the backend on the wayr;)rTrc3s(|] }ttfddiVqdS)limitrN)r _recursive_backend_info)r,rU)rrhr+r-rVisz*_recursive_backend_info..rN)rrQrrDrrX)rhrrZ this_levelrrr+)rrhr-ribs ric Csrt|dd t}WdQRXtdkr2d}d}nd}d}|jd}|df|dfd|fd|fg}||ksntdS)Nr;)rTrrrZBackendr)r"rir$titler[)r6backend_types_and_levelsZsecond_level_backend_typeZ max_levelZtop_level_backend_typeZexpected_types_and_levelsr+r+r-test_nested_parallelism_limitns   rlzThis test requires daskc Cstjddd}tjtdtjd}xTtdD]H}tdt|d}WdQRXt |dks^t t dd |Ds,t q,Wtd t}WdQRXt |dkst t d d |Dst dS) Nr;)Z n_workersZthreads_per_workergcA)r2dask)dataricss|]\}}|dkVqdS)DaskDistributedBackendNr+)r,rfrxr+r+r-rVsz4test_nested_parallelism_with_dask..css|]\}}|dkVqdS)roNr+)r,rfrxr+r+r-rVs) distributedZClientr r3intr<rXr"riror[r{)clientrnrUrkr+r+r-!test_nested_parallelism_with_dasks  rscCstddtdDS)z6A horrible function that does recursive parallel callscss|]}ttVqdS)N)r _recursive_parallel)r,rUr+r+r-rVsz&_recursive_parallel..r;)rrX)Z nesting_limitr+r+r-rtsrtcCsht|dd tt }tWdQRXWdQRX|j}|dkrVt|trVtjdnt|t sdt dS)Nr;)rTr0z1Loky worker crash when serializing RecursionError) r"r BaseExceptionrtrhrrrxfailRecursionErrorr[)r6rexcr+r+r-test_thread_bomb_mitigations  rycCs.i}xd D]}tjj|||<q W|tdfS) NOMP_NUM_THREADSOPENBLAS_NUM_THREADSMKL_NUM_THREADSVECLIB_MAXIMUM_THREADSNUMEXPR_NUM_THREADSNUMBA_NUM_THREADS ENABLE_IPCrc)rzr{r|r}r~rr)renvironrr)Zenv_varsvarr+r+r-_run_parallel_sums rzNeed OpenMP helper compiledcCstd|dddtdD}ttdd}xf|D]^\}}||ksHtxH|jD]<\}}|jdrv|t|kstqR|dkst|dksRtqRWq4WdS) Nr;)rTr6css|]}ttVqdS)N)r r)r,rxr+r+r-rVsz-test_parallel_thread_limit..rZ_THREADSr1)rrXrr$r[itemsendswithrj)r6rrZexpected_num_threadsZworker_env_varsZomp_num_threadsrfrhr+r+r-test_parallel_thread_limits    rz%This test requires dask NOT installedc Cs$ttddtdWdQRXdS)NzPlease install dask)rrm)rrIr"r+r+r+r-)test_dask_backend_when_dask_not_installedsrcCsdGdddt}d}t|<tjt|d"tddddtdDWdQRXWdQRXdS) Nc@s&eZdZddZdddZddZdS) z3test_zero_worker_backend..ZeroWorkerBackendc_sdS)Nrr+)r>rtrr+r+r-rsz=test_zero_worker_backend..ZeroWorkerBackend.configureNcSs tddS)NzNo worker available)r)r>funccallbackr+r+r-rWsz?test_zero_worker_backend..ZeroWorkerBackend.apply_asynccSsdS)Nrr+)r>rTr+r+r-r#szDtest_zero_worker_backend..ZeroWorkerBackend.effective_n_jobs)N)rDrErFrrWr#r+r+r+r-ZeroWorkerBackends rz&ZeroWorkerBackend has no active worker)rr;)rTcss|]}tt|VqdS)N)r r)r,rUr+r+r-rVsz+test_zero_worker_backend..)rr"rrrrrX)rZ expected_msgr+r+r-test_zero_worker_backends   rcsdadddksttddfddtdD}t|dhksLtdadks^ttddfd dtdD}t|dhkstdS) Nzoriginal valuecSstS)N)MY_GLOBAL_VARIABLEr+r+r+r- check_globalssz@test_globals_update_at_each_parallel_call..check_globalsr;)rTc3s|]}tVqdS)N)r )r,rU)rr+r-rVsz.z changed valuec3s|]}tVqdS)N)r )r,rU)rr+r-rVs)rr[rrXr)Zworkers_global_variabler+)rr-)test_globals_update_at_each_parallel_callsrcCs4ddl}|jjdd}|j||ddlm}|S)Nrrc)threadpool_info)Znumpyrr rZ threadpoolctlr)r r?rr+r+r-_check_numpy_threadpool_limits s   rcCs8x$|D]}|d|dkr|dSqWtdj|dS)Nfilepath num_threadsz,An unexpected module was loaded in child: {})rIr) child_module parent_info parent_moduler+r+r-_parent_max_num_threads_fors   rcCsHxB|D]:}x4|D],}t||}t|||h}|d|kstqWqWdS)Nr)rminr[)Z workers_inforrZchild_threadpool_inforZparent_max_num_threadsrur+r+r-check_child_num_threadss   rcCsbt}t|dkrtjddt|dddtdD}t|}tt|d}t |||dS) Nrz&Need a version of numpy linked to BLAS)msg)rTcss|]}ttVqdS)N)r r)r,rUr+r+r-rV5sz6test_threadpool_limitation_in_child..r;r) rrorrrrXr#rr$r)rTrworkers_threadpool_infosexpected_child_num_threadsr+r+r-#test_threadpool_limitation_in_child(s  rinner_max_num_threadsc Cst}t|dkrtjddtd|d"t|dddtd D}WdQRXt|}|dkrttt |d }n|}t |||dS) Nrz&Need a version of numpy linked to BLAS)rr0)r)rTcss|]}ttVqdS)N)r r)r,rUr+r+r-rVMsz>test_threadpool_limitation_in_child_context..r;r) rrorrr"rrXr#rr$r)rTrrrrr+r+r-+test_threadpool_limitation_in_child_context>s   rvar_namer{r|rzcstddjddtjj}zdtj<t|dfddtd D}|ddgks`ttd d d (t|dfd dtd D}WdQRX|ddgkstWd|dkrtj=n |tj<XdS)NT)reusecSs tjj|S)N)rrr)rr+r+r-_get_envfsz>test_threadpool_limitation_in_child_override.._get_env4)rTc3s|]}tVqdS)N)r )r,rU)rrr+r-rVnsz?test_threadpool_limitation_in_child_override..r;r0r)rc3s|]}tVqdS)N)r )r,rU)rrr+r-rVssr) r shutdownrrrrrXr[r")rTrZoriginal_var_valuerrr+)rrr-,test_threadpool_limitation_in_child_overrideYs   $ rc Cs(ttddt|ddWdQRXdS)Nz#does not acc.*inner_max_num_threads)rr)r)rr[r")r6r+r+r-1test_threadpool_limitation_in_child_context_error}srcCsNdd}||tdd}x.tdD]"}||tdd}||ks$tq$WdS)Ncs,tdt|dfddtdDdS)Nr)rTc3s|]}ttVqdS)N)r sum)r,rU)r8r+r-rVszAtest_loky_reuse_workers..parallel_call..)rXr)rTr+)r8r- parallel_callsz.test_loky_reuse_workers..parallel_callT)rr)r rXr[)rTrZfirst_executorrxexecutorr+r+r-test_loky_reuse_workerss  r)F)rr)rWrWrr)Nrr)r0r/T)r0r0F)r/r/T)r/r0T)r1r/T)r1r0Trr)r;r;r{r)r;r;rTr)rr;rTr)rr;rTr)rr;rTr)rrirTr)rrrTr)rrrr)rrr{r)rrrrH)rrrrrr)rW)Nrrrrr)rGrrrLr>r1 tracebackrmathrrrrr/rr importlibrr rr r Zjoblib.externals.lokyr Zjoblib.test.commonr r rZjoblib.testingrrrrrrZ&joblib.externals.loky.process_executorrrrposix ImportErrorZ _openmp_test_helper.parallel_sumrrpZjoblib._parallel_backendsrrrrrrZjoblib.parallelrr r!r"r#r$r%r&r'r(Zjoblib.my_exceptionsr)r*sortedkeysZALL_VALID_BACKENDSZPROCESS_BACKENDSZPARALLEL_BACKENDShasattrrr2rr7r:r<rr=rKrNrPrSrZr\r]markr`rersrwr~rrrrrrrrrrrrrrrrrrrrrvrrrrrrrrextendrXrrrrrrZ_parallel_backendsrrrrrrrrrrrr rrrrZ SQUARE_LOCALZ SQUARE_LAMBDAr%rrrr!r&r(r*r+r-r1r9r;rDrGrJrLrNrPrVr_rardrgrirlrsrtryrrrrrrrrrrrrrr+r+r+r-s                               J             %+5  #     $.4        !  &"