3 du(@sdZddlZddlZddlZddlZddlZddlZddlmZddl m Z ddl m Z Gddde Zdd d Zdd d ZGddde ZGddde ZdS)z Using nipype with persistence and lazy recomputation but without explicit name-steps pipeline: getting back scope in command-line based programming. N) BaseInterface)Node) modify_pathsc@s*eZdZdZd ddZddZddZdS) PipeFuncaCallable interface to nipype.interface objects Use this to wrap nipype.interface object and call them specifying their input with keyword arguments:: fsl_merge = PipeFunc(fsl.Merge, base_dir='.') out = fsl_merge(in_files=files, dimension='t') NcCst|tot|ts(td|t|f||_tjj|}tjj | r\tjj |r\td||_ d|jj |jj ddf}||_ ||_dS)ag Parameters =========== interface: a nipype interface class The interface class to wrap base_dir: a string The directory in which the computation will be stored callback: a callable An optional callable called each time after the function is called. zWthe interface argument should be a nipype interface class, but %s (type %s) was passed.z(base_dir should be an existing directoryz%s %sT)Z returnhelpN) isinstancetype issubclassr ValueError interfaceospathabspathexistsisdirbase_dir__doc__helpcallback)selfr rrdocr7/tmp/pip-build-7vycvbft/nipype/nipype/caching/memory.py__init__s zPipeFunc.__init__c Kst|dd}|j}|jjf||jj}tjd}|jtj |d|j j j dd|j j f}|j}t||d}tjj|j||_tj}z |j} Wdtj|X|jdk r|j||| S)NF)Zrelativemd5z%s-%s.-)name)rr inputsZ trait_setZ get_hashvalhashlibnewupdatepickledumps __class__ __module__replace__name__ hexdigestrr r joinrgetcwdrunchdirr) rkwargsr rZhasherdir_namejob_namenodecwdoutrrr__call__<s&         zPipeFunc.__call__cCsdj|jj|jj|jj|jS)Nz{}({}.{}), base_dir={}))formatr$r'r r%r)rrrr__repr__Xs zPipeFunc.__repr__)N)r'r% __qualname__rrr3r5rrrrrs rc Csl|dkrt}t|dJ}xB|D]:}|ddjd\}}|j|t}|j||||<q WWdQRX|S)Nr/)dictopensplitgetsetadd)filenameZrun_dictlogfileliner.r/jobsrrrread_logfs   rEFc Csytj|}Wntk r"dSXdd|D}t|j|}x@|D]8}tjj||}tjj|rF|rttd|t j |qFWdS)zRemove all the sub-directories of base_dir, but those listed Parameters ============ base_dir: string The base directory dirs_to_keep: set The names of the directories to keep NcSsg|]}|jds|qS)zlog.) startswith).0drrr szrm_all_but..zremoving directory: %s) r listdirOSErrorlistsymmetric_differencer r)rprintshutilrmtree)rZ dirs_to_keepwarnall_dirsZ dirs_to_rmr.rrr rm_all_butss    rSc@s eZdZdZddZddZdS)_MemoryCallbackz6An object to avoid closures and have everything picklecCs ||_dS)N)memory)rrUrrrrsz_MemoryCallback.__init__cCs|jj||dS)N)rU _log_name)rr.r/rrrr3sz_MemoryCallback.__call__N)r'r%r6rrr3rrrrrTsrTc@sNeZdZdZddZddZddZdd d Zdd d ZdddZ ddZ d S)MemoryaMemory context to provide caching for interfaces Parameters ========== base_dir: string The directory name of the location for the caching Methods ======= cache Creates a cacheable function from an nipype Interface class clear_previous_runs Removes from the disk all the runs that where not used after the creation time of the specific Memory instance clear_previous_runs Removes from the disk all the runs that where not used after the given time cCsdtjjtjj|d}tjj|s.tj|ntjj|sBtd||_t tjj|ddj dS)NZ nipype_memzbase_dir should be a directoryz log.currenta) r r r)rrmkdirrr rr<close)rrrrrrs   zMemory.__init__cCst||jt|S)aReturns a callable that caches the output of an interface Parameters ========== interface: nipype interface The nipype interface class to be wrapped and cached Returns ======= pipe_func: a PipeFunc callable object An object that can be used as a function to apply the interface to arguments. Inputs of the interface are given as keyword arguments, bearing the same name as the name in the inputs specs of the interface. Examples ======== >>> from tempfile import mkdtemp >>> mem = Memory(mkdtemp()) >>> from nipype.interfaces import fsl Here we create a callable that can be used to apply an fsl.Merge interface to files >>> fsl_merge = mem.cache(fsl.Merge) Now we apply it to a list of files. We need to specify the list of input files and the dimension along which the files should be merged. >>> results = fsl_merge(in_files=['a.nii', 'b.nii'], ... dimension='t') # doctest: +SKIP We can retrieve the resulting file from the outputs: >>> results.outputs.merged_file # doctest: +SKIP '...' )rrrT)rr rrrcaches'z Memory.cachec "Cs|j}ttjj|dd}|jd||fWdQRXtj}tjj|d|j}ytj |Wnt k rvYnXtjj|d|j }ytj |Wnt k rYnXttjj|d|j d}|jd||fWdQRXdS)z?Increment counters tracking which cached function get executed.z log.currentrXz%s/%s Nzlog.%iz%02iz%02i.log) rr<r r r)writetime localtimetm_yearrYrKtm_montm_mday) rr.r/rZ currentlogtZyear_dirZ month_dirZ rotatefilerrrrVs zMemory._log_nameTcCs*|j}ttjj|d}|j||ddS)a;Remove all the cache that where not used in the latest run of the memory object: i.e. since the corresponding Python object was created. Parameters ========== warn: boolean, optional If true, echoes warning messages for all directory removed z log.current)rQN)rrEr r r)_clear_all_but)rrQrZ latest_runsrrrclear_previous_runss zMemory.clear_previous_runsNc Cstj}|dk r|n|j}|dk r&|n|j}|dk r8|n|j}|j}d||||f}t}t} x4tjd|D]"} | |kr|j | qpt | | } qpW|j | |dx|D]} t j | qWdS)aRemove all the cache that where not used since the given date Parameters ========== day, month, year: integers, optional The integers specifying the latest day (in localtime) that a node should have been accessed to be kept. If not given, the current date is used. warn: boolean, optional If true, echoes warning messages for all directory removed Nz%s/log.%i/%02i/%02i.logz%s/log.*/*/*.log)rQ)r]r^rar`r_rrLr;globappendrErcr remove) rdaymonthyearrQrbrZ cut_off_fileZ logs_to_flushZ recent_runsZlog_namerrrclear_runs_sinces   zMemory.clear_runs_sincecCsPt|jt|j|dx2t|jD]"\}}ttjj|j|||dq&WdS)zSRemove all the runs appart from those given to the function input. )rQN) rSrr?keysrLitemsr r r))rrunsrQr.Z job_namesrrrrc!szMemory._clear_all_butcCsdj|jj|jS)Nz{}(base_dir={}))r4r$r'r)rrrrr5)szMemory.__repr__)T)NNNT)T) r'r%r6rrr[rVrdrkrcr5rrrrrWs )   rW)N)F)rr rr"r]rOreZinterfaces.baserZpipeline.enginerZpipeline.engine.utilsrobjectrrErSrTrWrrrrs   Q