3 d5@sLdZddlZddlZddlmZddlmZmZddZGd d d eZ dS) z&Parallel workflow execution via SLURM N) CommandLine)GraphPluginBaseloggerc CsX|jdkp|jdko|jj }d}y|j\}}}}Wntk rNd}YnX|oV|S)z A function to determine if a node has previously completed it's work :param checknode: The node to check the run status :return: boolean value True indicates that the node does not need to be run. FN) overwriteZ _interfaceZ always_run hash_exists Exception)Z checknodeZ%node_state_does_not_require_overwriter_r D/tmp/pip-build-7vycvbft/nipype/nipype/pipeline/plugins/slurmgraph.pynode_completed_status s  r cs,eZdZdZdZfddZddZZS)SLURMGraphPlugina0Execute using SLURM The plugin_args input to run can be used to control the SGE execution. Currently supported options are: - template : template to use for batch job submission - qsub_args : arguments to be prepended to the job execution script in the qsub call z #!/bin/bashc sd|_d|kr|drd|dkr0|dd|_d|dkrJ|dd|_d|dkr|dd|_tjj|jrt|jj|_d|dkr|dd|_d|dkr|dd|_ nd|_ t t |j f|dS) NZ plugin_argsZ retry_timeoutZ max_triestemplate sbatch_argsZdont_resubmit_completed_jobsF) _sbatch_argsZ_retry_timeoutZ _max_tries _templateospathisfileopenread_dont_resubmit_completed_jobssuperr__init__)selfkwargs) __class__r r r+s"      zSLURMGraphPlugin.__init__c Csdd}tjj|d\}}tjj|d}t}|jrxpt|D]d\} } || } t| } | r| |krx6|| D]*} | |kr|| }n t|| }| o|} qnW| || <q@Wt|d}|j d|j dxt|D]\} } || } |j | drqq|j | d d g\}}tjj| \}}d j|jd dd#}d j|dt j | ff}tjj|d|}|d}|d}t|d}|j ||jWdQRXd}| |krd}x<|| D]0}|j s|| r|dj|||7}qW|dkr|jd}d|}|| |}d}|jjddkr.dj|d}d}|jjddkrPdj|d}dj||||||d}|j |qWWdQRXtdttjddd }d!||j_|jtjd"dS)$NcSs2dj|||j}|jddjddjdd}|S)z - jobnumber: The index number of the job to create - nodeslist: The name of the node being processed - return: A string representing this job to be displayed by SLURM zj{0}_{1}-r .:)format_idreplace)Z jobnumberZ nodeslistZjob_namer r r make_job_nameAsz5SLURMGraphPlugin._submit_graph..make_job_namerzsubmit_jobs.shwtz#!/usr/bin/env bash z# Condense format attempted Frrr r z%s %szbatchscript_%s.shz.oz.erz ${{{0}}}:r!z--dependency=afterok:%sz-e z -e {errFile})ZerrFilez-o z -o {outFile})ZoutFilez{jobNm}=$(sbatch {outFileOption} {errFileOption} {extraSBatchArgs} {dependantIndex} -J {jobNm} {batchscript} | awk '/^Submitted/ {{print $4}}') )ZjobNmZ outFileOptionZ errFileOptionZextraSBatchArgsZdependantIndex batchscriptbashZ allatonce)environZresource_monitorZterminal_outputz%szsubmitted all jobs to queue)rrsplitjoindictr enumerater r writelinesget _get_argssys executablecloser"rstriprcountrr*inputsargsrunrinfo) rZpyfilesZ dependenciesZnodesr%Z batch_dirr ZsubmitjobsfileZcache_doneness_per_nodeidxZpyscriptnodeZnode_status_doneZ child_idxZchild_status_donefprrnamer(ZbatchscriptfileZbatchscriptoutfileZbatchscripterrfileZbatchfpdepsvaluesZjobidZjobnameZ stderrFileZ stdoutFileZ full_linecmdr r r _submit_graph@s                     zSLURMGraphPlugin._submit_graph)__name__ __module__ __qualname____doc__rrrC __classcell__r r )rr rs  r) rGrr3Zinterfaces.baserbaserrr rr r r r s