3 dX/@sdZddlZddlZddlZddlmZy ddlZWn&ek rZZz dZWYddZ[XnXddl m Z m Z e j dZ e jZdZGdd d ZGd d d ejZd d ZddZdddZddZdddZddZddZdS)z( Utilities to keep track of performance N)time)configloggingz nipype.utilsg@c@s6eZdZdZeddZd ddZd d Zd d ZdS)ResourceMonitorMockz1A mock class to use when the monitor is disabled.cCsdS)zGet/set the internal filenameN)selfrr7/tmp/pip-build-7vycvbft/nipype/nipype/utils/profiler.pyfnameszResourceMonitorMock.fnameNTcCsdS)Nr)rpidfreqr pythonrrr __init__#szResourceMonitorMock.__init__cCsdS)Nr)rrrr start&szResourceMonitorMock.startcCsiS)Nr)rrrr stop)szResourceMonitorMock.stop)r NT) __name__ __module__ __qualname____doc__propertyr rrrrrrr rs   rc@s@eZdZdZdddZeddZd d Zdd d Zd dZ dS)ResourceMonitorzW A ``Thread`` to monitor a specific PID with a certain frequence to a file r NTcCsddl}|dkrtd||dkr4d|t|f}tjj||_t|jd|_||_ ||_ |j ||_ |j ddtjj|tj|_dS)Nrg?z,Frequency (%0.2fs) cannot be lower than 0.2sz.proc-%d_time-%s_freq-%0.2fw) cpu_interval)psutil RuntimeErrorrospathabspath_fnameopen_logfile_freqZ_pythonProcess_process_sample threadingThreadrEvent_event)rr r r rrrrr r3s    zResourceMonitor.__init__cCs|jS)zGet/set the internal filename)r)rrrr r IszResourceMonitor.fnamecCs|jjs8|jj|j|j|jj|jjddd}tj |j dd}|j rtj |}|dddfj d|d<|dddfj |d <|ddd fj|dddfj|dddfdj|ddd fdjd |d <|S)zStop monitoring.N) mem_peak_gb cpu_percent,) delimiterrir*r+r)rZcpusZrss_GiBZvms_GiBZ prof_dict)r)is_setsetjoinr%r!flushclosenpZloadtxtrsizeZ atleast_2dmaxtolist)rretvalvalsrrr rNs$      zResourceMonitor.stopc%Cs4d}d}d}yJ|jj6||jj|d7}|jj}||j7}||j7}WdQRXWntjk rlYnXy|jjdd}Wntjk rg}YnXxd|D]\}y@|j.||j7}|j}||j7}||j7}WdQRXWqtjk rYqXqWt dt ||t |t f|j d|j j dS)Ng)intervalT) recursivez %f,%f,%f,%f)file)r$Zoneshotr+ memory_inforssvmsr NoSuchProcesschildrenprintr_MBr!r3)rrcpur?r@Zmem_inforBchildrrr r%js4         $zResourceMonitor._samplecCsHt}|}x8|jjsB|j||j7}|jjtd|tq WdS)z+Core monitoring function, called by start()rN)rr)r0r%r"waitr7)r start_timeZwait_tilrrr runs   zResourceMonitor.run)r NT)N) rrrrrrr rr%rIrrrr r-s    rc Cs|dkr dSddl}ddl}|j|jt|jjdt|jjdt|jjdt|jjddt|jjd d|j|jd }|d dks|d dkrd |d<|j dj |j |dS)aFunction to record node run statistics to a log file as json dictionaries Parameters ---------- node : nipype.pipeline.engine.Node the node being logged status : string acceptable values are 'start', 'end'; otherwise it is considered and error Returns ------- None this function does not return any values, it logs the node status info to the callback logger endNrZ startTimeZendTimedurationr+zN/Ar*) nameidrfinishrKZruntime_threadsZruntime_memory_gbZestimated_memory_gb num_threadsrrNTerrorcallback) rjsonrL_idgetattrresultZruntimemem_gbn_procs getLoggerdebugdumps)nodestatusrrRZ status_dictrrr log_nodes_cbs     r]c Csddl}ddl}d|jkrdtdd8}|j}dd|Dd}t|jd}|d}WdQRXn@d |jkr|jd jj jd d}t|d}n d}t ||S)zC Function to get the total RAM of the running system in GB rNlinuxz /proc/meminforcSsg|]}d|kr|qS)ZMemTotalr).0linerrr sz.get_system_total_memory_gb..r.g@rdarwinzsysctl hw.memsize r/z$System platform: %s is not supportedg0AgA) rsysplatformr readlinesfloatsplitpopenreadstrip Exception) rrfZf_inZ meminfo_linesZmem_total_line mem_totalZ memory_gbZmem_strerr_msgrrr get_system_total_memory_gbs   rqFcCshts tdy$t|t||d}t|t|}Wn.tk r^}ztjd|WYdd}~XnX||fS)a Function to get the RAM and threads utilized by a given process Parameters ---------- pid : integer the process ID of process to profile mem_mb : float the high memory watermark so far during process execution (in MB) num_threads: int the high thread watermark so far during process execution Returns ------- mem_mb : float the new high memory watermark of process (MB) num_threads : float the new high thread watermark of process zHAttempted to measure resources with option "monitoring.enabled" set off.)pyfuncz+Could not get resources used by process. %sN)resource_monitorrr7 _get_ram_mb_get_num_threadsrn profloggerinfo)r mem_mbrOrrexcrrr get_max_resources_usedsrzc Cs ytj|}|jtjkr$|j}n:|jdkrZdd|jD}dd|D}t|}nd}d}x~|jddD]n}t|jdkrp|jtjkr|j}n:|jdkrdd|jD}d d|D}t|}nd}||7}qpWWntjk rd SXt ||}|S) a Function to get the number of threads a process is using Parameters ---------- pid : integer the process ID of process to profile Returns ------- num_threads : int the number of threads that the process is using r.cSsg|]}tj|jqSr)rr#rM)r`thrrrr rbsz$_get_num_threads..cSsg|]}|jtjkr|qSr)r\rSTATUS_RUNNING)r`tprocrrr rbsrT)r<cSsg|]}tj|jqSr)rr#rM)r`r{rrr rb-scSsg|]}|jtjkr|qSr)r\rr|)r`r}rrr rb/sN) rr#r\r|rOthreadslenrBrAr7)r procrOZtprocsZ alive_tprocsZ child_threadsrFZ child_thrrrr rus2        ruc CstyXtj|}|jj}|t}x6|jddD]&}|jj}|rF||8}||t7}q,WWntjk rndSX|S)a Function to get the RAM usage of a process and its children Reference: http://ftp.dev411.com/t/python/python-list/095thexx8g/multiprocessing-forking-memory-usage Parameters ---------- pid : integer the PID of the process to get RAM usage of pyfunc : boolean (optional); default=False a flag to indicate if the process is a python function; when Pythons are multithreaded via multiprocess or threading, children functions include their own memory + parents. if this is set, the parent memory will removed from children memories Returns ------- mem_mb : float the memory RAM in MB utilized by the process PID T)r<N)rr#r>r?rDrBrA)r rrparentZ parent_memrxrFZ child_memrrr rtDs   rtcCs&d}x|dkr |d7}||qWdS)NrgcAr.r)xZctrrrr _use_cpuns rcsddl}ddlddl}ddlm}ddlm}ddlm}|j d}j dj dj d dfd d }|j |j } | j j} ||| } |d| j j} ~ |jd|j | | |dkr||} | j|t|dS)zG Function to execute multiple use_gb_ram functions in parallel rN)Pool)r)rznipype.interfacez rdg@r/cs2t|}d|}j||ks.t|S)z+A test function to consume mem_gb GB of RAMrd)int getsizeofAssertionError)rV num_bytesZgb_str)BOFFSETBSIZE_GBrfrr _use_gb_rams z#_use_resources.._use_gb_ramr z)[%d] Memory offset %0.2fGB, total %0.2fGBr.TgA)rrfrmultiprocessingrZnipyperZnipype.utils.profilerrrXrr#getpidr>r?rwmaprange)rWrVrrrrrZifloggerrpZ mem_offsetZbig_strropoolr)rrrrfr _use_resourcesvs.       rg0A)F)F)rrnumpyr5r&rr ImportErrorryrrrrXrvrsrDrr'rr]rqrzrurtrrrrrr s*   h. %@ *