3 Ud @s~dZddlmZddlmZddlZddlZddlZ ddlm Z ddl j Z ddZejjdd Zd d Zejjd d ZdS)zd Tests multithreading behaviour for reading and parsing files for each parser defined in parsers.py )BytesIO) ThreadPoolN) DataFramecCs\ttjj|dtdd}d|d<d|d<d|d<tjd|dd |d <tj|d d |d <|S)z Construct a DataFrame for testing. Parameters ---------- num_rows : int The number of rows for our DataFrame. Returns ------- df : DataFrame Zabcde)columnsZfoobarZbazz20000101 09:00:00s)ZperiodsfreqdateZint64)Zdtypeint)rnprandomZrandlistpdZ date_rangeZarange)num_rowsdfr\/var/www/html/virt/lib64/python3.6/site-packages/pandas/tests/io/parser/test_multi_thread.py_construct_dataframes rc sl|}dd}fddt|D}dd|D}td}|j|j|}|d}x|D]}tj||qTWdS)Ni'dcs(g|] }djddtDjqS) cSs&g|]}|dd|dd|dqS)d,r).0irrr .szCtest_multi_thread_string_io_read_csv...)joinrangeencode)r_) max_row_rangerrr.sz8test_multi_thread_string_io_read_csv..cSsg|] }t|qSr)r)rbrrrr1sr)rrmapread_csvtmassert_frame_equal) all_parsersparserZ num_filesZ bytes_to_dffilespoolresultsZ first_resultresultr)r r$test_multi_thread_string_io_read_csv&s   r-c spfdd}fddtD}td}|j||}|dj}x|ddD] } || _qTWtj|} | S) a Generate a DataFrame via multi-thread. Parameters ---------- parser : BaseParser The parser object to use for reading the data. path : str The location of the CSV file to read. num_rows : int The number of rows to read per task. num_tasks : int The number of tasks to use for reading this DataFrame. Returns ------- df : DataFrame csB|\}}|s"jdd|dgdSjddt|d|dgdS)aj Create a reader for part of the CSV. Parameters ---------- arg : tuple A tuple of the following: * start : int The starting row to start for parsing CSV * nrows : int The number of rows to read. Returns ------- df : DataFrame rr ) index_colheadernrows parse_datesN )r.r/Zskiprowsr0r1)r$r )argstartr0)r(pathrrreaderQs z0_generate_multi_thread_dataframe..readercs g|]}|fqSrr)rr)r num_tasksrrrtsz4_generate_multi_thread_dataframe..) processesrr2N)rrr#rrconcat) r(r6rr8r7tasksr*r+r/rfinal_dataframer)rr8r(r6r _generate_multi_thread_dataframe=s#     r>c CsVd}d}|}d}t|}tj|*}|j|t||||}tj||WdQRXdS)Niz__thread_pool_reader__.csv)rr%Z ensure_cleanZto_csvr>r&)r'r8rr( file_namerr6r=rrr)test_multi_thread_path_multipart_read_csvs   rA)__doc__iorZmultiprocessing.poolrZnumpyr ZpytestZpandasrrZpandas._testingZ_testingr%rmarkZslowr-r>rArrrrs    F