3 qdd* @sdZddlmZmZmZmZmZddlmZddl m Z ddl m Z ddl mZddlmZmZddlmZmZmZed d d d ZGd d d ZGdddeZGdddeZdee eeeeeeeeedddZdedddZdS)z parquet compat )AnyAnyStrDictListOptional)catch_warnings)FilePathOrBuffer)import_optional_dependency)AbstractMethodError) DataFrame get_option) _expand_userget_filepath_or_buffer is_fsspec_urlBaseImpl)enginereturncCs|dkrtd}|dkrzttg}d}xF|D]>}y|Stk rf}z|dt|7}WYdd}~Xq*Xq*Wtd||dkrtS|dkrtStd dS) z return our implementation autozio.parquet.enginez - NzUnable to find a usable engine; tried using: 'pyarrow', 'fastparquet'. A suitable version of pyarrow or fastparquet is required for parquet support. Trying to import the above resulted in these errors:pyarrow fastparquetz.engine must be one of 'pyarrow', 'fastparquet')r PyArrowImplFastParquetImpl ImportErrorstr ValueError)rZengine_classesZ error_msgsZ engine_classerrr3/tmp/pip-build-7vycvbft/pandas/pandas/io/parquet.py get_engines" & rc@s6eZdZeedddZedddZd ddZdS) r)dfcCsLt|tstd|jjdkr&tdtdd|jjD}|sHtddS) Nz+to_parquet only supports IO with DataFramesstringemptyz%parquet must have string column namescss |]}|dk rt|tVqdS)N) isinstancer).0namerrr >sz.BaseImpl.validate_dataframe..z!Index level names must be strings>r!r")r#r rcolumnsZ inferred_typeallindexnames)r Z valid_namesrrrvalidate_dataframe2s  zBaseImpl.validate_dataframecKs t|dS)N)r )selfr path compressionkwargsrrrwriteCszBaseImpl.writeNcKs t|dS)N)r )r,r-r'r/rrrreadFsz BaseImpl.read)N)__name__ __module__ __qualname__ staticmethodr r+r0r1rrrrr1sc@sJeZdZddZd eeeeeee ee edddZ d dd Z dS) rcCs&tdddddl}ddl}||_dS)Nrz(pyarrow is required for parquet support.)extrar)r Zpyarrow.parquetZpandas.core.arrays._arrow_utilsapi)r,rpandasrrr__init__Ks  zPyArrowImpl.__init__snappyN)r r-r.r)partition_colsc Ks|j|d|jddi}|dk r*||d<|jjj|f|}t|rvd|krvtdddl} | jj |\} }| |d<nt |}|dk r|jj j ||f||d|n|jj j ||fd|i|dS)NZschemaZpreserve_index filesystemfsspecr)r.r;r.)r+popr7ZTableZ from_pandasrr fsspec.corecore url_to_fsr parquetZwrite_to_datasetZ write_table) r,r r-r.r)r;r/Zfrom_pandas_kwargstabler=fsrrrr0Vs&   zPyArrowImpl.writec Kst|r6d|kr6tdddl}|jj|\}}d}n|jdd}d}t|}|sbt|\}}}}d|d<|jj j |f||d|j }|r|j |S)Nr<r=rFTZuse_pandas_metadata)r'r<) rr r?r@rAr>r rr7rBZ read_table to_pandasclose) r,r-r'r/r=rDZ should_close_resultrrrr1}s  zPyArrowImpl.read)r:NN)N) r2r3r4r9r rrrrboolrr0r1rrrrrJs ,!rc@s.eZdZddZd edddZd dd ZdS) rcCstddd}||_dS)Nrz,fastparquet is required for parquet support.)r6)r r7)r,rrrrr9s zFastParquetImpl.__init__r:N)r c s|j|d|kr$|dk r$tdnd|kr6|jd}|dk rFd|d<t|rhtdfdd|d<nt|\}}}}td d $|jj||f|||d |WdQRXdS) N partition_onzYCannot use both partition_on and partition_cols. Use partition_cols for partitioning dataZhiveZ file_schemer=csj|djS)Nwb)open)r-rG)r=rrsz'FastParquetImpl.write.. open_withT)record)r.Z write_indexrJ) r+rr>rr rrr7r0)r,r r-r.r)r;r/rGr)r=rr0s(   zFastParquetImpl.writec s^t|r.tdfdd}|jj||d}nt|\}}}}|jj|}|jfd|i|S)Nr=csj|djS)Nrb)rL)r-rG)r=rrrMsz&FastParquetImpl.read..)rNr')rr r7Z ParquetFilerrE)r,r-r'r/rNZ parquet_filerGr)r=rr1s  zFastParquetImpl.read)r:NN)N)r2r3r4r9r r0r1rrrrrs  %rrr:N)r r-rr.r)r;cKs4t|tr|g}t|}|j||f|||d|S)a Write a DataFrame to the parquet format. Parameters ---------- df : DataFrame path : str or file-like object If a string, it will be used as Root Directory path when writing a partitioned dataset. By file-like object, we refer to objects with a write() method, such as a file handler (e.g. via builtin open function) or io.BytesIO. The engine fastparquet does not accept file-like objects. .. versionchanged:: 0.24.0 engine : {'auto', 'pyarrow', 'fastparquet'}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. compression : {'snappy', 'gzip', 'brotli', None}, default 'snappy' Name of the compression to use. Use ``None`` for no compression. index : bool, default None If ``True``, include the dataframe's index(es) in the file output. If ``False``, they will not be written to the file. If ``None``, similar to ``True`` the dataframe's index(es) will be saved. However, instead of being saved as values, the RangeIndex will be stored as a range in the metadata so it doesn't require much space and is faster. Other indexes will be included as columns in the file output. .. versionadded:: 0.24.0 partition_cols : str or list, optional, default None Column names by which to partition the dataset. Columns are partitioned in the order they are given. Must be None if path is not a string. .. versionadded:: 0.24.0 kwargs Additional keyword arguments passed to the engine )r.r)r;)r#rrr0)r r-rr.r)r;r/implrrr to_parquets4 rR)rcKst|}|j|fd|i|S)a Load a parquet object from the file path, returning a DataFrame. Parameters ---------- path : str, path object or file-like object Any valid string path is acceptable. The string could be a URL. Valid URL schemes include http, ftp, s3, and file. For file URLs, a host is expected. A local file could be: ``file://localhost/path/to/table.parquet``. A file URL can also be a path to a directory that contains multiple partitioned parquet files. Both pyarrow and fastparquet support paths to directories as well as file URLs. A directory path could be: ``file://localhost/path/to/tables`` or ``s3://bucket/partition_dir`` If you want to pass in a path object, pandas accepts any ``os.PathLike``. By file-like object, we refer to objects with a ``read()`` method, such as a file handler (e.g. via builtin ``open`` function) or ``StringIO``. engine : {'auto', 'pyarrow', 'fastparquet'}, default 'auto' Parquet library to use. If 'auto', then the option ``io.parquet.engine`` is used. The default ``io.parquet.engine`` behavior is to try 'pyarrow', falling back to 'fastparquet' if 'pyarrow' is unavailable. columns : list, default=None If not None, only these columns will be read from the file. **kwargs Any additional kwargs are passed to the engine. Returns ------- DataFrame r')rr1)r-rr'r/rQrrr read_parquets$rS)rr:NN)rN)__doc__typingrrrrrwarningsrZpandas._typingrZpandas.compat._optionalr Z pandas.errorsr r8r r Zpandas.io.commonr rrrrrrrrIrRrSrrrrs     "LD.;