7|h`XddlmZddlmZmZmZmZmZer ddlm Z m Z m Z GddZ y)) annotations) TYPE_CHECKINGAnyIterableListOptional) DataFrameRow SparkSessionceZdZdZ d ddZe d ddZddZddZddZ dddZ dd Z dd Z dd Z ddd Zddd ZdddZy)SparkSQLz;SparkSQL is a utility class for interacting with Spark SQL.NcH ddlm}|r|n|jj |_|%|j j j||%|j j j|t|j|_ |r t|n t|_ |jr*|j|jz }|rtd|d|r t|n t|_|jr*|j|jz }|rtd|d|j} | r t| n |j|_t#|t$s t'd||_y#t$r tdwxYw) aInitialize a SparkSQL object. Args: spark_session: A SparkSession object. If not provided, one will be created. catalog: The catalog to use. If not provided, the default catalog will be used. schema: The schema to use. If not provided, the default schema will be used. ignore_tables: A list of tables to ignore. If not provided, all tables will be used. include_tables: A list of tables to include. If not provided, all tables will be used. sample_rows_in_table_info: The number of rows to include in the table info. Defaults to 3. rr Fpyspark is not installed. Please install it with `pip install pyspark`Nzinclude_tables  not found in databasezignore_tables z,sample_rows_in_table_info must be an integer) pyspark.sqlr ImportErrorbuilder getOrCreate_sparkcatalogsetCurrentCatalogsetCurrentDatabaseset_get_all_table_names _all_tables_include_tables ValueError_ignore_tablesget_usable_table_names_usable_tables isinstanceint TypeError_sample_rows_in_table_info) self spark_sessionrschema ignore_tablesinclude_tablessample_rows_in_table_infor missing_tables usable_tabless f/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_community/utilities/spark_sql.py__init__zSparkSQL.__init__ s2  0+M 0D0D0P0P0R    KK   1 1' :   KK   2 26 :t88:;6Ds>2#%   !11D4D4DDN %n%55KL5Bc-0su   !0043C3CCN $^$44JK335 4Ac-0tGWGW3S9JK K*C'E X  s F F!c  ddlm}|jj |j }||fi|S#t$r tdwxYw)zzCreating a remote Spark Session via Spark connect. For example: SparkSQL.from_uri("sc://localhost:15002") rrr)rr rrremoter)cls database_uri engine_argskwargsr sparks r.from_urizSparkSQL.from_uriKs\  0 $$++L9EEG5#F##  X  s :Acv|jr |jSt|j|jz S)zGet names of tables available.)rsortedrr)r&s r.r zSparkSQL.get_usable_table_names\s3   '' 'd&&)<)<<==c|jjdjdj}t t d|S)Nz SHOW TABLES tableNamec|jSN)r<)rows r.z/SparkSQL._get_all_table_names..es CMMr:)rsqlselectcollectlistmap)r&rowss r.rzSparkSQL._get_all_table_namescs;{{}-44[AIIKC14899r:c|jjd|jdj}|j d}|d|dzS)NzSHOW CREATE TABLE rUSING;)rrArCcreatetab_stmtfind)r&table statementusing_clause_indexs r._get_create_table_stmtzSparkSQL._get_create_table_stmtgsX KKOO08 9 A A CA F U U '^^G4,,-33r:cX|j}|-t|j|}|rtd|d|}g}|D]R}|j |}|j r"|dz }|d|j |dz }|dz }|j|Tdj|}|S)Nz table_names rz /* z*/z ) r r differencerrOr%_get_sample_spark_rowsappendjoin)r& table_namesall_table_namesr,tables table_name table_info final_strs r.get_table_infozSparkSQL.get_table_infoos557  " -88IN </??U!VWW)O) &J44Z@J..h& 4#>#>z#J"K2NN d" MM* %  &KK' r:cd|d|j}|jj|}djt t d|j j} |j|}dj|Dcgc]}dj|c}}|jd|d|d|Scc}w#t$rd}Y)wxYw) NzSELECT * FROM z LIMIT  c|jSr>)name)fs r.r@z1SparkSQL._get_sample_spark_rows..s 166r:rQz rows from z table: ) r%rrArUrDrEr(fields_get_dataframe_results Exception)r&rLquerydf columns_str sample_rowsr?sample_rows_strs r.rSzSparkSQL._get_sample_spark_rowss wt/N/N.OP [[__U #iiS)9299;K;K%L MN  !55b9K"ii;(OC3(OPO ../{5'm2 ! )P ! O !s$( CB> C>C CCclttt|jj Sr>)tuplerEstrasDictvalues)r&r?s r._convert_row_as_tuplezSparkSQL._convert_row_as_tuples#Scjjl113455r:c\tt|j|jSr>)rDrErprC)r&rgs r.rdzSparkSQL._get_dataframe_resultss C22BJJLABBr:c|jj|}|dk(r|jd}t|j |S)None)rrAlimitrmrd)r&commandfetchrgs r.runz SparkSQL.runs> [[__W % E>!B4..r233r:c^ |j|S#t$r} d|cYd}~Sd}~wwxYw)afGet information about specified tables. Follows best practices as specified in: Rajkumar et al, 2022 (https://arxiv.org/abs/2204.00498) If `sample_rows_in_table_info`, the specified number of sample rows will be appended to each table description. This can increase performance as demonstrated in the paper. Error: N)r\r)r&rVes r.get_table_info_no_throwz SparkSQL.get_table_info_no_throws7 !&&{3 3 ! *QC=  !s ,',,c` |j||S#t$r} d|cYd}~Sd}~wwxYw)a*Execute a SQL command and return a string representing the results. If the statement returns rows, a string of the results is returned. If the statement returns no rows, an empty string is returned. If the statement throws an error, the error message is returned. rzN)rxre)r&rvrwr{s r. run_no_throwzSparkSQL.run_no_throws7 !88GU+ + ! *QC=  !s -(--)NNNNN) r'zOptional[SparkSession]r Optional[str]r(rr)Optional[List[str]]r*rr+r#r>)r3rmr4zOptional[dict]r5rreturnr )rz Iterable[str])rLrmrrm)rVrrrm)r?r rrl)rgr rrD)all)rvrmrwrmrrm)__name__ __module__ __qualname____doc__r/ classmethodr7r rrOr\rSrprdrxr|r~r:r.r r sE15!% $-1.2)*=D-=D=D =D + =D , =D$'=D~>B$$-;$NQ$ $$ >:4$ "6C4 ! !r:r N) __future__rtypingrrrrrrr r r r rr:r.rs#"??88q!q!r: