7|hQddlZddlZddlZddlmZmZddlmZmZm Z m Z m Z ddl m Z ddlmZGddeeZGdd eZGd d eZy) N)ABCabstractmethod)DictIteratorOptionalTupleUnion)Document) BaseLoaderc!@eZdZdZdddddddd d dddd d d d ed ededeeefdededededededeeefdeeefdeeefdeedeedd f dZ de e fdZ e defdZdedefd Zd!ed"ede e fd#Z d(d!ed ed$eede e fd%Zd&edeeeffd'Zy ))DedocBaseLoadera Base Loader that uses `dedoc` (https://dedoc.readthedocs.io). Loader enables extracting text, tables and attached files from the given file: * `Text` can be split by pages, `dedoc` tree nodes, textual lines (according to the `split` parameter). * `Attached files` (when with_attachments=True) are split according to the `split` parameter. For attachments, langchain Document object has an additional metadata field `type`="attachment". * `Tables` (when with_tables=True) are not split - each table corresponds to one langchain Document object. For tables, Document object has additional metadata fields `type`="table" and `text_as_html` with table HTML representation. documentTF auto_tabbyrus+eng:autoN)split with_tableswith_attachmentsrecursion_deep_attachmentspdf_with_text_layerlanguagepagesis_one_column_documentdocument_orientationneed_header_footer_analysisneed_binarizationneed_pdf_table_analysis delimiterencoding file_pathrrrrrrrrrrrrr r!returncltjDcic] \}}|dvr||c}}|_hd|_||jvrt d|d|jd||_||_||_|j dk(rdnd}||jd <||jd <y cc}}w) a Initialize with file path and parsing parameters. Args: file_path: path to the file for processing split: type of document splitting into parts (each part is returned separately), default value "document" "document": document text is returned as a single langchain Document object (don't split) "page": split document text into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document text into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document text into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV >selfrr"r>linenodepagerGot $ for `split`, but should be one of ``r'treelinearstructure_typeneed_content_analysisN)localsitemsparsing_parametersvalid_split_values ValueErrorrrr")r%r"rrrrrrrrrrrrr r!keyvaluer.s i/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_community/document_loaders/dedoc.py__init__zDedocBaseLoader.__init__#sF%hnn.# UGG J#  #G // /ug++,A/  &"#'::#7X4B 01;K 78## sB0c#Kddl} ddlm}||j }d|j d_|j5}|j|ji|jd|i }ddd|jjj|j Ed{y#t$r tdwxYw#1swYcxYw7)w) Lazily load documents.rN) DedocManagerzE`dedoc` package not found, please install it with `pip install dedoc`)manager_configTloggerattachments_dir)r" parameters document_treer)tempfilededocr; ImportError _make_configconfigdisabledTemporaryDirectoryparser"r2_split_document to_api_schemadictr)r%rBr; dedoc_managertmpdirrAs r7 lazy_loadzDedocBaseLoader.lazy_loadws  * %D4E4E4GH 26 X&/  ( ( * f)//..Qd55Q7H&Q0M  '''557<<>djj(    W      s@C'C;C' .C7AC';C%<C'CC'C"C'cy)zu Make configuration for DedocManager according to the file extension and parsing parameters. Nr%s r7rEzDedocBaseLoader._make_configs paragraphcdj|dDcgc]}|j|c}}|r |dd|}|S|d}|Scc}w)z1Get text (recursively) of the document tree node. subparagraphstext)join _json2txt)r%rT subparagraphsubparagraphs_textrXs r7rZzDedocBaseLoader._json2txts{!YY%.o$>  |,  " !$6#7 8  6"   sArAdocument_metadatac#Kt|ddkDr&|dD]}|j||Ed{yt|di||dy7!w)z4Parse recursively document tree obtained by `dedoc`.rWrrAr]NrXmetadata page_contentr`)len_parse_subparagraphsr )r%rAr]r[s r7rdz$DedocBaseLoader._parse_subparagraphss }_- . 2 -o >  44".BS5  *62K-Kz1JK  s0AA"Aadditional_metadatac#K|d}|ri||}|dk(r)|j|dd}t||n|dk(r|ddd}|d dd }d }|D]P} | dd |k(r||j| z }#t|i|d |i| dd }|j| }Rt|i|d |in|d k(r8|dddD])} | d} t|j| i|| +nC|d k(r"|j|dd|Ed{ntd|d|jd|j r:|ddD]/} |j | \} } t| i| dd| d1|dD]*}|j||jddiEd{,y77 w)z=Split document into parts according to the `split` parameter.r`rcontent structure)rTrar(rWrpage_idr&r'r_Nr)r*r+tablestable)type text_as_html attachmentsrm attachment)rArre) rZr rdr4r3r _get_tablerJr)r%rArrer]rXnodesri page_textr' line_metadatarl table_text table_htmlrps r7rJzDedocBaseLoader._split_documents**5  L#4 L8K L  J >>M),D[,Q>RD7HI I f_!),[9/JEAhz*95GI 5 #I.'9!55I"%.!J$5!Jy'!J#:.y9G $t 4I 5&B-By'B  f_%i0=oN  $Z 0 !%!5C 1C]C f_00+I6{C"31   ug++,A/    &y1(; )-)?& J!+ + '(2 ( 6 J++(jj%+\$:,   / 0 s%D"G$F?%BG7G8GGrlc ld}|dD]1}|D]%}|djd|dDz }|dz }'|dz }3d}|dD]g}|d z }|D]V}djd |dD}tj|}|d z }|d r|d z }|d|dd|dd|dz }X|dz }i|dz }||fS)z.Get text and HTML representation of the table.rjcells c3&K|] }|d ywrXNrQ.0r&s r7 z-DedocBaseLoader._get_table..s&NtF|&Nlines rVzK z c3&K|] }|d ywr{rQr|s r7r~z-DedocBaseLoader._get_table..s%Mtd6l%Mrz z z
z
)rYhtmlescape)r%rlrurowcellrv cell_texts r7rqzDedocBaseLoader._get_tables- > C #chh&NW &NNN d"  # $ J    > $C ( "J  II%MtG}%MM  KK 2 e#  $";;J i 12Y(9+W>  ) #J $ ** :%%rSN)__name__ __module__ __qualname____doc__strboolr intrr8rr rOrrLrErZrdrJrrqrQrSr7r r s(  -2*,#/!&,$*8=.348#'"&#RLRL RL  RL  T * RL%(RL!RLRLRL!$RL"RL&+39%5RL!d+RL"'sDy!1RL C=!RL"3-#RL$ %RLh 8H- , d   4 C  ! 6:  (  &/3 III&d^ I (  IV&&sCx&rSr ceZdZdZdefdZy)DedocFileLoaderaw DedocFileLoader document loader integration to load files using `dedoc`. The file loader automatically detects the file type (with the correct extension). The list of supported file types is gives at https://dedoc.readthedocs.io/en/latest/index.html#id1. Please see the documentation of DedocBaseLoader to get more details. Setup: Install ``dedoc`` package. .. code-block:: bash pip install -U dedoc Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocFileLoader loader = DedocFileLoader( file_path="example.pdf", # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } r#c`ddlm}||j|j|jS)Nr)make_manager_config)r"parsing_paramsr)dedoc.utils.langchainrr"r2r)r%rs r7rEzDedocFileLoader._make_config`s)="nn22**  rSN)rrrrrLrErQrSr7rrs@D d rSrc# eZdZdZdddddddd d d dddd d d d ededededeeefdededededededeeefdeeefdeeefdeedeedd f"fdZ de e fdZ de fd Zded ed!e deeeee efffd"ZxZS)#DedocAPIFileLoaderaU Load files using `dedoc` API. The file loader automatically detects the file type (even with the wrong extension). By default, the loader makes a call to the locally hosted `dedoc` API. More information about `dedoc` API can be found in `dedoc` documentation: https://dedoc.readthedocs.io/en/latest/dedoc_api_usage/api.html Please see the documentation of DedocBaseLoader to get more details. Setup: You don't need to install `dedoc` library for using this loader. Instead, the `dedoc` API needs to be run. You may use Docker container for this purpose. Please see `dedoc` documentation for more details: https://dedoc.readthedocs.io/en/latest/getting_started/installation.html#install-and-run-dedoc-using-docker .. code-block:: bash docker pull dedocproject/dedoc docker run -p 1231:1231 Instantiate: .. code-block:: python from langchain_community.document_loaders import DedocAPIFileLoader loader = DedocAPIFileLoader( file_path="example.pdf", # url=..., # split=..., # with_tables=..., # pdf_with_text_layer=..., # pages=..., # ... ) Load: .. code-block:: python docs = loader.load() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Some text { 'file_name': 'example.pdf', 'file_type': 'application/pdf', # ... } zhttp://0.0.0.0:1231rTFrrrrrN)urlrrrrrrrrrrrrr r!r"rrrrrrrrrrrrrr r!r#cnt||||||||| | | | | |||||_d|jd<y)a Initialize with file path, API url and parsing parameters. Args: file_path: path to the file for processing url: URL to call `dedoc` API split: type of document splitting into parts (each part is returned separately), default value "document" "document": document is returned as a single langchain Document object (don't split) "page": split document into pages (works for PDF, DJVU, PPTX, PPT, ODP) "node": split document into tree nodes (title nodes, list item nodes, raw text nodes) "line": split document into lines with_tables: add tables to the result - each table is returned as a single langchain Document object Parameters used for document parsing via `dedoc` (https://dedoc.readthedocs.io/en/latest/parameters/parameters.html): with_attachments: enable attached files extraction recursion_deep_attachments: recursion level for attached files extraction, works only when with_attachments==True pdf_with_text_layer: type of handler for parsing PDF documents, available options ["true", "false", "tabby", "auto", "auto_tabby" (default)] language: language of the document for PDF without a textual layer and images, available options ["eng", "rus", "rus+eng" (default)], the list of languages can be extended, please see https://dedoc.readthedocs.io/en/latest/tutorials/add_new_language.html pages: page slice to define the reading range for parsing PDF documents is_one_column_document: detect number of columns for PDF without a textual layer and images, available options ["true", "false", "auto" (default)] document_orientation: fix document orientation (90, 180, 270 degrees) for PDF without a textual layer and images, available options ["auto" (default), "no_change"] need_header_footer_analysis: remove headers and footers from the output result for parsing PDF and images need_binarization: clean pages background (binarize) for PDF without a textual layer and images need_pdf_table_analysis: parse tables for PDF without a textual layer and images delimiter: column separator for CSV, TSV files encoding: encoding of TXT, CSV, TSV )r"rrrrrrrrrrrrr r!json return_formatN)superr8rr2)r%r"rrrrrrrrrrrrrr r! __class__s r7r8zDedocAPIFileLoader.__init__s^B #-'A 3#9!5(C/$;  "390rSc#K|j|j|j|j}|j ||j Ed{y7w)r:)rr"r?r@N) _send_filerr"r2rJr)r%doc_trees r7rOzDedocAPIFileLoader.lazy_load sM??DNNt?V?V# ''hdjj'QQQsAAAAciSrrQrRs r7rEzDedocAPIFileLoader._make_configs rSr?cddl}tjj|}t |d5}d||fi}|j |d||}dddj dk7r&td|jjtj|jj} | S#1swYmxYw) z7Send POST-request to `dedoc` API and return the resultsrNrbfilez/upload)filesdatazError during file handling: ) requestsospathbasenameopenpost status_coder4rgdecoderloads) r%rr"r?r file_namerrrresults r7rzDedocAPIFileLoader._send_files GG$$Y/ )T " Mdi./E WoU LA M ==C ;AIIrsQ #.@H&j#H&VJ oJ ZxxrS