7|hfddlmZddlZddlmZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZGd d eZy) ) annotationsN)Path)AnyDictIterableListOptional)CallbackManagerForRetrieverRun)Document) BaseRetriever) ConfigDictceZdZUdZdZded< ded< dZded< dZd ed < ed Z e d dd Z e dd ddZ ddZ d ddZe ddd ddZy)TFIDFRetrieverz`TF-IDF` retriever. Largely based on https://github.com/asvskartheek/Text-Retrieval/blob/master/TF-IDF%20Search%20Engine%20(SKLEARN).ipynb Nr vectorizerList[Document]docs tfidf_arrayintkT)arbitrary_types_allowedc  ddlm}|xsi}|di|}|j|}|xs d|D}t ||D cgc]\}} t || } }} |d|| |d|S#t$r tdwxYwcc} }w)Nr)TfidfVectorizerzNCould not import scikit-learn, please install with `pip install scikit-learn`.c3 K|]}iywN).0_s c/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_community/retrievers/tfidf.py z,TFIDFRetriever.from_texts..4s!4"!4s  page_contentmetadatarrrr)sklearn.feature_extraction.textr ImportError fit_transformzipr ) clstexts metadatas tfidf_paramskwargsrrrtmrs r from_textszTFIDFRetriever.from_texts!s  G$)r $4|4  ..u5 4!4e!4 ADUIAVWAa!4WWWjtWPVWW !  XsA)B)A>)r,c Rtd|D\}}|jd|||d|S)Nc3LK|]}|j|jfywrr!)rds rr z0TFIDFRetriever.from_documents..@s Q!!..!**!= Qs"$)r*r,r+r)r(r0)r) documentsr,r-r*r+s rfrom_documentszTFIDFRetriever.from_documents8s@ Qy QRys~~ li KQ  cddlm}|jj|g}||j|j d}|j |j ddddDcgc]}|j|}}|Scc}w)Nr)cosine_similarity)r9) sklearn.metrics.pairwiser8r transformrreshapeargsortrr)selfquery run_managerr8 query_vecresultsi return_docss r_get_relevant_documentsz&TFIDFRetriever._get_relevant_documentsEs ?OO-- G $D$4$4i@HH  .5__->wy-I$B$-OPtyy|P PQs+Btfidf_vectorizercd ddl}t|}|jdd|j |j ||dz t ||dz d5}tj|j|jf|dddy#t$r tdwxYw#1swYyxYw)NrBCould not import joblib, please install with `pip install joblib`.T)exist_okparents.joblib.pklwb) joblibr&rmkdirdumpropenpicklerr)r> folder_path file_namerNpathfs r save_localzTFIDFRetriever.save_localSs    K  D$ /  DOOTyk,A%AB$I;d++T 2 :a KKD$4$45q 9 : : T   : :sB-B&B#&B/F)allow_dangerous_deserializationrTc. ddl}|s tdt|}|j ||dz }t ||dz d5}t j|\}} ddd|| S#t$r tdwxYw#1swY,xYw) aLoad the retriever from local storage. Args: folder_path: Folder path to load from. allow_dangerous_deserialization: Whether to allow dangerous deserialization. Defaults to False. The deserialization relies on .joblib and .pkl files, which can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine. You will need to set this to `True` to use deserialization. If you do this, make sure you trust the source of the file. file_name: File name to load from. Defaults to "tfidf_vectorizer". Returns: TFIDFRetriever: Loaded retriever. rNrHaThe de-serialization of this retriever is based on .joblib and .pkl files.Such files can be modified to deliver a malicious payload that results in execution of arbitrary code on your machine.You will need to set `allow_dangerous_deserialization` to `True` to load this retriever. If you do this, make sure you trust the source of the file, and you are responsible for validating the file came from a trusted source.rKrLrbr$)rNr& ValueErrorrloadrQrR) r)rSrXrTrNrUrrVrrs r load_localzTFIDFRetriever.load_localis0   /.  K [[9+W(=!=> $I;d++T 2 /a!' A D+ / jtMM9 T  . / /sA3B 3B B)NN) r*z Iterable[str]r+zOptional[Iterable[dict]]r,Optional[Dict[str, Any]]r-rreturnr)r4zIterable[Document]r,r^r-rr_r)r?strr@r r_r)rF)rSr`rTr`r_None)rSr`rXboolrTr`r_r)__name__ __module__ __qualname____doc__r__annotations__rrr model_config classmethodr0r5rErWr]rr6rrr sb J K AsJ( $L/315 XX,X/ X  X  XX, 26  %  /          *H   ",:::  :, 16+ 5N5N*. 5N  5N  5N5Nr6r) __future__rrRpathlibrtypingrrrrr langchain_core.callbacksr langchain_core.documentsr langchain_core.retrieversr pydanticr rrr6rrqs0" 66C-3RN]RNr6