9|h~)*dZddlZddlmZddlmZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZddlmZmZddlmZdd lmZmZdd lmZdd lmZmZdd lm Z m!Z!dd l"m#Z#edZ$edeZ%de e$de e$ge%fde e$fdZ&GddeZ'y)zo Ensemble retriever that ensemble the results of multiple retrievers by using weighted Reciprocal Rank Fusion N) defaultdict)Hashable)chain) AnyCallableDictIterableIteratorListOptionalTypeVarcast)#AsyncCallbackManagerForRetrieverRunCallbackManagerForRetrieverRun)Document) BaseRetriever RetrieverLike)RunnableConfig) ensure_config patch_config)ConfigurableFieldSpecget_unique_config_specs)model_validatorTH)bounditerablekeyreturnc#rKt}|D]$}||x}|vs|j||&yw)a Yield unique elements of an iterable based on a key function. Args: iterable: The iterable to filter. key: A function that returns a hashable key for each element. Yields: Unique elements of the iterable based on the key function. N)setadd)rrseeneks \/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain/retrievers/ensemble.py unique_by_keyr'(s> 5D QKA $ HHQKGs77c eZdZUdZeeed<eeed<dZe ed<dZ e e ed<e deefd Zed ed ee efdefd Z dde de ededeefdZ dde de ededeefdZde dedeefdZde dedeefdZddde dede edeefdZddde dede edeefdZdeeedeefdZy)EnsembleRetrieveraeRetriever that ensembles the multiple retrievers. It uses a rank fusion. Args: retrievers: A list of retrievers to ensemble. weights: A list of weights corresponding to the retrievers. Defaults to equal weighting for all retrievers. c: A constant added to the rank, controlling the balance between the importance of high-ranked items and the consideration given to lower-ranked items. Default is 60. id_key: The key in the document's metadata used to determine unique documents. If not specified, page_content is used. retrieversweights<cNid_keyrc:td|jDS)z+List configurable fields for this runnable.c3BK|]}|jD]}|ywN) config_specs).0 retrieverspecs r& z1EnsembleRetriever.config_specs..Qs-' i>T>T' 6:D' ' s)rr*)selfs r&r2zEnsembleRetriever.config_specsNs"'' "&//'   before)modevaluesc\|jdst|d}d|z g|z|d<|S)Nr+r*)getlen)clsr; n_retrieverss r& set_weightszEnsembleRetriever.set_weightsUs;zz)$vl34L!"\!1 2\ AF9  r8inputconfigkwargsc ddlm}t|}|j|j dd|j dd|j dg|j |j di|j }|jd|fd |j d xs|ji|} |j||| }|j|fi||S#t$r}|j||d}~wwxYw) Nr)CallbackManager callbacksverboseFtagsmetadatarIinheritable_tags local_tagsinheritable_metadatalocal_metadatanamerun_name run_managerrD) langchain_core.callbacksrGr configurer>rJrKon_retriever_startget_name rank_fusionon_retriever_end Exceptionon_retriever_error) r7rCrDrErGcallback_managerrTresultr$s r&invokezEnsembleRetriever.invoke]s =v&*44 JJ{ # JJy%0#ZZ3yy!'J!;==5 :&99   J':4==?   %%eV%TF )K ( (  M   * *1 -G s5C C?'C::C?c ZKddlm}t|}|j|j dd|j dd|j dg|j |j di|j }|jd|fd |j d xs|ji|d{} |j||| d{}|j|fi|d{|S7>7#7 #t$r!}|j|d{7|d}~wwxYww) Nr)AsyncCallbackManagerrHrIFrJrKrLrQrRrS) rUrarrVr>rJrKrWrX arank_fusionrZr[r\) r7rCrDrErar]rTr^r$s r&ainvokezEnsembleRetriever.ainvoke~sK Bv&/99 JJ{ # JJy%0#ZZ3yy!'J!;==: @,??   J':4==?    ,,;v-F/+..  M%    003 3 3G sfB7D+9C8:D+?C>C:C>D+1C<2D+:C><D+> D(D#DD##D((D+queryrTc*|j||}|S)z Get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. )rYr7rdrTfused_documentss r&_get_relevant_documentsz)EnsembleRetriever._get_relevant_documentss"**5+>r8cFK|j||d{}|S7w)z Asynchronously get the relevant documents for a given query. Args: query: The query to search for. Returns: A list of reranked documents. N)rbrfs r&_aget_relevant_documentsz*EnsembleRetriever._aget_relevant_documentss)"!% 1 1% EEFs !!)rDct|jDcgc]8\}}|j|t||j d|dz:}}}t t |D]B}||Dcgc].}t|trttt|n|0c}||<D|j|}|Scc}}wcc}w)z Retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. retriever_r=tagrH page_content) enumerater*r_r get_childranger? isinstancestrrrweighted_reciprocal_rank r7rdrTrDir4retriever_docsdocrgs r&rYzEnsembleRetriever.rank_fusions4!*$// : 9   k&;&;*QQRUG@T&;&U   s>*+ A*!,!:DC9Md3n5SVV!N1  77G) !s =C73C cKtjt|jDcgc]8\}}|j |t ||j d|dz:c}}d{}tt|D]4}||Dcgc] }t|ts t|n|"c}||<6|j|}|Scc}}w7hcc}ww)z Asynchronously retrieve the results of the retrievers and use rank_fusion_func to get the final result. Args: query: The query to search for. Returns: A list of reranked documents. rlr=rmroNrp) asynciogatherrrr*rcrrsrtr?rurrwrxs r&rbzEnsembleRetriever.arank_fusions& '~~%.doo$> !Ay!! "-"7"7jQ*+ A*!,!3=S(2Kc*QTT!N1  77G-   !s.(C=C 'C/C0"C%C7"CC doc_listsct|tjk7r tdttt |jD]f\}}t |dD]Q\}}j |jn|jjxx||jzz z cc<Shtj|}tt|fddfd}|S)a Perform weighted Reciprocal Rank Fusion on multiple rank lists. You can find more details about RRF here: https://plg.uwaterloo.ca/~gvcormac/cormacksigir09-rrf.pdf Args: doc_lists: A list of rank lists, where each rank list contains unique items. Returns: list: The final aggregated list of items sorted by their weighted RRF scores in descending order. zz.Fs4{{*$$dkk2r8Tcpj|jS|jjSr1r)r{ rrf_scorer7s r&rz.Ms5I$(KK$7  =@\\$++=Vr8)reverser)r?r+ ValueErrorrfloatziprrr.rqrKr-r from_iterablesortedr') r7rdoc_listweightrankr{all_docs sorted_docsrs ` @r&rwz*EnsembleRetriever.weighted_reciprocal_rank s y>S. .N  '2%&8 #It|| < . Hf&xq9 . c ;;.(( \\$++6  tdff}- . . .&&y1    r8r1) __name__ __module__ __qualname____doc__r r__annotations__rr-intr.r rvpropertyrr2r classmethodrrrBrrr_rcrrhrrjrYrbrwr8r&r)r)9s ]## %[ AsK FHSM   d#89  (#c3hC$>B"*>":MP hD>B!!"*>":!MP! h!F4  h *9  h 4,0 ''4' ( ' h '\,0 **9* ( * h *X1d8n-1 h1r8r))(rr} collectionsrcollections.abcr itertoolsrtypingrrrr r r r r rrUrrlangchain_core.documentsrlangchain_core.retrieversrrlangchain_core.runnablesrlangchain_core.runnables.configrrlangchain_core.runnables.utilsrrpydanticrrrr'r)rr8r&rs #$   .B3G% CL Cx HQKhsAv.>8A;"X Xr8