6|hoydZddlmZddlZddlZddlZddlmZddlm Z m Z m Z m Z m Z mZmZmZmZmZddlmZddlmZ ddlmZmZmZmZddlZdd lm Z dd l!m"Z"m#Z#m$Z$m%Z%ejLe'Z(Gd d eZ)Gd dedZ*GddeZ+GddedZ,GddZ-ee+e,e.fZ/GddeZ0ee0e.fZ1Gdde-Z2 d&dZ3dZ4d'dZ5d(dZ6Gdd Z7 d)d!Z8 d*d"Z9 d+d#Z: d,d$Z;ee eejJeejHgee+e,ffe e ejJe ejHgee+e,fffZds + r'r@T)prectd|vs|d.t|ttfrtj d||S)z$Check that the value is not numeric.r3zJNumeric values should be provided in the 'score' field, not 'value'. Got: ) isinstanceintfloatloggerwarning)clsvvaluess r(check_value_non_numericz(EvaluationResult.check_value_non_numericisE & F7O$;!c5\*C! r')r!r"r#r$r%r3rr4r5rdictr7r8r:r;r<r@rrKr&r'r(r1r1Is H@E:0E:8!G]!2!%J%: 6ND65=AO:A;59M29659M29!E> ) wD! " r'r1ceZdZUdZded<y)EvaluationResultszqBatch evaluation results. This makes it easy for your evaluator to return multiple metrics at once. zList[EvaluationResult]resultsNr r&r'r(rNrNxs $#!r'rNcFeZdZdZe d ddZ d ddZy) RunEvaluatorzEvaluator interface class.Ncy)zEvaluate an example.Nr&selfrunexamples r( evaluate_runzRunEvaluator.evaluate_runsr'c~Ktjjd|j||d{S7w)z#Evaluate an example asynchronously.N)asyncioget_running_looprun_in_executorrWrSs r( aevaluate_runzRunEvaluator.aevaluate_runs=--/?? $##S'    s 4=;=NrUrrVOptional[Example]return*Union[EvaluationResult, EvaluationResults])r!r"r#r$rrWr\r&r'r(rQrQsT$59##!2# 3## 6:  !2 3 r'rQcHeZdZUdZded< ded< dZded< dZd ed <y) ComparisonEvaluationResultzFeedback scores for the results of comparative evaluations. These are generated by functions that compare two or more runs, returning a ranking or other feedback. rr2z'Dict[Union[uuid.UUID, str], SCORE_TYPE]scoresNr9r:z6Optional[Union[str, Dict[Union[uuid.UUID, str], str]]]r4)r!r"r#r$r%r:r4r&r'r(rcrcs8 H@ 33459M296FJG CJ:r'rcceZdZdZ d d dZ d ddZ ddZ ddZeddZ d ddZ d dfd Z d dd Z dd Z xZS)DynamicRunEvaluatoraA dynamic evaluator that wraps a function and transforms it into a `RunEvaluator`. This class is designed to be used with the `@run_evaluator` decorator, allowing functions that take a `Run` and an optional `Example` as arguments, and return an `EvaluationResult` or `EvaluationResults`, to be used as instances of `RunEvaluator`. Attributes: func (Callable): The function that is wrapped by this evaluator. ct|}|r t|}t||ddlm}|.|j |t |_t|dd|_tj|r<| td|j |t |_t|dd|_y|j ttttt gt"f|t |_t|dd|_y)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns a dict or `ComparisonEvaluationResult`. r run_helpersNprocess_inputsr!rfFunc was provided as a coroutine function, but afunc was also provided. If providing both, func should be a regular function to avoid ambiguity.)_normalize_evaluator_funcr langsmithriensure_traceable_serialize_inputsafuncgetattr_nameinspectiscoroutinefunction TypeErrorrrrr r_RUNNABLE_OUTPUTfuncrTrxrqris r(__init__zDynamicRunEvaluator.__init__s(). -e4Ed D)  $55&76DJ! 4IJDJ  & &t , 3 %55%66DJ!z3HIDJ#44XsHW$568HHI4P05DI!z3HIDJr'c<ttrjs|_S stddvr|r|jd<t fddDrtdtdid|iS#t $r}td|d}~wwxYw) NziExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got empty result: r2c3&K|]}|v ywr]r&).0kresults r( z@DynamicRunEvaluator._coerce_evaluation_result..sJq1F?J)r3rr4zrExpected an EvaluationResult object, or dict with a metric 'key' and optional 'score' or categorical 'value'; got r:z[Expected an EvaluationResult object, or dict with a metric 'key' and optional 'score'; got r&)rCr1r: ValueErrorrsallr)rTrr: allow_no_keyes ` r(_coerce_evaluation_resultz-DynamicRunEvaluator._coerce_evaluation_results f. /'''4$M  FFLXOF"| $ u J,IJJ OOUhX$Q &P&PQ Q 44:8=  sAA>> BBBcd|vrB|j}|dDcgc]}|j||c}|d<tdi|S|jtt||dScc}w)NrO)r:T)r:rr&)copyrrNrrL)rTrOr:cprs r(_coerce_evaluation_resultsz.DynamicRunEvaluator._coerce_evaluation_resultss  B!+..q .NByM%*r* *-- w }4.  sA)ct|tr|js||_|St|}|j ||Sr])rCr1r:_format_evaluator_resultr)rTrr:s r(_format_resultz"DynamicRunEvaluator._format_resultsB f. /'''4$M)&1..v}EEr'ct|dSzCheck if the evaluator function is asynchronous. Returns: bool: True if the evaluator function is asynchronous, False otherwise. rqhasattrrTs r(is_asynczDynamicRunEvaluator.is_async'tW%%r'ct|dsPtj}|jr t d|j |j ||Stj}d|ji}t|ddrt|j|d<|j||||d}|j||S) aEvaluate a run using the wrapped function. This method directly invokes the wrapped function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rxtCannot call `evaluate_run` on an async run evaluator from within an running event loop. Use `aevaluate_run` instead.r; session_idN experimentrun_idmetadatalangsmith_extra)rrYget_event_loop is_running RuntimeErrorrun_until_completer\uuiduuid4idrrrrrxr)rTrUrV running_loopr:rrs r(rWz DynamicRunEvaluator.evaluate_run0stV$"113L&&("R $66t7I7I#w7WXX $3SVV#< 3 d +%(%8H\ "  '4(K  ""6=99r'cLKt|dst| ||d{Stj}d|j i}t |ddrt|j|d<|j||||dd{}|j||S7|7w)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. rqNr;rrrr) rsuperr\rrrrrrrrqr)rTrUrVr:rr __class__s r(r\z!DynamicRunEvaluator.aevaluate_runRstW%.sG<< < $3SVV#< 3 d +%(%8H\ "zz  '4(K"   ""6=99=  s" B$B A%B$ B" B$"B$c&|j||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: Union[EvaluationResult, EvaluationResults]: The result of the evaluation. )rWrSs r(__call__zDynamicRunEvaluator.__call__ms  g..r'c"d|jdS))Represent the DynamicRunEvaluator object.zrsrs r(__repr__zDynamicRunEvaluator.__repr__~s&tzzl!44r'r])rxXCallable[[Run, Optional[Example]], Union[_RUNNABLE_OUTPUT, Awaitable[_RUNNABLE_OUTPUT]]]rqzIOptional[Callable[[Run, Optional[Example]], Awaitable[_RUNNABLE_OUTPUT]]])F)rzUnion[EvaluationResult, dict]r: uuid.UUIDrboolr`r1)rOzUnion[dict, EvaluationResults]r:rr`ra)rzMUnion[EvaluationResult, EvaluationResults, dict, str, int, bool, float, list]r:rr`rar`rr^)rUrrVr_r`r)r!r"r#r$rzrrrpropertyrrWr\rr __classcell__)rs@r(rfrfs, 0J 0J 0Jl# -!   < / !  4  " F  F ! F 4 F&&6: : :!2 : 3 :D:86://!2/ 3/"5r'rfct|S)zmCreate a run evaluator from a function. Decorator that transforms a function into a `RunEvaluator`. )rfrxs r( run_evaluatorrs t $$r'i'c^t|}t|tkDr|dtdz dz}|S)Nz...))reprlen_MAXSIZE)objss r( _maxsize_reprrs1 S A 1v n1  & Hr'ctt|jd}t|jd}||dS)NrUrV)rUrV)rget)inputs run_truncatedexample_truncateds r(rprps5!&**U"34M%fjj&;< -> ??r'ceZdZdZ d d dZed dZ d ddZ d ddZ d ddZ ddZ e dd Z dd Z y)DynamicComparisonRunEvaluatorz4Compare predictions (as traces) from 2 or more runs.Nct|}|r t|}t||ddlm}|.|j |t |_t|dd|_tj|r<| td|j |t |_t|dd|_y|j ttttt t"gt$f|t |_t|dd|_y)zInitialize the DynamicRunEvaluator with a given function. Args: func (Callable): A function that takes a `Run` and an optional `Example` as arguments, and returns an `EvaluationResult` or `EvaluationResults`. rrhNrjr!rfrl)$_normalize_comparison_evaluator_funcrrnrirorprqrrrsrtrurvrrr rr r_COMPARISON_OUTPUTrxrys r(rzz&DynamicComparisonRunEvaluator.__init__s(4D9 8?Ed D)  $55&76DJ! 4IJDJ  & &t , 3 %55%66DJ!z3HIDJ#44!#(9:*,  15 DI!z3HIDJr'ct|dSrrrs r(rz&DynamicComparisonRunEvaluator.is_asyncrr'cXt|dsPtj}|jr t d|j |j ||Stj}|j|}|j||||d}|j|||S)zCompare runs to score preferences. Args: runs: A list of runs to compare. example: An optional example to be used in the evaluation. rxrrtagsr) rrYrrrr acompare_runsrr _get_tagsrx_format_results)rTrunsrVrr:rrs r( compare_runsz*DynamicComparisonRunEvaluator.compare_runsstV$"113L&&("R $66&&tW5 ~~d#  '4dC  ##FM4@@r'cKt|ds|j||Stj}|j |}|j ||||dd{}|j |||S7w)aEvaluate a run asynchronously using the wrapped async function. This method directly invokes the wrapped async function with the provided arguments. Args: runs (Run): The runs to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. rqrrN)rrrrrrqr)rTrrVr:rrs r(rz+DynamicComparisonRunEvaluator.acompare_runss tW%$$T73 3 ~~d#zz  '4dC"   ##FM4@@  sAA: A8!A:c&|j||S)aMake the evaluator callable, allowing it to be used like a function. This method enables the evaluator instance to be called directly, forwarding the call to `evaluate_run`. Args: run (Run): The run to be evaluated. example (Optional[Example]): An optional example to be used in the evaluation. Returns: ComparisonEvaluationResult: The result of the evaluation. )r)rTrrVs r(rz&DynamicComparisonRunEvaluator.__call__s  w//r'c"d|jdS)rz>r'cg}|D]^}|jdt|jzt|dds8|jdt|jz`|S)zExtract tags from runs.zrun:rNz experiment:)appendrrrrr)rrrUs r(rz'DynamicComparisonRunEvaluator._get_tags3s` AC KKSVV, -sL$/ MC,??@ A r'ct|tr|js||_|St|tr9t ||Dcic]\}}|j |c}}|j |d}n4t|trd|vr |j |d<nd|}t| tdid|i|Scc}}w#t$r}td||d}~wwxYw)N)rdr2r:r2zXExpected 'dict', 'list' or 'ComparisonEvaluationResult' result object. Received: result=r:zExpected a dictionary with a 'key' and dictionary of scores mappingrun IDs to numeric scores, or ComparisonEvaluationResult object, got r&) rCrcr:listziprrsrLrr)rTrr:rrUr3msgrs r(rz-DynamicComparisonRunEvaluator._format_results>s f8 9'''4$M  %;>tV;LMZS%3665=Mzz!.F  %F" $ u -%+I/ S/ ! -"M 6J 6J 6Jp&&AEA!A,=A #A@AEA!A,=A #A:AE0!0,=0 #0"?"="!" " $ "r'rct|S)z.Create a comaprison evaluator from a function.)rrs r(comparison_evaluatorrcs ) ..r'cdtj}|jjDcgc]+\}}|j|j |j fvr|-c}}r"tfdDstdk7rdd}t|tfdDrddgk(rStjr? d fd }td rtd |_ |S|j|_ |Sd fd }td rtd |_ |S|j|_ |Scc}}w)N)rUrVroutputsreference_outputs attachmentsc3&K|]}|v ywr]r&r}pnamesupported_argss r(rz,_normalize_evaluator_func..EE'ErkInvalid evaluator function. Must have at least one positional argument. Supported positional arguments are . Please see https://docs.smith.langchain.com/evaluation/how_to_guides/evaluation/evaluate_llm_application#use-custom-evaluatorsc3&K|]}|v ywr]r&rs r(rz,_normalize_evaluator_func..$)rrUrVcK|||r |jni|jxsi|r|jxsini|r|jxsinidfdD}|d{S7w)NrUrVrrrrc3(K|] }| ywr]r&r}argarg_maps r(rz>_normalize_evaluator_func..awrapper..@ @rrrrUrVargsrrxpositional_argss @r(awrapperz+_normalize_evaluator_func..awrappersl&07gnnR"{{0b@G7#6#6#<"RBI)>Br A@!4[(((sAA(!A&"A(r!c|||r |jni|jxsi|jxsi|r|jxsinidfdD}|S)Nrc3(K|] }| ywr]r&rs r(rz=_normalize_evaluator_func..wrapper..rrrrs @r(wrapperz*_normalize_evaluator_func..wrappers\&07gnnR"{{0b#*#6#6#<"BI)>Br A@T{"r')rUrrVr_r`rw)rUrrVrr`rwrt signature parametersitemskindPOSITIONAL_OR_KEYWORDPOSITIONAL_ONLYrrrrurrrr! rxsigrprrrrrs ` @@r(rmrmms N   D !C,,. E1 66a--q/@/@A A O  E_E E  A %<.rrrrrc3&K|]}|v ywr]r&rs r(rz7_normalize_comparison_evaluator_func..rrrrVcK|||r |jni|Dcgc]}|jxsic}|r|jxsinidfdD}|d{Scc}w7 w)Nrc3(K|] }| ywr]r&rs r(rzI_normalize_comparison_evaluator_func..awrapper..rrrrrrVrUrrrxrs @r(rz6_normalize_comparison_evaluator_func..awrappersp!&07gnnR=ABc 1r 1BBI)>Br A@!4[(( C)sA(A! +A(A& A(r!c|||r |jni|Dcgc]}|jxsic}|r|jxsinidfdD}|Scc}w)Nrc3(K|] }| ywr]r&rs r(rzH_normalize_comparison_evaluator_func..wrapper..rrrrs @r(rz5_normalize_comparison_evaluator_func..wrappersb &07gnnR=ABc 1r 1BBI)>Br A@T{" CsA )rrrVr_r`r)rrrVrr`rrr s ` @@r(rrs SN   D !C,,. E1 66a--q/@/@A A O  E_E E  A %<.s71:a&7sz8Expected a list of dicts or EvaluationResults. Received .rOrzZExpected a dict, str, bool, int, float, list, EvaluationResult, or EvaluationResults. Got ) rCrrErDrrrrrL)rs r(rr s&4,-6"* M);;A( D   FD !777J6(RST V$ M FC 6" M FD ! M &&,X /  r'cdtj}|jjDcgc]+\}}|j|j |j fvr|-c}}r"tfdDs*tdk7rdd}r |ddz }t|tfdDrdd gk(rS d fd }td rtd |_ |S|j|_ |Scc}}w) Nrexamplesrrrc3&K|]}|v ywr]r&rs r(rz/_normalize_summary_evaluator..;rrrrrz Received positional arguments c3&K|]}|v ywr]r&rs r(rz/_normalize_summary_evaluator..Grrrrc .|||Dcgc]}|jc}|Dcgc]}|jxsic}|Dcgc]}|jxsic}dfdD}|}t|tr|St |Scc}wcc}wcc}w)Nrc3(K|] }| ywr]r&rs r(rz@_normalize_summary_evaluator..wrapper..Ws.wrapperMs$9ABg7>>B9=>#CKK-2->KS%Tgoo&;&;%T G=O%TsBB B r!)rzSequence[schemas.Run]rzSequence[schemas.Example]r`ra) rtrrrrrrrrrrrrr!)rxr rr rrrrs` @@r(_normalize_summary_evaluatorr"2sHSN   D !C,,. E1 66a--q/@/@A A O  E_E E  A %<r$ __future__rrYrtrabcrtypingrrrrr r r r r rtyping_extensionsrrnr pydantic.v1rrrr ImportErrorpydanticlogging functoolsrlangsmith.schemasrrrr getLoggerr!rFrr*r1rNrQrLrwrcrrfrrrrprrrmrrr#r"r&r'r(r/sE"    ( BB   8 $'y' 6Ye 6,y,^" "  $)+