ih7*dZddlmZddlZddlZddlmZmZmZm Z m Z ddl m Z ddl mZddlmZddlmZdd lmZdd lmZmZmZdd lmZmZdd lmZdd ZddZGddeeeZ GddeeeZ!Gdde!Z"y)z-LLM Chains for evaluating question answering.) annotationsN)AnyListOptionalSequenceTuple) Callbacks)BaseLanguageModel)PromptTemplate) ConfigDict)LLMChain)CONTEXT_PROMPT COT_PROMPTPROMPT) LLMEvalChainStringEvaluator)RUN_KEYctjd|jtj}|rF|j dj dk(ry|j dj dk(ry |jj djtjddtj}|j dk(ry|j dk(ry|jj d jtjddtj}|j dk(ry|j dk(ry y#t$rYywxYw) Nzgrade:\s*(correct|incorrect)CORRECT)rr INCORRECT)rrr) researchstrip IGNORECASEgroupuppersplit translatestr maketransstring punctuation IndexError)textmatch first_word last_words `/var/www/html/dev/engine/venv/lib/python3.12/site-packages/langchain/evaluation/qa/eval_chain.py _get_scorer,sC II5tzz|R]] SE ;;q>   !Y . [[^ ! ! #{ 2! JJL    # - -cmmBFDVDV.W X      *    ; .! JJL UWR  Ys}}RV-?-?@ A  ??  ) __ + -!.     s&=A&E5$E58A&E5E55 FFc^|j}t|}|d\}}n|\}}|||dS)zParse the output text. Args: text (str): The output text to parse. Returns: Any: The parsed output. )NN) reasoningvaluescore)rr,)r'r. parsed_scoresr/r0s r+_parse_string_eval_outputr21sD Iy)M! u$ u c2eZdZUdZdZded<edZeddZ e ddZ e dd Z e dd Z e d dd Z dd d  ddZddZd d d dd ddZd d d dd ddZy ) QAEvalChainz,LLM Chain for evaluating question answering.resultsr" output_keyignoreextracyNFclss r+is_lc_serializablezQAEvalChain.is_lc_serializablePr3cy)N correctnessr=selfs r+evaluation_namezQAEvalChain.evaluation_nameTsr3cyNTr=rDs r+requires_referencezQAEvalChain.requires_referenceXr3cyrHr=rDs r+requires_inputzQAEvalChain.requires_input\rJr3Nc |xst}hd}|t|jk7rtd|d|j|d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'input', 'answer' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: QAEvalChain: the loaded QA eval chain. >queryanswerresultInput variables should be , but got llmpromptr=)rsetinput_variables ValueError)r?rTrUkwargsexpected_input_varss r+from_llmzQAEvalChain.from_llm`sk,!6; #f&<&<"= =,-@,AB!1124 4s64V44r3 callbacksct|Dcgc]\}}|||||||d} }}|j| |Scc}}w)5Evaluate question answering examples and predictions.rNrOrPr\ enumerateapply) rEexamples predictions question_key answer_keyprediction_keyr]iexampleinputss r+evaluatezQAEvalChain.evaluatesd"(1  7 !.!*-%a.8   zz&Iz66 Acft||j}t|vr|t|t<|SNr2r7rrErP parsed_results r+_prepare_outputzQAEvalChain._prepare_output11&2IJ f %+G_M' "r3F referenceinputr]include_run_infoc B||||d||}|j|S)aEvaluate Chain or LLM output, based on optional input and label. Args: prediction (str): the LLM or chain prediction to evaluate. reference (Optional[str], optional): the reference label to evaluate against. input (Optional[str], optional): the input to consider during evaluation callbacks (Callbacks, optional): the callbacks to use for tracing. include_run_info (bool, optional): whether to include run info in the returned results. **kwargs: additional keyword arguments, including callbacks, tags, etc. Returns: dict: The evaluation results containing the score or value. r`r]rxrsrE predictionrvrwr]rxrYrPs r+_evaluate_stringszQAEvalChain._evaluate_stringss70#$   - ##F++r3cpK|j|||d||d{}|j|S7w)Nr`rkr]rxacallrsr|s r+_aevaluate_stringszQAEvalChain._aevaluate_stringssHzz"i:N-"   ##F++  646returnboolrr"ro)rTr rUOptional[PromptTemplate]rYrrr5r`)rdSequence[dict]rerrfr"rgr"rhr"r]r r List[dict]rPdictrrr}r"rv Optional[str]rwrr]r rxrrYrrr)__name__ __module__ __qualname____doc__r7__annotations__r model_config classmethodr@propertyrFrIrLr[rlrsr~rr=r3r+r5r5Gs6JL,05 5)5 5  55D$"& 7 $7 7$7 7  7  77 7,$(##!&!,!,! !,  !,  !,!,!, !,N$(##!&,,! ,  ,  ,,, ,r3r5c4eZdZdZeddZeddZeddZe dZ eddZ eddZ e d dd Z dd d  dd Zdd Zd d d dd ddZd d d dd ddZy )ContextQAEvalChainz3LLM Chain for evaluating QA w/o GT based on contextcyr<r=r>s r+r@z%ContextQAEvalChain.is_lc_serializablerAr3cy)z.Whether the chain requires a reference string.Tr=rDs r+rIz%ContextQAEvalChain.requires_referencer3cy)z+Whether the chain requires an input string.Tr=rDs r+rLz!ContextQAEvalChain.requires_inputrr3r8r9crhd}|t|jk7rtd|d|jy)N>rNrPcontextrQrR)rVrWrX)r?rUrZs r+_validate_input_varsz'ContextQAEvalChain._validate_input_varssM< #f&<&<"= =,-@,AB!1124  >r3cy)NzContextual Accuracyr=rDs r+rFz"ContextQAEvalChain.evaluation_names$r3Nc N|xst}|j||d||d|S)aLoad QA Eval Chain from LLM. Args: llm (BaseLanguageModel): the base language model to use. prompt (PromptTemplate): A prompt template containing the input_variables: 'query', 'context' and 'result' that will be used as the prompt for evaluation. Defaults to PROMPT. **kwargs: additional keyword arguments. Returns: ContextQAEvalChain: the loaded QA eval chain. rSr=)rrr?rTrUrYs r+r[zContextQAEvalChain.from_llms1,)>   (4s64V44r3r\ct|Dcgc]\}}|||||||d} }}|j| |Scc}}w)r_rNrrPr\ra) rErdrerf context_keyrhr]rirjrks r+rlzContextQAEvalChain.evaluate sd"(1  7 !.";/%a.8   zz&Iz66 rmcft||j}t|vr|t|t<|Srorprqs r+rsz"ContextQAEvalChain._prepare_output"rtr3Fruc B||||d||}|j|S)Nrrzr{r|s r+r~z$ContextQAEvalChain._evaluate_strings(s7$$   - ##F++r3cpK|j|||d||d{}|j|S7w)Nrrrr|s r+rz%ContextQAEvalChain._aevaluate_strings=sHzz"yJO-"   ##F++  rr)rUr rNonerro)rTr rUrrYrrrr)rdrrerrfr"rr"rhr"r]r rrrr)rrrrrr@rrIrLr rrrFr[rlrsr~rr=r3r+rrs=L%%,05 5)5 5  55:$$& 7 $77 7 7  7  77 7,$(##!&,,! ,  ,  ,,, ,2$(##!&,,! ,  ,  ,,, ,r3rcXeZdZdZeddZeddZe d d dZy) CotQAEvalChainz=LLM Chain for evaluating QA using chain of thought reasoning.cyr<r=r>s r+r@z!CotQAEvalChain.is_lc_serializableRrAr3cy)NzCOT Contextual Accuracyr=rDs r+rFzCotQAEvalChain.evaluation_nameVs(r3Nc N|xst}|j||d||d|S)zLoad QA Eval Chain from LLM.rSr=)rrrs r+r[zCotQAEvalChain.from_llmZs1%:   (4s64V44r3rrro)rTr rUrrYrrr) rrrrrr@rrFr[r=r3r+rrOsjG)),0 5  5) 5 5  5 5r3r)r'r"rzOptional[Tuple[str, int]])r'r"rr)#r __future__rrr$typingrrrrr langchain_core.callbacks.managerr langchain_core.language_modelsr langchain_core.promptsr pydanticr langchain.chains.llmr #langchain.evaluation.qa.eval_promptrrrlangchain.evaluation.schemarrlangchain.schemarr,r2r5rrr=r3r+rso3" 776<1)RRE$:,F,(O\F,R|,?L|,~5'5r3