7|hk d dl Z d dlZd dlZd dlZd dlZd dlZd dlmZ d dlm Z d dl
m Z m Z m
Z
mZmZmZ d dlmZmZ d dlmZ d dlmZ d dlmZ d d lmZm Z d d
l m Z m Z m Z m!Z!m"Z"m#Z# e jH e% Z&d e fd Z'd e
e( fd
Z)d e
e( fdZ* dde(de de d e+fdZ,de(de(d e fdZ- G d d Z. G d de e Z/y) N)deepcopy)Path)AnyDictListOptionalSequenceUnion) AgentAction AgentFinish)BaseCallbackHandler)Document) LLMResult)get_from_dict_or_env guard_import)BaseMetadataCallbackHandler flatten_dict hash_string
import_pandas import_spacyimport_textstatreturnc t d S )zKImport the mlflow python package and raise an error if it is not installed.mlflow)r l/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_community/callbacks/mlflow_callback.py
import_mlflowr s
!!r c
g dS )z!Get the metrics to log to MLFlow.)stepstartsendserrorstext_ctr chain_starts
chain_ends
llm_startsllm_ends llm_streams tool_starts tool_ends
agent_endsretriever_startsretriever_endsr r r r mlflow_callback_metricsr/ " s r c
g dS )z.Get the text complexity metrics from textstat.)flesch_reading_easeflesch_kincaid_grade
smog_indexcoleman_liau_indexautomated_readability_index dale_chall_readability_scoredifficult_wordslinsear_write_formula gunning_fogfernandez_huertaszigriszt_pazosgutierrez_polinicrawfordgulpease_indexosmanr r r r get_text_complexity_metricsr@ 7 s r textnlptextstatc i }|Lt D ci c] }| t || | }}|j d|i |j | |ft } || }|j j |ddd }|j j |ddd } || d}
|j |
|S c c}w )a Analyze text using textstat and spacy.
Parameters:
text (str): The text to analyze.
nlp (spacy.lang): The spacy language model to use for visualization.
textstat: The textstat library to use for complexity metrics calculation.
Returns:
(dict): A dictionary containing the complexity metrics and visualization
files serialized to HTML string.
text_complexity_metricsdepFT)stylejupyterpageent)dependency_treeentities)r@ getattrupdater displacyrender) rA rB rC respkeyrE spacydocdep_outent_outtext_visualizationss r analyze_textrX M s D9T9V#
25C'3'- -#
#
.0GHI +,
$i..''5%d'S..''5%d'S '
'( K+#
s B;prompt
generationc \ | j dd }|j dd }d| d| dS )zConstruct an html element from a prompt and a generation.
Parameters:
prompt (str): The prompt.
generation (str): The generation.
Returns:
(str): The html string.
z
z
z>:
z"
)replace)rY rZ formatted_promptformatted_generations r )construct_html_from_prompt_and_generationr` w sO ~~dF3%--dF; - . /
r c e Zd ZdZdefdZ d dedeeef dee ddfd Z d d
Z
d ed e ddfd
Z d de
eee f eeef f dee ddfdZdeeef deddfdZdededdfdZdededdfdZdededdfdZdeddfdZdeddfd Zy) MlflowLoggera Callback Handler that logs metrics and artifacts to mlflow server.
Parameters:
name (str): Name of the run.
experiment (str): Name of the experiment.
tags (dict): Tags to be attached for the run.
tracking_uri (str): MLflow tracking server uri.
This handler implements the helper functions to initialize,
log metrics and artifacts to the mlflow server.
kwargsc t | _ dt j v ry| j j d | j j
j j | _ | j j | j | _
nt |ddd }| j j | |j d x}r5| j j | j j | _ nut |dd }| j j! | | _
| j | j j | _ n | j j# | | _ | j% |d |d
|j dd |j d d | _ y ) NDATABRICKS_RUNTIME_VERSION
databricks tracking_uriMLFLOW_TRACKING_URI run_idexperiment_nameMLFLOW_EXPERIMENT_NAMErun_namerun_tags
artifacts_dir)r r osenvironset_tracking_uritrackingfluent_get_experiment_id mlf_expidget_experimentmlf_expr getget_runinfo
experiment_idget_experiment_by_namecreate_experiment start_rundir)selfrc rg rj rk s r __init__zMlflowLogger.__init__ s[ #o '2:: 5 KK ( ( 6 ![[ 1 1 8 8 K K MDN ;;55dnnEDL/(=r L
KK ( ( 6 H--v-!%!4!4V!}}T"NN6#9#9FMM#IQO CRy5(++**,77D 8 C XX__F r c 8 | j j y)zTo finish the run.N)r end_runr s r
finish_runzMlflowLogger.finish_run s r rR valuec T | j j ||| j y)z To log metric to mlflow server.rj N)r
log_metricrj )r rR r s r metriczMlflowLogger.metric s sE$++ >r datar c R | j j || j y)z%To log all metrics in the input dict.r N)r log_metricsrj )r r r s r metricszMlflowLogger.metrics s
T[[ 9r filenamec | j j |t j j | j
| d | j y)z,To log the input data as json file artifact.z.jsonr N)r log_dictrp pathr r rj )r r r s r jsonfzMlflowLogger.jsonf > "'',,txxH:U); "'',,txxH:T):;DKK
r r c R | j j || j y)z/To upload the file from given path as artifact.r N)r log_artifactrj )r r s r artifactzMlflowLogger.artifact s dkk :r chainc h | j j j |d| j y )Nzlangchain-modelr )r langchain log_modelrj )r r s r langchain_artifactz MlflowLogger.langchain_artifact s& ''/@'Ur Nr N)r )__name__
__module__ __qualname____doc__r r strr r r r floatr r
intr r r r rA r r r r r rb rb sG
3 3B HL #CH~ 7?}
& ?# ?e ? ?
TU:$sEz*DcN:;:CKC=:
:
$sCx.
C
D
8# 8# 8$ 8
;S ;T ;V V Vr rb c L e Zd ZdZ d/dee dee dee dee dee ded df fd
Zd0d Zd eee f d
e
e de d dfdZ dede d dfdZ de
de d dfdZdede d dfdZd eee f deee f de d dfdZdeeee f ee
e f de d dfdZdede d dfdZd eee f dede d dfd Zd e de d dfd Zdede d dfd Zd ede d dfd!Zd"ede d dfd#Zd$ede d e fd%Z d eee f d&ede d e fd'Z d(e e de d e fd)Z dede d e fd*Z!d e fd+Z"d e#fd,Z$d1d-e d"e#d dfd.Z% xZ&S )2MlflowCallbackHandleraO Callback Handler that logs metrics and artifacts to mlflow server.
Parameters:
name (str): Name of the run.
experiment (str): Name of the experiment.
tags (dict): Tags to be attached for the run.
tracking_uri (str): MLflow tracking server uri.
This handler will utilize the associated callback method called and formats
the input of each callback function with metadata regarding the state of LLM run,
and adds the response to the list of records for both the {method}_records and
action. It then logs the response to mlflow server.
Nr
experimentr rg rj ro r c t t t
|
|| _ || _ |xs i | _ || _ || _ || _ t j | _ t | j | j
| j | j | j | j | _ g | _ d| _ t# } |j% d | _ t1 | _ t5 D ci c] } | d c} | _ g g g g g g g g g g g g g d
| _ y# t& $ r t( j+ d Y dw xY w# t, $ r)}t( j+ |j. Y d}~d}~ww xY w# t, $ r0}t( j+ |j. d| _ Y d}~d}~ww xY wc c} w )z Initialize callback handler.)rg rk rm rn rj ro Nen_core_web_smzfRun `python -m spacy download en_core_web_sm` to download en_core_web_sm model for text visualization.r )
on_llm_start_recordson_llm_token_recordson_llm_end_recordson_chain_start_recordson_chain_end_recordson_tool_start_recordson_tool_end_recordson_text_recordson_agent_finish_recordson_agent_action_recordson_retriever_start_recordson_retriever_end_recordsaction_records) r r superr r r r rg rj ro tempfileTemporaryDirectorytemp_dirrb mlflgr rB r loadOSErrorloggerwarning ImportErrormsgr rC r/ r records) r r r r rg rj ro rS erR __class__s r r z MlflowCallbackHandler.__init__ s
$JB ( * 335
! ** OOYYYY;; ,,
%' NE
::&67 ! + -DM
+B*CD3QD %'$&"$&($&%'#% !')')*,(* (
O
" NN155 ! ! " ! NN155 ! DMM ! EsH
E D1 .F
G1 EE F FF
G&F>>Gc | j j D ] \ }}d| j |< | j j D ] \ }}g | j |< y )Nr )r itemsr )r r vs r _resetz MlflowCallbackHandler._resetO s\ LL&&( DAq DLLO LL&&( !DAq DLLO !r
serializedpromptsrc c | j dxx dz
cc< | j dxx dz
cc< | j dxx dz
cc< | j d }i }|j ddi |j t | |j | j | j j | j | j d t | D ]s \ }}t | }||d< | j d j | | j d
j | | j j |d | d | u y
)zRun when LLM starts.r r' r! action on_llm_startr rY r r
llm_start__prompt_N) r rN r r enumerater r appendr ) r r r rc r' rQ idxrY prompt_resps r r z"MlflowCallbackHandler.on_llm_startU s'
V ! \"a'" X !# \\,/
! X~./ L,- DLL!
4< JJ [Jzl(3%*P Q Rr tokenc | j dxx dz
cc< | j dxx dz
cc< | j d }i }|j d|d |j | j | j j | j | j d | j d j | | j d j | | j j |d | y
) z#Run when LLM generates a new token.r r r) on_llm_new_token)r r r r r llm_new_tokens_Nr rN r r r r )r r rc r) rQ s r r z&MlflowCallbackHandler.on_llm_new_tokenm s V ! ]#q(#ll=1 ! 1EBC DLL!
4<?r responsec | j dxx dz
cc< | j dxx dz
cc< | j dxx dz
cc< | j d }i }|j ddi |j t |j xs i |j | j | j j | j | j d |j
D ] }t
| D ] \ }}t | }|j t |j |j t |j | j | j d |v r;|j d } | j j | | j d | j d
j | | j d j | | j j! |d | d
| d|v r7|d }
| j j# |
dt% |j z d|v sS|d } | j j# | dt% |j z y)zRun when LLM ends running.r r r( r" r
on_llm_endr )rB rC rE r r llm_end_ _generation_rK zdep-rL zent-N)r rN r
llm_outputr generationsr r dictrX rA rB rC popr r r r r ) r r rc r( rQ r r rZ generation_respcomplexity_metricsrK rL s r r z MlflowCallbackHandler.on_llm_end~ s4 V ! Z A% V ! <<
+ ! X|,- L!4!4!:;< DLL!
4<#J41aqc1#J#JKK
% ((F#C5CJ#CDK f+K d^
*
8 -.55jA %&--j9
|L>%BC $K#Cs G
:Goutputsc > | j dxx dz
cc< | j dxx dz
cc< | j dxx dz
cc< | j d }i }t |t r8dj |j D cg c] \ }}| d| c}} }n;t |t
r dj t
t | }n t | }|j d|d |j | j | j j | j | j d | j d
j | | j d j | | j j |d | y
c c}}w )z Run when chain ends running.r r r& r" r r
on_chain_end)r r r r r
chain_end_N)
r r r r r r mapr rN r r r r )r r rc r& rQ r r chain_outputs r r z"MlflowCallbackHandler.on_chain_end sH
V ! \"a'" V ! \\,/
! gt $ 88GMMO$LDAqs!A3Z$LML
& 88CW$56L wz9MlflowCallbackHandler.on_retriever_end.. s %8c!f%8s ) page_contentmetadataon_retriever_end)r r9 r r r retriever_end_N)
r r? r@ r r r r r rN r r r r ) r r9 rc r. rQ rT r r retriever_documentss r rA z&MlflowCallbackHandler.on_retriever_endr s{
V ! %&!+& V ! &67 ! !
!$ 0 0 !$ 2 2 4
1 )!T 2 A XX%8a%8 89
1@STU DLL!
4<FFFc ` | j dxx dz
cc< | j dxx dz
cc< y)zRun when Retriever errors.r r r# Nr r s r on_retriever_errorz(MlflowCallbackHandler.on_retriever_error r r c t }|j | j d }|j | j d }ddg}d|j v r|j d n7d|j v r)|d j d |d< |j d || j
d j dd
id }| j
t ng }| j d d gng }g d
}|D cg c] } | |j v s| }} |ddg|z |z |z j
d j dddd }
|j ||
gd } | ddg j d d | d< | S c c} w )z=Create a dataframe with all the information from the session.r r r rY r idc | d S )Nr r )id_s r zCMlflowCallbackHandler._create_session_analysis_df.. s
CG r r )axis prompt_steprK rL )token_usage_total_tokenstoken_usage_prompt_tokens token_usage_completion_tokensrA output_stepr! )r rA c & t | d | d S )NrY r! )r` )rows r rJ zCMlflowCallbackHandler._create_session_analysis_df.. s AH
s8} r chat_html) r DataFramer columnsr applydropnarenamerC r@ rB concat) r pdon_llm_start_records_dfon_llm_end_records_dfllm_input_columnsllm_input_prompts_dfcomplexity_metrics_columnsvisualizations_columnstoken_usage_columnsr= llm_outputs_dfsession_analysis_dfs r _create_session_analysis_dfz1MlflowCallbackHandler._create_session_analysis_df s
_"$,,t||
x ,
%
K( #"5
s <E=E=c 2 t | j d S )Nr )boolr r s r _contain_llm_recordsz*MlflowCallbackHandler._contain_llm_records s DLL!7899r langchain_assetc t }| j j d|j | j d | j r| j
}|j d }|j ddd }| j j d|j | | j j dj |j d |rdt t | v r | j j | nVt t | j j" d } |j% | | j j' | |r+| j j7 | j9 y y # t( $ r |j+ | | j j' | nW# t, $ r" t/ d
t1 j2 Y n-t4 $ r" t/ d
t1 j2 Y nw xY wY t4 $ r" t/ d
t1 j2 Y w xY w) Nr rS r\ ri T)regexsession_analysisz langchain.chains.llm.LLMChainz
model.jsonzCould not save model.) r r r rT r rg rd r r] r r tolistr typer r r r saver
ValueError
save_agentAttributeErrorprint traceback print_excNotImplementedErrorr r )r rh r* rZ rc rS langchain_asset_paths r
flush_trackerz#MlflowCallbackHandler.flush_tracker s
_
)2<< EU8V+WX $ $ &"&"B"B"D +// Q1R S JJOOBGGI$4$4$6 7 E /#d?6K2LL
--o>'*4
0B0BL+Q'R$#(()=>JJ''(<=" JJ ! ! # KKM ! "
'223GH
++,@A) 56!++- . 56!++- + 12 '')sH
,F$ $ I .,GI (H/I 'H/,I .H//I 4'I I )zlangchainrun-%r NNNri r )NF)'r r r r r r r r r r r r r r r
BaseExceptionr r r
r r r r# r& r( r r, r r1 r7 r r rA rE rd rf rg rw
__classcell__)r s @r r r s /$/ #&* $ E
smE
SME
tn E
sm E
E
E
E
N! R sCx.R379RHKR
R0@c @S @T @",U9 ,U ,U ,U\$- $3 $4 $
D sCx.D26sCx.DLOD
D:: T#s(^S$s) ;<:HK:
:4$M $S $T $
< sCx.<58k >S >S >,FcNF F F
F0"BH%"B "B
"BH$
$ $ $
;#S ;#z:d :&S & &RV &r r )NN)0loggingrp r r r rs copyr pathlibr typingr r r r r r
langchain_core.agentsr r langchain_core.callbacksr
langchain_core.documentsr langchain_core.outputsr langchain_core.utilsr r #langchain_community.callbacks.utilsr r r r r r getLoggerr r r r r/ r@ r rX r` rb r r r r r s
= = : 8 - , C
8 $"s "
c *T#Y 0 '
' ' '
'Tc s s ,iV iVXH 79L Hr