7|hsZddlmZddlZddlZddlZddlmZmZmZm Z m Z m Z m Z m Z mZmZmZmZddlZddlmZddlmZddlmZmZmZddlmZmZmZm Z ddl!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'dd l(m)Z)ejTe+Z,dd Z-dd Z.ddd Z/ddZ0ddZ1edddGdd eeZ2y)) annotationsN) AnyCallableDictListLiteralMappingOptionalSequenceSetTupleUnioncast) deprecated) Embeddings)get_from_dict_or_envget_pydantic_field_namespre_init) BaseModel ConfigDictFieldmodel_validator) AsyncRetryingbefore_sleep_logretryretry_if_exception_typestop_after_attemptwait_exponential) is_openai_v1OpenAIEmbeddingsc ddl}tdt|jt d|j |j t|jjt|jjzt|jjzt|jjzt|jjzttt j"S)NrT multiplierminmaxreraisestopwaitr before_sleep)openairr max_retriesrretry_min_secondsretry_max_secondsrerrorTimeoutAPIErrorAPIConnectionErrorRateLimitErrorServiceUnavailableErrorrloggerloggingWARNING) embeddingsr,s d/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_community/embeddings/openai.py_create_retry_decoratorr;,s   6 6 7 ,,,, $FLL$8$8 9%fll&;&;< =%fll&E&EF G&fll&A&AB C&fll&J&JK  L &fgoo> c ddl}tdt|jt d|j |j t|jjt|jjzt|jjzt|jjzt|jjzttt j"dfd }|S)NrTr"r#r'cdfd }|S)NcfK23d{}|i|d{cS776tdw)Nzthis is unreachable)AssertionError)argskwargs_async_retryingfuncs r: wrapped_fz7_async_retry_decorator..wrap..wrapped_fbsD) 3 3a!426222 32* !67 7s(1%!% 1#1%1 1)rArrBrreturnr)rErFrDs` r:wrapz$_async_retry_decorator..wrapas 8 r<)rErrGr)r,rrr-rr.r/rr0r1r2r3r4r5rr6r7r8)r9r,rIrDs @r:_async_retry_decoratorrJGs#  6 6 7 ,,,, $FLL$8$8 9%fll&;&;< =%fll&E&EF G&fll&A&AB C&fll&J&JK  L &fgoo>N$ Kr<crtd|dDr!|sddl}|jjd|S)Nc3>K|]}t|ddk(yw) embeddingr"N)len).0ds r: z"_check_response..ns >3q~ ! # >sdatarz&OpenAI API returned an empty embedding)anyr,r0r2)response skip_emptyr,s r:_check_responserVms5 >Xf-= >>zll##$LMM Or<c trjjdi|St}|dfd }|di|S))Use tenacity to retry the embedding call.cjjjdi|}t|jSN)rUrH)clientcreaterVrUrBrTr9s r:_embed_with_retryz+embed_with_retry.._embed_with_retry{s1+:$$++5f5xJ4I4IJJr<rHrBrrGr)rr[r\r;)r9rBretry_decoratorr^s` r:embed_with_retryrausR~'z  ''1&11-j9OKK  &v &&r<cKtr$jjdi|d{Stdfd }|di|d{S7(7w)rXNcKjjdi|d{}t|jS7wrZ)r[acreaterVrUr]s r:_async_embed_with_retryz7async_embed_with_retry.._async_embed_with_retrys=2**22 3s!*AA#AAAAz0.0.9z1.0z!langchain_openai.OpenAIEmbeddings)sinceremovalalternative_importc eZdZUdZeddZded<eddZded<dZd ed <eZ d ed <edd Z d ed< eddZ d ed< dZ d ed<dZ d ed<dZded< eddZd ed< eddZd ed< eZded<dZded<d Zded!< d"Zded#< edd$Zd%ed&< dZded'<dZd(ed)< dZd ed*< d+Zd(ed,< ee-Zd.ed/< d+Zd(ed0< dZd1ed2<dZd3ed4<d5Z ded6< d7Z!ded8< dZ"d9ed:< e#dd;d<=Z$e%d>?e&dJd@Z'e(dKdAZ)e*dLdBZ+ddC dMdDZ,ddC dMdEZ- dN dOdFZ. dN dOdGZ/dPdHZ0dPdIZ1y)Qr ajOpenAI embedding models. To use, you should have the ``openai`` python package installed, and the environment variable ``OPENAI_API_KEY`` set with your API key or pass it as a named parameter to the constructor. Example: .. code-block:: python from langchain_community.embeddings import OpenAIEmbeddings openai = OpenAIEmbeddings(openai_api_key="my-api-key") In order to use the library with Microsoft Azure endpoints, you need to set the OPENAI_API_TYPE, OPENAI_API_BASE, OPENAI_API_KEY and OPENAI_API_VERSION. The OPENAI_API_TYPE must be set to 'azure' and the others correspond to the properties of your endpoint. In addition, the deployment name must be passed as the model parameter. Example: .. code-block:: python import os os.environ["OPENAI_API_TYPE"] = "azure" os.environ["OPENAI_API_BASE"] = "https://Build extra kwargs from additional params that were passed in.rzFound z supplied twice.z WARNING! z/ is not default parameter. zJ was transferred to model_kwargs. Please confirm that z is what you intended.z Parameters za should be specified explicitly. Instead they were passed in as part of `model_kwargs` parameter.) rgetlist ValueErrorwarningswarnpop intersectionkeys)clsvaluesall_required_field_namesr field_nameinvalid_model_kwargss r: build_extrazOpenAIEmbeddings.build_extras$2.v, ;JU" 6*5E!FGG!99 !*.L!))3 4JN %+JJz$:j! ; 8DDUZZ\R 234ST  "'~ r<c Nt|dd|d<|dxstjd|d<t|ddd|d<t|d d d|d <|dd vrd }t|d d|d <nd}t|dd||d<|dxs,tjdxstjd|d< ddl}t r|dd vrt jd|d|d|d|d|d|d|d|dd}|jds|jd i|j|d<|jdsD|jd i|j|d<|S|jds|j|d<|S |S#t$r tdwxYw)!z?Validate that api key and python package exists in environment.r{OPENAI_API_KEYruOPENAI_API_BASErvOPENAI_API_TYPE)rlrw OPENAI_PROXYazureazure_adazureadz 2023-05-15rirsOPENAI_API_VERSIONr} OPENAI_ORG_IDOPENAI_ORGANIZATIONrNzfIf you have openai>=1.0.0 installed and are using Azure, please use the `AzureOpenAIEmbeddings` class.rr-rrr)rzr|rtrr-rrrr[rfTCould not import openai python package. Please install it with `pip install openai`.rH)rosgetenvr%r,rrrrOpenAIr9 AsyncOpenAI Embedding ImportError)rrdefault_api_versionr, client_paramss r:validate_environmentz%OpenAIEmbeddings.validate_environment s@$8 $&6$  %++<$=%  B  !%9    %  ! "6    " ~ # $(H H". $'vl';T#BF< "$ ';  ' ( #$ ( ) 0yy) 0yy./ $% !  ~+,0PPMMH &&67$*+@$A &'8 9%&78#)-#8'-.?'@%+O%<#)-#8 ! zz(+'4v}}'E}'E'P'PF8$zz.1-?V-?-?.'. j>* ZZ)#)#3#3x   A ?  s ;FF$c trd|ji|j}|S|j|j|j|j |j |j|j|jd|j}|jdvr|j|d<|jr# ddl }|j|jd|_|S#t$r tdwxYw)Nro)rorrrzr|api_baseapi_typerqrenginerr)httphttps)rrorrrr{r}rurvrsrprwr,rproxy)self openai_argsr,s r:_invocation_paramsz#OpenAIEmbeddings._invocation_paramsms >!($** J8I8I JK:5#'#7#7<<.. $ 8 8 00 00#66 ## K##'GG(, H%  !!--!..  #%Gs 8CC1)rc . g}g}|jxs |j}|xs |j}|js ddlm}|j|} t|D]\} } | j| d} tdt| |jD]G} | | | |jz}| j|}|j||j| In ddl} |j!|}t|D]\} } |jj+d r| j-d d } |j| |j.|j0}tdt||jD]4} |j|| | |jz|j| 6|j2r$ ddlm}|tdt||}ntdt||}g}|D][} t9|fd|| | |zi|j:}t=|t>s|j?}|jAd|dD]tt|Dcgc]}g}}tt|Dcgc]}g}}tt|D]]} |jBrt|| dk(r!||| j|| ||| jt|| _tt|Dcgc]}g}}tt|D]} || }t|dk(rDt9|fddi|j:}t=|t>s|j?}|ddd}ntEjF|d|| }|tDjHjK|z jM|| <|S#t $r t dwxYw#t $r t dwxYw#t"$r,t$j'd d }|j)|}YbwxYw#t $rtdt||}YwxYwcc}wcc}wcc}w)al Generate length-safe embeddings for a list of texts. This method handles tokenization and embedding generation, respecting the set embedding context length and chunk size. It supports both tiktoken and HuggingFace tokenizer based on the tiktoken_enabled flag. Args: texts (List[str]): A list of texts to embed. engine (str): The engine or model to use for embeddings. chunk_size (Optional[int]): The size of chunks for processing embeddings. Returns: List[List[float]]: A list of embeddings for each input text. r AutoTokenizerzCould not import transformers python package. This is needed in order to for OpenAIEmbeddings without `tiktoken`. Please install it with `pip install transformers`. pretrained_model_name_or_pathFadd_special_tokensNCould not import tiktoken python package. This is needed in order to for OpenAIEmbeddings. Please install it with `pip install tiktoken`.5Warning: model not found. Using cl100k_base encoding. cl100k_base001  textr~r)tqdminputc3&K|] }|d ywrMNrHrOrs r:rQz.%Oa n%OrRr"rrMaxisweights)'rrorr transformersrrfrom_pretrained enumerateencoderangerNrydecodeappendtiktokenencoding_for_modelKeyErrorr6warning get_encodingendswithreplacer~rr tqdm.autorrar isinstancedictextendrUnpaveragelinalgnormtolist)rtextsrrtokensindices model_name _chunk_sizer tokenizerir tokenizedj token_chunk chunk_textrencodingrotokenr_iterbatched_embeddingsrTrCresultsnum_tokens_in_batchr9_resultaverage_embeddedrs r:_get_len_safe_embeddingsz)OpenAIEmbeddings._get_len_safe_embeddingss&--;  3DOO $$ 6&55.86I%U+ &4%,,Te,L q#i.$2K2KL&A"+AD4M4M0M"NK"+!1!1+!>JMM*-NN1% & &  8#66zB %U+ &4::&&u- <<c2D $($8$8'+'>'>(q#e*d.G.GH&AMM%A0I0I,I"JKNN1%& &$  ! ! ;*U1c&k;?@!S[+6E02 PA'Q[1))H h-#==?  % %%Ohv>N%O O P9>c%j8I+J1B+J+J B  C 6;3u:5F(G(G (Gs5z" IAajG7|q #3$$--$ ""2D9'7'<'<'>$*6215kB**W1>QRS>TU$ryy~~g'>>FFHJqM IK !V 2 !E  8VW%#007 86 ;aVk: ; ,K/N)HsMO:&P+P*"Q"9 R R / R:PP'*1QQ"RRc Kg}g}|jxs |j}|xs |j}|js ddlm}|j|} t|D]\} } | j| d} tdt| |jD]G} | | | |jz}| j|}|j||j| In ddl} |j!|}t|D]\} } |jj+d r| j-d d } |j| |j.|j0}tdt||jD]4} |j|| | |jz|j| 6g}|xs |j}tdt||D]c} t3|fd|| | |zi|j4d{}t7|t8s|j9}|j;d|dDett|Dcgc]}g}}tt|Dcgc]}g}}tt|D]?} ||| j|| ||| jt|| Att|Dcgc]}g}}tt|D]} || }t|dk(rLt3|fddi|j4d{}t7|t8s|j9}|ddd}nt=j>|d|| }|t<j@jC|z jE|| <|S#t $r t dwxYw#t $r t dwxYw#t"$r,t$j'd d }|j)|}Y2wxYw7+cc}wcc}wcc}w7w)a Asynchronously generate length-safe embeddings for a list of texts. This method handles tokenization and asynchronous embedding generation, respecting the set embedding context length and chunk size. It supports both `tiktoken` and HuggingFace `tokenizer` based on the tiktoken_enabled flag. Args: texts (List[str]): A list of texts to embed. engine (str): The engine or model to use for embeddings. chunk_size (Optional[int]): The size of chunks for processing embeddings. Returns: List[List[float]]: A list of embeddings for each input text. rrzCould not import transformers python package. This is needed in order to for OpenAIEmbeddings without `tiktoken`. Please install it with `pip install transformers`.rFrNrrrrrrrrc3&K|] }|d ywrrHrs r:rQz=OpenAIEmbeddings._aget_len_safe_embeddings..urrrRrrMr)#rrorrrrrrrrrrNryrrrrrr6rrrrr~rrgrrrrrrrrr)rrrrrrrrrrrrrrrrrrrorrrTrCrr r9r r rs r:_aget_len_safe_embeddingsz*OpenAIEmbeddings._aget_len_safe_embeddingssp&--;  3DOO $$ 6&55.86I%U+ &4%,,Te,L q#i.$2K2KL&A"+AD4M4M0M"NK"+!1!1+!>JMM*-NN1% & &  8#66zB %U+ &4::&&u- <<c2D $($8$8'+'>'>(q#e*d.G.GH&AMM%A0I0I,I"JKNN1%& &$13 3DOO q#f+{3 PA3Q[1))H h-#==?  % %%Ohv>N%O O P9>c%j8I+J1B+J+J B C6;3u:5F(G(G (Gs5z" IAajG7|q )?**--*$ ""2D9'7'<'<'>$*6215kB**W1>QRS>TU$ryy~~g'>>FFHJqM Iw !V 2 !E  8VW%#007 82,K/N )H$s;QO B#Q(O$-O<>D Q P4 AQ! P7*Q P< A.Q9 QAQ Q BQ O!!Q$O99Q<1P1-Q0P11Q7Qc\tt|j}|j||S)aMCall out to OpenAI's embedding endpoint for embedding search docs. Args: texts: The list of texts to embed. chunk_size: The chunk size of embeddings. If None, will use the chunk size specified by the class. Returns: List of embeddings, one for each text. r)rrnrpr rrrrs r:embed_documentsz OpenAIEmbeddings.embed_documentss*c4??+,,U6,BBr<cxKtt|j}|j||d{S7w)aSCall out to OpenAI's embedding endpoint async for embedding search docs. Args: texts: The list of texts to embed. chunk_size: The chunk size of embeddings. If None, will use the chunk size specified by the class. Returns: List of embeddings, one for each text. rN)rrnrprrs r:aembed_documentsz!OpenAIEmbeddings.aembed_documentss4c4??+33E&3IIIIs 1:8:c,|j|gdS)zCall out to OpenAI's embedding endpoint for embedding query text. Args: text: The text to embed. Returns: Embedding for the text. r)r)rrs r: embed_queryzOpenAIEmbeddings.embed_querys##TF+A..r<cLK|j|gd{}|dS7 w)zCall out to OpenAI's embedding endpoint async for embedding query text. Args: text: The text to embed. Returns: Embedding for the text. Nr)r)rrr9s r: aembed_queryzOpenAIEmbeddings.aembed_querys, 00$88 !}9s $" $)rrrGr)rrrGr)rGr)r List[str]rrnr Optional[int]rGList[List[float]])r)rrrrrGr)rrnrGz List[float])2__name__ __module__ __qualname____doc__rr[__annotations__rfrorprsrurvrwryr{r}setr~rrr-rrrrrrrrUrrr.r/rr model_configr classmethodrrrpropertyrr rrrrrrHr<r:r r s 'Rd3FC3dD9L#9)E3) %J %(-d-(P PS%*4z%JO]J&*O])"&L-& $#$8$)$i$HNMHO).t>)RRN7:uO4L.>VJ!6:O3:7;M4;s7s7$(K!( XBL(#$2JJXHMQAA+.ACC,9C C&=>JJ,9J J$ / r<)r9r rGzCallable[[Any], Any])r9r rGr)F)rTrrUrrGr)r9r rBrrGr)3 __future__rr7rrtypingrrrrrr r r r r rrnumpyrlangchain_core._api.deprecationrlangchain_core.embeddingsrlangchain_core.utilsrrrpydanticrrrrtenacityrrrrrr langchain_community.utils.openair getLoggerrr6r;rJrVrargr rHr<r:r0s"     60 CB:   8 $6"L ' 3  : ry*r  rr<