7|hg$ddlmZddlmZmZmZmZddlmZddl m Z ddl m Z ddl mZmZddlmZGdd eeZGd d eZGd d e Zy))Enum)AnyIteratorListOptional)CallbackManagerForLLMRun)LLM)GenerationChunk) BaseModel ConfigDict)enforce_stop_tokensceZdZdZdZdZy)Devicez,The device to use for inference, cuda or cpucudacpuN)__name__ __module__ __qualname____doc__rre/var/www/html/test/engine/venv/lib/python3.12/site-packages/langchain_community/llms/titan_takeoff.pyrr s6 D CrrceZdZUdZedZeed< ejZ eed< dZ eed< dZ e eed < d Zeed < d Zeed <y) ReaderConfigzAConfiguration for the reader to be deployed in Titan Takeoff API.r)protected_namespaces model_namedeviceprimaryconsumer_groupNtensor_parallelimax_seq_lengthmax_batch_size)rrrrr model_configstr__annotations__rrrrr rintr!r#rrrrrseKLO&[[FF 6#NC#5%)OXc])INCKNC@rrc eZdZUdZdZeed< dZeed< dZ eed< dZ e ed < d Z e ed < ddddgfdededed e d eef fd ZedefdZ ddedeeedeede def dZ ddedeeedeede deef dZxZS) TitanTakeoffaTitan Takeoff API LLMs. Titan Takeoff is a wrapper to interface with Takeoff Inference API for generative text to text language models. You can use this wrapper to send requests to a generative language model and to deploy readers with Takeoff. Examples: This is an example how to deploy a generative language model and send requests. .. code-block:: python # Import the TitanTakeoff class from community package import time from langchain_community.llms import TitanTakeoff # Specify the embedding reader you'd like to deploy reader_1 = { "model_name": "TheBloke/Llama-2-7b-Chat-AWQ", "device": "cuda", "tensor_parallel": 1, "consumer_group": "llama" } # For every reader you pass into models arg Takeoff will spin # up a reader according to the specs you provide. If you don't # specify the arg no models are spun up and it assumes you have # already done this separately. llm = TitanTakeoff(models=[reader_1]) # Wait for the reader to be deployed, time needed depends on the # model size and your internet speed time.sleep(60) # Returns the query, ie a List[float], sent to `llama` consumer group # where we just spun up the Llama 7B model print(embed.invoke( "Where can I see football?", consumer_group="llama" )) # You can also send generation parameters to the model, any of the # following can be passed in as kwargs: # https://docs.titanml.co/docs/next/apis/Takeoff%20inference_REST_API/generate#request # for instance: print(embed.invoke( "Where can I see football?", consumer_group="llama", max_new_tokens=100 )) zhttp://localhostbase_urli porti mgmt_portF streamingNclientmodelsc t||||| ddlm}||j |j |j|_|D]}|jj|y#t$r t dwxYw)aInitialize the Titan Takeoff language wrapper. Args: base_url (str, optional): The base URL where the Takeoff Inference Server is listening. Defaults to `http://localhost`. port (int, optional): What port is Takeoff Inference API listening on. Defaults to 3000. mgmt_port (int, optional): What port is Takeoff Management API listening on. Defaults to 3001. streaming (bool, optional): Whether you want to by default use the generate_stream endpoint over generate to stream responses. Defaults to False. In reality, this is not significantly different as the streamed response is buffered and returned similar to the non-streamed response, but the run manager is applied per token generated. models (List[ReaderConfig], optional): Any readers you'd like to spin up on. Defaults to []. Raises: ImportError: If you haven't installed takeoff-client, you will get an ImportError. To remedy run `pip install 'takeoff-client==0.4.0'` )r*r+r,r-r) TakeoffClientzjtakeoff-client is required for TitanTakeoff. Please install it with `pip install 'takeoff-client>=0.4.0'`.)r+r,N) super__init__takeoff_clientr1 ImportErrorr*r+r,r. create_reader) selfr*r+r,r-r/r1model __class__s rr3zTitanTakeoff.__init__os< DI    4 $ MM T^^   -E KK % %e , - P  s A--Breturncy)zReturn type of llm. titan_takeoffr)r7s r _llm_typezTitanTakeoff._llm_typesrpromptstop run_managerkwargsc |jr,d}|j|||D]}||jz }|S|jj|fi|}|d}| t ||}|S)aCall out to Titan Takeoff (Pro) generate endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. run_manager: Optional callback manager to use when streaming. Returns: The string generated by the model. Example: .. code-block:: python model = TitanTakeoff() prompt = "What is the capital of the United Kingdom?" # Use of model(prompt), ie `__call__` was deprecated in LangChain 0.1.7, # use model.invoke(prompt) instead. response = model.invoke(prompt) )r>r?r@text)r-_streamrDr.generater ) r7r>r?r@rA text_outputchunkresponserDs r_callzTitanTakeoff._calls: >>K'& * uzz)  *  '4;;''9&9  &tT2D rc+K|jj|fi|}d}|D]}||jz }d|vs|jdrd}t |j dddk(r&|j dd\}} |j d}|sqt|} d}|r|j| j| |r?t|jdd} |r|j| j| y y w) aCall out to Titan Takeoff (Pro) stream endpoint. Args: prompt: The prompt to pass into the model. stop: Optional list of stop words to use when generating. run_manager: Optional callback manager to use when streaming. Yields: A dictionary like object containing a string token. Example: .. code-block:: python model = TitanTakeoff() prompt = "What is the capital of the United Kingdom?" response = model.stream(prompt) # OR model = TitanTakeoff(streaming=True) response = model.invoke(prompt) rCzdata: )rD)tokenzN) r.generate_streamdata startswithlensplitrstripr on_llm_new_tokenrDreplace) r7r>r?r@rArIbufferrDcontent_rHs rrEzTitanTakeoff._streams@/4;;..v@@ D dii F& $$W-Fv||GQ/0A5!'gq!9JGQ$^^D1F+8EF"#445::4FK $ #)CDE,,5::,>K s8D AD A4D )NN)rrrrr*r%r&r+r'r,r-boolr.rrrr3propertyr=rrrJrr rE __classcell__)r9s@rr)r)-sQ0d'Hc&WD#EIsPIt8FCE+%' ,-,-,- ,-  ,- \" ,-\3%):> ,,tCy!,67 ,  , ,b%):> 88tCy!867 8  8 / " 8rr)N)enumrtypingrrrrlangchain_core.callbacksr#langchain_core.language_models.llmsr langchain_core.outputsr pydanticr r langchain_community.llms.utilsr r%rrr)rrrresG00=32*>S$A9A4[3[r