ih dZddlZddlmZmZmZmZddlmZddl m Z ddl m Z ejeZGddeZy) zScrapfly Web Reader.N)IteratorListLiteralOptional) BaseLoader)Document) get_from_envc feZdZdZddddddeedeeded d eed e d df d Z d e e fdZ y)ScrapflyLoaderzTurn a url to llm accessible markdown with `Scrapfly.io`. For further details, visit: https://scrapfly.io/docs/sdk/python NmarkdownT)api_key scrape_format scrape_configcontinue_on_failureurlsr r)r textrrreturnc ddlm}|s td|xs t dd}|||_||_||_||_||_y#t$r tdwxYw) aInitialize client. Args: urls: List of urls to scrape. api_key: The Scrapfly API key. If not specified must have env var SCRAPFLY_API_KEY set. scrape_format: Scrape result format, one or "markdown" or "text". scrape_config: Dictionary of ScrapFly scrape config object. continue_on_failure: Whether to continue if scraping a url fails. r)ScrapflyClientzC`scrapfly` package not found, please run `pip install scrapfly-sdk`zURLs must be provided.r SCRAPFLY_API_KEY)keyN) scrapflyr ImportError ValueErrorr rrrr)selfrr rrrrs k/var/www/html/dev/engine/venv/lib/python3.12/site-packages/langchain_community/document_loaders/scrapfly.py__init__zScrapflyLoader.__init__sy&  / 56 6H\)5GH&73  **#6  U  s AA%c#Kddlm}|j |jni}|jD]O} |jj ||fd|j i|}t |jdd|iQy#t$r4}|jrtjd|d|n|Yd}~d}~wwxYww) Nr) ScrapeConfigformatcontenturl) page_contentmetadatazError fetching data from z , exception: ) rrrrscraperr scrape_result Exceptionrloggererror)rrrr"responsees r lazy_loadzScrapflyLoader.lazy_load5s).2.@.@.L**RT 99 C ==// QT-?-?Q=Q!)!7!7 !B#S\  ++LL#r=s8446--   8 $9Z9r5