ihP0ZddlZddlmZmZmZddlmZddlmZddl m Z GddeZ y)N)IteratorLiteralOptional) BaseLoader)Document) get_from_envc eZdZdZdedefdZdedefdZdddddd ed eed eed e d deef dZ de e fdZ y)FireCrawlLoadera! FireCrawlLoader document loader integration Setup: Install ``firecrawl-py``,``langchain_community`` and set environment variable ``FIRECRAWL_API_KEY``. .. code-block:: bash pip install -U firecrawl-py langchain_community export FIRECRAWL_API_KEY="your-api-key" Instantiate: .. code-block:: python from langchain_community.document_loaders import FireCrawlLoader loader = FireCrawlLoader( url = "https://firecrawl.dev", mode = "crawl" # other params = ... ) Lazy load: .. code-block:: python docs = [] docs_lazy = loader.lazy_load() # async variant: # docs_lazy = await loader.alazy_load() for doc in docs_lazy: docs.append(doc) print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} Async load: .. code-block:: python docs = await loader.aload() print(docs[0].page_content[:100]) print(docs[0].metadata) .. code-block:: python Introducing [Smart Crawl!](https://www.firecrawl.dev/smart-crawl) Join the waitlist to turn any web {'ogUrl': 'https://www.firecrawl.dev/', 'title': 'Home - Firecrawl', 'robots': 'follow, index', 'ogImage': 'https://www.firecrawl.dev/og.png?123', 'ogTitle': 'Firecrawl', 'sitemap': {'lastmod': '2024-08-12T00:28:16.681Z', 'changefreq': 'weekly'}, 'keywords': 'Firecrawl,Markdown,Data,Mendable,Langchain', 'sourceURL': 'https://www.firecrawl.dev/', 'ogSiteName': 'Firecrawl', 'description': 'Firecrawl crawls and converts any website into clean markdown.', 'ogDescription': 'Turn any website into LLM-ready data.', 'pageStatusCode': 200, 'ogLocaleAlternate': []} paramsreturncd}gd}|D]}|j|sd}n|rtjdtd|vr|ddur|d|d<|d=d|vr|ddur|d|d<|d=d |vr|d dur|d |d <|d =d |vr|d dur|d |d <|d =d |vr-t |d t r|j |d |d<|d =|S)NF)includesexcludesallowBackwardCrawlingallowExternalContentLinks pageOptionsTBDeprecated parameters detected. See Firecrawl v1 docs for updates.r includePathsr excludePathsrallowBackwardLinksrallowExternalLinksr scrapeOptions)getwarningswarnDeprecationWarning isinstancedictlegacy_scrape_options_adapter)selfr use_legacy_options legacy_keyskeys l/var/www/html/dev/engine/venv/lib/python3.12/site-packages/langchain_community/document_loaders/firecrawl.pylegacy_crawler_options_adapterz.FireCrawlLoader.legacy_crawler_options_adapterCsH"  Czz#%)"   MMT" V#*%--3J-?F>*:&V#*%--3J-?F>*:&&&012d:39:Q3RF/023*f456$>39:U3VF/067&f]3T:.2.P.P}-/F?+=) cd}dg}d|vrd|dvr|dddk(s|dddk(s |dddk(rld}d |dvr,|dd r |dd |d <n|djd d |d <d |dvr|dd r |dd |d <d|dvr|ddr |dd|d <|d=gd}|D]}|j|sd}n|rtjdtd|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|j d|d=d|vr|ddur|d|d<|d=d|vr|ddur|d|d <|d=d!|vr||d!<|S)"NFmarkdownextractorOptionsmodezllm-extractionzllm-extraction-from-raw-htmlzllm-extraction-from-markdownTextractionPromptpromptz-Extract page information based on the schema.extractionSchemaschema userPrompt) includeMarkdown includeHtmlincludeRawHtmlincludeExtract includeLinks screenshotfullPageScreenshotonlyIncludeTags removeTagsrr0r1htmlr2rawHtmlr3extractr4linksr5r6zscreenshot@fullPager7 includeTagsr8 excludeTagsformats)rrrrremoveappend)r r r!r? scrape_keysr#s r$rz-FireCrawlLoader.legacy_scrape_options_adapterss",  ' 233-.v6:JJ01&95601&956*.&)V4F-GG!"456HI/56H/I 20F8,066H/I/M/M 2 O0F8, *V4F-GG!"456HI/56H/I 20F8,$v.@'AA!"45lC/56H/I,/WF8,12   Czz#%)"   MMT" !F*+,5NN:.,-&-(D0NN6*=)6)*+t3NN9-+,6)*+t3NN9-+,'.)T1NN7+>*v%,'4/NN<0<(#v-./47NN#89/0 F*+,4,23D,EF=),-v%,'4/,2<,@F=)<( F " 'F9  r&Ncrawl)api_keyapi_urlr*r urlrDrEr*)rCscrapemapc ddlm}|dvrtd|d|s td|xs t dd }||| |_||_||_|xsi|_y #t$r tdwxYw) aRInitialize with API key and url. Args: url: The url to be crawled. api_key: The Firecrawl API key. If not specified will be read from env var FIRECRAWL_API_KEY. Get an API key api_url: The Firecrawl API URL. If not specified will be read from env var FIRECRAWL_API_URL or defaults to https://api.firecrawl.dev. mode: The mode to run the loader in. Default is "crawl". Options include "scrape" (single url), "crawl" (all accessible sub pages), "map" (returns list of links that are semantically related). params: The parameters to pass to the Firecrawl API. Examples include crawlerOptions. For more details, visit: https://github.com/mendableai/firecrawl-py r) FirecrawlAppzD`firecrawl` package not found, please run `pip install firecrawl-py`)rCrGsearchrHInvalid mode 'z/'. Allowed: 'crawl', 'scrape', 'search', 'map'.zUrl must be providedrDFIRECRAWL_API_KEY)rDrEN) firecrawlrJ ImportError ValueErrorrrFr*r )r rFrDrEr*r rJs r$__init__zFireCrawlLoader.__init__s4  . ; ; &UV 34 4I\)5HI%gwG l ! V  s A!!A6c#K|jdk(rC|jj|j|j |j g}n|jdk(rj|js t d|jj|j|j|j }|jdg}n|jdk(rI|js t d|jj|j|j }n3|jdk(r t d t d |jd |D]r}|jdk(r|}i}nJ|jd xs%|jd xs|jdd}|jdi}|sdt||tyw)NrG)r rCzURL is required for crawl modedatarHzURL is required for map moderKz?Search mode is not supported in this version, please downgrade.rLz%'. Allowed: 'crawl', 'scrape', 'map'.r(r9r:metadata) page_contentrU) r*rN scrape_urlrFrr rP crawl_urlr%rmap_urlr)r firecrawl_docscrawl_responsedocrVrUs r$ lazy_loadzFireCrawlLoader.lazy_load s 99 ))HHT%G%G %T*N YY' !88 !ABB!^^55!D!DT[[!Q6N,//;N YY% 88 !?@@!^^33DHHT[[3QN YY( "Q   +PQ " CyyE!" GGJ'T3776?TcggiQS>T77:r2)!  sGG)__name__ __module__ __qualname____doc__rr%rstrrrrQrrr]r&r$r r s7r.T.d.`jDjTj`"&!%29!%,# ,## ,# # ,# ./ ,#,#\(8H-(r&r ) rtypingrrrlangchain_core.document_loadersrlangchain_core.documentsrlangchain_core.utilsrr rcr&r$rhs%..6--ljlr&