- Integrations
- /
- Tavily
- /
- Actions
- /
- Crawl Site
ActionTavilyUpdated May 2026
How do I crawl a whole site with Tavily?
Short answer: Drop the "Tavily → Crawl Site" action anywhere in your workflow, map the inputs from upstream nodes, and publish.
Inputs
The fields this action accepts.
Every field can be mapped from an upstream trigger, AI step, table row, or hard-coded literal.
| Field | Type | Required | Description |
|---|---|---|---|
Starting URL url | string | Required | Starting URL (required) |
Max Depth max_depth | string | Optional | Max Depth |
Max Breadth max_breadth | string | Optional | Max Breadth |
Page Limit limit | string | Optional | Page Limit |
Sample request
{"url": "e.g. https://example.com/path","max_depth": "{{trigger.max_depth}}","max_breadth": "{{trigger.max_breadth}}","limit": "{{trigger.limit}}"}
Returns
{"results": [{"url": "https://example.com/p","raw_content": "..."}],"base_url": "https://example.com","response_time": 2.5}
Use these fields in downstream nodes for routing, logging, or error handling.
Triggered by
Apps that pair well as the trigger for Crawl Site.
Any of these apps can fire this action as part of a workflow.
FAQ
Questions about Crawl Site.
What does the Crawl Site action do in Tavily?
Crawls a website starting from the given URL and returns clean Markdown for every reachable page. Best for RAG ingestion, knowledge-base imports, or competitive content scrapes.
What inputs does Crawl Site require?
Required: Starting URL. Every input accepts a static value or a variable from any upstream node in your workflow.
Can I use dynamic inputs from earlier workflow nodes?
Yes. Any field on this action can pull values from upstream nodes, whether that's a form response, a trigger payload, an AI output, or a lookup result.
What happens if Tavily returns an error?
The workflow pauses on the failed node, the error message is captured in the run log, and you can retry the run with one click. Auto-retry policies are configurable per workflow with exponential backoff up to 5 attempts.
Does Crawl Site support batch operations?
Yes. Run Crawl Site inside a Loop node to process arrays. Tiny Command handles Tavily's rate limits automatically so you don't have to throttle manually.
More actions
Other Tavily actions.
Action
Extract URLs
Fetches a list of URLs and returns clean, structured Markdown for each, stripping nav, ads, and boilerplate. Use this in RAG pipelines or when you need to feed pages into an LLM context.
ActionMap Site
Discovers URLs on a site without downloading or parsing page content. Much faster than Crawl when you only need the URL graph (sitemaps, audits, route inventories).
ActionWeb Search
Runs a web search through Tavily and returns fresh, AI-optimized results tuned for LLM agents and RAG. Returns ranked snippets with URLs so the model can cite sources.
Send crawl site from your workflows.
Triggered by anything in the catalog. Free tier available. No credit card.