Start free
ActionTavilyUpdated June 2026

How do I crawl a whole site with Tavily?

Short answer: You can crawl site in Tavily by hand from its own interface, but it won’t repeat itself. On TinyCommand, add the Tavily Crawl Site action to a workflow, map its 4 inputs from any upstream app, and it runs automatically every time the trigger fires. No code, and a free tier to start.

Crawl Site in Tavily — start free All 4 Tavily actions
Crawl Site in Tavily — start free
Inputs

The fields this action accepts.

Every field can be mapped from an upstream trigger, AI step, table row, or hard-coded literal.

FieldTypeRequiredDescription
Starting URL
url
stringRequiredStarting URL (required)
Max Depth
max_depth
stringOptionalMax Depth
Max Breadth
max_breadth
stringOptionalMax Breadth
Page Limit
limit
stringOptionalPage Limit
Sample request
{
  "url": "e.g. https://example.com/path",
  "max_depth": "{{trigger.max_depth}}",
  "max_breadth": "{{trigger.max_breadth}}",
  "limit": "{{trigger.limit}}"
}
Returns
{
  "results": [
    {
      "url": "https://example.com/p",
      "raw_content": "..."
    }
  ],
  "base_url": "https://example.com",
  "response_time": 2.5
}

Use these fields in downstream nodes for routing, logging, or error handling.

Triggered by

Apps that pair well as the trigger for Crawl Site.

Any of these apps can fire this action as part of a workflow.

Gmail → Tavily
1 Gmail triggers
Slack → Tavily
13 Slack triggers
Google Sheets → Tavily
2 Google Sheets triggers
HubSpot → Tavily
18 HubSpot triggers
FAQ

Questions about Crawl Site.

What does the Crawl Site action do in Tavily?
Crawls a website starting from the given URL and returns clean Markdown for every reachable page. Best for RAG ingestion, knowledge-base imports, or competitive content scrapes.
What inputs does Crawl Site require?
Required: Starting URL. Every input accepts a static value or a variable from any upstream node in your workflow.
Can I use dynamic inputs from earlier workflow nodes?
Yes. Any field on this action can pull values from upstream nodes, whether that's a form response, a trigger payload, an AI output, or a lookup result.
What happens if Tavily returns an error?
The workflow pauses on the failed node, the error message is captured in the run log, and you can retry the run with one click. Auto-retry policies are configurable per workflow with exponential backoff up to 5 attempts.
Does Crawl Site support batch operations?
Yes. Run Crawl Site inside a Loop node to process arrays. TinyCommand handles Tavily's rate limits automatically so you don't have to throttle manually.
More actions

Other Tavily actions.

Action
Extract URLs
Fetches a list of URLs and returns clean, structured Markdown for each, stripping nav, ads, and boilerplate. Use this in RAG pipelines or when you need to feed pages into an LLM context.
Action
Map Site
Discovers URLs on a site without downloading or parsing page content. Much faster than Crawl when you only need the URL graph (sitemaps, audits, route inventories).
Action
Web Search
Runs a web search through Tavily and returns fresh, AI-optimized results tuned for LLM agents and RAG. Returns ranked snippets with URLs so the model can cite sources.
Crawl Site in Tavily — start free