Crawlee

Crawlee

An Apache-2.0-licensed crawling library you can run independently of Apify.

Visit Website
All time
— views
— clicks
Last 30 days
— views
— clicks

Loading activity…

What is Crawlee?

Crawlee is an open-source web crawling and browser automation framework developed by Apify. Its JavaScript and TypeScript library supports HTTP-based and browser-based crawling with tools such as Playwright and Puppeteer. Developers can manage request queues, sessions, retries, configurable storage and proxy integrations within their own crawler applications. Crawlee uses the Apache License 2.0 and does not require an Apify subscription for independent use. It is software rather than a complete managed hosting platform. A self-operated deployment still needs compute, scheduling, persistent storage, monitoring, proxies and maintenance. Existing Actors that already use Crawlee may reuse parts of their crawling logic, but Apify-specific inputs, storage and runtime calls need inspection and adaptation. Crawlee does not include the Actor Store or managed platform services merely because the library is free.

Key Features

HTTP and browser-based crawling

Playwright and Puppeteer integrations

Request queues, sessions and retries

Configurable storage and proxy integrations

Use Cases

  • Building custom JavaScript or TypeScript crawlers
  • Operating crawler code on owned infrastructure
  • Adapting reusable Crawlee logic from custom Actors

How to Use Crawlee

  1. 1Install Crawlee in a compatible Node.js project.
  2. 2Implement and test a crawler against a permitted source.
  3. 3Configure hosting, storage, scheduling, monitoring and proxies as required.

Best For

Developers prepared to operate their crawler infrastructure

FAQ

Can I use Crawlee without Apify?

Yes. The Apache-2.0-licensed library can run independently and does not require an Apify subscription.

Is Crawlee completely free to operate?

The software has no required library subscription. Compute, browser runtime, proxies, storage, monitoring and maintenance can still incur costs.

Will my Apify Actor run unchanged?

Not necessarily. Reusable Crawlee logic may transfer, while Apify-specific initialization, inputs, storage, scheduling and runtime APIs may need replacing.

Related Tools

More active tools in Automation.

View Automation
Apify

Apify

The web scraping and automation cloud — 4,000+ ready-made Actors plus infrastructure for your own crawlers.

FreemiumAutomationAPI+1
Axiom.ai

Axiom.ai

A no-code browser RPA bot — automate clicks, typing, and scraping in any web app from a Chrome extension.

PaidAutomationNo-Code+1
Browse AI

Browse AI

Train a robot in two minutes to scrape and monitor any website — no code required.

FreemiumAutomationNo-Code+1
Zyte

Zyte

Host Scrapy spiders in Scrapy Cloud or use Zyte API for managed web extraction.

FreemiumAPIAutomation+1
Bright Data

Bright Data

Managed scraping APIs and structured datasets for supported website sources.

FreemiumAPIAutomation
ScrapingBee

ScrapingBee

Retrieve and extract web pages through an API with managed rendering and proxies.

PaidAPIAutomation