# Orchestrate Hundreds of Scrapers in Parallel

> Distributed scraping across 300+ edge locations with intelligent load balancing, automatic failover, and centralized monitoring. Scrape multiple sites simultaneously, aggregate results, and detect changes in real-time with <500ms P99 latency per site.

Canonical: https://scraper.do/

## In code

```ts
import { Scrapers } from 'scrapers.do'

// Orchestrate hundreds of scrapers in parallel
const fleet = Scrapers({ concurrency: 50, rateLimits: { perDomain: 10 } })

fleet.every.day(async $ => {
  const sites = ['amazon.com', 'walmart.com', 'target.com']
  const results = await fleet.scrapeAll(sites, { selector: '.product' })
  $.Analytics.detect.PriceChanges(results).alert.OnChanges()
})
```

## Frequently asked questions

### How many sites can I scrape simultaneously?

Run unlimited scrapers in parallel with intelligent load balancing across 300+ edge locations. Default concurrency of 50 scrapers per fleet, configurable up to 1000+ for enterprise. Each scraper operates independently with isolated proxy pools, rate limits, and error handling.

### How does load balancing work across scrapers?

Automatic distribution based on site latency, proxy availability, and rate limit status. Scrapers dynamically route to optimal edge locations—European sites scrape from EU regions, Asian sites from APAC, etc. Failed scrapers automatically failover to healthy instances with zero data loss.

### Can I aggregate and compare data across sites?

Yes. Built-in aggregation, deduplication, and normalization across different site schemas. Compare prices, detect changes, merge datasets, and analyze trends across multiple sources. Full-text search, fuzzy matching, and schema mapping for heterogeneous data.

### What happens when individual scrapers fail?

Automatic retry with exponential backoff, proxy rotation, and browser fingerprint changes. Failed scrapers quarantine without affecting the fleet. Detailed error logs, success rate metrics, and alerts when success rate drops below threshold. Manual replay of failed scrapes available.

### How do I monitor scraper health and performance?

Real-time dashboard shows per-scraper metrics: success rate, latency, data quality, proxy health, and rate limit status. Automatic alerts for anomalies, downtime, or schema changes. Historical analytics track scraping trends and optimize performance over time.

## Part of Platform.do

scraper.do is one of the primitives of Platform.do. Every .do shares one grammar, Business-as-Code, and one SDK.

- What is Business-as-Code? https://platform.do/business-as-code
- Platform.do: https://platform.do/
- Docs: https://docs.platform.do