Scrape It Now!
Web scraper made for AI and simplicity in mind
Listed in categories:
Artificial IntelligenceTechGitHub
Description
Scrape It Now is a web scraper designed for AI and simplicity, operating as a command-line interface (CLI) that can be parallelized to produce high-quality markdown content. It efficiently scrapes web pages, extracts relevant data, and stores it in various formats, making it ideal for developers and data scientists.
How to use Scrape It Now!?
To use Scrape It Now, download the latest release, configure the CLI with your Azure or local storage settings, and run the command to scrape a website. You can specify options for saving images, screenshots, and more.
Core features of Scrape It Now!:
1️⃣
Decoupled architecture with Azure Queue Storage or local SQLite
2️⃣
Idempotent operations that can be run in parallel
3️⃣
Extract markdown content from a page using Pandoc
4️⃣
Load dynamic JavaScript content with Playwright and Chromium
5️⃣
Store images and screenshots collected from the page
Why could be used Scrape It Now!?
| # | Use case | Status | |
|---|---|---|---|
| # 1 | Scraping news articles for data analysis | ✅ | |
| # 2 | Indexing web pages for AI search applications | ✅ | |
| # 3 | Extracting content for content management systems | ✅ | |
Who developed Scrape It Now!?
Clem Lesnesne is the creator of Scrape It Now, focusing on developing tools that simplify web scraping and data extraction for AI applications.