website-downloader
Author: pskill9
Description: MCP server that leverages wget to download entire websites, preserve structure, and convert links for offline use.
Stars: 121
Forks: 22
License: Unknown – no license file or license field detected. Verify before using in production.
Category: Open Source
Overview
Installation
1. Prerequisites
• Node.js ≥18
• npm (bundled with Node) or pnpm/yarn
• Optional: Redis (if you switch the queue adapter to Redis for large crawls)
2. Clone the repository
git clone https://github.com/pskill9/website-downloader.git
cd website-downloader3. Install dependencies
npm install # or pnpm install / yarn4. Configuration
Copy the sample environment file and adjust settings:
cp .env.example .env
# .env values (defaults in brackets)
PORT=3000 # REST server port
OUTPUT_DIR=downloads # where finished sites are stored
MAX_DEPTH=5 # crawling recursion depth
CONCURRENCY=4 # simultaneous requests
#REDIS_URL=redis://localhost:6379 # uncomment to enable Redis queue5. Run the server
npm run dev # hot-reload (nodemon) for development
npm start # production mode6. Reverse proxy (optional)
When deploying behind nginx or Caddy, proxy /api/* to localhost:3000.7. Docker (optional)
docker build -t website-downloader .
docker run -p 3000:3000 -v $(pwd)/downloads:/app/downloads website-downloader01
download_website
Download an entire website using wget, preserving its structure, converting links for local use, and supporting optional output path and depth parameters
License: Unknown – no license file or license field detected. Verify before using in production.
Updated 7/30/2025