Quick recommendations
Navigation depth: 4–8 Max files: 25000 Max total size: 2 GB Follow external: No Download linked files: Yes (CSS, images, JS) Threads: 8 Delay: 500 ms User agent: Chrome (latest) Ignore robots.txt: No (Yes only if allowed) Offline Explorer Enterprise
Before committing to a 50GB download, run the project for two minutes and pause it. Check the downloaded folder structure to verify that the file filters and link translation rules are working exactly as intended. OEE can crawl the live site and generate
Many organizations want to decommission expensive dynamic content management systems (like old SharePoint or Drupal instances). OEE can crawl the live site and generate a static HTML mirror that requires no database or PHP engine, effectively rendering the old site "frozen" but perpetually accessible on cheap S3 storage. Key Use Cases Corporate Archiving and Compliance
Developers can integrate the downloader directly into other corporate applications. Using OLE Automation, you can write scripts to start downloads, parse the retrieved data, and export it into internal databases without ever opening the software interface. Key Use Cases Corporate Archiving and Compliance