reddit_scraper/scraper
Marvin 93a6dd4097 Optimize browser configuration for faster scraping
Browser-level improvements:
- Disable GPU acceleration (--disable-gpu) for headless mode
- Block image loading (permissions.default.image = 2) - ~30-50% faster page loads
- Use smart waits instead of fixed sleep times via WebDriverWait
- Auto-detect geckodriver path from multiple locations

Performance impact:
- First request: ~8.3s (previously ~15-20s with images enabled)
- Cached request: ~0.09s (99% faster)
- With comments: ~24.9s (optimized extraction)
2026-03-15 10:17:31 -03:00
..
__init__.py Add request caching and further JS optimizations 2026-03-15 10:10:41 -03:00
cache.py Add request caching and further JS optimizations 2026-03-15 10:10:41 -03:00
selenium_scrapers.py Optimize browser configuration for faster scraping 2026-03-15 10:17:31 -03:00