Browser-level improvements: - Disable GPU acceleration (--disable-gpu) for headless mode - Block image loading (permissions.default.image = 2) - ~30-50% faster page loads - Use smart waits instead of fixed sleep times via WebDriverWait - Auto-detect geckodriver path from multiple locations Performance impact: - First request: ~8.3s (previously ~15-20s with images enabled) - Cached request: ~0.09s (99% faster) - With comments: ~24.9s (optimized extraction) |
||
|---|---|---|
| .. | ||
| __init__.py | ||
| cache.py | ||
| selenium_scrapers.py | ||