Web Scraping

I ran a web scraping agency before I moved into AI work, so this tag reaches the furthest back on the blog. It covers browser automation, headless drivers, crawler architecture, and the growing friction between publishers and AI crawlers. The defensive side of that story is under bot protection.

7 posts

Cloudflare Launches a Scraping API That Respects Bot Protection

Cloudflare's new /crawl endpoint built on Browser Rendering won't bypass CAPTCHAs or CDN bot protection - and that's exactly the point. Here's why this could be a bigger disruption than the technology itself.

AI in Web Scraping and Web Crawling

After 8+ years in web scraping, learn where AI and LLMs genuinely help — from writing selectors and fallback parsing to deduplication and self-healing scrapers — and what to avoid.

Pay Per Crawl - Cloudflare VS AI Crawling

Discover how Cloudflare’s new bot-blocking and Pay Per Crawl features aim to protect content, fight AI scrapers, and help creators earn from their data.

WebDriver vs NoDriver in Web Automation

Compare web automation with WebDriver and no driver implementations

AI Labyrinth - AI vs AI in Web Scraping

Will AI Labyrinth, Cloudflare's new antibot feature, disrupt web scraping and crawling?

Web Scraping in Selenium

Learn the basics of Web Scraping with Selenium and Python

How Does Google Find Websites? A Deep Dive into Web Crawling

Explore web crawling, indexing, JavaScript challenges, and the future of AI-driven search engines.