Soft directives don’t stop crawlers from ingesting deprecated content. Redirects for AI Training allows anybody on Cloudflare to redirect verified crawlers to canonical pages with one toggle and no or
AI Summary
To address the issue of AI training crawlers consuming outdated content, Cloudflare is launching Redirects for AI Training. This feature leverages existing canonical tags in HTML to enforce 301 redirects for verified AI training crawlers to up-to-date content, making it easier to keep AI-generated answers accurate. By enabling this feature on Cloudflare plans, users can automatically redirect AI crawlers to current content without requiring manual configuration or additional maintenance.
Get the top 10 engineering articles delivered every Monday.