OpenAI Bots: The Complete Guide to Identifying and Controlling Them
What each OpenAI bot does, how to identify and verify its requests, how to control its access, and what blocking each one changes for search, training and ChatGPT.
Technical SEO & GEO guides and articles
Here I write about what I’ve learned in 20+ years of SEO at scale: how to build technical SEO and GEO that work across millions of pages, how to solve problems no one has solved yet, and how to translate SEO into language product and engineering teams understand.
Add to preferred sourcesArticles
The Complete Guide to WebMCP
The complete guide to WebMCP: what it is, how it works, who supports it, how to expose tools on your site and test them, and whether it is time to implement it. Includes a demo, a measurement of ChatGPT with and without tools, and everything I tested myself.
Podcast SEO: My Guide to Getting Found on Google and in AI Search
A practical guide to podcast SEO: show and episode pages, seasons, transcripts, video players, RSS feeds, distribution and measurement.
AI Adoption in Israel: Usage Rates, Tools and Workplace Data
I examine AI adoption in Israel through local surveys: tool usage, workplace adoption and high-tech data, with methods and limits for each study.
The list of bots that support Markdown Content Negotiation
A living list of bots, coding agents, and AI crawlers: who asks for text/markdown, and who still gets regular HTML.
Google's new Preferred Sources button: why content sites should add it now
Google added an interactive Preferred Sources button for content sites. The new button keeps the reader on the site instead of sending them to an external preferences page.
Platform properties in Search Console: Google starts measuring presence beyond your website
Platform properties in Search Console are now open to everyone: connect social and video accounts, and give SEO and social teams their first shared search data layer.
The Complete Guide to Google’s Search and AI Bots (Updated September 2026)
What Googlebot, Google-Extended and Google’s other bots do, how they relate to Search and AI, and how to identify them and control their access to your site.
What is server response time, and how it affects Google's crawl rate
Server response time sets how much Googlebot crawls. On one large site, dropping from 500ms to 120ms took crawling from 15,000 to 100,000 URLs a day.
Query Fan-Out Guide: How to Analyze the Searches ChatGPT Runs, with a Free Checker Extension
What ChatGPT searches for on the web to answer a question, and how those searches can help you improve the content on your site. A guide built on a test I ran in ChatGPT, explaining how to collect the queries, compare them and work out what they can teach you.
What is llms.txt, and why you probably don't need it
You don't need an llms.txt file for AI visibility. Google's docs say Search doesn't use it, and the logs of real sites show almost no crawler fetches it.
web-bot-auth: The Bot Authentication Protocol Explained
The definitive guide to web-bot-auth, the IETF protocol for cryptographic bot identity. How it works, who built it (Cloudflare + Google), who's using it (OpenAI, AWS WAF, Akamai), and what it means for your site.
I tested if AI agents can read Lovable, Base44, Bolt, and v0 websites so you don't have to
I built the same page on Lovable, Base44, Bolt, and v0, then tested whether ChatGPT and Claude could retrieve it. All exposed the topic. Only some exposed enough HTML for reliable body retrieval.
Googlebot IP Checker
To verify Googlebot, match the request's IP against the ranges Google publishes for its crawlers, or run a reverse DNS lookup followed by a forward lookup that returns the same IP. This page has a Googlebot IP checker and the full list of Googlebot IP ranges, checked against Google's files daily.