Short answer: A web crawler is a program that browses the web on its own to collect pages, and search engines use crawlers to build their indexes. Google’s main crawler is Googlebot and Bing’s is Bingbot. They find new pages through links and sitemaps, and you can check whether a visitor that claims to be Googlebot is real.
What a web crawler does
MDN describes a web crawler as a program, often called a bot or robot, that systematically browses the web to collect data from pages. Search engines use crawlers to build the indexes they show results from.
Google explains that there’s no central list of all web pages, so it keeps looking for new ones. It finds some because it visited them before, some through links from pages it already knows, and some through the sitemaps site owners submit.
Once Google knows a URL, Googlebot may crawl it. An algorithm decides which sites to crawl, how often and how many pages to fetch, and Googlebot tries not to overload a site. Google adds that Googlebot doesn’t crawl every page it discovers: owners block some, and others need a login.
Googlebot, Bingbot and the other search crawlers
| Crawler | Run by | What it does |
|---|---|---|
| Googlebot | Google’s main crawler; it always respects robots.txt for automatic crawls | |
| Special-case crawlers, such as AdsBot | Crawl for one product under an agreement with the site, so some skip the general robots.txt rules | |
| User-triggered fetchers | Fetch a page when a person asks a Google tool for it | |
| Bingbot | Microsoft Bing | Bing’s standard crawler, which does most of its crawling |
| AdIdxBot | Microsoft Bing | Crawls Bing ads and follows the sites in them for quality control |
From Google’s crawler overview and Bing’s overview of its crawlers, read November 4, 2026.
Search engines aren’t the only crawlers. Our AI crawlers guide covers the bots that AI companies send. Our robots.txt guide covers how to tell each crawler what it may visit.
How to tell a real Googlebot from a fake one
Anyone can send a request that claims to be Googlebot, so the name in your logs proves nothing. Google gives two ways to check.
By hand, run a reverse DNS lookup on the IP address in your logs and check the name ends in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that name and confirm it returns the same IP address.
For many requests, Google suggests matching each IP address automatically against the lists of IP addresses it publishes for its crawlers.
How to help crawlers find your pages
- Don’t block the pages you want found in your robots.txt file, since that’s a common reason Googlebot can’t reach a page.
- Keep those pages open to visitors, without a login.
- Link every new page from a page crawlers already know, such as a category or hub page.
- List your pages in a sitemap and submit it.
- Fix server errors quickly, because Google slows its crawling when it gets them.
- Check any suspicious “Googlebot” in your logs with a reverse and a forward DNS lookup.
- For an important new page, request indexing with the URL Inspection tool in Search Console.
Our XML sitemap guide and our URL Inspection tool guide cover the last two tools, and our IndexNow guide covers another way to announce a new page.
How crawlers find our new guides
Our robots.txt has a single group of rules for every crawler. It doesn’t name Googlebot or Bingbot, and it points crawlers to our sitemap.
On November 4, 2026 we published our guide to the Facebook Sharing Debugger at 12:35:16 PM, and three signals went out. Our sitemap listed both language versions, with a last change of 12:35:16 PM and 12:35:17 PM. Our SEO plugin sent both to IndexNow, which answered 200 a second later.
Two minutes after publishing, an older guide on the same subject linked to the new one, so a crawler reading the older page finds it too.
Before you publish a new page
- Robots.txt doesn’t block the page.
- The page opens without a login.
- A page crawlers already know links to it.
- Your sitemap lists it.
- Your server answers without errors.
- If it matters, you requested indexing for it.
RememberCrawlers find pages through links and sitemaps, not by guessing. A page nothing links to and no sitemap lists may never be crawled.
Crawl checks like these are part of our technical SEO service.
Quick answers about web crawlers
What is a web crawler?
A program that browses the web on its own to collect pages. MDN notes that search engines such as Google and Bing use crawlers to build their indexes, the lists of pages they can show in results.
Can I pay Google to crawl my site more often?
No. Google says it doesn’t accept payment to crawl a site more frequently or to rank it higher. It also says it doesn’t guarantee it will crawl, index or show a page, even when the page follows its rules.
How often does Googlebot visit a website?
Google doesn’t give a fixed schedule. Googlebot uses an algorithm to decide which sites to crawl, how often and how many pages to fetch. It also tries not to crawl a site so fast that it overloads it.
Does Googlebot follow robots.txt?
Yes. Google says its common crawlers, Googlebot among them, always respect robots.txt rules for automatic crawls. Fetchers a person triggers through a Google tool are a separate group, so check Google’s list for each one.
How do I know if a visit from Googlebot is real?
Run a reverse DNS lookup on the visiting IP address and check the name ends in googlebot.com, google.com or googleusercontent.com. Then run a forward lookup on that name and confirm it returns the same address.
Sources
Read on November 4, 2026. MDN: Crawler. Google Search Central: In-Depth Guide to How Google Search Works. Google Crawling Infrastructure: Google Crawler (User Agent) Overview and Verify Requests from Google Crawlers and Fetchers. Bing Webmaster Tools: Overview of Bing crawlers. Search Console Help: URL Inspection tool. Our case: our robots.txt, our sitemap and the publication of one of our guides, on November 4, 2026.