AdsBot and Landing Page Crawlability in 2026: When Google Cannot See Your Page

The ad is approved, the campaign is enabled, bids look fine — and nothing serves. Or the opposite: a disapproval lands overnight saying the destination is not accessible, while the page opens perfectly in your browser. Nine times out of ten, the gap between “works for you” and “works for Google” is AdsBot — a separate crawler that checks ad landing pages and follows rules that do not match the ones you set for the search crawler.

This guide covers how that crawler behaves, why a standard robots.txt block does not stop it, which server and WAF settings break ads most quietly, and how to diagnose in 30–40 minutes a problem that usually lingers for weeks.

What AdsBot is and how it differs from Googlebot

AdsBot is the crawler serving Google’s advertising products. It checks three things on a landing page: that it is reachable, that it complies with advertising policy, and that its content matches the ad. Two agents are active today:

• AdsBot-Google — desktop checks;
• AdsBot-Google-Mobile — mobile web checks.

The difference that causes half the trouble:

AdsBot ignores the User-agent: * group in robots.txt. The classic “disallow everything for everyone” block stops the search crawler and leaves the ads crawler untouched. To block AdsBot you must name it explicitly — and conversely, “allow all” guarantees it nothing.

Google classifies AdsBot as a special-case crawler: by running ads, you submitted the page for review and implicitly granted permission to fetch it. Two practical consequences follow. First, if someone on the team disallowed a section “for everyone”, your ads probably still run. Second, if an engineer decided to trim bot traffic and blocked AdsBot by name, ads stop — and nothing in your site’s indexing reports will change to explain it.

When the crawler actually shows up

AdsBot arrives on account events, not on a site schedule:

• an ad enters review (new or edited);
• new dynamic ad targets are generated;
• routine re-checks of destinations already running — Google’s own documentation gives “roughly every two weeks” in one place and “every three weeks” in another. Treat it as regular and unannounced.

That last point explains the nastiest scenario: a campaign runs quietly for three weeks with no edits, then suddenly the ads are disapproved. Nothing changed in the account. Something changed on the site or in its protection layer, and the re-check found it first.

Three disapproval types the crawler sits behind

Status	What happened	Where to look
Destination not crawlable	The crawler could not retrieve the content	robots.txt, WAF, firewall, rate limiting, CMS security plugin
Destination not working	The server returned an error or the page failed to load	404/403/5xx, redirect chains, timeouts, certificates
Destination mismatch	The crawler saw something different from what users see	Geo redirects, device detection, personalisation, CAPTCHAs

Formal technical limits that get breached without anyone noticing:

• fewer than ten redirects in the chain (in practice, keep it to two);
• URL length between 12 and 2,048 characters — long tracking-parameter URLs occasionally hit the ceiling;
• no sign-in walls, CAPTCHAs or interstitials between the ad and the content;
• the page must open not only in your target region but also when requested from the United States, where checks frequently originate.

The full list of landing page disapproval reasons and how to clear them is covered separately in Google Ads destination requirements. This article stays on the technical layer: why the crawler never reaches the content in the first place.

Five reasons the crawler cannot see your page

1. WAF and anti-bot protection

The most common and least visible cause. Cloudflare, cloud WAFs and “anti-scraping” toggles in hosting panels classify traffic heuristically: request rate, absence of JavaScript execution, IP reputation, behaviour. AdsBot looks exactly like a suspicious client — fast, script-free, with a non-browser user agent.

Crucially, you rarely block AdsBot on purpose. It gets caught by a general rule: “challenge all non-browser agents”, “protect everything under /promo/”, a managed bot-fighting mode. The outcome is the same — the crawler gets a 403 or a challenge page, and the ad is disapproved.

Fix: allowlist AdsBot-Google and AdsBot-Google-Mobile by user agent, and verify authenticity by reverse DNS rather than by maintaining an IP list that changes.

2. robots.txt blocking the crawler by name

Check for an explicit block:

• User-agent: AdsBot-Google with Disallow: /;
• a disallow on the specific directory holding promo pages;
• a robots.txt that returns a 5xx — that is also read as a refusal to allow crawling.

A subtle trap: after a migration or CDN swap, robots.txt can still be served from the old origin. If you changed infrastructure recently, work through the site migration checklist for paid traffic, which covers this alongside redirects and tracking.

3. Rate limiting and load spikes

Dynamic targets and large accounts generate heavy crawling — on a site with thousands of URLs, the load is noticeable. When the server starts answering 429, 500 or 503, Google reads that as unavailability, and repeated failures can pause campaigns.

The tell is disapprovals arriving in batches, correlated with traffic peaks or large account edits. The fix is not blocking the crawler but raising limits for its agent and making pages cheaper to serve — see landing page speed and Core Web Vitals.

4. Geo redirects and “smart” personalisation

Automatic country redirects are a classic trap. Checks often originate from US addresses: a user in your target market sees the right page, while the crawler gets redirected to an international version, a “not available in your region” notice, or the homepage. Formally, that is a destination mismatch.

Cookie-based personalisation and A/B tests that serve the bot an empty variant break it the same way. The rule is simple: every ad landing page needs a version that works with no cookies, no session and no geolocation.

5. JavaScript rendering and empty HTML

If content is assembled client-side and the raw HTML contains only an empty container, the crawler may see a page with no copy, price or offer. That rarely causes an outright disapproval, but it does damage landing page experience — one of the three components of Quality Score alongside expected CTR and ad relevance.

The minimum bar: headline, offer, price and main copy present in the HTML response without executing scripts.

How to allow access properly: the concrete layers

“Allowlist AdsBot” runs into a practical question: where exactly? Work from the outer perimeter inward, because the first blocking layer ends the chain.

robots.txt

An explicit allow is its own group. If you maintain a blanket disallow for all agents, add a dedicated block for the ads crawler:

• User-agent: AdsBot-Google + Allow: /
• User-agent: AdsBot-Google-Mobile + Allow: /

Strictly speaking, with no explicit block in place these lines are unnecessary — the crawler ignores the wildcard group anyway. They serve a documentation purpose: the next person editing the file sees the agents were considered deliberately and does not sweep them up in a tidy-up.

CDN and WAF layer

Rule order matters more than rule content. An allow rule for AdsBot must sit above the general bot-protection rules, or it never evaluates. Things to check:

• bot-fight modes and managed rule sets — confirm an exception exists for these agents;
• geo restrictions — access must survive requests originating in the United States;
• path-level protection (/promo/, /lp/, /sale/) — promo directories are usually locked down separately and then forgotten;
• rate limiting — move the crawler agents into a separate, more generous policy.

Web server and application layer

Where a request can still die after passing the WAF:

• user-agent blocklists in nginx or Apache config — leftovers from an old anti-scraper effort;
• CMS security plugins with their own bot blacklists;
• content that requires a cookie or session to render;
• HTTP basic auth left on a directory after testing.

The rule that saves time: after any configuration change, verify with an actual crawler-agent request rather than an assumption. Distributed infrastructure applies config unevenly, and “we allowed it” frequently means “we allowed it on one node out of four”.

Dynamic formats: why crawl volume is rising now

A distinctly 2026 issue is growing crawl volume from formats that decide for themselves which page to send a user to. This began with dynamic search ads; the same principle now drives broader content-based targeting modes.

For a site, that means the crawler works through the entire discoverable URL set rather than a dozen landing pages — repeatedly. A site with a few thousand pages can see a material load increase, and these are precisely the accounts where rate limits start firing.

What to do about it:

1. Constrain the perimeter on the ads side, not the server side. If you do not need the whole site as targets, narrow the page set in campaign settings — that reduces load without any risk of blocking.
2. Exclude junk URLs: internal search results, filter combinations with unbounded parameters, utility pages. Every one of them is a separate address to a crawler.
3. Cache more aggressively. Pages assembled from a dozen database queries become a performance problem under bulk crawling.
4. Track response codes for the crawler agent separately. A rising share of 5xx for AdsBot is an early warning that will only surface in review several days later.

If you are moving campaigns onto formats that pick landing pages automatically, put this on the migration plan alongside settings — see DSA to AI Max migration.

A 30–40 minute diagnosis

Step 1. Reproduce the request as the crawler

Request the page with the AdsBot-Google and AdsBot-Google-Mobile user agents and inspect the status code and body. Compare against a normal browser request. A mismatch (200 versus 403) is your diagnosis.

Step 2. Read robots.txt literally

Open the file and find every AdsBot mention. Remember that User-agent: * does not apply to it — neither as a block nor as permission.

Step 3. Trace the redirect chain

Follow the path from the ad’s final URL to the destination and count the hops. Look for country redirects, http→https followed by another www hop, and tracking parameters dropped along the way. The last one breaks analytics too — see GCLID, GBRAID and WBRAID.

Step 4. Read 14 days of server logs

Filter on the AdsBot string and look at the distribution of response codes. This is the most honest source available: it shows blocks, 5xx spikes, and whether the crawler came at all.

Step 5. Audit the WAF rules

Do not search for “block AdsBot”. Search for general rules: challenges for non-browser agents, geo restrictions, path-level protection, managed bot modes. None of them mention Google in the configuration.

Step 6. Cross-reference with account change history

If disapprovals started on a specific day, line that date up against account edits and site releases. The method is described in using Google Ads change history to diagnose a drop.

Telling crawl problems apart from other “no impressions” causes

Signal	Likely crawling	Likely something else
Ad status	Disapproved, destination-related reason	Approved but not serving
Scope	Every ad on one domain or directory	Specific ad groups or keywords
Timing	After a site release, CDN change, security rollout	After bid, budget or targeting edits
Server logs	403/429/5xx for AdsBot	Crawler present, 200s returned

If ads are approved and still not serving, the cause lies elsewhere — rank, competition, limited serving status or settings. Those paths are covered in ads not showing: 12 causes and the order to check them and limited ad serving.

Six prevention rules that save weeks

1. Allowlist AdsBot explicitly in WAF and CDN rules — not “allow bots”, these two agents specifically. Document it so the rule survives the next configuration refresh.
2. Add a crawl check to the release checklist. Before shipping a promo section, fetch a handful of URLs as AdsBot.
3. Keep robots.txt under version control. Most sudden disapprovals trace back to an edit nobody remembers making.
4. No geo redirects on ad landing pages. If geo logic is required, make it soft: a banner offering to switch region instead of a forced hop.
5. Watch response codes under load, especially during sales when traffic and re-check frequency climb together — a point that intersects with planning peak windows in promotion mode in Google Ads.
6. Validate new URLs before launch, not after disapproval. Five minutes in preparation beats two days in review.

What to do when ads are already disapproved

1. Read the exact reason in the status column — it determines what you fix.
2. Reproduce the crawler’s request and capture the status code. That is your evidence.
3. Fix the cause on the site side (WAF rule, robots.txt, redirect, server error).
4. Confirm the fix propagated to every CDN node, not just the origin.
5. Resubmit: open the ad, make an edit, or appeal if you are confident the disapproval is wrong.
6. Do not delete and recreate the campaign to “speed things up” — you lose history and keep the problem.

Worth remembering that destination accessibility affects more than review. Landing page experience is one of three Quality Score components, so chronic access problems gradually erode rank and raise cost per click even when no disapproval ever fires. If you are working on both quality and price of traffic, the logical next step is a full Google Ads account audit.

Once the infrastructure side is stable, what remains is the advertising itself: structure, bidding and accounts sized for your volume. For the last part, see Google Ads agency accounts and the other PPC Rebels services.

FAQ

Does User-agent: * / Disallow: / block AdsBot?

No. AdsBot does not follow the wildcard group. To block it you must name it. The reverse is also true: a blanket “allow” guarantees nothing.

How do I verify a request really comes from AdsBot?

Use reverse DNS on the requesting IP: it should resolve to a Google domain, and a forward lookup on that hostname should return the same address. The user-agent string alone is trivially spoofed.

How often are live landing pages re-checked?

Google’s public documentation gives “roughly every two weeks” in one place and “every three weeks” in another. The practical conclusion: re-checks are regular, unannounced, and your site needs to be reachable continuously, not just on launch day.

Can I rate-limit AdsBot?

There is no crawl-rate control for it. Reduce load on your side instead — caching, lighter pages, avoiding expensive dynamic generation per URL. Blocking it outright stops your ads.

Why does the page work for me but “not” for the crawler?

Most often: the WAF serves you the page and the bot a challenge; a geo redirect fires because the check came from another country; content is script-assembled and absent from the HTML; or the failure is intermittent and only appears under load.

Does destination accessibility affect Quality Score?

Yes. Landing page experience is one of the three Quality Score components, and recurring access problems degrade it, which shows up in Ad Rank and cost per click.

What does “destination mismatch” mean?

The crawler saw content that differs from what a user sees at the same URL, or that does not match the ad. Typical causes are geo and device redirects, cookie-based personalisation, and A/B tests without a stable default variant.

Should AdsBot have access to staging environments?

No. Ad URLs should never point at closed environments. Keep staging locked down, but confirm no ad references it by mistake.

Does CDN caching affect the review outcome?

Yes. A fix at origin does nothing until edge caches refresh. After changing robots.txt or access rules, purge the cache before resubmitting for review.

How long does re-review take after a fix?

Usually a few hours to one business day. If the status does not move, verify the fix actually applied: repeat the crawler request and confirm a 200 comes back.

Should I appeal immediately?

Only if you have confirmed the page is reachable by the crawler, with a request and logs to prove it. Appealing without fixing the cause leads to another disapproval and wasted days.

Can I see in advance whether the crawler visited and what it got?

The most reliable source is your own server or CDN logs filtered by agent name. Search Console exposes AdsBot activity only to verified owners of the root property, which agencies managing an account often do not have.