How to Check If Your Delhi WordPress Site Is Blocked from ChatGPT and Perplexity in 5 Minutes

To check if your Delhi WordPress site is blocked from ChatGPT and Perplexity, open a new browser tab and type your domain followed by /robots.txt — for example, yourdomain.com/robots.txt. If you see “User-agent: GPTBot” with “Disallow: /” below it, ChatGPT cannot crawl your site. Same for PerplexityBot. The fix takes under 10 minutes.

Why Does This Check Matter for Delhi Businesses?

Most Delhi business owners I speak to have never opened their robots.txt file. That’s understandable — it’s a technical file that most developers set up once and never touch again. But in 2026, it has become one of the most important files on your entire WordPress site for a very specific reason.

ChatGPT uses a crawler called GPTBot to read websites and then cite them in answers. Perplexity uses PerplexityBot. If either of these is blocked in your robots.txt — which happens more often than you’d think, especially on WordPress sites running Wordfence or set up by developers using blanket security templates — your content simply does not exist in those AI systems. It doesn’t matter how good your content is. It doesn’t matter if you rank on page 1 of Google. Blocked means invisible.

I’m Amrinder Singh, founder of Digi Ustad. I’ve been auditing Delhi NCR WordPress sites for AI visibility since early 2025, and the robots.txt block is the most common issue I find — present on more than half the sites I review. Here is exactly how to check yours and fix it if needed.

Step 1 — Open Your robots.txt File Right Now

Type this into your browser address bar, replacing “yourdomain” with your actual domain:

yourdomain.com/robots.txt

This file is always publicly accessible. No login needed. It should load within 2 to 3 seconds and show plain text.

If the page comes back with a 404 error — your site has no robots.txt file. That’s a separate issue to fix, but it does mean AI crawlers are not explicitly blocked either.

Step 2 — Look for These Specific Lines That Block ChatGPT and Perplexity

Scan the file for any of these combinations:

ChatGPT is blocked if you see:

User-agent: GPTBot

Disallow: /

or

User-agent: ChatGPT-User

Disallow: /

Perplexity is blocked if you see:

User-agent: PerplexityBot

Disallow: /

ClaudeBot (Anthropic) is blocked if you see:

User-agent: ClaudeBot

Disallow: /

Google AI Overviews training is blocked if you see:

User-agent: Google-Extended

Disallow: /

Any of these with Disallow: / means that AI system cannot read your site.

One important distinction worth knowing: GPTBot and ChatGPT-User are two separate OpenAI crawlers. GPTBot handles training data and offline indexing. ChatGPT-User handles real-time browsing when someone asks ChatGPT to look at a specific page. Some Delhi businesses choose to block GPTBot (training) but allow ChatGPT-User (real-time search) — this is a valid configuration that lets your content appear in ChatGPT search results without contributing to OpenAI’s training dataset. Both need to be checked separately.

Step 3 — Check If Cloudflare or Your Security Plugin Is the Cause

If your robots.txt looks clean but you’re still not appearing in ChatGPT searches for your category — check these three common causes specific to Delhi WordPress setups:

Cloudflare’s AI Scraper Blocking toggle. Cloudflare added a one-click “Block AI Scrapers and Crawlers” feature in 2024. It’s easy to enable by accident, and it modifies your robots.txt and firewall rules automatically. Log into your Cloudflare dashboard, go to Security → Bots, and check whether AI bot blocking is toggled on. Many Delhi web designers enable this during setup without telling the client.

Wordfence Firewall rules. Wordfence is the most commonly installed security plugin on Indian WordPress sites. Its default firewall rules can block AI crawlers at the server level — meaning even if your robots.txt looks fine, Wordfence may be refusing requests from GPTBot before they even reach your content. Go to Wordfence → Firewall → Blocking and check for any rules targeting AI bots.

Your hosting provider’s WAF (Web Application Firewall). Some Delhi web hosting providers — especially managed WordPress hosts — have server-level WAF rules that block AI crawlers by default. If you’re on a managed host, contact their support and ask specifically whether GPTBot and PerplexityBot are permitted.

[INSERT SCREENSHOT: Wordfence firewall panel showing bot blocking rules — ALT text: “Wordfence firewall settings on Delhi WordPress site showing AI bot blocking configuration”]

Step 4 — Fix the Block in Under 10 Minutes

If you found a block, here is the correct fix. The fastest method on WordPress is using the WPCode plugin (free) or editing robots.txt directly via RankMath or Yoast SEO.

Via RankMath: Go to RankMath → General Settings → Edit robots.txt. Add these lines:

User-agent: GPTBot

Allow: /

User-agent: ChatGPT-User

Allow: /

User-agent: PerplexityBot

Allow: /

User-agent: ClaudeBot

Allow: /

User-agent: Google-Extended

Allow: /

Via Yoast SEO: Go to Yoast SEO → Tools → File Editor → robots.txt. Add the same lines above.

Via cPanel File Manager: Log into your hosting cPanel, open File Manager, navigate to public_html, find robots.txt, and edit it directly. Add the Allow lines for each AI crawler.

After making changes, verify immediately by re-opening yourdomain.com/robots.txt in your browser and confirming the Allow lines appear. If the Disallow lines are still present and conflicting, remove them entirely.

Step 5 — Verify in 48 Hours Using Server Access Logs

The fastest way to confirm AI crawlers are now accessing your Delhi WordPress site is checking your server access logs 48 to 72 hours after making the fix. Look for entries containing “GPTBot” and “PerplexityBot” — their appearance in your logs confirms they are successfully crawling your pages.

In cPanel, go to Metrics → Raw Access → Download your latest access log. Open the file and search for “GPTBot”. If you see entries like:

123.45.67.89 – – [15/Aug/2026:09:14:22 +0530] “GET /your-page/ HTTP/1.1” 200 – “-” “GPTBot/1.1”

GPTBot is successfully crawling. If you see no entries after 72 hours, there is likely a server-level block still in place — contact your hosting provider.

The Unique Insight — Two Different GPTBot Configurations

Most guides treat “allow GPTBot” as a binary choice. It’s not. The smart configuration for most Delhi businesses — especially those concerned about content being used for AI training without compensation — is what’s called a hybrid robots.txt policy.

Allow ChatGPT-User (real-time search, gets your content cited in ChatGPT answers) while blocking GPTBot (training data collection). This means your content appears in ChatGPT search results when potential customers ask for recommendations, but your site content isn’t fed into OpenAI’s training pipeline for future models.

# Allow real-time ChatGPT search citations

User-agent: ChatGPT-User

Allow: /

# Block training data collection only

User-agent: GPTBot

Disallow: /

# Allow Perplexity citations

User-agent: PerplexityBot

Allow: /

# Allow ClaudeBot citations

User-agent: ClaudeBot

Allow: /

This is the configuration I recommend for Delhi professional services businesses — clinics, law firms, consultancies — where the content is proprietary but the citation in ChatGPT answers is commercially valuable.

What Comes After Fixing the robots.txt Block

Unblocking AI crawlers is step one. It’s the prerequisite. But a Delhi WordPress site that AI crawlers can access will still not get cited in ChatGPT unless it has two additional elements in place:

FAQPage schema via RankMath or Yoast. Every service page needs a FAQ section with structured schema so Google AI Overviews and ChatGPT can extract and cite specific answers. This is covered in detail on the Delhi WordPress SEO, AEO and GEO pillar page.

llms.txt file. A plain text file at yourdomain.com/llms.txt that tells AI systems what your Delhi business does, which pages are most important, and what queries you want to be cited for. Think of it as a sitemap specifically for AI models.

For Delhi businesses wanting to understand the full methodology — from robots.txt to FAQPage schema to GA4 AI referral tracking — the WordPress SEO AEO GEO pillar page covers each step in sequence.

How to Know If the Fix Is Working — GA4 AI Referral Sessions

The definitive proof that AI crawlers are successfully accessing your Delhi WordPress site and referring traffic is GA4 with a custom AI referral channel. Set up a channel group in GA4 → Reports → Acquisition → Traffic Acquisition → Channel Groups → Create custom channel group.

Add these sources as a new channel called “AI Referrals”:

• Source exactly matches: chatgpt.com
• Source exactly matches: perplexity.ai
• Source exactly matches: gemini.google.com
• Source exactly matches: copilot.microsoft.com

Once this is set up, you’ll see AI referral sessions appear within 60 to 90 days of correctly fixing your robots.txt and adding FAQPage schema. Digi Ustad’s own WordPress site — digiustad.com — shows utm_source=chatgpt.com sessions growing month-over-month in live GA4 since the fix was applied in early 2025.

FAQ — Delhi WordPress Site Blocked from ChatGPT