Robots.txt for Ecommerce: What It Is and How to Get It Right

2 min read

Every website has a robots.txt file — it's the first thing search engines look at when they visit your site. It tells them which pages they're allowed to crawl and which ones to skip. For ecommerce stores, getting this wrong can mean your products disappear from Google entirely.

What Is robots.txt?

Robots.txt is a plain text file that lives at the root of your website (yourstore.com/robots.txt). Search engine crawlers read it before crawling any other page. It contains rules like: `User-agent: *` (applies to all search engines) `Disallow: /admin/` (don't crawl the admin area) `Allow: /products/` (crawl product pages) `Sitemap: https://yourstore.com/sitemap.xml` (here's my sitemap) Think of it as a bouncer at the door — it tells search engines where they can and can't go. But it's advisory, not enforced — some crawlers may ignore it.

Common robots.txt Mistakes

**Blocking everything:** `Disallow: /` blocks your entire site from search engines. This is surprisingly common after site migrations or developer mistakes. **Blocking product pages:** `Disallow: /products/` or `Disallow: /shop/` prevents Google from indexing your products. Some staging environments have this and it accidentally carries over to production. **Missing sitemap reference:** Without a `Sitemap:` line, search engines rely on other methods to find your sitemap. Adding it helps them discover all your pages faster. **Blocking CSS/JS:** Some old guides recommend blocking CSS and JavaScript files. Don't — Google needs to render your pages to evaluate them properly.

robots.txt on Shopify

Shopify generates robots.txt automatically with sensible defaults. It blocks admin pages, checkout, and cart, while allowing all product, collection, and content pages. Since Shopify 2021, you can customize robots.txt by creating a `robots.txt.liquid` template in your theme. This lets you add custom rules or additional sitemap references. You can view your current robots.txt at: yourdomain.com/robots.txt

robots.txt on WooCommerce

WordPress generates a virtual robots.txt by default. SEO plugins like Yoast and RankMath let you customize it through their settings. You can also create a physical robots.txt file in your WordPress root directory, which overrides the virtual one. RankRipper analyzes your robots.txt on every scan — showing the raw content with syntax highlighting, flagging any rules that block important pages, and checking that your sitemap is referenced.

buildRelated Fix Guides

build

How to Fix Robots.txt Blocking Important Pages

Your robots.txt is blocking Google from crawling important pages. Learn how to fix robots.txt on Shopify and WooCommerce to stop losing organic traffic.

build

How to Add Your Sitemap to robots.txt on Shopify & WooCommerce

Your robots.txt doesn't reference your sitemap. This simple fix helps Google discover and crawl all your pages faster.

Keep Reading

article

What Is a Canonical Tag? A Plain-English Guide for Store Owners

Learn what canonical tags are, why they matter for your online store's SEO, and how to make sure yours are set up correctly — no coding knowledge required.

article

The Complete WooCommerce SEO Checklist for 2026

A practical WooCommerce SEO checklist covering permalinks, schema markup, page speed, meta descriptions, and more. Built for store owners, not developers.

Check your store now

Scan your site for free — get copy-paste fixes in 60 seconds.

searchScan Now — It's Free