robots.txt generator

free robots.txt generator
marwvengat.com
site details
website url (optional)
sitemap url will be auto-generated and added to your robots.txt
added automatically to the bottom of your robots.txt
search engine access
allow these bots full access
googlebot
google search crawler
bingbot
microsoft bing crawler
slurp
yahoo search crawler
duckduckbot
duckduckgo crawler
yandexbot
yandex search crawler
baiduspider
baidu search crawler
block ai training bots
select bots to block from crawling your content
gptbot
openai training crawler
claudebot
anthropic training crawler
ccbot
common crawl (ai datasets)
google-extended
google ai training crawler
meta-externalagent
meta ai training crawler
omgilibot
used for llm training data
block common paths
block these paths from all bots
/wp-admin/
wordpress admin panel
/wp-login.php
wordpress login page
/wp-includes/
wordpress core files
/wp-content/plugins/
plugin files
/?s=
search result pages
/tag/
tag archive pages
/author/
author archive pages
/cart/
woocommerce cart page
/checkout/
woocommerce checkout
/my-account/
woocommerce account
/feed/
rss feed pages
/cgi-bin/
server cgi scripts
custom disallow rules (optional)
add any additional paths you want to block from all bots
your robots.txt file
!
upload this file to your website root directory (e.g. https://yoursite.com/robots.txt). replace any existing robots.txt file. always test with google search console after uploading.

about this tool

your robots.txt file tells search engine crawlers which pages on your site they're allowed to visit — and which ones they should skip. it's a small but important file that affects how your site gets indexed. get it wrong and you could accidentally block google from crawling pages you actually want ranked, or leave sensitive pages unnecessarily exposed.

this free robots.txt generator makes it easy to build a properly formatted robots.txt file without having to write a single line of code. just fill in your website url, add your sitemap location, and configure which crawlers and directories you want to allow or disallow. the tool generates the correct syntax instantly.

it's ideal for marketers, bloggers, and site owners who want control over their crawl settings but don't want to dive into technical seo documentation to get there. once your file is generated, you simply upload it to your site's root directory and you're done. a properly configured robots.txt is one of those foundational seo basics that's easy to get right when you have the right tool.