WordPress Robots.txt Generator and Editor

Take full control over the instructions you give web crawlers about what folders and files to crawl on your WordPress site.

wordpress robots.txt editor

Easily Make Changes to Robots.txt From Your WordPress Dashboard

Need an easy way to edit robots.txt in WordPress?

With All in One SEO, you can easily take control of your site and configure a robots.txt file to override the default WordPress file.

Explore All Our Powerful SEO Features

Here are all the features that make All in One SEO the best WordPress SEO plugin.

TruSEO On-Page Analysis

Easily add title tags, meta descriptions, keywords, and everything else you need for proper on-page SEO optimization.Continue reading

Rich Snippets Schema

All in One SEO Pro offers complete support for schema markup so you can get more clicks and traffic with rich snippets.Continue reading

Local SEO

All in One SEO gives you all the tools you need to improve your local SEO and rank higher on Google Maps.Continue reading

Smart XML Sitemaps

Automatically generate a WordPress XML sitemap and notify all search engines of any updates.Continue reading

Video SEO Sitemaps

All in One SEO includes a video sitemap generator, so you can rank in Google’s video carousel widget and grow your traffic.Continue reading

WooCommerce SEO

Advanced eCommerce SEO support for WooCommerce to optimize product pages, product categories, and more.Continue reading

SEO Audit Checklist

Analyze your entire WordPress site to detect critical errors and get actionable insights to boost your SEO and get more traffic.Continue reading

Redirection Manager

Easily create and manage redirects for broken links to avoid confusing search engines or lose valuable backlinks.Continue reading

Google News Sitemaps

Get higher rankings and unlock more traffic by submitting your latest news articles to Google News.Continue reading

Robots.txt Editor

Take full control over the instructions you give web crawlers about what folders and files to crawl on your WordPress site.Continue reading

Search Statistics

Optimize your content with precision, and maximize conversions and revenue. Take control of your SEO strategy and start ranking higher in search results.Continue reading

SEO Custom User Roles

SEO user roles allow you to manage access to important SEO features without handing over control of your website.Continue reading

RSS Content

Take control of your RSS feed to add credits, custom text and links, or even advertisements to help monetize your blog.Continue reading

Post Index Status

Check whether your posts have been indexed by Google inside the Search Statistics reports.Continue reading

Image SEO

Automatically set image alt text and title attributes for images with our powerful image SEO addon.Continue reading

Improve your WordPress SEO Rankings with AIOSEO

Get the most powerful All in One WordPress SEO plugin and improve your website search rankings today.

(Instant Download)

Even more powerful SEO features

  • Easily add title, meta description, keywords, and everything you need for on-page SEO optimization
  • Complete support for Google Knowledge Graph and Schema rich snippets
  • Easily connect with Google Search Console, Bing, Yandex, Baidu, and Pinterest webmaster tools
  • Advanced eCommerce SEO support for WooCommerce, so you can optimize product pages, product categories, and more.
  • Automatically notify search engines like Google and Bing about changes to your site.

How to Create and Optimize Robots.txt File on WordPress

Are you having trouble creating or editing a robots.txt file for your WordPress website? If yes, then you don’t have to look any further. This guide will teach you how to create a robots.txt in WordPress that’s perfectly optimized by using All in One SEO, the best WordPress plugin for SEO in the market.

What Is Robots.txt?

Robots.txt is a text file that has instructions, also known as directives, for the search engine bots. Website owners use it to prevent search engines from crawling unnecessary pages on their website. It also gives tips to search engines on how to best navigate and index your website.

Here’s a typical WordPress robots.txt example:

User-agent: *
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-admin/
Disallow: /wp-content/uploads/wpforms/
Disallow: /xmlrpc.php
Sitemap: https://aioseo.com/video-sitemap.xml
Sitemap: https://aioseo.com/sitemap.xml
Sitemap: https://aioseo.com/sitemap.rss

Before we discuss the robots.txt file in the example above, let’s take a look at the main components of a robots.txt file.

  • User-agent: User-agent is what search bots use to identify themselves. You can use this directive to target specific search engine bots. For example, if you only want Google to crawl your website, then you should add googlebot in place of *. The * indicates that all of the search engines like Google, Bing, Yahoo, and others are allowed to crawl your website.
  • Allow: This indicates that the search engine is allowed to crawl this specific section of the website.
  • Disallow: Using this directive, you can tell search engines not to access certain pages or files on your website.

Going back to the robots.txt example, we have allowed all search engines to crawl our website using the user-agent: * directive. We have also allowed and disallowed URLs based on how we want Google to crawl our website.

By using the robots.txt file, you can also submit multiple sitemaps. In our robots.txt file, we have submitted a separate video sitemap we have on our website. This makes it easier for search engines to find, crawl, and index videos of our website.

A sitemap is a navigational map of your website. It tells the search engines of how many pages you have on your website. With AIOSEO’s WordPress Sitemap Generator you can easily create and submit your sitemaps.

A lot of people also ask, “Where is robots.txt in WordPress?” in case they want to make edits.

The WordPress robots.txt location is in the root directory of your WordPress website. You can view the file by connecting to your site using an FTP client or cPanel.

But you won’t have to go that far if you’re using AIOSEO. It lets you edit and view your robots.txt file from your WordPress dashboard. No need for a separate WordPress robots.txt plugin. We will discuss more about it in detail later in this guide.

You can also view it online by typing in your domain name followed by the text “robots.txt.” For example, https://www.example.com/robots.txt.

Now that you know what a robots.txt file is and what it looks like, let’s talk about why it’s important for your website.

Why Do You Need a Robots.txt File For Your WordPress Website?

Robots.txt allows you to control how search engines crawl and index your website. Without the robots.txt file, search engines will crawl all of your website, even the pages that aren’t important.

If you don’t stop the search bots from crawling unnecessary pages, they will use up all of your crawl budget and might not crawl the pages you want them to crawl.

Crawl budget is the level of attention a search engine like Google gives to your website. This is based on how frequently search engines like to crawl your website and how frequently your website can be crawled.

If you’re wasting your crawl budget, Google and other search engines won’t be able to efficiently crawl your website and that could hurt your website’s performance.

Each website has a limited crawl budget. Search engine bots only crawl a certain number of pages during their crawl session.

If the bot doesn’t crawl all of your website in the first crawl session, then it will come back and crawl the remaining pages in the next session.

Depending on your website, the next crawl sessions might not be for days. This can slow down your website’s indexation, and your important pages will not get crawled and indexed on search engines. This can lead to a huge loss in traffic and business.

You can quickly fix this by disallowing your website’s unnecessary pages, like your WordPress admin page, plugin files, and themes folder. This will help save your crawl budget and improve your indexation rate.

Now let’s get to the part where we tell you how to view, edit, create, and submit robots.txt file using AIOSEO.

How to Create a Robots.txt file for your WordPress Website?

All in One SEO (AIOSEO) makes it very easy to create and edit the robots.txt file on WordPress.

All you need to do is head over to your WordPress dashboard and go to All in One SEO » Tools.

AIOSEO is the best WordPress robots.txt plugin and SEO plugin

Then, you will be then taken to the Robots.txt Editor page. From here, you can easily add or edit the robots.txt file using the form.

allow or disallow in WordPress robots.txt editor

With AIOSEO, you won’t have to worry about formatting the robots.txt file. All you have to do is add directives to the fields, and AIOSEO will do the rest.

Once you’ve added the directives, click on Save Changes to complete the setup.

Now that you’ve created your robots.txt file, you need to test it to see if it has any errors.

Test Your WordPress Robots.txt File

To test your robots.txt file, simply go to the robots.txt tester tool and sign in with your Google search console account.

Then type “robots.txt” in the URL field and click on Test.

If the robots.txt is file validated, then you will get the Allowed message. If not, then you will get the errors and warnings highlighted in the box.

validate your robots.txt wordpress file

And you’re done! You’ve now successfully created and validated your WordPress robots.txt file.

We hope that this guide helped you learn how to create an optimized robots.txt file for your WordPress website using AIOSEO.

Frequently Asked Questions and Resources

Do you have a question about AIOSEO? See the list below for our most frequently asked questions. If your question is not listed here, then please contact us.

Who should use AIOSEO?

All in One SEO is perfect for business owners, bloggers, designers, developers, photographers, and basically everyone else. If you want to optimize your WordPress SEO, then you need to use All in One SEO Pack.

What’s required to use AIOSEO?

All in One SEO is a WordPress Plugin. In order to use All in One SEO pack, you must have a self-hosted WordPress site. That’s all.

Do I need coding skills to use AIOSEO?

Absolutely not. You can optimize your WordPress SEO without any coding knowledge. All in One SEO is one of the most beginner friendly WordPress SEO plugin.

Will All in One SEO slow down my website?

Absolutely not. All in One SEO is carefully built with performance in mind. We have developed everything with best practices and modern standards to ensure things run smooth and fast.

Is All in One SEO translation ready?

Yes, All in One SEO has full translation and localization support. All in One SEO is translated into 10 languages with many more on the way.

Does All in One SEO work on non-WordPress sites?

No. All in One SEO is a WordPress SEO plugin, so it will NOT work on sites that do not use WordPress. Additionally, All in One SEO is not compatible with the WordPress.com platform. You must be using a self-hosted version of WordPress to utilize All in One SEO.

Can I use All in One SEO on client sites?

Yes, you can use All in One SEO on client sites in two ways. Either you can purchase the Pro / Agency license OR purchase an appropriate license for each client sites. You can also refer your clients to purchase directly from our site using your affiliate link to earn a 20% commission.