Feb 26, 2019 By: drive

Robots.txt is a text file that gives permission to the search engine bots, to whether or not to crawl the website content. Basically, the search engine robots before crawling the website content, checks if it has the permission to do so.

(When a search engine bot start crawling your domain. it will first look for robots.txt if found then it will follow the instructions mention in the file. and if robots.txt is not available it will crawl the entire domain including admin urls.)

The code for these instruction is written simple in a notepad and saved as a txt file. It looks something like this:

User-agent: * 
Allow: /
Disallow: /cgi-bin/
Disallow: /wp-admin
Sitemap: https://www.yourdomain.com/sitemap.xml

In order to solve the Robot.txt test error, simply follow two steps:

  • Create a Robot.txt file (Test the code suggested below)
  • Add the file in the cpanel –> public folder of the domain.

How to create robots.txt using Google Webmaster?

webmaster test for robots

You can download the robots.txt file from the google webmaster panel itself by clicking ‘submit’.

download robot.txt file

In this simple code each word has its Specification

  • User-agent defines the search engine. * sign by default means all search engines.
    For specific search engine it can be like:
    # Only for Google:- “User-agent: Googlebot ”
    # Only for Bing :- “User-agent: Bingbot “
  • Allow: / means all the search engines robots are allowed to crawl all the website directories/Pages (links).
  • Code ‘Disallow’ is used to disallow bot(s) to not crawl the directorie(s) or page(s) Disallow: /cgi-bin :- means the robots are not allowed to visit the cgi files. Disallow: /wp-admin :- means the robots are not allowed to crawl the wordpress admin urls. It will not show in SERPs

These urls can be different in your case. It depends on the platform which you are using to develop your website.

Note:- some smart Search engine (bad search engines) can skip your robots.txt and crawl your entire domain. But most popular search engine like google, bing etc gives priority to it.

Share It On:-

Tags :

Leave a Reply

Your email address will not be published. Required fields are marked *

20 − 8 =