Differences between Disallow: /test and Disallow: /test/
Recently, I found that some friends still can't understand the difference between them. todayseoLet's give you a detailed introduction
The value of this item is used to describe a URL that the Robot does not want to access. This URL can be a complete path or a partial path. Any URL that starts with Disallow will not be accessed by the Robot;
Note the difference between "Disallow: /test" and "Disallow: /test/". Although it is only different from a backslash "/", the meaning is completely different. "Disallow: /test" means that the URLs that can be prohibited include: /test,/testabc. html, and/test/abc;"Disallow: /test/" allows the Robot to access/test,/testabc. html, and prohibits access to the form/test/abc.
If the value of the Disallow record is null, in the format of "Disallow: ", it means that all content of the website can be crawled by any search engine Robot; in the robots.txt file, if there is a declaration User-agent, there must be at least one Disallow record.