How to write robots.txt and what can robots.txt do?
Allow all robot access
User-agent: *
Disallow:
Or you can create an empty file "/robots.txt" file
Ban all search engines from accessing any part of the website
User-agent: *
Disallow: /
Block all search engines from accessing several parts of the website (directories 01, 02, 03 in the following example)
User-agent: *
Disallow: /01/
Disallow: /02/
Disallow: /03/
Disable access to a search engine (BadBot in the following example)
User-agent: BadBot
Disallow: /
Allow access to only one search engine (Crawler in the following example)
User-agent: Crawler
Disallow:
User-agent: *
Disallow: /
Usage of robots.txt
First, use robots to block duplicate pages on websites
Many websites provide multiple browsing versions of one content. Although it is very convenient for users, it causes difficulties for spiders, because it cannot distinguish which is the main and which is the second. Once it thinks that you are repeating it maliciously, brother, you will be in trouble.
User-agent： *
Disallow: /sitemap/"Ban spiders from grabbing text pages"
Second, use robots to protect website security
Many people wonder, why are robots related to website security? In fact, it still has a lot to do. Many low-level hackers search for default background login to achieve the goal of invading websites.
User-agent： *
Disallow: /admin/"Prevent spiders from grabbing all files in the admin directory"
Third, prevent chain theft
Generally, there are only a few people who steal links, but once you are "hacked" by a search engine, you can't afford to have 100M broadband. If you are not building a photo website and you don't want to be "hacked" by a search engine, your website pictures are not hacked "by a search engine.
User-agent： *
Disallow： .jpg$
Fourth, submit a website map
Nowadays, everyone who does optimizations knows how to make site maps, but few will submit them. Most people just add a link to the webpage. In fact, robots support this feature.
Sitemap：https://www.zolfox.com/sitemap.html
Fifth, it is prohibited to crawl a certain second-level domain name
Some websites provide special services to VIP members but do not want this service to be retrieved by search engines
User-agent： *
Disallow： /
The above five robots are enough to give you a lot of control over search engine spiders. As Baidu said: We and search engines should be friends. Only by increasing some communication can we eliminate some gaps.
Let's talk about nofollow
Nofollow was first proposed by a tag that led the creation to minimize the impact of spam links on search engines. The meaning of this tag is to tell search engines that the link was not edited by the author himself, so the link is not a vote of confidence. When search engines see this tag, they may reduce or completely cancel the voting weight of the link. The initial reason was that with the popularity of online blogs, spam comments and spam links gradually spread, which attracted Google's attention and took the lead in proposing a protest plan. Google's measure is to effectively block such spam comments and spam links through a new label. Since then, once Google discovers that hyperlinks contain this attribute ("rel=nofollow"), such links will not be used as a basis for evaluation of a site in search result rankings; at the same time, the tag will not become a negative vote on the blog or web page where the spam comment is located. It is only used to prevent public areas such as spammers 'blog comments, feedback forms or recommendation lists from benefiting from confusion. The Nofollow tag is important to search engines.
But after so much talk about nofollow, what is the nofollow label? How to use the nofollow label? Here's Google's explanation of rel="nofollow": When Google sees the attribute rel="nofollow" on hyperlinks, those links won't get any credit when we rank websites in our search results. That is, once Google discovers that a hyperlink contains this attribute ("rel=nofollow"), such links will not be used as a basis for evaluation of a site in search result rankings.
The nofollow tag has another purpose. If you are selling ads on your website, you can use this tag. Because the original intention of buying and selling web ads should be traffic, not PR value or trying to influence search engine rankings. Adding this tag will not affect traffic at all, but it may reduce the impact on search engine rankings.
Now talk about the role of the nofollow tag on links. Some people will ask whether adding this tag really has an impact on the voting weight and PR value of links? This is questionable-links in blog comments and forum posts are different. These links are added by users and readers themselves, not by site owners or authors. In many cases, authors will not look at what websites these links go to. So of course, I don't know the quality of these websites. I have never said that links to comments from my website are important, and then charged people to add links, or asked readers to link back. Although the links in blog comments are external links, they are of little use to search engine rankings and PR values.
Why talk about these two separately? I heard a buddy say that it has been proven through practice that neither robots.txt nor nofollow work. I haven't tried them specifically, but interested friends can try them. More exciting content is inBeijing seo