Reasons why website content cannot be fully included
Beijing seoLet everyone reveal the real reasons why it cannot be fully included.
First: It is blocked by the robots.txt file. When analyzing website logs, we found that spiders crawl on the website every day, but the website is still not fully included. At this time, it is necessary for us to test the robot files of the website. Webmasters all know that when spiders come to the website to crawl, they will first search for the website to see if there are robots files on the website to check whether there are any pages that do not need to be crawled, and whether the files block certain parts of the website. Many webmasters cannot write robot files correctly, resulting in the website not being fully included. For many novice webmasters who don't know how to write files, you can use the robots file function of Baidu webmaster tool to detect whether your file is written correctly, or you can directly help you generate files, requiring you to fill in the blocked file path. Take the following example:
User-agent: *
Disallow: /news /Not allow all search engines to crawl content under the news path
If the above files are added to a website, the spider will not crawl the content on the website under the news path. The articles updated by the website under the news directory will never be included. No matter how many articles the website updates, the spider will crawl every day, but it will still not include these content. But the content under news is very important for websites. Such erroneous robots files are not the mastermind behind the website's inability to be fully included.
Second: The robots meta tag is prohibited. During the diagnosis process, the website found that the actual inclusion rate of the website is very low. Some column pages of the website can be included completely, but some column pages that update original articles will not be included. Later, when checking the website code, I found that the noindex tag was used on the page to tell the spider that this page was not allowed to index this page. Obviously, the column page protecting this code will not be included, and the updated content will not be crawled no matter how high the quality is. At the same time, the nofollow tag tells spiders that links on this page do not convey weight. If all page links on the website carry nofollow, this obviously tells search engines that this page has no value. If you encounter a site that cannot be fully included, check the meta tags to see if there are any wrong tags.
Third: The page has not been visited. Especially for some large content-based websites, there are many content pages on the website. If internal links cannot be established well between websites, many pages may face being sunk and not included. Most of these pages are deep away from the home page of the website, and spiders cannot crawl the pages, resulting in them not being included by spiders. Or the links to this page are all links with nofollow tags and do not convey any weight. Because of this reason, it is unfair that the website is not fully included. For website construction, it is best not to use nofollow tags, JS code, or directions that spiders cannot recognize on the website. For the case where website pages are relatively deep, webmasters can improve the website's internal chain construction, prevent the website pages from becoming independent pages, establish navigation and internal links, or add external links to the pages they want to include, and improve the page's weight in search engines.
Fourth: Spiders regard content as cheating, and a large number of black hats are used on the websiteSEO, using cheating techniques to optimize pages, such page spiders will not include them. Hidden words have been used on websites for a long time to accumulate keywords on the page. The website retrieves hidden words and links for spiders. At this time, your website is likely to be deleted from the index by the spider and will not appear in the search page again. Webmasters may ask what cheating content is. For example, the original cheating method uses text with the same color as the background, accumulating a large number of keywords, which is easy to detect; using the noscript tag to tell the browser when there is JS on the page, when the browser closes, what content should be displayed on the page. In fact, a lot of keywords are piled up in the noscript tag. This practice is related and dangerous and can easily lead to content being deleted.
Fifth: Low-quality content spam sites. Apart from the above-mentioned four-point optimization methods, there is also an important problem that cannot be ignored. The search engine's recognition capabilities are gradually strengthened. For non-high-quality and non-original content, spiders have certain recognition capabilities. Whether it is duplicate content on their own website or duplicate content obtained by the website from the outside, spiders can recognize it to a certain extent. For duplicate content pages, spiders will not continue to put them into their own database, and even delete the index on some low-weight websites. In the era of the Internet garbage bubble, if you want to truly achieve sufficient inclusion and high weight of websites, you still need to insist on making high-quality content so that you can survive in the industry for a long time.