Why study page value
First of all, the pages on the Internet are endless, and the hardware resources of search engines are limited. If we want to use limited resources to cover the endless Internet, we need to make a judgment on the value of the pages and not include those with no search value. Pages, include less pages with low search value. This is the application of page value in inclusion control.
Second, the crawling capabilities of search engine spider are limited. For accessibility reasons, there needs to be an upper limit on the crawling rate of a website or an IP. Under this limit, there is a priority order for crawling or page updates, and the main reference basis for this ranking is the page value, or prediction of the page value (when not crawled). This is the application of page value in spider scheduling.
Third, for some pages, the content of the page changes, resulting in its retrieval value from existence to non-existence. Typically, it becomes a "dead chain" or "hacked". For these pages, good search engines will exclude them from the index as soon as possible, or block them during retrieval to ensure that the results returned to users are more "good pages" with high search value. For other pages, it not only has high retrieval value, but also has strong "timeliness". Being able to allow users to retrieve these pages as soon as possible will greatly improve the search experience. For search engines, the faster the inclusion and index of pages means more additional resource overhead. How fast the inclusion and how short the index is updated needs to be guided by the analysis of page value. These two aspects are the application of page value in improving the two major search engine indicators of dead chain rate and timeliness.
Finally, the value of a page in the general sense issearch engineThere is also guidance in sorting the results returned to the user. Ideally, search engine results are ranked according to their relevance to the query request. When the relevance is roughly equivalent, users are more likely to browse web pages with high page value in the general sense. This is the application of page value in ranking.
It can be said that the study of page search value is a relatively basic task in search engines. The understanding and accuracy of the judgment of page value directly affects several major indicators such as coverage rate, dead chain rate, and timeliness of search engines.