seed collection

URL queue

Multithreading

politeness strategy

robot.txt

HEAD request reduces access overhead

Predict the frequency of changes based on historical updates

Site map provides update frequency

Topic-specific pages (authoritative sites)

Classify content text

Classify link title text

Features

Classification

Amortize compute and storage resource requirements

Multiple URL queue

Hash distribution tasks

Contains a set of homologous documents

Regularly updated

Rarely modified after release

Subscribe to Push

Active crawling (Pull)

multiple definitions

There are many standards

Has expiration time (ttl)

There is a summary of web page content

Large scale, high efficiency, low cost

Row storage, supports column groups

Hash server

hash file

Positioning within the file

Most compression algorithms do not support random access

Single file compression

Small amount of document compression

Plagiarism

Rubbish

Features

Document fingerprint

Not relevant to the main content

Generally includes logos, advertisements, pictures, navigation, etc.

Text can interfere with the index’s word frequency statistics

Based on label distribution

Filtering based on DOM tree

Based on DOM visual features