Different websites have different anti-crawler strategies and different restrictions on crawlers. Generally, they can be divided into the following three categories:
1. Set not to return web pages or delay return time
The traditional anti-crawler method is not to return web pages, that is, the crawler sends the request to the corresponding website, and the website returns a 404 page, indicating that the server cannot provide information normally, or the server has no response; the website may also not return data for a long time, which means that the crawler is banned.
2. The returned web page is not the target web page
In addition to not returning pages, there are also some crawlers that return non-target pages, that is, the website returns false data, for example, when returning a blank page or crawling back multiple pages, the same page is returned. If your crawler runs smoothly, you will be happy to do other things. After searching for half an hour, you will find that the search results for each page are the same, that is, fake websites.
For example, the price page of Qunar.com, the price marked online is different from the html source code. For example, the air ticket price marked online is 530 yuan, and the ticket price in the html source code is 538 yuan. In addition to Qunar.com, Maoyan Movies and Douyu Live also adopted this method, and the numbers crawled down are different from the real numbers.
3. Increase the difficulty of access
The website will also prevent crawlers by increasing the difficulty of obtaining data. Generally speaking, you can see the data by logging in and setting a verification code. In order to limit crawlers, the website may require you to log in and enter the verification code to access, regardless of whether you are a real user. For example, in order to limit automatic ticket grabbing, 12306 adopts a strict verification code function, requiring users to correctly select from 8 pictures.
These three situations are common in the crawler world. Crawlers need to formulate different anti-crawling strategies according to different actual situations in order to work smoothly.