When we use Python crawlers to collect information, we are often blocked. Sometimes it prompts that the access is too frequent, and sometimes some error codes are returned. The reason for this situation is that the crawler IP is detected and restricted by the website. So how does the website know that the crawler is collecting information?

1. IP detection

The website will detect the speed of user IP access. If the access speed reaches the set threshold, the restriction will be opened, the IP will be blocked, the crawler will stop, and it will not be able to obtain data again. To deal with IP detection, you can use proxy IP and switch a large number of IP addresses to break through the restrictions.

2. Verification code detection

Set login verification code restrictions and set verification code restrictions for access too fast. If you do not enter the correct verification code, you will not be able to obtain information again. Because crawlers can use other tools to identify verification codes, websites continue to deepen the difficulty of verification codes, from ordinary pure data research verification codes to mixed verification codes, or sliding verification codes, picture verification codes, etc.

3. Request header detection

Crawlers are not users and have no other features when accessing. Websites can detect whether the other party is a user or a crawler by detecting the request header of the crawler.

4. Cookie detection

The browser will save cookies, so the website will detect cookies to identify whether you are a real user. If the crawler is not well disguised, it will trigger access restrictions.