Python脚本爬虫如何设置请求头

wen 实用脚本 2026-07-19 1

本文目录导读：

Python脚本爬虫如何设置请求头

使用 requests 库设置请求头
使用 urllib 库设置请求头
使用随机User-Agent（反反爬虫技巧）
完整的请求头示例（模拟真实浏览器）
使用 scrapy 框架设置请求头
重要提示
获取真实浏览器请求头的方法

在Python爬虫中设置请求头（Headers）是为了模拟浏览器行为，避免被服务器识别为爬虫而拒绝访问,以下是几种常见的方法：

使用 `requests` 库设置请求头

基本用法

import requests
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}
response = requests.get('https://example.com', headers=headers)

使用Session对象（推荐用于多次请求）

import requests
session = requests.Session()
session.headers.update({
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
})
# 后续所有使用session的请求都会自动带上这个User-Agent
response1 = session.get('https://example.com/page1')
response2 = session.get('https://example.com/page2')

使用 `urllib` 库设置请求头

from urllib.request import Request, urlopen
url = 'https://example.com'
headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
req = Request(url, headers=headers)
response = urlopen(req)

使用随机User-Agent（反反爬虫技巧）

使用fake-useragent库

pip install fake-useragent

from fake_useragent import UserAgent
import requests
ua = UserAgent()
headers = {
    'User-Agent': ua.random,  # 随机生成User-Agent
    'Accept': '*/*'
}
response = requests.get('https://example.com', headers=headers)

自定义User-Agent列表

import requests
import random
user_agents = [
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36',
    'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0',
    'Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.1 Mobile/15E148 Safari/604.1'
]
headers = {
    'User-Agent': random.choice(user_agents),
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
    'Accept-Encoding': 'gzip, deflate, br',
    'Connection': 'keep-alive',
    'Upgrade-Insecure-Requests': '1'
}
response = requests.get('https://example.com', headers=headers)

完整的请求头示例（模拟真实浏览器）

import requests
headers = {
    # 基本浏览器标识
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36',
    # 接受的内容类型
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8',
    # 接受的语言
    'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',
    # 接受的编码方式
    'Accept-Encoding': 'gzip, deflate, br',
    # 连接类型
    'Connection': 'keep-alive',
    # 缓存控制
    'Cache-Control': 'max-age=0',
    # 来源页面（重要）
    'Referer': 'https://www.google.com/',
    # 是否升级不安全的请求
    'Upgrade-Insecure-Requests': '1',
    # Cookie（如果需要）
    'Cookie': 'your_cookie_here'
}
response = requests.get('https://example.com', headers=headers)

使用 `scrapy` 框架设置请求头

# settings.py
DEFAULT_REQUEST_HEADERS = {
    'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en',
    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
}
# 或者在spider中动态设置
import scrapy
class MySpider(scrapy.Spider):
    name = 'my_spider'
    def start_requests(self):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'
        }
        yield scrapy.Request(url='https://example.com', headers=headers, callback=self.parse)
    def parse(self, response):
        pass