Scrapy如何支持URL自定义过滤

以下是一个简单的示例，演示如何实现一个自定义的过滤器中间件来过滤URL：

```python

from scrapy import signals

from scrapy.downloadermiddlewares.httpproxy import HttpProxyMiddleware

class CustomFilterMiddleware(HttpProxyMiddleware):

def __init__(self, settings):

super().__init__(settings)

# 自定义的URL过滤规则

self.allowed_domains = settings.getlist('ALLOWED_DOMAINS')

@classmethod

def from_crawler(cls, crawler):

middleware = super(CustomFilterMiddleware, cls).from_crawler(crawler)

crawler.signals.connect(middleware.spider_opened, signal=signals.spider_opened)

return middleware

def spider_opened(self, spider):

self.allowed_domains.extend(getattr(spider, 'allowed_domains', []))

def process_request(self, request, spider):

if not any(domain in request.url for domain in self.allowed_domains):

self.logger.debug(f"URL {request.url} is not allowed by custom filter")

return None

return None

```

然后，在Scrapy的settings.py文件中添加以下配置：

```python

DOWNLOADER_MIDDLEWARES = {

'myproject.middlewares.CustomFilterMiddleware': 543,

}

ALLOWED_DOMAINS = ['example.com', 'example.org']

```

在这个示例中，CustomFilterMiddleware类继承自Scrapy内置的HttpProxyMiddleware，并在process_request方法中检查请求的URL是否属于ALLOWED_DOMAINS列表中的任何一个域名。如果不属于任何一个域名，则该请求将被过滤掉。

通过实现这样一个自定义的过滤器中间件，您可以灵活地定义URL的过滤规则，以满足您的需求。