Python的实现方式
设计思路和原理本思路就是从目前提供代理服务的网站获取可使用的IP、端口、代理类型信息，并检测可用性，然后对外提供服务。功能模块私信小编007即可获取神秘大礼包一份哦！自行下载
ProxyWebsite - 目标抓取的代理服务网站Crawler - 抓取模块，通过HTTP来抓取定向代理服务网站内容Extrator - 抽取模块，将HTML页面内容，抽取成结构化数据Data - 数据模块，为结构化数据存储服务Validator - 检验模块，检查代理的可用性Service - 对外提供REST API服务核心代码实现示例
ProxyWebsiteclass ProxyWebsite(object): def __init__(self, url, pattern, ip_pos, port_pos): self.url = url self.pattern = pattern self.ip_pos = ip_pos self.port_pos = port_pos Crawlerclass Crawler(object): @staticmethod def get_html(proxy_website): try: rsp = requests.get(proxy_website.url) return (0, rsp.text) except Exception as e: return (-1, e) Extratorclass Extractor(object): @staticmethod def get_data(proxy_website, html): try: pattern = re.compile(proxy_website.pattern, re.M|re.S ) return map(lambda x:(x[proxy_website.ip_pos], x[proxy_website.port_pos]), pattern.findall(html)) except Exception as e: return (-1, e) Dataclass Data(object): def __init__(self, ip, port, http_enable, https_enable): self.ip = ip self.port = port self.http_enable = http_enable self.https_enable = https_enable Validatorclass Validator(object): @staticmethod def get_baidu(ip, port): try: proxies = {http: http://%s:%s %(ip, port), https: http://%s:%s %(ip, port)} http_valid_result = False rsp = requests.get( proxies = proxies, verify=False, timeout=(10, 60)) if rsp.status_code == 200: http_valid_result = True rsp = requests.get( proxies = proxies, verify=False, timeout=(10, 60)) if rsp.status_code == 200: https_valid_result = True return (0, (http_valid_result, https_valid_result)) except Exception as e: return (-1, e) Service 请移步