[issue28756] robotfileparser always uses default Python user-agent

2016-11-20 Thread Ezio Melotti
Changes by Ezio Melotti : -- stage: -> resolved ___ Python tracker ___ ___ Python-bugs-list mailing list Unsubscribe: https://mail.p

[issue28756] robotfileparser always uses default Python user-agent

2016-11-20 Thread Xiang Zhang
Xiang Zhang added the comment: Hi, John. This issue of robotparser has been reported in #15851. I'll close this as duplicate and you can discuss in that thread. -- nosy: +xiang.zhang resolution: -> duplicate status: open -> closed superseder: -> Lib/robotparser.py doesn't accept setti

[issue28756] robotfileparser always uses default Python user-agent

2016-11-20 Thread John Nagle
John Nagle added the comment: (That's from a subclass I wrote. As a change to RobotFileParser, __init__ should start like this.) def __init__(self, url='', user_agent=None): self.user_agent = user_agent# save user agent ... -- _

[issue28756] robotfileparser always uses default Python user-agent

2016-11-20 Thread John Nagle
John Nagle added the comment: Suggest adding a user_agent optional parameter, as shown here: def __init__(self, url='', user_agent=None): urllib.robotparser.RobotFileParser.__init__(self, url) # init parent self.user_agent = user_agent# save user agent

[issue28756] robotfileparser always uses default Python user-agent

2016-11-20 Thread John Nagle
New submission from John Nagle: urllib.robotparser.RobotFileParser always uses the default Python user agent. This agent is now blacklisted by many sites, and it's not possible to read the robots.txt file at all. -- components: Library (Lib) messages: 281314 nosy: nagle priority: norma