On Wed, Jul 11, 2018 at 01:34:29PM +0200, Julien Cristau wrote: > On 07/11/2018 10:18 AM, Andreas Tille wrote: > > Hi, > > > > I'm running a daily cron job on host blends.debian.net to gather machine > > readable data from all blends packages. The cron job fetches only the > > following files > > > > debian/changelog > > debian/control > > debian/copyright > > debian/README.Debian > > debian/upstream/edam > > debian/upstream/metadata > > > > (if the latter two exist) from about 2000 repositories. These data are > > consumed in UDD from where they are used in the Blends web sentinel. The > > script which is running can be found in Git[1]. > > > > Unfortunately the cron job seems to stop with > > > > Traceback (most recent call last): > > File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 251, in > > wrapped_f > > return f(*args, **kwargs) > > File "/usr/lib/python3/dist-packages/gitlab/mixins.py", line 48, in get > > server_data = self.gitlab.http_get(path, **kwargs) > > File "/usr/lib/python3/dist-packages/gitlab/__init__.py", line 728, in > > http_get > > streamed=streamed, **kwargs) > > File "/usr/lib/python3/dist-packages/gitlab/__init__.py", line 706, in > > http_request > > response_body=result.content) > > gitlab.exceptions.GitlabHttpError: 429: b'Retry later\n' > > > > During handling of the above exception, another exception occurred: > > > > Traceback (most recent call last): > > File > > "/srv/blends.debian.org/misc/machine_readable/fetch-machine-readable_salsa.py", > > line 106, in <module> > > project = gl.projects.get(pr.attributes['id']) # without this extra get > > repository_tree() fails > > File "/usr/lib/python3/dist-packages/gitlab/exceptions.py", line 253, in > > wrapped_f > > raise error(e.error_message, e.response_code, e.response_body) > > gitlab.exceptions.GitlabGetError: 429: b'Retry later\n' > > > You could probably save yourself some trouble by not polling repos that > have had no activity since you last looked at them.
Or by setting up a webhook[0] so the relevant repos can notify you when there's a commit that changes the files you care about. [0]: https://salsa.debian.org/help/user/project/integrations/webhooks.md Cheers, -- James GPG Key: 4096R/91BF BF4D 6956 BD5D F7B7 2D23 DFE6 91AE 331B A3DB
