On 2021-07-14 14:51, Michael Niedermayer wrote:
Signed-off-by: Michael Niedermayer <[email protected]>
---
htdocs/robots.txt | 13 ++++++++++++-
1 file changed, 12 insertions(+), 1 deletion(-)
diff --git a/htdocs/robots.txt b/htdocs/robots.txt
index eb05362..4bbc395 100644
--- a/htdocs/robots.txt
+++ b/htdocs/robots.txt
@@ -1,2 +1,13 @@
User-agent: *
-Disallow:
+Crawl-delay: 10
+Disallow: /gitweb/
+Disallow: /*a=search*
+Disallow: /*/search/*
+Disallow: /*a=blobdiff*
+Disallow: /*/blobdiff/*
+Disallow: /*a=commitdiff*
+Disallow: /*/commitdiff/*
+Disallow: /*a=snapshot*
+Disallow: /*/snapshot/*
+Disallow: /*a=blame*
+Disallow: /*/blame/*
LGTM based on my own personal experiences. But the robots.txt has to be
applied for git.ffmpeg.org as well, and not just ffmpeg.org. Or else
they will just do the same for git.ffmpeg since there are treated
separately.
_______________________________________________
ffmpeg-devel mailing list
[email protected]
https://ffmpeg.org/mailman/listinfo/ffmpeg-devel
To unsubscribe, visit link above, or email
[email protected] with subject "unsubscribe".
