Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-08 Thread Robert Tiberius Johnson
On Sun, 2002-04-07 at 19:36, Adam Heath wrote: > 54 days with of Packages(sid/main/i386) gives 900k of xdeltas. Thanks for the info. I think that keeping 54 days of diffs (or xdeltas) is unnecessary -- most of the benefit is accrued by keeping only 20 days or so. But I need real stats on frequen

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Adam Heath
On 7 Apr 2002, Robert Tiberius Johnson wrote: > On Sun, 2002-04-07 at 11:16, Otto Wyss wrote: > > What amazes me is that nobody is able or willing to provide any figures. > > So I guess no provider of an rsync server is interested in this subject > > and therefore it can't be a big problem. Btw,

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Adam Heath
On 7 Apr 2002, Robert Tiberius Johnson wrote: > On Sun, 2002-04-07 at 11:16, Otto Wyss wrote: > > What amazes me is that nobody is able or willing to provide any figures. > > So I guess no provider of an rsync server is interested in this subject > > and therefore it can't be a big problem. > > He

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Nathan E Norman
On Sun, Apr 07, 2002 at 09:11:27PM -0500, Jeff Licquia wrote: > On Sun, 2002-04-07 at 13:16, Otto Wyss wrote: > > > A large mirror in Australia does provide an rsync server to access debian > > > packages. When redhat 7.0 came out so many people tried to rsync it at the > > > same time, the machine

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Jeff Licquia
On Sun, 2002-04-07 at 13:16, Otto Wyss wrote: > > A large mirror in Australia does provide an rsync server to access debian > > packages. When redhat 7.0 came out so many people tried to rsync it at the > > same time, the machine promptly fell over. > > > What amazes me is that nobody is able or

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Richard Atterer
On Sun, Apr 07, 2002 at 08:16:28PM +0200, Otto Wyss wrote: > What amazes me is that nobody is able or willing to provide any > figures. So I guess no provider of an rsync server is interested in > this subject and therefore it can't be a big problem. It is a problem on cdimage.d.o, which is also f

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Robert Tiberius Johnson
On Sun, 2002-04-07 at 11:16, Otto Wyss wrote: > What amazes me is that nobody is able or willing to provide any figures. > So I guess no provider of an rsync server is interested in this subject > and therefore it can't be a big problem. Here are some experiments, and a mathematical analysis of d

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Otto Wyss
> A large mirror in Australia does provide an rsync server to access debian > packages. When redhat 7.0 came out so many people tried to rsync it at the > same time, the machine promptly fell over. > What amazes me is that nobody is able or willing to provide any figures. So I guess no provider o

Re: Package metadata server

2002-04-07 Thread Sam Couter
Glenn McGrath <[EMAIL PROTECTED]> wrote: > It could probably be done with HTTP, using cgi scripts (i dont know much > about this), that way standard clients can be used to retrieve pieces of > the Packages's file by putting the querry in the url. And then you get the solution which has been mentio

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-07 Thread Martijn van Oosterhout
On Sat, Apr 06, 2002 at 10:19:21AM -0500, Jeff Licquia wrote: > On Sat, 2002-04-06 at 03:13, Otto Wyss wrote: > > Please show use any figures first before you assert this. > > > > I know rsync imposes some load for the computing of the md5sum but > > sendind only the difference outweighs it repeat

Re: Package metadata server

2002-04-06 Thread Adam Heath
On Fri, 5 Apr 2002, Glenn McGrath wrote: > (picked up from http://www.debianplanet.org/article.php?sid=633) > > The scalability problems of the Packages file is a recognised problem that > has been discused many times on this list, i think the following idea > could go a long way to solving it. >

Re: Package metadata server

2002-04-06 Thread Glenn McGrath
On 06 Apr 2002 19:45:29 -0500 "Colin Walters" <[EMAIL PROTECTED]> wrote: > On Fri, 2002-04-05 at 06:37, Glenn McGrath wrote: > > > An idea that i havent heard mentioned here is to create a > > client/server application for specifically handling our metadata, the > > server can be queried by clien

Re: Package metadata server

2002-04-06 Thread Colin Walters
On Fri, 2002-04-05 at 06:37, Glenn McGrath wrote: > An idea that i havent heard mentioned here is to create a client/server > application for specifically handling our metadata, the server can be > queried by clients to send only the required metadata. debian-corba already implements most of the

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-06 Thread Jeff Licquia
On Sat, 2002-04-06 at 03:13, Otto Wyss wrote: > Please show use any figures first before you assert this. > > I know rsync imposes some load for the computing of the md5sum but > sendind only the difference outweighs it repeatedly. It's my understanding that rsync imposes a large computational b

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-06 Thread Otto Wyss
> > Some questions that need to be asked: > > Howmany of our mirrors are rsyncable? > How much load can the servers handle? > How much more load does rsync do than a fast http server like tux? > Please show use any figures first before you assert this. I know rsync imposes some load for the compu

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-05 Thread Rob Bradford
> I think the proposed way of providing diff's for certain "common" > versions is much nicer. Sending a request for > Packages-diff-mymd5sum.gz I like the sound of this way, as you say rsync could cause load problems on the servers. Now someone just needs to hack apt =) Cheers -- Rob 'robster' B

Re: Rsyncable GZIP (was Re: Package metadata server)

2002-04-05 Thread Erich Schubert
> Some questions that need to be asked: > Howmany of our mirrors are rsyncable? How much load can the servers handle? How much more load does rsync do than a fast http server like tux? I think the proposed way of providing diff's for certain "common" versions is much nicer. Sending a request for P

Rsyncable GZIP (was Re: Package metadata server)

2002-04-05 Thread Rob Bradford
On Fri, 2002-04-05 at 12:37, Glenn McGrath wrote: > (picked up from http://www.debianplanet.org/article.php?sid=633) > Hehe. > The current method of checking for updates is to retrieve a new > Packages.gz file and discard the old Packages.gz file. The problem with > this method is that commonly