Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 8:59 PM, Junio C Hamano wrote: > Felipe Contreras writes: > How? >>> >>> * if we teach fast-import to optionally not write marks for blobs >>>and trees out, your remote-bzr can take advantage of it, >> >> I already said remote-bzr is irrelevant. *Everybody* benefi

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Felipe Contreras writes: >>> How? >> >> * if we teach fast-import to optionally not write marks for blobs >>and trees out, your remote-bzr can take advantage of it, > > I already said remote-bzr is irrelevant. *Everybody* benefits. Everybody who does _not_ need to look at marks for non-comm

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 3:58 PM, Junio C Hamano wrote: > Felipe Contreras writes: > >>> The story is different on the fast-import side, where we do say we >>> dump the full table and a later run can depend on these marks. >> >> Yes, and gaining nothing but increased disk-space. > > I thought that

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Felipe Contreras writes: >> The story is different on the fast-import side, where we do say we >> dump the full table and a later run can depend on these marks. > > Yes, and gaining nothing but increased disk-space. I thought that the "gaining nothing" has already been refuted by the discussion

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 2:11 PM, Jeff King wrote: > On Mon, May 06, 2013 at 02:02:13PM -0500, Felipe Contreras wrote: > >> > I did a double-take on reading this subject line and first paragraph, >> > thinking "surely fast-export needs to actually output blobs?". >> >> If you think that, then you ar

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 11:20 AM, Jeff King wrote: > On Mon, May 06, 2013 at 08:08:45AM -0700, Junio C Hamano wrote: > >> > I'm also not sure why your claim "we don't care about blobs" is true, >> > because naively we would want future runs of fast-export to avoid having >> > to write out the whole

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 02:02:13PM -0500, Felipe Contreras wrote: > > I did a double-take on reading this subject line and first paragraph, > > thinking "surely fast-export needs to actually output blobs?". > > If you think that, then you are not familiar with the code. > > --export-marks=:: > [

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 10:08 AM, Junio C Hamano wrote: > Jeff King writes: > >> On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: >> >>> We don't care about blobs, or any object other than commits, but in >>> order to find the type of object, we are parsing the whole thing, which

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Felipe Contreras
On Mon, May 6, 2013 at 7:31 AM, Jeff King wrote: > On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: > >> We don't care about blobs, or any object other than commits, but in >> order to find the type of object, we are parsing the whole thing, which >> is slow, specially in big repo

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 01:19:35PM -0400, Jeff King wrote: > > Here is what I tentatively queued. > > [...] > > Yeah, that is much for to understand (to me, at least). Ugh. That was supposed to be "much easier to understand". Perhaps I will learn to type one day. -Peff -- To unsubscribe from th

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 10:17:41AM -0700, Junio C Hamano wrote: > Jeff King writes: > > > So yes, I think this is an obviously correct optimization. Thanks for > > clarifying, and sorry to be so slow. > > No need to be sorry. It just shows that the log message could have > been more helpful. >

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Jeff King writes: > So yes, I think this is an obviously correct optimization. Thanks for > clarifying, and sorry to be so slow. No need to be sorry. It just shows that the log message could have been more helpful. Here is what I tentatively queued. commit 83582e91d22c66413b291d4d6d45bbeafddc

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 09:32:41AM -0700, Junio C Hamano wrote: > > OK. If the argument is "we do not write them, so do not bother reading > > them back in", I think that is reasonable. > > The way I read builtin/fast-export.c::import_marks() is that it is > more like "we do not write them, and w

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Jeff King writes: > On Mon, May 06, 2013 at 08:08:45AM -0700, Junio C Hamano wrote: > >> > I'm also not sure why your claim "we don't care about blobs" is true, >> > because naively we would want future runs of fast-export to avoid having >> > to write out the whole blob content when mentioning t

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Mon, May 06, 2013 at 08:08:45AM -0700, Junio C Hamano wrote: > > I'm also not sure why your claim "we don't care about blobs" is true, > > because naively we would want future runs of fast-export to avoid having > > to write out the whole blob content when mentioning the blob again. > > The ex

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Junio C Hamano writes: > By discarding marks on blobs, we may be robbing some optimization > possibilities, and by discarding marks on tags, we may be robbing > some features, from users of fast-export; we might want to add an > option "--use-object-marks={blob,commit,tag}" or something to both >

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Junio C Hamano
Jeff King writes: > On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: > >> We don't care about blobs, or any object other than commits, but in >> order to find the type of object, we are parsing the whole thing, which >> is slow, specially in big repositories with lots of big file

Re: [PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-06 Thread Jeff King
On Sun, May 05, 2013 at 05:38:53PM -0500, Felipe Contreras wrote: > We don't care about blobs, or any object other than commits, but in > order to find the type of object, we are parsing the whole thing, which > is slow, specially in big repositories with lots of big files. I did a double-take on

[PATCH v2 2/3] fast-export: improve speed by skipping blobs

2013-05-05 Thread Felipe Contreras
We don't care about blobs, or any object other than commits, but in order to find the type of object, we are parsing the whole thing, which is slow, specially in big repositories with lots of big files. There's no need for that, we can query the object information with sha1_object_info(); Before