Re: Please Take Attention To This BUG.

[林作健]

I have found the cause of this bug.
In 5.3, the function strip_typedefs only use
result = cp_build_type_attribute_variant (result, TYPE_ATTRIBUTES (t));
but in 6.3 remove_attributes prediction get invovled:
  if (TYPE_ATTRIBUTES (t))
{
  if (remove_attributes)
result = apply_identity_attributes (result, TYPE_ATTRIBUTES (t),
   remove_attributes);
  else
result = cp_build_type_attribute_variant (result, TYPE_ATTRIBUTES (t));
}
and unfortunately the pcs attribute is declared as
  { "pcs",  1, 1, false, true,  true,  arm_handle_pcs_attribute,
false },
which affects_type_identity field is set to false, cause
apply_identity_attributes
drop this attribute.
--
Lin Zuojian

2017-06-07 9:39 GMT+08:00 林作健 :
> Hi,
> Please help me with the following bug. It's a blocker.
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=80986
> Thanks.
> --
> Lin Zuojian

Re: Please Take Attention To This BUG.

[Jonathan Wakely]

On 7 June 2017 at 08:38, 林作健 wrote:
> I have found the cause of this bug.
> In 5.3, the function strip_typedefs only use
> result = cp_build_type_attribute_variant (result, TYPE_ATTRIBUTES (t));
> but in 6.3 remove_attributes prediction get invovled:
>   if (TYPE_ATTRIBUTES (t))
> {
>   if (remove_attributes)
> result = apply_identity_attributes (result, TYPE_ATTRIBUTES (t),
>remove_attributes);
>   else
> result = cp_build_type_attribute_variant (result, TYPE_ATTRIBUTES (t));
> }
> and unfortunately the pcs attribute is declared as
>   { "pcs",  1, 1, false, true,  true,  arm_handle_pcs_attribute,
> false },
> which affects_type_identity field is set to false, cause
> apply_identity_attributes
> drop this attribute.

Please put this information in bugzilla where it belongs, thanks.

Also if it is a regression since 5.3 please mention that in the comments too.

"Uninitialized array" warnings by c++ with -O2

[Kirill Yu Berezin]

Hello!

I have a code snippet (actually it is a part of larger project):


#include 

#define UDP_PROTO_NUMBER 17

uint32_t calc_16bit_checksum_part(uint8_t* buf, int len, uint32_t ret) {
uint16_t *ptr = reinterpret_cast(buf);
int i;

for( i = 0; i < (len / 2); ++i) {
ret = ret + ptr[i];
}
if ( len%2) {
buf[len] = 0;
ret += ptr[len/2];
}

return ret;
}

uint32_t calc_ch_udp_pseudo(uint32_t src, uint32_t dst, uint16_t len) {
struct udp_pseudo {
uint32_t src;
uint32_t dst;
 uint8_t  z;
 uint8_t  proto;
 uint16_t len;
};
uint8_t tbuf[sizeof(udp_pseudo)];
udp_pseudo* tmp=reinterpret_cast(tbuf);

tmp->src = src;
tmp->dst = dst;
tmp->z = 0;
tmp->proto = UDP_PROTO_NUMBER;
tmp->len = len;

return calc_16bit_checksum_part(tbuf, sizeof(tbuf), 0);
}

int main() {
return calc_ch_udp_pseudo(0x01020304, 0x12131415, 320);
}


When I try to compile it with -O2 and higher I got the following

#c++ -DHAVE_CONFIG_H -I. -I.././include -g -O3 -Wall -I../include/  
-o snippet snippet.cc
snippet.cc: In function ‘uint32_t calc_ch_udp_pseudo(uint32_t, uint32_t, 
uint16_t)’:
snippet.cc:10:34: warning: ‘tbuf’ is used uninitialized in this function 
[-Wuninitialized]

 ret = ret + ptr[i];
 ~^
snippet.cc:10:34: warning: ‘*((void*)& tbuf +2)’ is used uninitialized 
in this function [-Wuninitialized]
snippet.cc:10:34: warning: ‘*((void*)& tbuf +4)’ is used uninitialized 
in this function [-Wuninitialized]
snippet.cc:10:34: warning: ‘*((void*)& tbuf +6)’ is used uninitialized 
in this function [-Wuninitialized]

snippet.cc: In function ‘int main()’:
snippet.cc:10:34: warning: ‘tbuf’ is used uninitialized in this function 
[-Wuninitialized]

 ret = ret + ptr[i];
 ~^
snippet.cc:10:34: warning: ‘*((void*)& tbuf +2)’ is used uninitialized 
in this function [-Wuninitialized]
snippet.cc:10:34: warning: ‘*((void*)& tbuf +4)’ is used uninitialized 
in this function [-Wuninitialized]
snippet.cc:10:34: warning: ‘*((void*)& tbuf +6)’ is used uninitialized 
in this function [-Wuninitialized]


Compiler :

#gcc --versio
gcc (GCC) 7.1.1 20170528
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

My question is. Is this an expected behaviour or I must report a bug ?


Thank you!

Kirill

Re: "Uninitialized array" warnings by c++ with -O2

[Jakub Jelinek]

On Wed, Jun 07, 2017 at 12:15:54PM +0300, Kirill Yu Berezin wrote:
> Hello!
> 
> I have a code snippet (actually it is a part of larger project):

That snippet invokes undefined behavior at runtime (violates C++ aliasing 
rules),
so just fix it, rather than bother with bugreports.  E.g. look for 
-fstrict-aliasing
in GCC documentation, or read the C++ standard.  With -fno-strict-aliasing,
which is a workaround for broken code you won't get any warnings about
uninitialized uses.

Jakub

Re: "Uninitialized array" warnings by c++ with -O2

[Andrew Haley]

On 07/06/17 10:15, Kirill Yu Berezin wrote:
> My question is. Is this an expected behaviour or I must report a bug ?

It's not a bug: your code displays undefined behaviour: you're casting
a pointer to struct udp_pseudo fields to an array of uint16_t.  This
is never well-defined in C++, but if you really want to do this kind
of thing, use -fno-strict-aliasing.

See also
http://www.microhowto.info/howto/calculate_an_internet_protocol_checksum_in_c.html

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: "Uninitialized array" warnings by c++ with -O2

[David Brown]

On 07/06/17 11:33, Andrew Haley wrote:
> On 07/06/17 10:15, Kirill Yu Berezin wrote:
>> My question is. Is this an expected behaviour or I must report a bug ?
> 
> It's not a bug: your code displays undefined behaviour: you're casting
> a pointer to struct udp_pseudo fields to an array of uint16_t.  This
> is never well-defined in C++, but if you really want to do this kind
> of thing, use -fno-strict-aliasing.
> 
> See also
> http://www.microhowto.info/howto/calculate_an_internet_protocol_checksum_in_c.html
> 

Two other ideas spring to mind.

Use the "may_alias" attribute on the udp_pseudo struct:

struct udp_pseudo {
uint32_t src;
uint32_t dst;
 uint8_t  z;
 uint8_t  proto;
 uint16_t len;
} __attribute__((may_alias));

This tells the compiler that this type may alias other types (such as
the uint8_t array).  I think you will also need a "may_alias" attribute
on the uint16_t pointer in calc_16bit_checksum_part as well.  You can
imagine "may_alias" attribute as giving you the effect of
-fno-strict-aliasing, but only for particular types.


The other is to make a union type that has your struct as a field, and
your uint8_t array as another field.  If you access everything through
the union, the compiler knows they alias.  Again, you would want to
include the uint16_t pointer in this solution in order to get all the
aliasing correct.

How to create new functions with a gcc plugin?

[Benxi Liu]

Hi all,

I'm using a gcc plugin to do some instrument during compilation.

Instrument in functions is simple. But how can I create new functions,
and append it to executables? I want to instrument in this way: to
create new functions,  add my codes into them, then instrument some
calls to them.

foo:
call to new_function;  //instrument a call

new_function: //created function
instrument  codes here

I think the most difficult part is to create functions. If it's
possible to do so, can I create functions at any phase during
compilation(with a gimple, or a rtl pass)?

 Any tips?

Re: "Uninitialized array" warnings by c++ with -O2

[K]

That snippet invokes undefined behavior at runtime (violates C++ aliasing 
rules),
so just fix it, rather than bother with bugreports.  E.g. look for 
-fstrict-aliasing
in GCC documentation, or read the C++ standard.  With -fno-strict-aliasing,
which is a workaround for broken code you won't get any warnings about
uninitialized uses.

Jakub



Yes this code has problems with aliasing. But anyway warning messages 
are extremely misleading.
And I found that that a version which I beleve mustn't have aliasing 
problems still generates same warnings.


The code:

 static uint32_t calc_16bit_checksum_part(uint8_t* buf, int len, 
uint32_t ret) {

struct ui16{
uint16_t d;
};
ui16 *ptr = (ui16*)buf;
for( int i = 0; i < (len / 2); ++i) {
ret += ptr[i].d;
}
if ( len%2) {
ret += buf[len-1];
}

return ret;
}

static uint32_t calc_ch_udp_pseudo(uint32_t src, uint32_t dst, uint16_t 
len) {

struct udp_pseudo {
uint32_t src;
uint32_t dst;
uint8_t  z;
uint8_t  proto;
uint16_t len;
} tmp;

tmp.src = src;
tmp.dst = dst;
tmp.z = 0;
tmp.proto = UDP_PROTO_NUMBER;
tmp.len = len;

auto ret = calc_16bit_checksum_part((uint8_t*)&tmp, 
sizeof(tmp), 0);

return ret;
}

and messages

static uint16_t calc_16bit_checksum(uint8_t* buf, int len) {
 ^~~
snippet.cc: In function ‘int main()’:
snippet.cc:66:31: warning: ‘tmp.calc_16bit_checksum_part(uint8_t*, int, 
uint32_t)::ui16::d’ is used uninitialized in this function [-Wuninitialized]

 ret += ptr[i].d;
~~~^
snippet.cc:66:31: warning: ‘*((void*)(& 
tmp)+2).calc_16bit_checksum_part(uint8_t*, int, uint32_t)::ui16::d’ is 
used uninitialized in this function [-Wuninitialized]
snippet.cc:66:31: warning: ‘*((void*)(& 
tmp)+4).calc_16bit_checksum_part(uint8_t*, int, uint32_t)::ui16::d’ is 
used uninitialized in this function [-Wuninitialized]
snippet.cc:66:31: warning: ‘*((void*)(& 
tmp)+6).calc_16bit_checksum_part(uint8_t*, int, uint32_t)::ui16::d’ is 
used uninitialized in this function [-Wuninitialized]
snippet.cc:66:31: warning: ‘*((void*)(& 
tmp)+8).calc_16bit_checksum_part(uint8_t*, int, uint32_t)::ui16::d’ is 
used uninitialized in this function [-Wuninitialized]
snippet.cc:66:31: warning: ‘*((void*)(& 
tmp)+10).calc_16bit_checksum_part(uint8_t*, int, uint32_t)::ui16::d’ is 
used uninitialized in this function [-Wuninitialized]

[


ui16 is an aggregate type and uint8_t is unsigned char, so there will be 
no undefined behaviour.


Kirill.

Re: "Uninitialized array" warnings by c++ with -O2

[Andrew Haley]

On 07/06/17 14:45, K wrote:
> And I found that that a version which I beleve mustn't have aliasing 
> problems still generates same warnings.

It still has aliasing problems: you can't make them magically go away
by using an intermediate uint8_t*.

You're doing this:

 struct udp_pseudo {
 uint32_t src;
 uint32_t dst;
 uint8_t  z;
 uint8_t  proto;
 uint16_t len;
 } tmp;
...

 auto ret = calc_16bit_checksum_part((uint8_t*)&tmp, sizeof(tmp), 0);

 static uint32_t calc_16bit_checksum_part(uint8_t* buf, int len,  
uint32_t ret) {
 struct ui16{
 uint16_t d;
 };
 ui16 *ptr = (ui16*)buf;

There's no need for any of this messing about with pointer casts, as has
been explained.

-- 
Andrew Haley
Java Platform Lead Engineer
Red Hat UK Ltd. 
EAC8 43EB D3EF DB98 CC77 2FAD A5CD 6035 332F A671

Re: "Uninitialized array" warnings by c++ with -O2

[K]

On 06/07/2017 04:56 PM, Andrew Haley wrote:

On 07/06/17 14:45, K wrote:

And I found that that a version which I beleve mustn't have aliasing
problems still generates same warnings.

It still has aliasing problems: you can't make them magically go away
by using an intermediate uint8_t*.

You're doing this:

  struct udp_pseudo {
  uint32_t src;
  uint32_t dst;
  uint8_t  z;
  uint8_t  proto;
  uint16_t len;
  } tmp;
...

  auto ret = calc_16bit_checksum_part((uint8_t*)&tmp, sizeof(tmp), 0);

  static uint32_t calc_16bit_checksum_part(uint8_t* buf, int len,  
uint32_t ret) {
  struct ui16{
  uint16_t d;
  };
  ui16 *ptr = (ui16*)buf;

There's no need for any of this messing about with pointer casts, as has
been explained.



Sorry, but I still can't get the idea. Cast from udp_pseudo to uint8_t 
doesn't have an aliasing problem (std 8.8) and a cast from uint8_t to 
ui16 still doesn't have an aliasing problem (std  8.6), or may be I 
missed something?



Kirill.

Re: "Uninitialized array" warnings by c++ with -O2

[Jakub Jelinek]

On Wed, Jun 07, 2017 at 05:16:44PM +0300, K wrote:
> >   ui16 *ptr = (ui16*)buf;
> > 
> > There's no need for any of this messing about with pointer casts, as has
> > been explained.
> > 
> 
> Sorry, but I still can't get the idea. Cast from udp_pseudo to uint8_t
> doesn't have an aliasing problem (std 8.8) and a cast from uint8_t to ui16
> still doesn't have an aliasing problem (std  8.6), or may be I missed
> something?

This is not the right list to learn C or C++, so this should be moved
to gcc-help.  The casts themselves are not the points of UB (unless the
pointer is misaligned for the pointee type you cast to), the problem is that
you have an object (tmp and its fields) that doesn't have unsigned short as
its dynamic type, nor cv-qualified version thereof, nor similar type, etc.
(see [basic.lval]/8 for details) and you are accessing that object
using a glvalue with unsigned short type.  And that invokes the UB:
"If a program attempts to access the stored value of an object through a glvalue
of other than one of the following types the behavior is undefined:"
... long list of what is allowed.

Accessing it e.g. using glvalue of char, unsigned char or std::byte would be
fine, or say using ui16 __attribute__((may_alias)) type (by having
  typedef ui16 ui16a __attribute__((may_alias));
  ui16a *ptr = (ui16a*)buf;
  ... *ptr ...
or similar, or union that includes the type of tmp and ui16 as type of its
members), etc.

Jakub

Re: libatomic IFUNC question (arm & libat_have_strexbhd)

[Jim Wilson]

On 06/06/2017 09:01 AM, Steve Ellcey wrote:

So the question remains, where is libat_have_strexbhd set?  As near as
I can tell it isn't set, which would make the libatomic IFUNC pointless
on arm.


libat_have_strexbhd isn't set anywhere.  It looks like this was a 
prototype that was never fully fleshed out.  See for instance the 
libatomic/config/x86/init.c file.  Finishing this means someone has to 
figure out how to use the arm cpuid equivalent to set the two variables 
appropriately.


Jim

Re: libatomic IFUNC question (arm & libat_have_strexbhd)

[Richard Henderson]

On 06/05/2017 10:50 PM, Florian Weimer wrote:

* Steve Ellcey:


I have a question about the use of IFUNCs in libatomic.  I was looking at the
arm implementation and in gcc/libatomic/config/linux/arm/host-config.h I see:

extern bool libat_have_strexbhd HIDDEN;
# define IFUNC_COND_1   libat_have_strexbhd

I also see that gcc/libatomic/config/linux/arm/init.c has:

bool libat_have_strexbhd;
static void __attribute__((constructor))
init_cpu_revision (void)
{
}

What I don't see is any place that libat_have_strexbhd would ever get
set.  What am I missing here?  init_cpu_revision is going to get called
when libatomic is first loaded since it is a constructor but it doesn't
seem to do anything and it isn't going to set libat_have_strexbhd as far
as I can see.


Setting the variable in the constructor wouldn't influence IFUNC
resolver behavior because those can run before ELF constructors
(even with lazy binding).


With lazy binding, the constructors of libraries should run in graph dependency 
order, which means this constructor should run before any users.


Without lazy binding, you're right that ifunc resolvers can run earlier, and 
this would be largely useless.  Suggestions for a better organization welcome.


That said, this clearly is lacking some actual guts.

Once upon a time, we didn't really have a way to get at either AT_HWCAP or 
AT_PLATFORM from here.  Since then, glibc has grown the getauxval function.


As far as I can see, there's no AT_HWCAP bit that can help.  We can get "v6" or 
"v7" from AT_PLATFORM, from which we can infer strexd.


We don't seem to be able to infer v6t2, but I suspect we don't *really* need 
that here, since the compiler *can* generate sub-word atomic accesses from 
strex and we don't necessarily require strex[bh].


That said, real distributions targeting armhf are really targeting armv7, and 
*all* of this IFUNC stuff is worthless overhead for that case.  So it would be 
better to avoid it all.



r~

Re: libatomic IFUNC question (arm & libat_have_strexbhd)

[Steve Ellcey]

On Wed, 2017-06-07 at 12:21 -0700, Richard Henderson wrote:

> > Setting the variable in the constructor wouldn't influence IFUNC
> > resolver behavior because those can run before ELF constructors
> > (even with lazy binding).
> With lazy binding, the constructors of libraries should run in graph 
> dependency 
> order, which means this constructor should run before any users.
> 
> Without lazy binding, you're right that ifunc resolvers can run earlier, and 
> this would be largely useless.  Suggestions for a better organization
> welcome.

Would defining  __builtin_cpu_init, __builtin_cpu_is,
and __builtin_cpu_supports for ARM help with this?  X86 seems to have
some special code to call __builtin_cpu_init
from dispatch_function_versions.  Is that early enough?  Then
__builtin_cpu_is or __builtin_cpu_supports could be used in the IFUNC
resolvers instead of checking the libat_have_strexbhd variable.

Steve Ellcey
sell...@cavium.com

Steering committee, please, consider using lzip instead of xz

[Antonio Diaz Diaz]

Dear GCC steering committee,

This has been recently asked in this list[1], but in case you have 
missed it because of a subject line not explicit enough, I would like to 
appeal to you directly.


[1] http://gcc.gnu.org/ml/gcc/2017-06/msg9.html

Since 2017-05-24 weekly snapshots use xz compression instead of bzip2. I 
suppose this means that release tarballs will also use xz at some point.


If this is the case, I politely request you to consider using lzip 
instead of xz. I have spent a lot of time during the last 9 years 
developing lzip and studying the xz format, and based on this experience 
I consider that lzip is a better choice than xz, now and in the long term.


I have been developing software since the early 80s, and I am a GNU 
maintainer since 2003. You are all experienced developers. All I ask is 
that you read carefully the following references and then consider lzip 
and xz based on their technical merits.


http://www.nongnu.org/lzip/xz_inadequate.html
http://www.nongnu.org/lzip/lzip_benchmark.html#xz1

Also note that 'lzip -9' produces a tarball a 1% smaller than xz, in 
spite of lzip using half the RAM to compress and requiring half the RAM 
to decompress than xz.


-rw-r--r-- 1 58765134 2017-06-07 09:13 gcc-8-20170604.tar.lz
-rw-r--r-- 1 59367680 2017-06-07 09:13 gcc-8-20170604.tar.xz


Thanks and regards,
Antonio.

Re: libatomic IFUNC question (arm & libat_have_strexbhd)

[Florian Weimer]

* Richard Henderson:

> On 06/05/2017 10:50 PM, Florian Weimer wrote:
>> * Steve Ellcey:
>> 
>>> I have a question about the use of IFUNCs in libatomic.  I was looking at 
>>> the
>>> arm implementation and in gcc/libatomic/config/linux/arm/host-config.h I 
>>> see:
>>>
>>> extern bool libat_have_strexbhd HIDDEN;
>>> # define IFUNC_COND_1   libat_have_strexbhd
>>>
>>> I also see that gcc/libatomic/config/linux/arm/init.c has:
>>>
>>> bool libat_have_strexbhd;
>>> static void __attribute__((constructor))
>>> init_cpu_revision (void)
>>> {
>>> }
>>>
>>> What I don't see is any place that libat_have_strexbhd would ever get
>>> set.  What am I missing here?  init_cpu_revision is going to get called
>>> when libatomic is first loaded since it is a constructor but it doesn't
>>> seem to do anything and it isn't going to set libat_have_strexbhd as far
>>> as I can see.
>> 
>> Setting the variable in the constructor wouldn't influence IFUNC
>> resolver behavior because those can run before ELF constructors
>> (even with lazy binding).
>
> With lazy binding, the constructors of libraries should run in graph
> dependency order, which means this constructor should run before any
> users.

Except when another shared object uses the function from its own ELF
constructor, and the libatomic constructor has not yet run.  This
should not happen if the object has a proper DT_NEEDED reference on
libatomic, but strange things happen once LD_PRELOAD is involved.

(There might also exist relocations which cannot be lazily bound, even
if the object wasn't linked with BIND_NOW.)

> Once upon a time, we didn't really have a way to get at either
> AT_HWCAP or AT_PLATFORM from here.  Since then, glibc has grown the
> getauxval function.

getauxval does not work from IFUNC handlers because it itself requires
a relocation (which may not have been processed at this point), and it
relies on global data which might not have been initialized.

I think we would have to pass ways to reach the relevant data as IFUNC
resolver arguments, and in addition fix the relocation processing in
ld.so to delay IFUNC resolution after all other relocation processing.
(I have an unfinished patch for the latter; I posted it to libc-alpha
a while back.)

Re: libatomic IFUNC question (arm & libat_have_strexbhd)

[Richard Henderson]

On 06/07/2017 01:31 PM, Florian Weimer wrote:

With lazy binding, the constructors of libraries should run in graph
dependency order, which means this constructor should run before any
users.


Except when another shared object uses the function from its own ELF
constructor, and the libatomic constructor has not yet run.  This
should not happen if the object has a proper DT_NEEDED reference on
libatomic, but strange things happen once LD_PRELOAD is involved.


True, but that's relatively rare.


(There might also exist relocations which cannot be lazily bound, even
if the object wasn't linked with BIND_NOW.)


Once upon a time, we didn't really have a way to get at either
AT_HWCAP or AT_PLATFORM from here.  Since then, glibc has grown the
getauxval function.


getauxval does not work from IFUNC handlers because it itself requires
a relocation (which may not have been processed at this point), and it
relies on global data which might not have been initialized.


Which is why I didn't actually call getauxval from the IFUNC resolver, but rely 
on a data member being initialized.


And when the data member isn't initialized, for the various ordering issues 
described above, the only consequence is that we use a less efficient (but not 
incorrect) mechanism for the atomic.


It's possible we could do slightly better vs ordering with DT_PREINIT, but far 
more common is DF_BIND_NOW, and no form of constructor helps with that.



I think we would have to pass ways to reach the relevant data as IFUNC
resolver arguments, and in addition fix the relocation processing in
ld.so to delay IFUNC resolution after all other relocation processing.
(I have an unfinished patch for the latter; I posted it to libc-alpha
a while back.)


That would be an improvement, yes.


r~

Re: Steering committee, please, consider using lzip instead of xz

[Matthias Klose]

On 07.06.2017 13:25, Antonio Diaz Diaz wrote:
> Dear GCC steering committee,
> 
> This has been recently asked in this list[1], but in case you have missed it
> because of a subject line not explicit enough, I would like to appeal to you
> directly.
> 
> [1] http://gcc.gnu.org/ml/gcc/2017-06/msg9.html
> 
> Since 2017-05-24 weekly snapshots use xz compression instead of bzip2. I 
> suppose
> this means that release tarballs will also use xz at some point.
> 
> If this is the case, I politely request you to consider using lzip instead of
> xz. I have spent a lot of time during the last 9 years developing lzip and
> studying the xz format, and based on this experience I consider that lzip is a
> better choice than xz, now and in the long term.
> 
> I have been developing software since the early 80s, and I am a GNU maintainer
> since 2003. You are all experienced developers. All I ask is that you read
> carefully the following references and then consider lzip and xz based on 
> their
> technical merits.
> 
> http://www.nongnu.org/lzip/xz_inadequate.html
> http://www.nongnu.org/lzip/lzip_benchmark.html#xz1
> 
> Also note that 'lzip -9' produces a tarball a 1% smaller than xz, in spite of
> lzip using half the RAM to compress and requiring half the RAM to decompress
> than xz.
> 
> -rw-r--r-- 1 58765134 2017-06-07 09:13 gcc-8-20170604.tar.lz
> -rw-r--r-- 1 59367680 2017-06-07 09:13 gcc-8-20170604.tar.xz

I proposed and implemented the change to use xz instead of bzip2 because of the
space savings compared to bzip2.  I'm not commenting on the "inadequateness" of
xz, but maybe it would better help lzip to address some project issues and
promoting it as an alternative rather than appealing to the GCC steering 
committee.

 - lzip is not a GNU project (afaics), same as for xz.
 - lzip doesn't have a public VCS.
 - lzip doesn't have a documented API, doesn't build as a library,
   and I can't find language bindings for lzip.
 - lzip isn't (yet) used for software distributions, while xz is (and afaics
   xz is used for GNU projects in addition to gz).

Matthias

gcc-6-20170607 is now available

[gccadmin]

Snapshot gcc-6-20170607 is now available on
  ftp://gcc.gnu.org/pub/gcc/snapshots/6-20170607/
and on various mirrors, see http://gcc.gnu.org/mirrors.html for details.

This snapshot has been generated from the GCC 6 SVN branch
with the following options: svn://gcc.gnu.org/svn/gcc/branches/gcc-6-branch 
revision 248998

You'll find:

 gcc-6-20170607.tar.xzComplete GCC

  SHA256=5084d593b8ffc7924e24ebe4f53d99e0c4a1ec4aea8a369e89bb94fdbba2eb76
  SHA1=92797d037faee2a5f42a15e361e483af3bd97946

Diffs from 6-20170531 are available in the diffs/ subdirectory.

When a particular snapshot is ready for public consumption the LATEST-6
link is updated and a message is sent to the gcc list.  Please do not use
a snapshot before it has been announced that way.

Re: Steering committee, please, consider using lzip instead of xz

[Jeff Law]

On 06/07/2017 02:25 PM, Antonio Diaz Diaz wrote:
> Dear GCC steering committee,
> 
> This has been recently asked in this list[1], but in case you have
> missed it because of a subject line not explicit enough, I would like to
> appeal to you directly.
> 
> [1] http://gcc.gnu.org/ml/gcc/2017-06/msg9.html
> 
> Since 2017-05-24 weekly snapshots use xz compression instead of bzip2. I
> suppose this means that release tarballs will also use xz at some point.
> 
> If this is the case, I politely request you to consider using lzip
> instead of xz. I have spent a lot of time during the last 9 years
> developing lzip and studying the xz format, and based on this experience
> I consider that lzip is a better choice than xz, now and in the long term.
> 
> I have been developing software since the early 80s, and I am a GNU
> maintainer since 2003. You are all experienced developers. All I ask is
> that you read carefully the following references and then consider lzip
> and xz based on their technical merits.
> 
> http://www.nongnu.org/lzip/xz_inadequate.html
> http://www.nongnu.org/lzip/lzip_benchmark.html#xz1
> 
> Also note that 'lzip -9' produces a tarball a 1% smaller than xz, in
> spite of lzip using half the RAM to compress and requiring half the RAM
> to decompress than xz.
> 
> -rw-r--r-- 1 58765134 2017-06-07 09:13 gcc-8-20170604.tar.lz
> -rw-r--r-- 1 59367680 2017-06-07 09:13 gcc-8-20170604.tar.xz
Given the breadth that xz already has in the distribution space, I'd
have a hard time supporting moving to lz over xz.

We've got far more important items to tackle than this.  But if you want
me to bring it up formally with the SC I can.

jeff

ps.  And just to be clear, I actually don't like xz and I'm always
annoyed when I run into something delivered in xz format.  But xz
support at the distro level is pretty ubiquitous at this point.

Re: Steering committee, please, consider using lzip instead of xz

[Matias Fonzo]

On Wed, 7 Jun 2017 15:25:26 -0700
Matthias Klose  wrote:

> On 07.06.2017 13:25, Antonio Diaz Diaz wrote:
> > Dear GCC steering committee,
> > 
> > This has been recently asked in this list[1], but in case you have
> > missed it because of a subject line not explicit enough, I would
> > like to appeal to you directly.
> > 
> > [1] http://gcc.gnu.org/ml/gcc/2017-06/msg9.html
> > 
> > Since 2017-05-24 weekly snapshots use xz compression instead of
> > bzip2. I suppose this means that release tarballs will also use xz
> > at some point.
> > 
> > If this is the case, I politely request you to consider using lzip
> > instead of xz. I have spent a lot of time during the last 9 years
> > developing lzip and studying the xz format, and based on this
> > experience I consider that lzip is a better choice than xz, now and
> > in the long term.
> > 
> > I have been developing software since the early 80s, and I am a GNU
> > maintainer since 2003. You are all experienced developers. All I
> > ask is that you read carefully the following references and then
> > consider lzip and xz based on their technical merits.
> > 
> > http://www.nongnu.org/lzip/xz_inadequate.html
> > http://www.nongnu.org/lzip/lzip_benchmark.html#xz1
> > 
> > Also note that 'lzip -9' produces a tarball a 1% smaller than xz,
> > in spite of lzip using half the RAM to compress and requiring half
> > the RAM to decompress than xz.
> > 
> > -rw-r--r-- 1 58765134 2017-06-07 09:13 gcc-8-20170604.tar.lz
> > -rw-r--r-- 1 59367680 2017-06-07 09:13 gcc-8-20170604.tar.xz  
> 
> I proposed and implemented the change to use xz instead of bzip2
> because of the space savings compared to bzip2.  I'm not commenting
> on the "inadequateness" of xz, but maybe it would better help lzip to
> address some project issues and promoting it as an alternative rather
> than appealing to the GCC steering committee.
> 
>  - lzip is not a GNU project (afaics), same as for xz.

The developer is part of the GNU project and the lzip project lives on
Savannah (nongnu).  The license of lzip is under the terms of the GPL.
While the author of xz is not part of the GNU project, the project
lives on tukaani.org, the license is public domain.

>  - lzip doesn't have a public VCS.

The code, license and the releases are a reflection of the "public".
A VCS can be a matter of personal choice, working preference.

>  - lzip doesn't have a documented API, doesn't build as a library,

Under "Related projects" (lzip.nongnu.org):

Lzlib[1] - A compression library for the lzip file format, written in
C. 

[1] http://lzip.nongnu.org/lzlib.html

>and I can't find language bindings for lzip.

Well.. this could be done.  lzip is *less* complex than xz.  See[2]:

http://lzip.nongnu.org/lzd.html

>  - lzip isn't (yet) used for software distributions, while xz is (and
> afaics xz is used for GNU projects in addition to gz).
> 

Mainstream distributions provides lzip: Fedora, Debian (and
derivatives), Slackware, ...



pgp8rwqDJDdao.pgp
Description: OpenPGP digital signature

Re: Steering committee, please, consider using lzip instead of xz

[Antonio Diaz Diaz]

Matthias Klose wrote:

I'm not commenting on the "inadequateness" of xz, but maybe it would
better help lzip to address some project issues and promoting it as
an alternative rather than appealing to the GCC steering committee.


It is not my intention to "help lzip", but to use lzip as a mean to help 
others avoid the "inadequateness" of xz. :-)




  - lzip is not a GNU project (afaics), same as for xz.


Lzip is GPL, xz is not.


  - lzip doesn't have a public VCS.


Why is this important?


  - lzip doesn't have a documented API, doesn't build as a library,
and I can't find language bindings for lzip.


- The lzip library (lzlib) is at http://www.nongnu.org/lzip/lzlib.html
- The lzlib API is documented at 
http://www.nongnu.org/lzip/manual/lzlib_manual.html
- Libarchive http://www.libarchive.org/ is a library that automatically 
handles lzip archives.
- There are language bindings for libarchive and I think that there are 
also some language bindings for lzip, but I don't have links for them.


BTW, lzip offers a parallel compressor (plzip) 
http://www.nongnu.org/lzip/plzip.html that uses lzlib and features 
multi-threaded decompression since 2010. Xz-utils still does not 
implement multi-threaded decompression because LZMA2 makes it difficult.



  - lzip isn't (yet) used for software distributions, while xz is (and afaics
xz is used for GNU projects in addition to gz).


Lzip is used for software distribution of GNU projects, and contrarily 
to xz, it has never caused problems for excessive memory requirements or 
wrong type of CRC.


http://gmplib.org/download/gmp/gmp-6.1.2.tar.lz
http://ftp.gnu.org/gnu/linux-libre/4.x/4.10.17-gnu/linux-libre-4.10.17-gnu.tar.lz
http://ftp.gnu.org/gnu/guile/guile-2.2.2.tar.lz
ftp://ftp.gnu.org/gnu/guile-sdl/guile-sdl-0.5.2.tar.lz
http://download.savannah.nongnu.org/releases/guile-www/guile-www-2.40.tar.lz
http://www.iana.org/time-zones/repository/releases/tzdb-2017b.tar.lz
http://ftp.gnu.org/gnu/gettext/gettext-0.19.8.tar.lz
http://ftp.gnu.org/gnu/octave/octave-4.2.1.tar.lz
http://ftp.gnu.org/gnu/gawk/gawk-4.1.4.tar.lz
http://ftp.gnu.org/gnu/mtools/mtools-4.0.18.tar.lz
http://ftp.gnu.org/gnu/serveez/serveez-0.2.2.tar.lz
ftp://ftp.graphicsmagick.org/pub/GraphicsMagick/GraphicsMagick-LATEST.tar.lz
http://www.imagemagick.org/download/ImageMagick.tar.lz
http://ftp.gnu.org/gnu/rcs/rcs-5.9.4.tar.lz
ftp://ftp.gnu.org/gnu/solfege/solfege-3.22.2.tar.lz
http://ftp.gnu.org/gnu/alive/alive-2.0.2.tar.lz
https://github.com/westes/flex/releases/download/v2.6.4/flex-2.6.4.tar.lz


Best regards,
Antonio.

Re: Steering committee, please, consider using lzip instead of xz

[Matias Fonzo]

On Wed, 7 Jun 2017 17:18:49 -0600
Jeff Law  wrote:

> On 06/07/2017 02:25 PM, Antonio Diaz Diaz wrote:
> > Dear GCC steering committee,
> > 
> > This has been recently asked in this list[1], but in case you have
> > missed it because of a subject line not explicit enough, I would
> > like to appeal to you directly.
> > 
> > [1] http://gcc.gnu.org/ml/gcc/2017-06/msg9.html
> > 
> > Since 2017-05-24 weekly snapshots use xz compression instead of
> > bzip2. I suppose this means that release tarballs will also use xz
> > at some point.
> > 
> > If this is the case, I politely request you to consider using lzip
> > instead of xz. I have spent a lot of time during the last 9 years
> > developing lzip and studying the xz format, and based on this
> > experience I consider that lzip is a better choice than xz, now and
> > in the long term.
> > 
> > I have been developing software since the early 80s, and I am a GNU
> > maintainer since 2003. You are all experienced developers. All I
> > ask is that you read carefully the following references and then
> > consider lzip and xz based on their technical merits.
> > 
> > http://www.nongnu.org/lzip/xz_inadequate.html
> > http://www.nongnu.org/lzip/lzip_benchmark.html#xz1
> > 
> > Also note that 'lzip -9' produces a tarball a 1% smaller than xz, in
> > spite of lzip using half the RAM to compress and requiring half the
> > RAM to decompress than xz.
> > 
> > -rw-r--r-- 1 58765134 2017-06-07 09:13 gcc-8-20170604.tar.lz
> > -rw-r--r-- 1 59367680 2017-06-07 09:13 gcc-8-20170604.tar.xz  
> Given the breadth that xz already has in the distribution space, I'd
> have a hard time supporting moving to lz over xz.
> 
> We've got far more important items to tackle than this.  But if you
> want me to bring it up formally with the SC I can.
> 
> jeff
> 
> ps.  And just to be clear, I actually don't like xz and I'm always
> annoyed when I run into something delivered in xz format.  But xz
> support at the distro level is pretty ubiquitous at this point.

All the important distributions have the lzip tool, besides GNU tar
has the support for it.  If not, it is easy to add lzip for decompress
the source.  This does not affect the preference of the distribution
for X format in the package redistribution.



pgptLJAKLx5ne.pgp
Description: OpenPGP digital signature