Re: GCC Optimisation, Part 0: Introduction

[Paolo Bonzini]

On 04/27/2011 03:28 PM, Yuan Pengfei wrote:

Any other advice will be appreciated.


I think you can look into llvm-clang. It compiles faster and uses
much less memory than gcc.


It is also a completely different compiler.  It doesn't make sense to 
compare the two, unless Dimitrios wants to rewrite GCC in 3 months.


Joe's suggestion (take a look at the "speedup areas" page, investigate 
which bullets can still be a priority) and talking to Nicola to help him 
with benchmarks and get up to speed with the code, are the best in this 
thread.


Regarding bitmaps, I have patches to use vector instructions.  The 
compile-time difference was basically a wash, but I can give them to you 
if you are interested.  Alternatively, the ebitmap.[ch] files in GCC are 
totally unused, and better in theory.  The main hurdle is that ebitmaps 
are not growable.  Still, they should be usable in many cases (or they 
can be made growable! :).


A nice exercise to get up to speed with the code is to make GCC record a 
set of bitmap operations from real compilation, and then have a driver 
that uses that set of operations for benchmarking.  This is the only 
sane way to extract the real performance of the data structure.  It 
doesn't avoid the need for validation in the compiler, of course, where 
other effects can come into play (caching etc.), but it is a useful start.


From the list in the webpage, the following items should be easier than 
the others:


* Split simulate_stmt into simulate_stmt and simulate_phi because the 
caller of simulate_stmt knows whether it's got a statement or a PHI node.


* Get rid of EXPR_LIST and INSN_LIST

* cxx_binding should be 16 bytes, not 20.

* Eliminate binfos for POD structs and other no-inheritance classes (not 
so easy)


* Put the string at the end of the IDENTIFIER_NODE using the trailing 
array hack (or possibly in the ht_identifier, see 
libcpp/include/symtab.h and libcpp/symtab.c)


Paolo

Re: Traversing typedef hierarchy in C/C++ tree

[Boris Kolpackov]

Hi Dodji,

Dodji Seketeli  writes:

> Boris Kolpackov  a =C3=A9crit:
>
> > template 
> > struct wrap
> > {
> >   typedef T w_s;
> > };
> >
> > typedef wrap::w_s w_s_t;
> >
> > Now if I traverse from w_s_t using DECL_ORIGINAL_TYPE I get:
> >
> > w_s_t->w_s->s
> >
> > Instead of:
> >
> > w_s_t->w_s->my_s_t->s_t->s
>
> Ah.  Indeed.  We strip typedefs from template arguments because G++
> keeps only one instance of each template specialization.  So it chooses
> to keep the "canonical" one.  In other words wrap and wrap
> ultimately representing the same specialization, G++ only construct one
> of them.  And it chooses to construct wrap because 's' is the
> canonical type here and not "my_s_t".  If did choose to keep "my_s_t",
> error messages would refer to wrap even for cases where it
> really is wrap that has actually been written by the user.  That
> would be confusing.

I see. I guess this is also the reason why we get verbose error messages
like:

error: ‘foo’ is not a member of ‘std::vector >’

Instead of:

error: ‘foo’ is not a member of ‘std::vector’

Do you know if there are any plans (or desire) to improve this? Creating
a separate tree_node for each instantiation would probably be too wasteful
so maybe we could keep only "distinct instantiations", i.e., those that
were created using distinct type nodes as template arguments. We could use
a hash based on the template node pointer plus all the argument type node
pointers to detect duplicates.

What do you think?

Boris

GCC 4.5.4 Status Report (2011-04-29)

[Richard Guenther]

Status
==

GCC 4.5.3 has been released, the release will be announced after
mirrors have catched up.  The branch is open again for regression
and documentation fixes.


Quality Data


Priority  #   Change from Last Report
---   ---
P10
P2  109   -  3
P35   +  5
---   ---
Total   114   +  2


Previous Report
===

http://gcc.gnu.org/ml/gcc/2011-04/msg00308.html

The next report will be sent by Jakub.

Re: GCC Optimisation, Part 0: Introduction

[Richard Guenther]

On Fri, Apr 29, 2011 at 9:18 AM, Paolo Bonzini  wrote:
> On 04/27/2011 03:28 PM, Yuan Pengfei wrote:
>>>
>>> Any other advice will be appreciated.
>>
>> I think you can look into llvm-clang. It compiles faster and uses
>> much less memory than gcc.
>
> It is also a completely different compiler.  It doesn't make sense to
> compare the two, unless Dimitrios wants to rewrite GCC in 3 months.
>
> Joe's suggestion (take a look at the "speedup areas" page, investigate which
> bullets can still be a priority) and talking to Nicola to help him with
> benchmarks and get up to speed with the code, are the best in this thread.
>
> Regarding bitmaps, I have patches to use vector instructions.  The
> compile-time difference was basically a wash, but I can give them to you if
> you are interested.  Alternatively, the ebitmap.[ch] files in GCC are
> totally unused, and better in theory.  The main hurdle is that ebitmaps are
> not growable.  Still, they should be usable in many cases (or they can be
> made growable! :).

If they are not growable then they are a candidate to replace sbitmaps
(or bitmaps where sbitmaps are not used because the bitmap will be only
sparsely populated).

As usual most of the time improving the container implementation isn't
the best kind of optimization.  Instead exchanging the container type
used or the algorithms will make more of a difference.

> A nice exercise to get up to speed with the code is to make GCC record a set
> of bitmap operations from real compilation, and then have a driver that uses
> that set of operations for benchmarking.  This is the only sane way to
> extract the real performance of the data structure.  It doesn't avoid the
> need for validation in the compiler, of course, where other effects can come
> into play (caching etc.), but it is a useful start.
>
> From the list in the webpage, the following items should be easier than the
> others:
>
> * Split simulate_stmt into simulate_stmt and simulate_phi because the caller
> of simulate_stmt knows whether it's got a statement or a PHI node.
>
> * Get rid of EXPR_LIST and INSN_LIST
>
> * cxx_binding should be 16 bytes, not 20.
>
> * Eliminate binfos for POD structs and other no-inheritance classes (not so
> easy)
>
> * Put the string at the end of the IDENTIFIER_NODE using the trailing array
> hack (or possibly in the ht_identifier, see libcpp/include/symtab.h and
> libcpp/symtab.c)

Not so easy because C++ uses TREE_TYPE of it and thus you'd lose
the nice sharing of identifiers.

* Embed gimple_seq_node in gimple itself.  This should simplify some
  iterators, avoid pointer indirection and eventually be more cache friendly.
  No idea why it wasn't done this way from the start.

Richard.

> Paolo
>

customizable warnings with a GCC plugin

[Pierre Vittet]

Hi,

My name is Pierre Vittet and my GSOC application as been selected. My 
project is about writing a plugin allowing GCC users to add some simple 
warnings, being useful in their particular project.
The user should be able to add rules like "when I got a call to a foo 
function, I would like to be sure that a check is made to monitor that 
it doesn't return somethings null" or "when I call this foo function, I 
would like it to be followed by that bar function". I will use MELT in 
order to realize this plugin.


You can read the detail of my application here: 
http://pvittet.com/GSOC/GSOC.pdf .


I have also started a page on the GCC wiki, it is pretty empty for now 
but is going to grow with the project: 
http://gcc.gnu.org/wiki/CustomizableWarningPlugin .


For now, I am able to write simple plugins, two exemples can be found 
here (https://github.com/Piervit/GMWarn). However, that is still 
difficult to test them on real size programs. Mainly because it can be 
difficult to build a complete program using GCC MELT (the branch). For 
exemple GCC MELT has mainly been tested on C programs, so if there is a 
C++ part, it mights fails, I have got also some problems when the 
configure scripts had to check GCCMELT. There is a fresh release of MELT 
as a plugin which will simplify things, I am going to report few issues 
coming from it. I think MELT is improving , and I am confident in being 
able to compile on real programs (I could already build readline for 
exemple).


So, I will be ready to start real coding of the plugin for the GSOC 
starting date (Mai 23).
For the moment I would like to give the following monitoring opportunity 
to the user:


- Checking that the returned value of a call to foo function is tested 
to be (not) NULL.
- Checking that the returned value of a call to foo function is tested 
to be (not) negative.
- Checking that the returned value of a call to foo function is tested 
to be (not) zero.
- Checking that a call to foo is immediately followed by a call to the 
bar function.
- Checking that a call to foo is followed in the same function by a call 
to the bar function.


My checks are limited to a function because I am going to work at the 
GIMPLE representation, and pass are run successively for each function. 
I am not sure that MELT is ready for IPA pass (or that might be 
experimental work). There is sufficient things to do with GIMPLE pass.


Do you see any others interesting cases to check?

Another big question: For this type of plugin, what would be the best 
position to insert my pass. For now, I try to add it just after the SSA 
pass and as it worked, I keep it at the position. Is it a good choice? I 
need a pass which is always run (or my plugin will not be inserted) and 
which has a correct GIMPLE representation.


I guess the most difficult point of my project might be to organise how 
my pass will run the different checks. The user will have the 
opportunity to run different checks in one compilation. I could choice 
to make one pass for every checks.
This would mean to have a first pass that parse the arguments and set 
the others pass (Is it possible to create new pass while we are already 
in one pass ?).


The others idea would be to have just one big pass, which is able to run 
for each GIMPLE statement the different checks in parallel. I am not 
sure that would be really nice.

What is your opinion on this?


If you have feedbacks from what is given here, I would enjoy to read 
them. If you have some documentation which could be related to the pass 
organisation (I already had a look at the very good GCC tutorial from 
Albert Cohen (http://www.hipeac.net/node/746)) or to anything looking 
interesting for my project, it would be great.


Thanks for your reading, I hope my English is not so bad.

Pierre Vittet

Re: customizable warnings with a GCC plugin

[Basile Starynkevitch]

On Fri, 29 Apr 2011 09:24:43 +0200
Pierre Vittet  wrote:

> Hi,
> 
> My name is Pierre Vittet and my GSOC application as been selected. My 
> project is about writing a plugin allowing GCC users to add some simple 
> warnings, being useful in their particular project.

[...]

Thanks to Pierre for his interesting message. I (Basile) am mentoring
his GSOC project.
>   
> If you have feedbacks from what is given here, I would enjoy to read 
> them. If you have some documentation which could be related to the pass 
> organisation (I already had a look at the very good GCC tutorial from 
> Albert Cohen (http://www.hipeac.net/node/746)) or to anything looking 
> interesting for my project, it would be great.

http://gcc.gnu.org/ml/gcc/2011-03/msg00106.html mentions Uday Khedker's
excellent tutorial slides
http://www.cse.iitb.ac.in/grc/index.php?page=gcc-tut given at CGO
http://www.cgo.org/cgo2011/

These are newer than Albert's ones.

Regards.
-- 
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basilestarynkevitchnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: customizable warnings with a GCC plugin

[David Brown]

On 29/04/2011 09:24, Pierre Vittet wrote:

Hi,

My name is Pierre Vittet and my GSOC application as been selected. My
project is about writing a plugin allowing GCC users to add some simple
warnings, being useful in their particular project.
The user should be able to add rules like "when I got a call to a foo
function, I would like to be sure that a check is made to monitor that
it doesn't return somethings null" or "when I call this foo function, I
would like it to be followed by that bar function". I will use MELT in
order to realize this plugin.

You can read the detail of my application here:
http://pvittet.com/GSOC/GSOC.pdf .

I have also started a page on the GCC wiki, it is pretty empty for now
but is going to grow with the project:
http://gcc.gnu.org/wiki/CustomizableWarningPlugin .

For now, I am able to write simple plugins, two exemples can be found
here (https://github.com/Piervit/GMWarn). However, that is still
difficult to test them on real size programs. Mainly because it can be
difficult to build a complete program using GCC MELT (the branch). For
exemple GCC MELT has mainly been tested on C programs, so if there is a
C++ part, it mights fails, I have got also some problems when the
configure scripts had to check GCCMELT. There is a fresh release of MELT
as a plugin which will simplify things, I am going to report few issues
coming from it. I think MELT is improving , and I am confident in being
able to compile on real programs (I could already build readline for
exemple).

So, I will be ready to start real coding of the plugin for the GSOC
starting date (Mai 23).
For the moment I would like to give the following monitoring opportunity
to the user:

- Checking that the returned value of a call to foo function is tested
to be (not) NULL.
- Checking that the returned value of a call to foo function is tested
to be (not) negative.
- Checking that the returned value of a call to foo function is tested
to be (not) zero.
- Checking that a call to foo is immediately followed by a call to the
bar function.
- Checking that a call to foo is followed in the same function by a call
to the bar function.

My checks are limited to a function because I am going to work at the
GIMPLE representation, and pass are run successively for each function.
I am not sure that MELT is ready for IPA pass (or that might be
experimental work). There is sufficient things to do with GIMPLE pass.

Do you see any others interesting cases to check?

Another big question: For this type of plugin, what would be the best
position to insert my pass. For now, I try to add it just after the SSA
pass and as it worked, I keep it at the position. Is it a good choice? I
need a pass which is always run (or my plugin will not be inserted) and
which has a correct GIMPLE representation.

I guess the most difficult point of my project might be to organise how
my pass will run the different checks. The user will have the
opportunity to run different checks in one compilation. I could choice
to make one pass for every checks.
This would mean to have a first pass that parse the arguments and set
the others pass (Is it possible to create new pass while we are already
in one pass ?).

The others idea would be to have just one big pass, which is able to run
for each GIMPLE statement the different checks in parallel. I am not
sure that would be really nice.
What is your opinion on this?


If you have feedbacks from what is given here, I would enjoy to read
them. If you have some documentation which could be related to the pass
organisation (I already had a look at the very good GCC tutorial from
Albert Cohen (http://www.hipeac.net/node/746)) or to anything looking
interesting for my project, it would be great.

Thanks for your reading, I hope my English is not so bad.

Pierre Vittet



There is a lot of interesting and useful work that could be done here. 
Melt is a nice idea, but the big barrier (for me, anyway) is the 
language - it's Lisp, which is very different to other languages that 
I've used.  Most gcc users are C or C++ programmers - while they might 
like the idea of having customised warnings or checks, it would take 
them a great deal of work to learn enough Lisp to get something useful 
from melt.  So if you manage to make something that C and C++ 
programmers can easily make use of, then that would be great.


Another problem with Melt is that it is inconvenient to install and use, 
and slow to run - if you make a layer on top of that, it will be worse. 
 This is not necessary a major issue, but it will typically mean that 
the extra checking will be done as a separate "make target" rather than 
as a standard compile option (such as is done with normal gcc warnings). 
 Of course, if Melt makes it easier to do this development then it's a 
good choice - speed issues can be looked at later, once you know the 
idea is good.


Have you also considered looking at other types of static checking?  The 
ideas you have so far are definitel

Re: customizable warnings with a GCC plugin

[Basile Starynkevitch]

On Fri, 29 Apr 2011 12:30:45 +0200
David Brown  wrote:
> 
> There is a lot of interesting and useful work that could be done here. 
> Melt is a nice idea, but the big barrier (for me, anyway) is the 
> language - it's Lisp, which is very different to other languages that 
> I've used.  

Pierre Vittet knows MELT quite well and has already successfully used
it (and he did found some bugs in MELT that I was happy to correct). He
is one of the most active posters on the
gcc-melt-fre...@googlegroups.com list. We (Pierre & me Basile) met
regularily (with Alexandre Lissy). Pierre is perhaps one of the best
(and few) MELT users. So MELT is definitely no more an obstacle for him!

And regarding the Lispy syntax, I do have a infix syntax in the works.
However, contrarily to many others, and conforming to Pierre's
experience, the hard part is not learning MELT syntax (which is looking
similar to e.g. Elisp or Scheme or Common Lisp code, so learning MELT
is fast if you did try Elisp or Scheme before), the hard part is
learning GCC internals. Pierre struggled much more understanding where
to insert his pass and what precise kind of Gimple he wants to detect
than with the MELT syntax. So my feeling is even more than before that
MELT lispy syntax is not important: providing an infix syntax to MELT
(which in my eyes would be parsed into exactly the same abstract MELT
trees as the current lisped syntax) is a significant amount of work,
and I am not sure at all it will buy new users to MELT. Hence a
question to David Brown: will he use MELT if it had an infix syntax
(but all the rest of MELT being the same)? [I am not sure, and this is
why making an infix syntax is not my priority]

> Most gcc users are C or C++ programmers - while they might 
> like the idea of having customised warnings or checks, it would take 
> them a great deal of work to learn enough Lisp to get something useful 
> from melt. 

No, the idea is that the users of Pierre's work will just invoke gcc
with appropriate (and indeed lenghty) option, perhaps something as
simple as gcc -fplugin=melt -fplugin-arg-melt-mode=give-posix-warnings
-c -O2 foobar.c so that means essentially adding
  CFLAGS +=  -fplugin=melt -fplugin-arg-melt-mode=give-posix-warnings 

in a Makefile. If Pierre coded a specific plugin directly in C (a
seperate plugin for each family of warnings), that would be much more
work for him (since he does know MELT!) and indeed the user of such a
plugin would have to pass 
  CFLAGS += -fplugin=some-plugin-to-give-posix-warnings

I (Basile) don't think it is such a big issue for the users of Pierre's
work. And Pierre (and of course me Basile) prefer to code in MELT
instead of C (because there is no pattern matching in C, while there is
a quite powerful pattern matching in MELT to match Gimple or Tree-s...).

> So if you manage to make something that C and C++ 
> programmers can easily make use of, then that would be great.

Again, the goal of Pierre's work is to give a set of extra contextual
warnings to users. Users of Pierre code don't need to understand the
MELT lispy syntax (and they don't need neither to understand what is a
Gimple or a Tree). But Pierre need to know all (MELT, what is a
Gimple... and understanding Gimple is much harder than understanding
MELT syntax).

> 
> Another problem with Melt is that it is inconvenient to install and use, 

Did you try the latest MELT plugin from http://gcc-melt.org/ (the 0.7
was released but I did not announce it yet, because I want to write a
small tutorial docuemntation) ? It is a plugin for GCC 4.6 (I put it
there a few days ago), and can be installed (by running a script given
inside the MELT plugin source tar ball) as a melt.so plugin. I am
surprised by the "inconvenient to use" phrase. MELT is just a [meta-]
plugin, it is as easy to use as any other plugin.

> and slow to run - if you make a layer on top of that, it will be worse. 

MELT is not that slow (what is slow is indeed the translation of C code
generated by MELT from a *.melt file; the translation from *.melt to
*.c is quite fast.). For the kind of checks that Pierre wish to
implement, MELT seems definitely the right tool, it was designed for
such use. For the simple pass that Pierre already coded in MELT, using
it on the entire readline6 source code to find untested calls to fopen
did not slow much the compilation time (but I have no figures for that
claim, just an impression). Besides, I want a lot to improve these
issues!

>   This is not necessary a major issue, but it will typically mean that 
> the extra checking will be done as a separate "make target" rather than 
> as a standard compile option (such as is done with normal gcc warnings). 

I dont understand that point. Once Pierre developped his GCC extensions
in MELT, using them is nearly as simple as using any other plugin,
except that one have to pass an extra -fplugin-melt-arg-mode=
argument. Is that such a big deal? 

But of course, adding extra checks is costly, and

Re: GCC Optimisation, Part 0: Introduction

[Dimitrios Apostolou]

Thank you all for your ideas, they are much appreciated. I will certainly 
investigate into the areas you mentioned, so do keep the feedback coming. 
I will certainly comment on them, once I have a better understanding. And 
I'd like to get in sync with existing work, so that duplicate effort is 
avoided (for example I'm already researching on finding an alternative 
hash function to replace htab_hash_string(), I don't know if Nicola has 
done that).


But for now I will be reading some tutorial slides I found from HiPEAC and 
CGO conferences, so that I understand GCC's architecture. Slides are good 
but are missing many things, so if you have video/audio transcripts of 
tutorials they would come in handy.


It is also a priority to standardise the way of measuring gcc's speed 
(which files to compile with what options), and dedicate some old 
machinery to that purpose. Perhaps this has already been done, and I can 
use existing setups? I found the following very useful pages:


http://gcc.opensuse.org/
http://gcc.gnu.org/wiki/PerformanceTesting

On another note, I can see that a similar project was accepted for 2007 
GSOC, titled as "Speeding up GCC for fun and profit". Where can I find 
more info?


Finally I'd appreciate any guidelines regarding submitting patches for 
review or development in general. For example is "diff -u" the proper 
format to use, or should I include Changelog diff in all patches, or 
should they all be sent to gcc-patches list even if the patch is initial 
version just for reviewing, etc.



Thanks,
Dimitris

Re: GCC Optimisation, Part 0: Introduction

[Diego Novillo]

On Fri, Apr 29, 2011 at 09:24, Dimitrios Apostolou  wrote:
> Thank you all for your ideas, they are much appreciated. I will certainly
> investigate into the areas you mentioned, so do keep the feedback coming. I
> will certainly comment on them, once I have a better understanding. And I'd
> like to get in sync with existing work, so that duplicate effort is avoided
> (for example I'm already researching on finding an alternative hash function
> to replace htab_hash_string(), I don't know if Nicola has done that).
>
> But for now I will be reading some tutorial slides I found from HiPEAC and
> CGO conferences, so that I understand GCC's architecture. Slides are good
> but are missing many things, so if you have video/audio transcripts of
> tutorials they would come in handy.

I don't think there are any such transcripts, sorry.  I remember one
year they had video going for the GCC summit, but I don't think they
were ever released.

> It is also a priority to standardise the way of measuring gcc's speed (which
> files to compile with what options), and dedicate some old machinery to that
> purpose. Perhaps this has already been done, and I can use existing setups?
> I found the following very useful pages:
>
> http://gcc.opensuse.org/
> http://gcc.gnu.org/wiki/PerformanceTesting

Both are useful.  My recommendation is to decide on a set of programs
that you will use as your build timing benchmark and operate on those.
 A good source of code may be a big project like KDE or OpenOffice.

> On another note, I can see that a similar project was accepted for 2007
> GSOC, titled as "Speeding up GCC for fun and profit". Where can I find more
> info?

I don't know what came of that, actually.  Eric?

> Finally I'd appreciate any guidelines regarding submitting patches for
> review or development in general. For example is "diff -u" the proper format
> to use, or should I include Changelog diff in all patches, or should they
> all be sent to gcc-patches list even if the patch is initial version just
> for reviewing, etc.

You should follow the guidelines in http://gcc.gnu.org/contribute.html.

The very first one is a copyright assignment.  Have you started that
process?  If not, I will send you the form.


Diego.

Re: GCC Optimisation, Part 0: Introduction

[Tom Tromey]

> "Paolo" == Paolo Bonzini  writes:

Paolo> * Put the string at the end of the IDENTIFIER_NODE using the trailing
Paolo> array hack (or possibly in the ht_identifier, see
Paolo> libcpp/include/symtab.h and libcpp/symtab.c)

I implemented this once:

http://gcc.gnu.org/ml/gcc-patches/2008-03/msg01293.html

It did not go in because a different approach (the one in the code now)
was deemed clearer.

Tom

Re: GCC Optimisation, Part 0: Introduction

[Dimitrios Apostolou]

On Fri, 29 Apr 2011, Diego Novillo wrote:


The very first one is a copyright assignment.  Have you started that
process?  If not, I will send you the form.



Thanks Diego, please send me the form. I'll sign it as soon as my 
contributions require it.



Dimitris

Re: GCC Optimisation, Part 0: Introduction

[Nathan Froyd]

On Fri, Apr 29, 2011 at 09:18:56AM +0200, Paolo Bonzini wrote:
> * Get rid of EXPR_LIST and INSN_LIST

This is reasonably difficult, though particular subprojects may be easy
enough.  Notable uses of EXPR_LIST:

- loop-iv.c

- the interface to TARGET_FUNCTION_VALUE

- the scheduler

- REG_NOTES

- var-tracking.c

- reload

Notable uses of INSN_LIST:

- the scheduler

- reload

- gcse.c

The biggest uses of each in the scheduler ought to be easy to deal with,
but the scheduler manipulates the lists in peculiar ways.

> * cxx_binding should be 16 bytes, not 20.

Not your fault, but comments like this on SpeedupAreas are so opaque as
to be useless.  *Why* should cxx_binding be 16 bytes?  Should we take
the next member out and have a VEC someplace instead of chaining?  Are
we duplicating information in the members themselves?  Etc.

-Nathan

Re: GCC Optimisation, Part 0: Introduction

[Paolo Bonzini]

On 04/29/2011 04:15 PM, Nathan Froyd wrote:

>  * cxx_binding should be 16 bytes, not 20.


Not your fault, but comments like this on SpeedupAreas are so opaque as
to be useless. *Why* should cxx_binding be 16 bytes?  Should we take
the next member out and have a VEC someplace instead of chaining?  Are
we duplicating information in the members themselves?  Etc.


Sorry, you're right.  It's about cache lines I guess, and moving the 
bitfields into one of the pointers.


Paolo

Re: customizable warnings with a GCC plugin

[David Brown]

On 29/04/2011 13:16, Basile Starynkevitch wrote:

On Fri, 29 Apr 2011 12:30:45 +0200
David Brown  wrote:


There is a lot of interesting and useful work that could be done here.
Melt is a nice idea, but the big barrier (for me, anyway) is the
language - it's Lisp, which is very different to other languages that
I've used.


Pierre Vittet knows MELT quite well and has already successfully used
it (and he did found some bugs in MELT that I was happy to correct). He
is one of the most active posters on the
gcc-melt-fre...@googlegroups.com list. We (Pierre&  me Basile) met
regularily (with Alexandre Lissy). Pierre is perhaps one of the best
(and few) MELT users. So MELT is definitely no more an obstacle for him!



I wasn't thinking of MELT's language as an obstacle for Pierre - but for 
us mere mortal C programmers.  That's why this is such a good idea for a 
project if it lets people like me get extra checks without having to 
learn MELT and Lisp.



And regarding the Lispy syntax, I do have a infix syntax in the works.
However, contrarily to many others, and conforming to Pierre's
experience, the hard part is not learning MELT syntax (which is looking
similar to e.g. Elisp or Scheme or Common Lisp code, so learning MELT
is fast if you did try Elisp or Scheme before), the hard part is
learning GCC internals. Pierre struggled much more understanding where
to insert his pass and what precise kind of Gimple he wants to detect
than with the MELT syntax. So my feeling is even more than before that
MELT lispy syntax is not important: providing an infix syntax to MELT
(which in my eyes would be parsed into exactly the same abstract MELT
trees as the current lisped syntax) is a significant amount of work,
and I am not sure at all it will buy new users to MELT. Hence a
question to David Brown: will he use MELT if it had an infix syntax
(but all the rest of MELT being the same)? [I am not sure, and this is
why making an infix syntax is not my priority]



Perhaps I wasn't clear here (or perhaps I misunderstood Pierre's first 
post).  I am not suggesting changing the language for MELT - I assume it 
uses Lisp because the developers thought that's a good language for such 
pattern-matching applications.  While my Lisp knowledge is very 
rudimentary, I can easily see it's a better choice than, for example, C 
- I am a big fan of choosing appropriate languages for the task in hand, 
rather than trying to do everything with the one language.  Of course, I 
am a bigger fan of people using languages I already know - I am sure 
Python is the ideal language for MELT :-)


As far as I understand Pierre's project, it will let users like me get 
configurable warnings using a simple syntax on the gcc command line - 
and that's great.  The language used by Pierre is then an implementation 
detail that doesn't affect me as a user (unless I want to fiddle with 
the source myself) - it doesn't matter whether I like Lisp or not, 
because I won't see the Lisp code.


As to whether I would be more likely to use MELT if it had another 
language - probably I would be a little more likely to try it.  But as 
you say, the language is only one part of the issue - understanding how 
gcc works and how to fit in with it is a far bigger challenge.  What 
would make me much more likely to try it is lots of examples, or 
library-style code so that I could have something useful (not "hello, 
world!") to start with, that I could modify easily.  At that point, the 
choice of language would be a much smaller matter.  Of course, it's 
quite possible that such easy-to-use examples are not practically 
possible - gcc is a very complicated piece of software.



For the longer term, it would be good if there were a way to use this 
with windows (and without installing all of cygwin).  I know these 
things are /much/ easier on Linux, but gcc is getting increasingly 
popular on windows.  I work as an embedded programmer, and steadily more 
embedded systems are programmed using gcc - often on windows hosts.




Most gcc users are C or C++ programmers - while they might
like the idea of having customised warnings or checks, it would take
them a great deal of work to learn enough Lisp to get something useful
from melt.


No, the idea is that the users of Pierre's work will just invoke gcc
with appropriate (and indeed lenghty) option, perhaps something as
simple as gcc -fplugin=melt -fplugin-arg-melt-mode=give-posix-warnings
-c -O2 foobar.c so that means essentially adding
   CFLAGS +=  -fplugin=melt -fplugin-arg-melt-mode=give-posix-warnings

in a Makefile. If Pierre coded a specific plugin directly in C (a
seperate plugin for each family of warnings), that would be much more
work for him (since he does know MELT!) and indeed the user of such a
plugin would have to pass
   CFLAGS += -fplugin=some-plugin-to-give-posix-warnings



Yes, I understand that now (I've read more since my first post).  That 
sounds excellent.



I (Basile) don't think it is such a big i

Re: GCC Optimisation, Part 0: Introduction

[Nathan Froyd]

On Fri, Apr 29, 2011 at 04:20:15PM +0200, Paolo Bonzini wrote:
> On 04/29/2011 04:15 PM, Nathan Froyd wrote:
>>> >  * cxx_binding should be 16 bytes, not 20.
>>
>> Not your fault, but comments like this on SpeedupAreas are so opaque as
>> to be useless. *Why* should cxx_binding be 16 bytes?  Should we take
>> the next member out and have a VEC someplace instead of chaining?  Are
>> we duplicating information in the members themselves?  Etc.
>
> Sorry, you're right.  It's about cache lines I guess, and moving the  
> bitfields into one of the pointers.

Gross. :)

-Nathan

Re: GCC Optimisation, Part 0: Introduction

[Laurynas Biveinis]

> Thanks Diego, please send me the form. I'll sign it as soon as my
> contributions require it.

Don't wait; sign it right away - it might take a while to process it.

-- 
Laurynas

Integration of transactional memory support into a data-flow extension of OpenMP

[ismail]

Hi, 

I have thought that my first email can be seen as too long.

I have done experiments regarding compatibility of OpenMP pramas +
trans-mem constructs.  (It is fine, working. I have a primitive
interface (combined constructs of trans-mem  and OpenMP))

Any suggestions for benchmarks would be great especially including: 
* transactions that can be  pipelined with concerns of targeting
data-flow parallelism.

* I have studied on LeeTM but it does not include pipelined
structure for transactions. I will be trying to structure 
  this benchmark as pipelined BUT any other suggestions would  
  be great!!


regards

Ismail KURU

Re: customizable warnings with a GCC plugin

[Pierre Vittet]

Using MELT is always heavily discussed :).

I thinks this project is really an opportunity to see how useful is 
MELT. Limitation doesn't really come from the syntax (infix or not) but 
from what is already implemented (cannot make code replacement, not a 
complete implementation of IPA). However I don't need those 
functionnalities in my plugin and I will be able to use some nice 
functionnalities of MELT (pattern matching on trees to just give one).


As said, user will not have to know about the Lisp syntax, he will just 
have to install MELT as a plugin and to add some rules (in the easiest 
possible syntax). For plugin develloper, it is an opportunity to see how 
MELT works.


About the security issue, I guess that if someone find something very 
good to detect potential buffer overflow, it does not have to be coded 
in a plugin but in GCC itself. Moreover, as Basile said, I don't know 
much about those questions.


I don't think it is possible to write a plugin able to help respecting 
coding rules for every project. I thinks there is always some particular 
issues comming from the specific project we are working on which make 
this quite impossible. That why I think, giving to the user an 
opportunity to easily set his on rules is the best choice. If the user 
knows that using the unsafe_foo function need a check or another 
function to be called, this is a way to improve security.


Thanks for your mails!

On 29/04/2011 16:25, David Brown wrote:

On 29/04/2011 13:16, Basile Starynkevitch wrote:

On Fri, 29 Apr 2011 12:30:45 +0200
David Brown wrote:


There is a lot of interesting and useful work that could be done here.
Melt is a nice idea, but the big barrier (for me, anyway) is the
language - it's Lisp, which is very different to other languages that
I've used.


Pierre Vittet knows MELT quite well and has already successfully used
it (and he did found some bugs in MELT that I was happy to correct). He
is one of the most active posters on the
gcc-melt-fre...@googlegroups.com list. We (Pierre& me Basile) met
regularily (with Alexandre Lissy). Pierre is perhaps one of the best
(and few) MELT users. So MELT is definitely no more an obstacle for him!



I wasn't thinking of MELT's language as an obstacle for Pierre - but for
us mere mortal C programmers. That's why this is such a good idea for a
project if it lets people like me get extra checks without having to
learn MELT and Lisp.


And regarding the Lispy syntax, I do have a infix syntax in the works.
However, contrarily to many others, and conforming to Pierre's
experience, the hard part is not learning MELT syntax (which is looking
similar to e.g. Elisp or Scheme or Common Lisp code, so learning MELT
is fast if you did try Elisp or Scheme before), the hard part is
learning GCC internals. Pierre struggled much more understanding where
to insert his pass and what precise kind of Gimple he wants to detect
than with the MELT syntax. So my feeling is even more than before that
MELT lispy syntax is not important: providing an infix syntax to MELT
(which in my eyes would be parsed into exactly the same abstract MELT
trees as the current lisped syntax) is a significant amount of work,
and I am not sure at all it will buy new users to MELT. Hence a
question to David Brown: will he use MELT if it had an infix syntax
(but all the rest of MELT being the same)? [I am not sure, and this is
why making an infix syntax is not my priority]



Perhaps I wasn't clear here (or perhaps I misunderstood Pierre's first
post). I am not suggesting changing the language for MELT - I assume it
uses Lisp because the developers thought that's a good language for such
pattern-matching applications. While my Lisp knowledge is very
rudimentary, I can easily see it's a better choice than, for example, C
- I am a big fan of choosing appropriate languages for the task in hand,
rather than trying to do everything with the one language. Of course, I
am a bigger fan of people using languages I already know - I am sure
Python is the ideal language for MELT :-)

As far as I understand Pierre's project, it will let users like me get
configurable warnings using a simple syntax on the gcc command line -
and that's great. The language used by Pierre is then an implementation
detail that doesn't affect me as a user (unless I want to fiddle with
the source myself) - it doesn't matter whether I like Lisp or not,
because I won't see the Lisp code.

As to whether I would be more likely to use MELT if it had another
language - probably I would be a little more likely to try it. But as
you say, the language is only one part of the issue - understanding how
gcc works and how to fit in with it is a far bigger challenge. What
would make me much more likely to try it is lots of examples, or
library-style code so that I could have something useful (not "hello,
world!") to start with, that I could modify easily. At that point, the
choice of language would be a much smaller matter. Of course, it's quit

Re: customizable warnings with a GCC plugin

[Basile Starynkevitch]

On Fri, 29 Apr 2011 16:25:39 +0200
David Brown  wrote:
[...]
I am skipping interesting parts of David reply (I might reply to them later)
> 
> For the longer term, it would be good if there were a way to use this 
> with windows (and without installing all of cygwin).  I know these 
> things are /much/ easier on Linux, but gcc is getting increasingly 
> popular on windows.  I work as an embedded programmer, and steadily more 
> embedded systems are programmed using gcc - often on windows hosts.

The major issue is to have GCC plugins working on Windows (and, when
that happens, to adapt existing plugins, including melt.so, to work
under Windows). I know nothing about the subject, but Levine's book on
linkers & loaders makes me believe it might not be fun.

I (Basile) personally don't care at all about Windows, since I never
used it (and my first program was, in 1974 - I was a 14 years old
teenager then - in PL/1 on punched cards, so I am not a very young
Linux fan boy), and probably I will never code for it. The last
Microsoft system I used on my PC was some MS-DOS thing. The DEBUG
debugger was not very sexy at that time. Microsoft products don't left
a good impression in my aging human memory!

All the [i386 or AMD64] PCs I had or have (both at work and at home) are
running Linux. I am a Linux enthusiast since kernel 0.99.15 (that was
an SLS distribution on more than 50? floppies) - and GCC was 1.x IIRC
at that time.

I don't know if some people are working today on adapting the current
GCC plugin machinery to Windows. I've seen no patch about this (and I
don't care much).

Maybe some future version of Windows might run under Linux :-) :-)

Regards.
-- 
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basilestarynkevitchnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: customizable warnings with a GCC plugin

[Kyle Girard]

> The major issue is to have GCC plugins working on Windows (and, when
> that happens, to adapt existing plugins, including melt.so, to work
> under Windows). I know nothing about the subject, but Levine's book on
> linkers & loaders makes me believe it might not be fun.
> 

Just a point of info

A few months ago I asked about the possibility/state of plugins on
windows. While having run-time plugins was basically impossible (because
of the work involved) link-time plugins were possible and I was given a
patch that did hacked the build system to include plugins at link time.

While not really support for plugins on windows it did allow me to
deploy a version of gcc with our unmodified plugin linked in.

The patch is probably still in the archives of this mailing list if
anyone is interested..



> I don't know if some people are working today on adapting the current
> GCC plugin machinery to Windows. I've seen no patch about this (and I
> don't care much).
> 

If there are people working on this I'd be interested.. save me some
serious compile time for every new release ;)

Cheers,

Kyle

Re: customizable warnings with a GCC plugin

[Basile Starynkevitch]

On Fri, 29 Apr 2011 14:30:32 -0400
Kyle Girard  wrote:
> Just a point of info
> 
> A few months ago I asked about the possibility/state of plugins on
> windows. While having run-time plugins was basically impossible (because
> of the work involved) link-time plugins were possible and I was given a
> patch that did hacked the build system to include plugins at link time.

Thnaks for the info. However, MELT requires a real dynamic plugin
ability, since the melt.so plugin may generate C code foo.c (from foo.melt),
run a make command to build foo.so (from generated foo.c), and dlopen
that foo.so. 

So I don't expect MELT to work under Windows very soon :-)


Cheers.

-- 
Basile STARYNKEVITCH http://starynkevitch.net/Basile/
email: basilestarynkevitchnet mobile: +33 6 8501 2359
8, rue de la Faiencerie, 92340 Bourg La Reine, France
*** opinions {are only mine, sont seulement les miennes} ***

Re: gcov/-ftest-coverage instrumentation for uninstantiated C++ function templates

[Andi Hellmund]

Hey Manuel,

I would like to be able to change this behaviour so non-instantiated 
code templates are considered as blocks (I think this is the term used 
by GCC/GCov). This would help me greatly to uncover unused/untested 
codes in a header/template-only library.


First of all: Is this feasible with the GCC infrastructure?

I am not sure about that. It could be harder to get this implemented 
since the required information might not be available when you require it.


For this to work, one would probably need access to the internal 
representation of the program. It is my understanding that there are 
at least two distinct representations of a C++ program: (1) The AST 
that only contains the parse tree, before any template instantiation 
and (2) the intermediate language representation (GIMPLE?) that will 
contain a representation of the instantiated templates. On which level 
is coverage instrumentation added right now?
Yes, that's true. You get the details of both representations by using 
the additional compiler flags


-fdump-translation-unit (prints out an extended version of GENERIC (= AST))
-fdump-tree-gimple-raw (prints out the second IR known as GIMPLE)

The instrumentation of the source code is finally done on the GIMPLE IR. 
You could print out the IR by specifiying -fdump-tree-tree_profile



Second: Where would I start looking?

So far, I have discovered gcov*.{h,c} in the gcc directory. However, I 
have not yet found where the .gcno files are written out in the compiler.
There was a recent discussion on the gcc-help mailing list about the 
basic internals of GCOV. Please find the thread here: 
http://gcc.gnu.org/ml/gcc-help/2011-04/msg00072.html


Third: How hard would it be and how would I proceed?

Naively, my first guess at an approach would be: Locate where the 
.gcno file is written out. After writing out all real blocks, identify 
all uninitialized template functions, or member functions that are 
members of templates, etc. etc. Add a "block" to the .gcno file for 
each such function.
As listed above, the .gcno file is written in coverage.c, though you 
should look at this one.


I hope that this helps at least a bit to get a step further ...

Best regards,
Andi

Re: My current idea for improving libgomp

[Andreas Prell]

Hi Sho,

I just came across your project and would like to add a few comments.

I think the biggest problem in libgomp is that tasks are stored in a
single queue. This may work for a few threads or for long-running tasks,
but certainly not for 48 threads. In fact, contention on the queue can
grow so high that performance starts to degrade when you add more
threads.

First of all, you need to think about data structures for storing tasks.
When you look at other parallel runtime systems/libraries, you will find
that many implement some form of work-stealing scheduling. In
work-stealing scheduling, each worker or group of workers have their own
queue on which they operate. Load balancing is done by stealing tasks
from the queues of other workers. The Nanos runtime is probably a good
place to start to get some ideas...  

Regarding Lazy Task Creation (LTC)... I agree with the advice here and
would say this is not what you should focus on. The most prominent use
of LTC is in the Cilk multithreaded language. Basically, we have two
options when we encounter a new task: (1) spawn the task as a child and
continue executing the parent, or, what LTC does, (2) start executing
the task and leave the continuation (the rest of the parent) for later
execution. Load balancing requires that the continuation is made
available so that it can be picked up and executed by other threads. The
Nanos runtime supports different scheduling strategies, including
work-first scheduling, which is similar in principle to LTC. Nanos
implements tasks as user-level threads (nano-threads), so it's easier to
switch between tasks or move tasks across threads. Something like
work-first scheduling that depends on untied tasks is nice to have. But
I think a decent scheduler (for tied tasks) is more important at this
point...

-Andreas

gcc-4.6-20110429 is now available

[gccadmin]

Snapshot gcc-4.6-20110429 is now available on
  ftp://gcc.gnu.org/pub/gcc/snapshots/4.6-20110429/
and on various mirrors, see http://gcc.gnu.org/mirrors.html for details.

This snapshot has been generated from the GCC 4.6 SVN branch
with the following options: svn://gcc.gnu.org/svn/gcc/branches/gcc-4_6-branch 
revision 173194

You'll find:

 gcc-4.6-20110429.tar.bz2 Complete GCC (includes all of below)

  MD5=0974ec5801d9f74bded61b5aac9b8fbb
  SHA1=225963b142f773a3b595750216431cab0947effd

 gcc-core-4.6-20110429.tar.bz2C front end and core compiler

  MD5=fa586a5634ae3b462c5fb2d55160b14f
  SHA1=4ad60387e58bd52d74812ec559454f572359727b

 gcc-ada-4.6-20110429.tar.bz2 Ada front end and runtime

  MD5=cf518c56b968925a2eca3aa2b4cdbd7b
  SHA1=fdeb16275dc79a6e8557257868df66131769996d

 gcc-fortran-4.6-20110429.tar.bz2 Fortran front end and runtime

  MD5=366f93c1867f35c8860b42965efde955
  SHA1=0db350a17f56132c88c4b9fb0b27e1f19bd19199

 gcc-g++-4.6-20110429.tar.bz2 C++ front end and runtime

  MD5=2591b1fad977f7bd6136c9ac1298ef04
  SHA1=97b228d649fc1c494ce237947e1806a48a8df0b0

 gcc-go-4.6-20110429.tar.bz2  Go front end and runtime

  MD5=f66ef8597b99961eda7573df505ea354
  SHA1=f683d8c3d7bdf4ea56903358ff0daedac5938cf1

 gcc-java-4.6-20110429.tar.bz2Java front end and runtime

  MD5=d4775781934c96c812b230cbe04f8f72
  SHA1=7bb6f0c9777b0b3c3a2681b4b3b6e36610055d0e

 gcc-objc-4.6-20110429.tar.bz2Objective-C front end and runtime

  MD5=21779792ef7e0ed7abef611160b0099c
  SHA1=d00bebe3f43c6a65415819a5c47886c8b8b03936

 gcc-testsuite-4.6-20110429.tar.bz2   The GCC testsuite

  MD5=3d89dd233ceb6a837d14ee6b940a43ed
  SHA1=eba2049430d6d38a29d6fd757a59e9540166ba61

Diffs from 4.6-20110422 are available in the diffs/ subdirectory.

When a particular snapshot is ready for public consumption the LATEST-4.6
link is updated and a message is sent to the gcc list.  Please do not use
a snapshot before it has been announced that way.

Re: customizable warnings with a GCC plugin

[Gabriel Dos Reis]

On Fri, Apr 29, 2011 at 1:49 PM, Basile Starynkevitch
 wrote:

> So I don't expect MELT to work under Windows very soon :-)

so, is this all a plot to get GCC melt down on windows? ;-p

-- Gaby

Re: My current idea for improving libgomp

[Sho Nakatani]

Hi Andreas,

Thank you for your comments.

> I think the biggest problem in libgomp is that tasks are stored in a
> single queue. This may work for a few threads or for long-running tasks,
> but certainly not for 48 threads. In fact, contention on the queue can
> grow so high that performance starts to degrade when you add more
> threads.
>
> First of all, you need to think about data structures for storing tasks.
> When you look at other parallel runtime systems/libraries, you will find
> that many implement some form of work-stealing scheduling. In
> work-stealing scheduling, each worker or group of workers have their own
> queue on which they operate. Load balancing is done by stealing tasks
> from the queues of other workers. The Nanos runtime is probably a good
> place to start to get some ideas...

I totally agree with this point.
Currently, I'm planning to implement tied task using breath-first
scheduler wrote in
section 3.1 of "Evaluation of OpenMP Task Scheduling Strategies" by Nanos Group.
http://www.sarc-ip.org/files/null/Workshop/1234128788173__TSchedStrat-iwomp08.pdf

That is:
* A team has one team queue which contains tasks.
* A team has some (user-level) threads.
* A thread can have one running task.
* A thread has private queue which contains tasks.
* When a task is created, it is queued in team queue.
* Each thread steals tasks from the team queue and inserts it in the
private queue.
* Once tied task is executed in a thread, it is queued only in the
private queue in the thread
  when it encounters `taskwait'.
* Each thread runs a task from its private queue.

But I'm not sure how to achieve good load-balancing and what kind of
cutoff strategy to take.
As for load-balancing, I'll read Nanos4 implementations and ask Nanos
Group for it.
(Of course your advice will do :-) )

As for cutoff, basically I can choose `max-tasks' strategy or
`max-levels' strategy.
When number of tasks or recursion levels exceed this value, the
scheduler stops its work
and execute each task as sentences in sequential programs.
But "Evaluation of OpenMP Task Scheduling Strategies" says better
cutoff strategy is different
from application to application.


> Regarding Lazy Task Creation (LTC)... I agree with the advice here and
> would say this is not what you should focus on. The most prominent use
> of LTC is in the Cilk multithreaded language. Basically, we have two
> options when we encounter a new task: (1) spawn the task as a child and
> continue executing the parent, or, what LTC does, (2) start executing
> the task and leave the continuation (the rest of the parent) for later
> execution. Load balancing requires that the continuation is made
> available so that it can be picked up and executed by other threads. The
> Nanos runtime supports different scheduling strategies, including
> work-first scheduling, which is similar in principle to LTC. Nanos
> implements tasks as user-level threads (nano-threads), so it's easier to
> switch between tasks or move tasks across threads. Something like
> work-first scheduling that depends on untied tasks is nice to have. But
> I think a decent scheduler (for tied tasks) is more important at this
> point...

In GSoC project, I will not implement LTC but just focus on schedulers
which Nanos4 has already has.
I'll first implement `tied task' using breadth-first scheduler, and after that
I'll implement `untied task' using work-first scheduler (if I have time).

Thanks,
--
Sho Nakatani