Support for export keyword to use with C++ templates ?
Hello Is there any progress or start yet in implemententing export for C++ templates ? A search in the mailing list archive shows the last time the issue was discussed was in 2006. Why is everybody such not interested in this ? It would be such a great feature, especially for a leading C++ implementation like gcc ... Why is it so hard to store template definitions (and the set of symbols visible to them) in an object file, probably as a representation of the parsed template source tree ? A compiler-provided tool (or even the compiler) would then be invoked as a separate step in the build process before linking, being passed all such object files (named translated translation units by the standard) in the program, same as the linker is given all the object files. This tool would then examine the translated units in order to: - identify the required template instantiations, - find the template definitions - instantiate them (event in a separate output file, later to be passed to the linker as an object file) It is a matter of giving the compiler (compiler-provided tool) access to all translation units in the program at some point before linking, in a new step in the build process. Such a tool would share attributes of: - a compiler, in that it generates object code, but only from the encoded template definitions, and - a linker, in that it has the entire program at its disposal and it resolves references, but only references between required template instantiations and template definitions. To put it simple though it is just: "the compiler having all the object files of a program", or like having gcc understand a some type of input file: parsed-template-definition object files, and then have gcc output some regular object file from this new input files. The linker in the end should stay out of all these and only get the instantiated, regular object files. So a Makefile to explicitly go through the build steps would be like array_tmpl.o: array_tmpl.cc g++ -c -o array_tmpl.o array_tmpl.cc hash_tmpl.o: hash_tmpl.cc g++ -c -o hash_tmpl.o hash_tmpl.cc main.o: main.cc g++ -c -o main.o main.cc templ_inst.o: main.o array_tmpl.o hash_tmpl.o g++ -instantiate-only -o templ_inst.o main.o array_tmpl.o hash_tmpl.o main: main.o array_tmpl.o hash_tmpl.o templ_inst.o ld main.o arrray_tmpl. hash_tmpl.o templ_inst.o Any thoughts ? Thank you, Timothy Madden
Re: Support for export keyword to use with C++ templates ?
On Fri, Jan 22, 2010 at 2:24 AM, Ian Lance Taylor wrote: > Timothy Madden writes: > >> Is there any progress or start yet in implemententing export for C++ >> templates ? > > Not to my knowledge. > > The C++0x standards committee considered deprecating export for C++0x, > but I think they have decided to retain it for now. > > >> Why is everybody such not interested in this ? It would be such a >> great feature, especially for >> a leading C++ implementation like gcc ... > > Why would it be useful? What advantage does it provide? > > >> Why is it so hard to store template definitions (and the set of >> symbols visible to them) in an >> object file, probably as a representation of the parsed template source tree >> ? > > I recommend reading "Why We Can't Afford Export." This link may work > for you: > > http://docs.google.com/viewer?a=v&q=cache:lmlZ1p7te3gJ:www.open-std.org/jtc1/sc22/wg21/docs/papers/2003/n1426.pdf+why+we+can't+afford+export+sutter&hl=en&gl=us&pid=bl&srcid=ADGEESgDdiuh8WxBHeWHGfnK_iJXXYj0XCWOwfnHQuja8Ihd5567rnFyRsF13uY-HN9NbjYJpwMsUo8VQNx5_ffYSiOfOMeIfKMW-d8s57IMr8B8vUN3UfpXLXVtlLd1C8UiUNNe-i7t&sig=AHIEtbQo42iDSyaR0-I4hQ1bHFV3WTeYfA > > or search for the PDF. > > Note in particular that namespace issues are hideously complex with > export. Thanks to two-phase name lookup, the meaning of a template > depends on the environment in which it is defined. That means that in > order to correctly instantiate an exported template, the export file > must contain not only the template definition, but the definition of > all names mentioned in the template definition, recursively. > > As far as I know, exactly one compiler implements export, and nobody > uses it. It's because it is not implemented yet that it is not used ... :) Even if I would buy Comeau I would not write code that I know will not compile with any other compiler. Many books though teach how to use a #define to turn your template code into export-ready templates, even if export is not widely available. Actually lots of books and articles talk about export. I have seen that paper in the link, N1426, and I find it to be biased against export ... Everyone will admit to the technical challenges when implementing export. Template code is not object code. It is something else, it is another level, another way of doing things... This is why templates in C++ are so great. For that the compiler has to adapt, there is a new step required in the compilation process, for template instantiation. It is like there are now two levels of compilation: for templates and for object code ... So yes, there would be quite some work for it. The requirement to store symbol table entries when generating pre-compiled templates is as true as it can be. Such an idea is known for quite some time: it is what a precompiled header does: it stores the symbols table. Or at least there are certain similarities or a common approach or point of view. Is this such a problem ? The export keyword is there in the standard with some purpose: it allows the programmer to separate the template definition from the declaration. Its benefits are organised and clear code (and template code) for end-users of the language. It allows anyone to use the template without knowledge of how it is implemented and where it is implemented. It is like in object code one can use external functions and variables without knowledge of where do they come from and how do they work. You just have the declarations and you can use them, templates or not. Many people will say now that it is not the same, that it can not be the same because template code is not like object code, it goes by different rules. Well then I will have to say it loud and clear, that for the end-user's view of things, yes it is ! For them it is the same thing, this is what export does! And it is the end users that matter, it is them the compiler is made for, right ? The differences that still exist (rightfully so, for technical reasons) are ... not important. Someone on that long thread on comp.lang.c++.moderated about 'export' has put things in a very clear light, 6 years ago, when asked what is export so good for: Imagine for a moment that C++ had no one-definition-rule problems, and that it /required/ you to have the body of each and every function you use (call or take address) included in every translation unit ! Imagine you would have to always include the function definition of all functions you call in a .cc file, and the functions that are called by them. How would that be ? Now imagine that this happened because compiler implementers decided it is a real lot of work to devise something like a linker that would link function definitions with references at some point after compilation... I know it
Re: Support for export keyword to use with C++ templates ?
On Mon, Jan 25, 2010 at 8:12 PM, Ian Lance Taylor wrote: > Timothy Madden writes: > >> On Fri, Jan 22, 2010 at 2:24 AM, Ian Lance Taylor wrote: >>> Timothy Madden writes: >>> >>>> Why is it so hard to store template definitions (and the set of >>>> symbols visible to them) in an >>>> object file, probably as a representation of the parsed template source >>>> tree ? >>> >>> I recommend reading "Why We Can't Afford Export." >> >> I have seen that paper in the link, N1426, and I find it to be biased >> against export ... > > It may be biased, but I think it does a nice job of explaining why it > is so hard to store template definitions and the set of symbols > visible to them. No, it only scares people off the subject ... So nobody here wants to try a big thing ? :( How long would it take for someone to understand how parsing works in g++ ? Or to understand the build system in gcc ? Thank you, Timothy Madden
Support for export keyword to use with C++ templates ?
On Sat, Jan 30, 2010 at 4:23 AM, Michael Witten [...] > However, I have a gut feeling that at least a restricted version of > 'export' (or a cousin of 'export') could be both useful and trivial to > implement: [...] > Those were my thoughts too. Since such a change must happen in small steps, I would be interested to know how 'acceptable' would a limited implementation be at first ? Like the command line options I have seen declared 'experimental' in the gcc manual before ... The idea of a "cousin of" or "variation of" is less important though, as I am interested in standard conformance, even if partly implemented. Timothy Madden
Re: Support for export keyword to use with C++ templates ?
On Sat, Jan 30, 2010 at 4:05 AM, Paolo Carlini wrote: [...] > Even for implementors knowing *very* well both the details of the C++ > standard and the internals of a specific front-end, implementing export > is an *highly* non-trivial task. [...] Yes, everyone is telling me that, but could someone please describe a little what would be the first problems that would have to be addressed for export with the current g++ implementation ? Or summarise the steps that would have to be taken for a first attempt at export ? Did someone here said they tried some things before ? Thank you, Timothy Madden
Re: Support for export keyword to use with C++ templates ?
On Mon, Feb 1, 2010 at 2:38 AM, Paolo Carlini wrote: > On 02/01/2010 01:26 AM, Timothy Madden wrote: [...] > As I see the issue, you should first check over the next months that the > feature is not deprecated by ISO. I know, I tried to talk about it on std.c++. I am afraid I can not see a consensus in the end, still I would not say the committee will just remove it. They were strongly against deprecation in 2003. > Then start learning about the > internals of GCC and eventually propose a detailed plan explaining how > you want to attack the problem, because before that it's extremely > unlikely that the C++ front-end maintainers could even consider > reviewing patches from a novice for such an hard to implement feature. Yes it is. I wish there were a nice and easy way about it, but an attempt at export is something quite big ... > That said, if you *really* plan contributing to GCC, maybe outside > export first (which seems a terribly good idea to me) first and > foremost, read the relevant web page: > > http://gcc.gnu.org/contribute.html > > and start immediately the paperwork for the Copyright assignment, > because it takes time. I see that what I need is an assignment for all future changes. If my employer is not involved with any contributions of mine, the employer disclaimer is not needed, right ? It is going to be a while until I will be able to submit any kind of patch; should I start with the paperwork ? Thank you, Timothy Madden
Re: Support for export keyword to use with C++ templates ?
On Wed, Feb 3, 2010 at 4:19 AM, Richard Kenner wrote: >> I see that what I need is an assignment for all future changes. If my >> employer is not involved with any contributions of mine, the employer >> disclaimer is not needed, right ? > > It's safest to have it. The best way to prove that your employer is > not involved with any contributions of yours is with such a disclaimer. The term "employer" is inappropriate for legal purposes in my case as I am a self-employed individual or a freelancer. I do not know the correct wording in English for this but I am legally licensed as an individual to offer software consulting services and to develop software. So the so-called employer is really my client, with whom I have signed a contract for consulting services. What legal papers would I need for this ? Thank you, Timothy Madden
-mfpmath=sse,387 is experimental ?
Hello Is -mfpmath=both for i386 and x86-64 still experimental in gcc 4.3, as the in the online manual page ? `sse,387'`sse+387'`both'Attempt to utilize both instruction sets at once. This effectively double the amount of available registers and on chips with separate execution units for 387 and SSE the execution resources too. Use this option with care, as it is still experimental, because the GCC register allocator does not model separate functional units well resulting in instable performance. Thank you, Timothy Madden
Re: -mfpmath=sse,387 is experimental ?
On Mon, Mar 9, 2009 at 11:13 AM, Paolo Bonzini wrote: > Timothy Madden wrote: >> Hello >> >> Is -mfpmath=both for i386 and x86-64 still experimental in gcc 4.3, as >> the in the online manual page ? > > Yes. It might (*might*) be better in GCC 4.4 thanks to the new register > allocator, but it's unlikely that the manual page will be changed before > the release. Thank you, I shall wait then. Timothy Madden
Re: -mfpmath=sse,387 is experimental ?
On Thu, Mar 12, 2009 at 1:15 AM, Jan Hubicka wrote: >> Timothy Madden wrote: >> > Hello >> > >> > Is -mfpmath=both for i386 and x86-64 still experimental in gcc 4.3, as >> > the in the online manual page ? [...] > > The fundamental problem here is that backend lies to compiler about the > fact that FP operation can not take one operand from SSE and other from > X87. This is something I want to look into once I have more time. With > new RA, perhaps we can drop all these fake constraints. That would be great ! I am sure having twice the number of registers (sse+387) would make a big difference. Even if SSE and FPU instructions set can not mix operands, using both at the same time (each with its registers) will be an improvement. Until then I would have a question: if I compile with -msse than using -mfpmath=387 would help floating-point operations not steal SSE registers that are already used by CPU operations ? And using -mfpmath=sse would make FPU and CPU share the SSE registers and compete on them ? How would I know if my AMD Sempron 2200+ has separate execution units for SSE and FPU instructions, with independent registers ? Thank you, Timothy Madden
Re: Support for export keyword to use with C++ templates ?
On 25.01.2010 20:12, Ian Lance Taylor wrote: Timothy Madden writes: [...] g++ is free software. A clean implementation of export would certainly be accepted. All it takes is for somebody to write one. Hello Is that statement above still true please ? I know export is now to be made deprecated by the next version of the C++ standard, but I am still interested in what does it take to implement it in gcc and how would a possible design look like. For this, I remember that in times long forgotten someone on this list actually tried to negotiate implementing export with some potential "client" or interested party. They must have quickly gave up back then since to my knowledge export has not been approached in gcc yet. But if you please have or you can remember any design approach/decisions for export made then, it will very much help me to know about it. Than I would like to get your opinion if possible with the sort of "design" I could come up with at a first attempt. First, I am interested in a design for export that does not require sources to be available for the instantiation phase. For this, .cc files would compile into several files file: - the regular object file .obj - some shared, project-global .sym file with a repository of immutable (read-only) symbols from the symbol table, for symbols encountered throughout the entire program. - some .tpl file with the parse tree for the template definitions, together with references (into the .sym file) for all symbols in the symbol table at the point of definition - some .tpi file listing the template instantiations (usages), or the s with references to the entire symbols table at every point of instantiation. This references are indexes into the symbols repository. The above are generated at translation phase, which needs to prepare everything for instantiation. After a complete compilation, the symbols repository will most likely collect symbols from the entire project, including static symbols. Each symbol in the repository has to be properly indexed and versioned, to allow both complete and incomplete definitions for the same symbol, and to allow multiple static symbols with the same name. Each symbol should also include a list of .tpl and .tpi files that reference them. In this way when a source file is re-compiled and it no longer uses a given symbol, it can be removed from the repository. Its index/version number is not re-used. The repository is extended/updated as each source file is being compiled/recompiled when the .tpl/.tpi files are generated. Any symbol referenced from the .tp? files is added to the repository if not already there. The instantiation phase then needs only the above three files (.sym, .tpl, .tpi), and from them it generates some additional object code for the linker. When instantiation triggers other instantiations, multiple .tpl files join the context. This is where the look-up in multiple symbol tables has to be done. It is here when the ODR has to be checked. Every template that gets instantiated keeps a record of all symbols that were looked up for it, together with the look-up result for each. I will need some symbol look-up log file .slk for that. Think of it like a replication log file for database replication. Then, when an instantiation is encountered for the same template-id (same template, same arguments), the template need not be re-compiled, but the sequence of symbol look-ups has to be replicated, to ensure the same results are produced at the new point of instantiation. If look-up results differ, a smart error message (suggestions welcome) will be output. This is where I got until now. My problems now are: - the old parallel-compilation-problem. A shared repository can not really be accessed by multiple gcc instances without database-like locking, which I think is not the domain of gcc. So I think I could allow like 4 such repositories to be created and used in parallel, for a quad-core CPU , and have the fifth gcc instance that fires up enter a locking wait state based on a lockfile, until one of the other four is done. - the repository file size. I hear pre-compiled header files are 50 to 200 MB in size. Since the repository includes symbols from the entire program, not only standard headers, I expect it will be up to twice as large. I guess each client will have to decide for themselves if this size is a good price to pay for the export feature. - it is still not clear to me how to keep the repository up-to-date when only some of the source files change. - I would like to know if it would be useful for all these files types to have a public format, so
