Re: [PATCH] Postpone __LINE__ evaluation to the end of #line directives

[Max Woodbury]

From 6c95593f684c120a0ea7ef6178401283f63250b7 Mon Sep 17 00:00:00 2001
From: Max TenEyck Woodbury 
Date: Sun, 24 Nov 2013 09:48:09 -0500
Subject: [PATCH] Postpone __LINE__ evaluation to the end of #line directives
To: gcc-patches@gcc.gnu.org

Copyright 2013 assigned to the Free Software Foundation.
---
 gcc/testsuite/gcc.dg/cpp/line4.c | 19 +++
 libcpp/directives.c  |  9 -
 libcpp/internal.h|  1 +
 libcpp/macro.c   |  3 +++
 4 files changed, 27 insertions(+), 5 deletions(-)

diff --git a/gcc/testsuite/gcc.dg/cpp/line4.c 
b/gcc/testsuite/gcc.dg/cpp/line4.c

index 84dbf96..0120a2b 100644
--- a/gcc/testsuite/gcc.dg/cpp/line4.c
+++ b/gcc/testsuite/gcc.dg/cpp/line4.c
@@ -13,7 +13,18 @@ enum { i = __LINE__ };
 enum { j = __LINE__ };

 #line 16  /* N.B. the _next_ line is line 16.  */
-
-char array1[i== 44 ? 1 : -1];
-char array2[j== 90 ? 1 : -1];
-char array3[__LINE__ == 19 ? 1 : -1];
+ /* __LINE__ should 
be 16 */
+char array1[i== 44 ? 1 : -1]; 
 /* 17 */
+char array2[j== 90 ? 1 : -1]; 
 /* 18 */
+char array3[__LINE__ == 19 ? 1 : -1]; 
 /* 19 */
+ 
 /* 20 */
+# line __LINE__ /* N.B. the __LINE__ sequence should _not_ change here. 
 */
+ 
 /* 22 */
+char array4[__LINE__ == 23 ? 1: -1]; 
 /* 23 */
+char array5[__LINE__ == 23 ? -1: 1]; 
 /* 24 */
+ 
 /* 25 */
+# line __LINE__ /* N.B. nor should a multi-line comment change the fact 

+   that the __LINE__ sequence should _not_ change here. 
*/
+ 
 /* 28 */
+char array6[__LINE__ == 29 ? 1: -1]; 
 /* 29 */
+char array7[__LINE__ == 27 ? -1: 1]; 
 /* 30 */

diff --git a/libcpp/directives.c b/libcpp/directives.c
index 65b2034..adb04a5 100644
--- a/libcpp/directives.c
+++ b/libcpp/directives.c
@@ -900,7 +900,9 @@ do_line (cpp_reader *pfile)
   bool wrapped;

   /* #line commands expand macros.  */
+  ++pfile->state.in_directive;  /* Request special __LINE__ 
handling. */

   token = cpp_get_token (pfile);
+  --pfile->state.in_directive;  /* Cancel 
request */

   if (token->type != CPP_NUMBER
   || strtolinenum (token->val.str.text, token->val.str.len,
   &new_lineno, &wrapped))
@@ -914,7 +916,9 @@ do_line (cpp_reader *pfile)
   return;
 }

-  if (CPP_PEDANTIC (pfile) && (new_lineno == 0 || new_lineno > cap || 
wrapped))

+  if (CPP_PEDANTIC (pfile) && (new_lineno == 0
+  || (new_lineno > cap && new_lineno != CUR__LINE__)
+  || wrapped))
 cpp_error (pfile, CPP_DL_PEDWARN, "line number out of range");
   else if (wrapped)
 cpp_error (pfile, CPP_DL_WARNING, "line number out of range");
@@ -936,6 +940,9 @@ do_line (cpp_reader *pfile)
 }

   skip_rest_of_line (pfile);
+  if ( new_lineno == CUR__LINE__ )  /* Postponed 
evaluation ? */

+new_lineno = linemap_get_expansion_line (pfile->line_table,
+ 
pfile->line_table->highest_line);

   _cpp_do_file_change (pfile, LC_RENAME_VERBATIM, new_file, new_lineno,
   map_sysp);
 }
diff --git a/libcpp/internal.h b/libcpp/internal.h
index 5321458..268de86 100644
--- a/libcpp/internal.h
+++ b/libcpp/internal.h
@@ -604,6 +604,7 @@ cpp_in_primary_file (cpp_reader *pfile)
 {
   return pfile->line_table->depth == 1;
 }
+#define CUR__LINE__ -1U

 /* In macro.c */
 extern void _cpp_free_definition (cpp_hashnode *);
diff --git a/libcpp/macro.c b/libcpp/macro.c
index e359d15..47e41b6 100644
--- a/libcpp/macro.c
+++ b/libcpp/macro.c
@@ -309,6 +309,9 @@ _cpp_builtin_macro_text (cpp_reader *pfile, 
cpp_hashnode *node)

   /* If __LINE__ is embedded in a macro, it must expand to the
 line of the macro's invocation, not its definition.
 Otherwise things like assert() will not work properly.  */
+  if (pfile->state.in_directive > 1)   /* In #line 
directive? */
+number = CUR__LINE__;  /* yes, postpone the 
lookup... */

+  else
   number = linemap_get_expansion_line (pfile->line_table,
   CPP_OPTION (pfile, traditional)
   ? 
pfile->line_table->highest_line

--
1.8.0.rc0.18.gf84667d

Re: [PATCH] Postpone __LINE__ evaluation to the end of #line directives

[Max Woodbury]

On 11/27/2013 05:46 AM, Marek Polacek wrote:> On Wed, Nov 27, 2013 at 
05:29:22AM -0500, mtewoodb...@gmail.com wrote:

>> From: Max TenEyck Woodbury 
>
> This patch is badly missing a description.  You also want to mention
> the PR number, if this fixes a bug.  I guess this is to fix PR58687.
>

I am new to GNU patching.  There is some description on Bugzilla.  At
least one other group I have worked with disliked bug number references
in patches.

Please point me to the formatting requirements for change log entries...

On 11/27/2013 11:21 AM, Joseph S. Myers wrote:

On Wed, 27 Nov 2013, mtewoodb...@gmail.com wrote:


Copyright 2013 assigned to the Free Software Foundation.


FWIW I don't see this in copyright.list yet.  If you sent the paperwork
(whether paper mail or scans) to the FSF over a week ago and haven't had
it acknowledged, please chase up ass...@gnu.org, and keep chasing them up
weekly until it's acknowledged and a maintainer confirms it's now listed
in copyright.list.


I am a new at GNU patching.  I had hoped that a simple assignment in
the document would be sufficient.  I really do not want to turn
Everything I do over to FSF...




diff --git a/gcc/testsuite/gcc.dg/cpp/line4.c b/gcc/testsuite/gcc.dg/cpp/line4.c


I think it's best to leave this test as-is and add a new test line9.c
specifically for this case of "#line __LINE__".  Changing existing tests
unnecessarily complicates establishing whether a newly seen failure is a
regression or not.


HMM?  Not sure I understand that line of reasoning.  The test is about
setting the line number.  This just adds the __LINE__ special case.



I think it would be worth including tests where the __LINE__ expansion
(which as you note should be the line number of the line *after* the
directive) is involved in concatenation or stringized, e.g.

#define xstr(x) #x
#define str(x) xstr(x)
#line 1 str(__LINE__)

which should set __FILE__ to be the stringized number of the line after
the directive.  (I hope these cases will just work given your patch, so
this is just a matter of adding more to the testcase.)



This patch would not do that.  It only fixes the use of __LINE__ when
used to reset the line number.  There are at least two ways this could
be implemented and keep with the standard:

One would be to set the new line value as soon as it is seen in the
'# line' directive and let the normal __LINE__ tracking take care of
the end-of-line increments.  That would require a larger change than
this patch and would have implications when it comes to error handling.

The other is to postpone macro expansion until the end of the directive
has been seen.  That would require two passes over the token sequence;
one to build the sequence and a second to evaluate it.  Doing that is
'the right thing' in my opinion but it should be applied to all the
other directives that permit  substitution as well.  I tried
to do it that way initially, but that requires a deeper understanding
of the current implementation than I was able to build quickly.  I
would like to see a 'theory of operation' document that contained such
details, as some commercial software shops require, but that does not
seem to be how it is done here.

So I punted...

Re: [PATCH] Postpone __LINE__ evaluation to the end of #line directives

[Max Woodbury]

On 11/27/2013 04:10 PM, Joseph S. Myers wrote:

On further consideration, I'm not convinced there's a bug here at all; I
don't think it's sufficiently defined in the standard what the current
token is for the purposes of line numbering when __LINE__ gets expanded to
be able to say that one or the other value in a directive is wrong.  As
far as I can tell after checking the reflector email archives, the only
time WG14 considered anything vaguely relevant was C90 DR#173, which
didn't receive a response.

is a statement by the standard editor that some cases were deliberately
left ambiguous.  So that suggests the bug should be closed as INVALID (but
if you think this case should be fully defined, you could always file a DR
with WG14).


PLEASE think about this a bit more.

There should be a way to change the __FILE__ value without changing the
line number sequencing.  Whatever that mechanism is, it should NOT
introduce maintenance problems that involve counting lines of code.

A little Googeling quickly turns up examples that make it clear that:

#line __LINE__ "new__FILE__value"

is that expected mechanism,

A little additional thought should make it clear that multi-line
comments should NOT screw up resetting the line number,  They are
supposed to be removed in a translation phase before the phase where
directives are identified.

There is also the way the standard defines the two fixed forms for
'#line' first and then allows for  substitution for cases
that do not match the prescribed forms as a separate form.  All three
forms require that the  be seen before the directive is
processed which means any substitution for __LINE__ should only take
place AFTER the  that ends the directive has been seen.

In other words, if you processed the text in multiple phases the way
the standard requires, you would not substitute the value for the
__LINE__ token until after the end of the directive has been seen.
Thus the problem only arises because this implementation folds the
translation phases into a single pass over the text and takes an
improper short-cut as it does so.  The standard explicitly warns
against this kind of mistake.

It looks to me like WG14 tried quite hard to make the expected form
work the way most people expect it to.

So I have STRONG objections to calling the bug report 'INVALID'!

Fwd: Re: [PATCH] Postpone __LINE__ evaluation to the end of #line directives

[Max Woodbury]

 Original Message 
Subject: Re: [PATCH] Postpone __LINE__ evaluation to the end of #line 
directives

Date: Thu, 28 Nov 2013 17:32:41 -0500
From: Max Woodbury 
To: Joseph S. Myers 

On 11/28/2013 11:34 AM, Joseph S. Myers wrote:

On Wed, 27 Nov 2013, Max Woodbury wrote:


There should be a way to change the __FILE__ value without changing the
line number sequencing.  Whatever that mechanism is, it should NOT
introduce maintenance problems that involve counting lines of code.


I think that #line is mainly intended for use by code generators that
generate C code, rather than directly by people writing C programs.  Such
a code generator can easily manage counting lines of code.


A little Googeling quickly turns up examples that make it clear that:

 #line __LINE__ "new__FILE__value"

is that expected mechanism,


You'll find any number of examples online based on misconceptions about
the C languages, possibly together with what one particular implementation
does.  Any recommendation to do things based on an area where the editor
of the standard has said the ambiguity in the standard is deliberate is
clearly a bad recommendation.  Recommendations on use of C should be based
on areas where the standard is clear and implementations agree.


Please try not to be deliberately obstructive.  While #line is indeed
used extensively by code generators to map generated code back to the
source code used by the generator, other uses are possible, and the
expectations associated with those uses are worthy of serious
consideration.  '#line __LINE__' is indeed a common idiom and it is
expected to leave the line numbering sequence unchanged.

As for the sequence of comments you point to, they are discussing the
use of __LINE__ in macros, not directives.  The standard is quite a bit
more explicit about token substitution in directives, making it fairly
clear that substitution is not to occur in directives until
specifically called for.  The elaboration of three distinct forms for
the '#line' directive with substitution only being called for in the
third and last form, indicates that something special is intended.

The standard was not created in a vacuum.  The ideas did not
materialize out of thin air.  The elaborate specification was intended
to codify actual usage.  That usage included the '#line __LINE__' idiom
with its intent to NOT break line sequencing.


In other words, if you processed the text in multiple phases the way
the standard requires, you would not substitute the value for the
__LINE__ token until after the end of the directive has been seen.
Thus the problem only arises because this implementation folds the
translation phases into a single pass over the text and takes an
improper short-cut as it does so.  The standard explicitly warns
against this kind of mistake.


The standard itself mixes up the phases.  Recall that the definition of
line number is "one greater than the number of new-line characters read or
introduced in translation phase 1 (5.1.1.2) while processing the source
file to the current token" (where "current token" is never defined).  If
the phases were completely separate, by your reasoning every newline has
been processed in phase 1 before any of phases 2, 3 or 4 do anything, and
so all line numbers relate to the end of the file.  There is absolutely
nothing to say that the newline at the end of the #line directive has been
read "while processing the source file to the current token" (if __LINE__
in the #line directive is the current token) but that the newline after it
hasn't been read; if anything, the phases imply that all newlines have
been read.


The standard also includes a mechanism for encoding s seen
in tokens, so that argument falls apart fairly easily.


This case is just as ambiguous as the case of a multi-line macro call,
where __LINE__ gets expanded somewhere in the macro arguments, and the
line number can be that of the macro name, or of the closing parenthesis
of the call, or somewhere in between, and the standard does not make a
conformance distinction between those choices.

So, I don't think we should make complicated changes to implement one
particular choice in an area of deliberate ambiguity without direction
from WG14 to eliminate the ambiguity in the standard.  Instead, we can let
the choices be whatever is most natural in the implementation.  If you
believe the standard is defective in not defining certain things, I advise
filing a DR (or, when next open for revisions, proposing a paper at a
meeting to change the definition as you think appropriate).


As pointed out above, this case is distinct from the macro CALL case.
The rules are much more explicitly spelled out for directives and is
only ambiguous if you start with the preconceived notion that it is.
The standard is explicit enough as it stands.

Further, the changes are not all that complicated.  O

[PATCH] Fix #line __LINE__ handling and added conformance tests.

[Max Woodbury]

>From 62ab7123e73563b43f1833842a419aa66eca7ce2 Mon Sep 17 00:00:00 2001
From: Max T Woodbury 
Date: Mon, 23 Oct 2017 16:58:49 -0400

Copyright 2017 Max TenEyck Woodbury, Durham North Carolina
all rights assigned to the Free Software Foundation, Inc., 23 Oct 2017

The Problem:

There is a problem associated with writing portable code; it is sometimes
desirable to change the string returned by the __FILE__ and related macros or
diagnostic messages.  The #line directive lets you do this.  The problem is that
the #line directive also changes the sequence numbers of subsequent lines and
that may be an unwanted side effect.  The obvious solution is to put a number
into the new line number slot of the directive that will leave the line number
sequencing unchanged.  This creates a maintenance issue: any changes to the
module that changes the number of lines before the #line directive ends will
require changing the line number value in the directive.
You would think that this issue could be avoided by using the __LINE__ macro
to insert the proper number.  There are examples in the programming literature
that suggest exactly that.  Unfortunately an ambiguity in the interpreting the
standard makes the exact value generated by __LINE__ in the #line directive
implementation dependent.  One fairly obvious implementation has the pre-
processor collect the entire text of any directive together, including the '\n'
that ends the directive, before processing a directive.  Since some directives
suppress macro expansion, the expansion of the __LINE__ macro in a #line dir-
ective is postponed until the #line directive has been identified.  In that case
the __LINE__ macro expands to the number of the next input line, which is
exactly the value wanted.
Unfortunately there is another way to proceed; in this alternative, the type
of directive is identified "on the fly" and directives that allow macro
expansion also do the macro expansions "on-the-fly".  This results in __LINE__
values for the #line directive itself instead of the number for the next line,
which changes the subsequent line sequence.

The solution:

In the interest of portability, a special check has been added to the #line
directive implementation to see if the __LINE__ macro is being used to specify
the new line number.  If it is, a flag is set that suppresses the line number
change.
Note that this solution is not robust; hiding the __LINE__ reference inside
another macro will bypass this special case test.

Test cases have been added to the test suite to check this behavior.
---
 gcc/testsuite/ChangeLog  |  4 
 gcc/testsuite/gcc.dg/cpp/line4.c |  7 +++
 libcpp/ChangeLog |  4 
 libcpp/directives.c  | 12 
 4 files changed, 27 insertions(+)

diff --git a/gcc/testsuite/ChangeLog b/gcc/testsuite/ChangeLog
index f14930b..a65328f 100644
--- a/gcc/testsuite/ChangeLog
+++ b/gcc/testsuite/ChangeLog
@@ -1,3 +1,7 @@
+2017-10-22  Max T. Woodbury 
+
+* gcc.dg/cpp/line4.c: New #line __LINE__ tests.
+
 2017-10-22  Uros Bizjak  

PR target/52451
diff --git a/gcc/testsuite/gcc.dg/cpp/line4.c b/gcc/testsuite/gcc.dg/cpp/line4.c
index 84dbf96..5902046 100644
--- a/gcc/testsuite/gcc.dg/cpp/line4.c
+++ b/gcc/testsuite/gcc.dg/cpp/line4.c
@@ -17,3 +17,10 @@ enum { j = __LINE__ };
 char array1[i== 44 ? 1 : -1];
 char array2[j== 90 ? 1 : -1];
 char array3[__LINE__ == 19 ? 1 : -1];
+
+#line __LINE__ /* N.B. this should not change the line sequence numbering.  */
+char array4[__LINE__ == 22 ? 1 : -1];
+#line __LINE__ /* N.B. extra lines in block comment should
+  not change the line sequence numbering.  */
+char array5[__LINE__ == 25 ? 1 : -1];
+
diff --git a/libcpp/ChangeLog b/libcpp/ChangeLog
index f2c0d4d..31e1991 100644
--- a/libcpp/ChangeLog
+++ b/libcpp/ChangeLog
@@ -1,3 +1,7 @@
+2017-10-23  Max T. Woodbury 
+
+* directives.c: Added #line __LINE__ special case handling.
+
 2017-10-10  Nathan Sidwell  

PR preprocessor/82506
diff --git a/libcpp/directives.c b/libcpp/directives.c
index 7cac653..a6243ac 100644
--- a/libcpp/directives.c
+++ b/libcpp/directives.c
@@ -964,6 +964,13 @@ do_line (cpp_reader *pfile)
   linenum_type cap = CPP_OPTION (pfile, c99) ? 2147483647 : 32767;
   bool wrapped;

+  /* #line __LINE__ should not change the line numbers. */
+  token = cpp_peek_token (pfile, 0);
+  bool no_change = (token->type == CPP_NAME &&
+token->val.node.node->type == NT_MACRO &&
+token->val.node.node->flags & NODE_BUILTIN &&
 char array1[i== 44 ? 1 : -1];
 char array2[j== 90 ? 1 : -1];
 char array3[__LINE__ == 19 ? 1 : -1];
+
+#line __LINE__ /* N.B. this should not change the line sequence numbering.  */
+char array4[__LINE__ == 22 ? 1 : -1];
+#line __LINE__ /* N.B. extra lines in block comment should
+  not change the line sequence numbering.  */
+char array5[__LINE__ == 25 ? 1 : -1];
+
d