from:"Dora, Sunil Kumar"

PING ^1 [PATCH] GCC Driver : Enable very long gcc command-line option

2024-08-27 Thread Dora, Sunil Kumar

Dear GCC Team,

Please consider this as a gentle reminder to review the patch I posted at the 
following link: [ 
https://gcc.gnu.org/pipermail/gcc-patches/2024-August/660223.html ].

BUG Link : [ https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527 ]

Your feedback or approval would be greatly appreciated.

Best Regards,
Sunil Dora

On 2024-08-13 2:30 a.m., 
deepthi.hem...@windriver.com wrote:

From: sunil dora 

For excessively long environment variables i.e >128KB
Store the arguments in a temporary file and collect them back together in 
collect2.

This commit patches for COLLECT_GCC_OPTIONS issue:
GCC should not limit the length of command line passed to collect2.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527

The Linux kernel has the following limits on shell commands:
I.  Total number of bytes used to specify arguments must be under 128KB.
II. Each environment variable passed to an executable must be under 128 KiB

In order to circumvent these limitations, many build tools support
response-files, i.e. files that contain the arguments for the executed
command. These are typically passed using @ syntax.

Gcc uses the COLLECT_GCC_OPTIONS environment variable to transfer the
expanded command line to collect2. With many options, this exceeds the limit II.

GCC : Added Testcase for PR111527

TC1 : If the command line argument less than 128kb, gcc should use
  COLLECT_GCC_OPTION to communicate and compile fine.
TC2 : If the command line argument in the range of 128kb to 2mb,
  gcc should copy arguments in a file and use FILE_GCC_OPTIONS
  to communicate and compile fine.
TC3 : If the command line argument greater thean 2mb, gcc shuld
  fail the compile and report error. (Expected FAIL)

Signed-off-by: Topi Kuutela 

Signed-off-by: sunil dora 


---
 
gcc/collect2.cc
   | 40 +++--
 
gcc/gcc.cc
| 37 +--
 gcc/testsuite/gcc.dg/longcmd/longcmd.exp  | 16 +
 gcc/testsuite/gcc.dg/longcmd/pr111527-1.c | 44 +++
 gcc/testsuite/gcc.dg/longcmd/pr111527-2.c |  9 +
 gcc/testsuite/gcc.dg/longcmd/pr111527-3.c | 10 ++
 gcc/testsuite/gcc.dg/longcmd/pr111527-4.c | 10 ++
 7 files changed, 160 insertions(+), 6 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/longcmd/longcmd.exp
 create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-1.c
 create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-2.c
 create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-3.c
 create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-4.c

diff --git 
a/gcc/collect2.cc
 
b/gcc/collect2.cc
index 902014a9cc1..564d8968648 100644
--- 
a/gcc/collect2.cc
+++ 
b/gcc/collect2.cc
@@ -376,6 +376,40 @@ typedef int scanfilter;

 static void scan_prog_file (const char *, scanpass, scanfilter);

+char* getenv_extended (const char* var_name)
+{
+  int file_size;
+  char* buf = NULL;
+
+  char* string = getenv (var_name);
+  if (!string)
+{
+  char* string = getenv ("FILE_GCC_OPTIONS");
+  FILE *fptr;
+  fptr = fopen (string, "r");
+  if (fptr == NULL)
+   return (0);
+  /* Copy contents from temporary file to buffer */
+  if (fseek (fptr, 0, SEEK_END) == -1)
+   return (0);
+  file_size = ftell (fptr);
+  rewind (fptr);
+  buf = (char *) xmalloc (file_size + 1);
+  if (buf == NULL)
+   return (0);
+  if (fread ((void *) buf, file_size, 1, fptr) <= 0)
+   {
+ free (buf);
+ fatal_error (input_location, "fread failed");
+ return (0);
+   }
+  buf[file_size] = '\0';
+  return buf;
+}
+  return string;
+}
+
+

 /* Delete tempfiles and exit function.  */

@

Re: PING ^1 [PATCH] GCC Driver : Enable very long gcc command-line option

2024-08-30 Thread Dora, Sunil Kumar

Thank you, Richard, for your feedback. I have removed the FILE_GCC_OPTION and 
used COLLECT_GCC_OPTION exclusively to manage the long command line option. The 
revised V2 patch has been posted and can be found at the following link:

https://gcc.gnu.org/pipermail/gcc-patches/2024-August/661842.html

I would appreciate it if you could review the updated patch.


Best Regards,
Sunil Dora

From: Richard Biener 
Sent: Tuesday, August 27, 2024 6:09 PM
To: Dora, Sunil Kumar 
Cc: gcc-patches@gcc.gnu.org ; pins...@gmail.com 
; jeffreya...@gmail.com ; 
josmy...@redhat.com ; Hemraj, Deepthi 
; MacLeod, Randy ; 
Gowda, Naveen ; Kokkonda, Sundeep 

Subject: Re: PING ^1 [PATCH] GCC Driver : Enable very long gcc command-line 
option

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.

On Tue, 27 Aug 2024, Dora, Sunil Kumar wrote:

> Dear GCC Team,
>
> Please consider this as a gentle reminder to review the patch I posted at the 
> following link: [ 
> https://gcc.gnu.org/pipermail/gcc-patches/2024-August/660223.html ].
>
> BUG Link : [ https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527 ]
>
> Your feedback or approval would be greatly appreciated.
>
> Best Regards,
> Sunil Dora
>
> On 2024-08-13 2:30 a.m., 
> deepthi.hem...@windriver.com<mailto:deepthi.hem...@windriver.com> wrote:
>
> From: sunil dora <mailto:sunil.dora1...@gmail.com>
>
> For excessively long environment variables i.e >128KB
> Store the arguments in a temporary file and collect them back together in 
> collect2.
>
> This commit patches for COLLECT_GCC_OPTIONS issue:
> GCC should not limit the length of command line passed to collect2.
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527<https://urldefense.com/v3/__https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527__;!!AjveYdw8EvQ!ZKeP8wCVwVcrJUs6gQlE4qeAjJ6qNDbZLBt8Smt01dIdiPvTI9X0LL6LPKtXkQtKzKRsfIVRqXGkPbj4CvwvC8ecOLOVBw$>
>
> The Linux kernel has the following limits on shell commands:
> I.  Total number of bytes used to specify arguments must be under 128KB.
> II. Each environment variable passed to an executable must be under 128 KiB
>
> In order to circumvent these limitations, many build tools support
> response-files, i.e. files that contain the arguments for the executed
> command. These are typically passed using @ syntax.
>
> Gcc uses the COLLECT_GCC_OPTIONS environment variable to transfer the
> expanded command line to collect2. With many options, this exceeds the limit 
> II.
>
> GCC : Added Testcase for PR111527
>
> TC1 : If the command line argument less than 128kb, gcc should use
>   COLLECT_GCC_OPTION to communicate and compile fine.
> TC2 : If the command line argument in the range of 128kb to 2mb,
>   gcc should copy arguments in a file and use FILE_GCC_OPTIONS
>   to communicate and compile fine.
> TC3 : If the command line argument greater thean 2mb, gcc shuld
>   fail the compile and report error. (Expected FAIL)

Just as a random comment - I'd prefer if COLLECT_GCC_OPTIONS would
allow response files instead of indirecting via a new fixed other
environment like the proposed FILE_GCC_OPTIONS.

That looks like a cleaner interface to me.

Richard.

> Signed-off-by: Topi Kuutela 
> <mailto:topi.kuut...@nokia.com>
> Signed-off-by: sunil dora 
> <mailto:sunil.dora1...@gmail.com>
>
> ---
>  
> gcc/collect2.cc<https://urldefense.com/v3/__http://collect2.cc__;!!AjveYdw8EvQ!ZKeP8wCVwVcrJUs6gQlE4qeAjJ6qNDbZLBt8Smt01dIdiPvTI9X0LL6LPKtXkQtKzKRsfIVRqXGkPbj4CvwvC8f5x6pChg$>
>| 40 +++--
>  
> gcc/gcc.cc<https://urldefense.com/v3/__http://gcc.cc__;!!AjveYdw8EvQ!ZKeP8wCVwVcrJUs6gQlE4qeAjJ6qNDbZLBt8Smt01dIdiPvTI9X0LL6LPKtXkQtKzKRsfIVRqXGkPbj4CvwvC8fDr-4p_g$>
> | 37 +--
>  gcc/testsuite/gcc.dg/longcmd/longcmd.exp  | 16 +
>  gcc/testsuite/gcc.dg/longcmd/pr111527-1.c | 44 +++
>  gcc/testsuite/gcc.dg/longcmd/pr111527-2.c |  9 +
>  gcc/testsuite/gcc.dg/longcmd/pr111527-3.c | 10 ++
>  gcc/testsuite/gcc.dg/longcmd/pr111527-4.c | 10 ++
>  7 files changed, 160 insertions(+), 6 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/longcmd.exp
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-1.c
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-2.c
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-3.c
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-4.c
>
> diff --git 
> a/gcc/collect2.cc<https://urldefense.com/v3/__http://collect2.cc__;!!AjveYdw8EvQ!ZKeP8wCVwVcrJUs6gQlE4qeAjJ6qNDbZLBt8Smt01dIdiPvT

Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

2024-09-06 Thread Dora, Sunil Kumar

Hi Andrew,

Thank you for your feedback. Initially, we attempted to address the issue by 
utilizing GCC’s response files. However, we discovered that the 
COLLECT_GCC_OPTIONS variable already contains the expanded contents of the 
response files.

As a result, using response files only mitigates the multiplication factor but 
does not bypass the 128KB limit.
I have included the response file usage logs and the complete history in the 
Bugzilla report for your reference: Bugzilla 
Link<https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527#c19>.
Following your suggestion, I have updated the logic to avoid hardcoding /tmp.
Please find the revised version of patch at the following link:

https://gcc.gnu.org/pipermail/gcc-patches/2024-September/662519.html

Thanks,
Sunil Dora

From: Andrew Pinski 
Sent: Friday, August 30, 2024 8:05 PM
To: Hemraj, Deepthi 
Cc: gcc-patches@gcc.gnu.org ; rguent...@suse.de 
; jeffreya...@gmail.com ; 
josmy...@redhat.com ; MacLeod, Randy 
; Gowda, Naveen ; 
Dora, Sunil Kumar 
Subject: Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.

On Fri, Aug 30, 2024 at 12:34 AM  wrote:
>
> From: Deepthi Hemraj 
>
> For excessively long environment variables i.e >128KB
> Store the arguments in a temporary file and collect them back together in 
> collect2.
>
> This commit patches for COLLECT_GCC_OPTIONS issue:
> GCC should not limit the length of command line passed to collect2.
> https://gcc.gnu.org/bugzilla/show_bug.cgi?id=111527
>
> The Linux kernel has the following limits on shell commands:
> I.  Total number of bytes used to specify arguments must be under 128KB.
> II. Each environment variable passed to an executable must be under 128 KiB
>
> In order to circumvent these limitations, many build tools support
> response-files, i.e. files that contain the arguments for the executed
> command. These are typically passed using @ syntax.
>
> Gcc uses the COLLECT_GCC_OPTIONS environment variable to transfer the
> expanded command line to collect2. With many options, this exceeds the limit 
> II.
>
> GCC : Added Testcase for PR111527
>
> TC1 : If the command line argument less than 128kb, gcc should use
>   COLLECT_GCC_OPTION to communicate and compile fine.
> TC2 : If the command line argument in the range of 128kb to 2mb,
>   gcc should copy arguments in a file and use FILE_GCC_OPTIONS
>   to communicate and compile fine.
> TC3 : If the command line argument greater thean 2mb, gcc shuld
>   fail the compile and report error. (Expected FAIL)
>
> Signed-off-by: sunil dora 
> Signed-off-by: Topi Kuutela 
> Signed-off-by: Deepthi Hemraj 
> ---
>  gcc/collect2.cc   | 39 ++--
>  gcc/gcc.cc| 37 +--
>  gcc/testsuite/gcc.dg/longcmd/longcmd.exp  | 16 +
>  gcc/testsuite/gcc.dg/longcmd/pr111527-1.c | 44 +++
>  gcc/testsuite/gcc.dg/longcmd/pr111527-2.c |  9 +
>  gcc/testsuite/gcc.dg/longcmd/pr111527-3.c | 10 ++
>  gcc/testsuite/gcc.dg/longcmd/pr111527-4.c | 10 ++
>  7 files changed, 159 insertions(+), 6 deletions(-)
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/longcmd.exp
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-1.c
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-2.c
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-3.c
>  create mode 100644 gcc/testsuite/gcc.dg/longcmd/pr111527-4.c
>
> diff --git a/gcc/collect2.cc b/gcc/collect2.cc
> index 902014a9cc1..1f56963b1ce 100644
> --- a/gcc/collect2.cc
> +++ b/gcc/collect2.cc
> @@ -376,6 +376,39 @@ typedef int scanfilter;
>
>  static void scan_prog_file (const char *, scanpass, scanfilter);
>
> +char* getenv_extended (const char* var_name)
> +{
> +  int file_size;
> +  char* buf = NULL;
> +  const char* prefix = "/tmp";
> +
> +  char* string = getenv (var_name);
> +  if (strncmp (var_name, prefix, strlen(prefix)) == 0)

This is not what was meant by saying using the same env and supporting
response files.
Instead what Richard meant was use `@file` as the option that gets
passed via COLLECT_GCC_OPTIONS and then if you see `@` expand the
options like what is done for the normal command line.
Hard coding "/tmp" here is wrong because TMPDIR might not be set to
"/tmp" and even more with -save-temps, the response file should stay
around afterwards and be in the working directory rather than TMPDIR.

Thanks,
Andrew Pinski

> +{
> +  FILE *fptr;
> +  fptr = fopen (string, "r");
> +  if (fptr =

Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

2024-12-10 Thread Dora, Sunil Kumar

Hi Andrew,

"Apologies, I just realized that my previous reply may not have gone through. 
I'm resending it for your reference."

Thank you for your response. I just wanted to clarify the issue I’m facing. The 
concern is not that new files are being generated,
but rather that the LTO debug files are not being generated at all when using 
the response file format inside
the set_collect_gcc_option(). (Attached : response_f.patch )
I understand that you’re suggesting adjusting the test case to accommodate the 
new files, but in this case,
The files themselves aren’t being created to begin with when following this 
approach.

For example : With host gcc if i pass "-flto" flag
$ gcc -O2 -flto -flto-partition=one -fdump-ipa-icf-optimized -fdump-rtl-final 
-fstack-usage a.c
a.c  a.c.079i.icf  a.ltrans0.ltrans.337r.final  a.ltrans0.ltrans.su  a.out  
a.wpa.079i.icf
The above debug files are not generating with Attached : response_f.patch 
approach.

Could you please advise on how we might resolve this, or if there’s something 
we might be missing in the configuration to
ensure the debug files are generated?

Moving the response file creation to xputenv resolves this. Please see 
wrapper_lto.patch.
Would it be appropriate to send this patch? Please advise.

$make check-gcc RUNTESTFLAGS="outputs.exp"
=== gcc Summary ===
# of expected passes2659
/home/sunild/snap/LTO_WRAPPER_gcc/build/gcc/xgcc  version 15.0.0 20241016 
(experimental) (GCC)
Thank you for your help!
Best regards,
Sunil Dora


From: Andrew Pinski 
Sent: Monday, November 18, 2024 11:14 AM
To: Dora, Sunil Kumar 
Cc: GCC Patches ; Richard Guenther 
; Jeff Law ; Joseph Myers 
; MacLeod, Randy ; Gowda, 
Naveen 
Subject: Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

CAUTION: This email comes from a non Wind River email account!
Do not click links or open attachments unless you recognize the sender and know 
the content is safe.


On Tue, Sep 17, 2024, 3:40 AM Dora, Sunil Kumar 
mailto:sunilkumar.d...@windriver.com>> wrote:
Hi Andrew,

Initially, I thought to address long command line options (when exceeding 
128KB) without disrupting the existing GCC driver behavior.

As you suggested, I implemented changes to use the response file format (@file) 
within the set_collect_gcc_options function and ensured that this was passed 
through COLLECT_GCC_OPTIONS.
However, these changes have introduced a side effect: they impact the behavior 
of the -save-temps switch by generating additional .args.N files. As a result, 
some existing test cases, including the one reported by the Linaro team, are 
now failing.
(File: Attached)
Could you please advise on how we should proceed? Specifically, should we 
adjust the test cases to accommodate the impact on the -save-temps switch, or 
is there an alternative approach you would recommend? Your guidance on how to 
address these issues while implementing the response file approach would be 
greatly appreciated.
Sounds like the test case need to adjusted for the new files that are saved 
now. Since we want to have this file around when using -saves-temps to be able 
to reproduce what is being invoked.

Thanks,
Andrew


Thank you for your support.



Thanks,
Sunil Dora

From: Andrew Pinski mailto:pins...@gmail.com>>
Sent: Friday, September 6, 2024 11:33 PM
To: Dora, Sunil Kumar 
mailto:sunilkumar.d...@windriver.com>>
Cc: Hemraj, Deepthi 
mailto:deepthi.hem...@windriver.com>>; GCC 
Patches mailto:gcc-patches@gcc.gnu.org>>; Richard 
Guenther mailto:rguent...@suse.de>>; Jeff Law 
mailto:jeffreya...@gmail.com>>; 
josmy...@redhat.com<mailto:josmy...@redhat.com> 
mailto:josmy...@redhat.com>>; MacLeod, Randy 
mailto:randy.macl...@windriver.com>>; Gowda, 
Naveen mailto:naveen.go...@windriver.com>>
Subject: Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

CAUTION: This email comes from a non Wind River email account!

Do not click links or open attachments unless you recognize the sender and know 
the content is safe.


On Fri, Sep 6, 2024, 9:38 AM Dora, Sunil Kumar 
mailto:sunilkumar.d...@windriver.com>> wrote:
Hi Andrew,

Thank you for your feedback. Initially, we attempted to address the issue by 
utilizing GCC’s response files. However, we discovered that the 
COLLECT_GCC_OPTIONS variable already contains the expanded contents of the 
response files.

As a result, using response files only mitigates the multiplication factor but 
does not bypass the 128KB limit.

I think you missed understood me fully. What I was saying instead of creating a 
string inside set_collect_gcc_options, create the response file and pass that 
via COLLECT_GCC_OPTIONS with the @file format. And then inside 
collect2.cc<https://urldefense.com/v3/__http://collect2.cc__;!!AjveYdw8EvQ!bB1wlMkDkX1X5ykXc7p8DUV8HqhVBevZy

PING ^1 [PATCH] GCC Driver : Enable very long gcc command-line option

Re: PING ^1 [PATCH] GCC Driver : Enable very long gcc command-line option

Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

Re: [PATCH v2] GCC Driver : Enable very long gcc command-line option

4 matches

Site Navigation

Mail list logo

Footer information