Re: [PATCH] ctf: emit CTF_K_ARRAY for GNU vector types

[Indu]

On 2025-05-01 2:34 p.m., Bruce McCulloch wrote:

Currently, there is a check in gen_ctf_array_type that prevents GNU vectors
generated by the vector attribute from being emitted (e.g. typedef int v8si
__attribute__ ((vector_size (32)));). Because this check happens in
dwarf2ctf.cc, this prevents GNU vectors from being emitted not only in CTF,
but also in BTF. This is a problem, as there are a handful of GNU vectors
present in the kernel that are not being accurately represented in the
vmlinux.{ctfa,btfa}. Additionally, BTF generated by clang emits these vectors
as arrays.

This patch solves the issue by simply removing the check that prevents
these types from being appropriately emitted. Additionally, a new test is
included that checks for the appropriate asm emission when generating CTF.



Hi Bruce,

(CC Nick)

Vector type is different from an array type.  A CTF consumer may want to 
distinguish between the two for various reasons.  I found this useful in 
this regard: https://dwarfstd.org/issues/230413.1.html.


If, for the case of BTF, it suffices to emit vectors with kind 
BTF_K_ARRAY (although I would assume BTF to have cared for the 
distinction for the same reasons as CTF..), we will need to add a new 
"internal" kind to CTF, say CTF_K_VECTOR, and not emit them in the 
output section when -gctf is in effect.  Any types, vars etc.  referring 
to the vector type will continue to be emitted as referring to a 
CTF_K_UNKNOWN, as CTF does not have representation for vector types in 
CTF V3.




gcc/ChangeLog:

* dwarf2ctf.cc (gen_ctf_array_type): Remove check for DW_AT_GNU_vector.

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-vector.c: New test.


Signed-off-by: Bruce McCulloch 
---
  gcc/dwarf2ctf.cc|  4 ---
  gcc/testsuite/gcc.dg/debug/ctf/ctf-vector.c | 32 +
  2 files changed, 32 insertions(+), 4 deletions(-)
  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-vector.c

diff --git a/gcc/dwarf2ctf.cc b/gcc/dwarf2ctf.cc
index fd326b320af..a3497d58504 100644
--- a/gcc/dwarf2ctf.cc
+++ b/gcc/dwarf2ctf.cc
@@ -417,10 +417,6 @@ gen_ctf_array_type (ctf_container_ref ctfc,
dw_die_ref first, last, array_elems_type;
ctf_dtdef_ref array_dtd, elem_dtd;
  
-  int vector_type_p = get_AT_flag (array_type, DW_AT_GNU_vector);

-  if (vector_type_p)
-return NULL;
-
/* Find the first and last array dimension DIEs.  */
last = dw_get_die_child (array_type);
first = dw_get_die_sib (last);
diff --git a/gcc/testsuite/gcc.dg/debug/ctf/ctf-vector.c 
b/gcc/testsuite/gcc.dg/debug/ctf/ctf-vector.c
new file mode 100644
index 000..368046db214
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/ctf/ctf-vector.c
@@ -0,0 +1,32 @@
+/* Tests for CTF SIMD vector type.
+   - Verify that there is a record of:
+ + int
+ + void
+ + int[8] -> int
+ + v8si -> int[8] -> int.  */
+
+/* { dg-do compile } */
+/* { dg-options "-O0 -gctf -dA" } */
+
+/* Check for presence of strings:  */
+/* { dg-final { scan-assembler-times "ascii \"int.0\"\[\t 
\]+\[^\n\]*ctf_string" 1 } } */
+/* { dg-final { scan-assembler-times "ascii \"void.0\"\[\t 
\]+\[^\n\]*ctf_string" 1 } } */
+/* { dg-final { scan-assembler-times "ascii \"v8si.0\"\[\t 
\]+\[^\n\]*ctf_string" 1 } } */
+
+/* Check for information about int.  */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0x600\[ \t\]+\[^\n\]*# 
ctt_info" 2 } } */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0x4\[ \t\]+\[^\n\]*# ctt_size or 
ctt_type" 1 } } */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0x120\[ \t\]+\[^\n\]*# 
ctf_encoding_data" 1 } } */
+
+/* Check for information about void.  */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0\[ \t\]+\[^\n\]*# ctt_size or 
ctt_type" 2 } } */
+
+/* Check for information about int[8] array.  */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0x1200\[ \t\]+\[^\n\]*# 
ctt_info" 1 } } */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0x8\[ \t\]+\[^\n\]*# 
cta_nelems" 1 } } */
+
+/* Check for information about v8si.  */
+/* { dg-final { scan-assembler-times ".long\[ \t\]+0x2a00\[ \t\]+\[^\n\]*# 
ctt_info" 1 } } */
+
+typedef int v8si __attribute__ ((vector_size (32)));
+v8si foo;

Re: Type representation in CTF and DWARF

[Indu Bhagat]

On 10/11/2019 04:41 AM, Jakub Jelinek wrote:

On Fri, Oct 11, 2019 at 01:23:12PM +0200, Richard Biener wrote:

(coreutils-0.22)
   .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | .ctf 
(uncompressed) | ratio (.ctf/(D1+D2+0.5*D4))
ls   30616   |1136   |21098   | 26240   
| 0.62
pwd  10734   |788|10433   | 13929   
| 0.83
groups 10706 |811|10249   | 13378   
| 0.80

(emacs-26.3)
   .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | .ctf 
(uncompressed) | ratio (.ctf/(D1+D2+0.5*D4))
emacs-26.3.1 674657  |6402   |   273963   |   273910
| 0.33

I chose to account for 50% of .debug_str because at this point, it will be
unfair to not account for them. Actually, one could even argue that upto 70%
of the .debug_str are names of entities. CTF section sizes do include the CTF
string tables.

Across coreutils, I see a geomean of 0.73 (ratio of
.ctf/(.debug_info + .debug_abbrev + 50% of .debug_str)). So, with the
"-gdwarf-like-ctf code stubs" and dwz, DWARF continues to have a larger
footprint than CTF (with 50% of .debug_str accounted for).

I'm not convinced this "improvement" in size is worth maintainig another
debug-info format much less since it lacks desirable features right now
and thus evaluation is tricky.

At least you can improve dwarf size considerably with a low amount of work.

I suspect another factor where dwarf is bigger compared to CTF is that dwarf
is recording typedef names as well as qualified type variants.  But maybe
CTF just has a more compact representation for the bits it actually implements.

Does CTF record automatic variables in functions, or just global variables?
If only the latter, it would be fair to also disable addition of local
variable DIEs, lexical blocks.  Does CTF record inline functions?  Again, if
not, it would be fair to not emit that either in .debug_info.
-gno-record-gcc-switches so that the compiler command line is not encoded in
the debug info (unless it is in CTF).


CTF includes file-scope and global-scope entities. So, CTF for a function
defined/declared at these scopes is available in .ctf section, even if it is
inlined.

To not generate DWARF for function-local entities, I made a tweak in the
gen_decl_die API to have an early exit when TREE_CODE (DECL_CONTEXT (decl))
is FUNCTION_DECL.

@@ -26374,6 +26374,12 @@ gen_decl_die (tree decl, tree origin, struct 
vlr_context *ctx,
   if (DECL_P (decl_or_origin) && DECL_IGNORED_P (decl_or_origin))
 return NULL;
 
+  /* Do not generate info for function local decl when -gdwarf-like-ctf is

+ enabled.  */
+  if (debug_dwarf_like_ctf && DECL_CONTEXT (decl)
+  && (TREE_CODE (DECL_CONTEXT (decl)) == FUNCTION_DECL))
+return NULL;
+
   switch (TREE_CODE (decl_or_origin))
 {
 case ERROR_MARK:


For the numbers in the email today:
1. CFLAGS="-g -gdwarf-like-ctf -gno-record-gcc-switches -O2". dwz is used on
   generated binaries.
2. At this time, I wanted to account for .debug_str entities appropriately (not
   50% as done previously). Using a small script to count chars for
   accounting the "path-like" strings, specifically those strings that start
   with a ".", I gathered the data in column named D5.

(coreutils-0.22)
 .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | path strings (D5) | 
.ctf (uncompressed) | ratio (.ctf/(D1+D2+D4-D5))
ls   14100   |994|16945   | 1328  | 
  26240 | 0.85
pwd   6341   |632| 9311   |  596  | 
  13929 | 0.88
groups 6410  |714| 9218   |  667  | 
  13378 | 0.85
Average geomean across coreutils = 0.84

(emacs-26.3)
 .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | path strings (D5) | 
.ctf (uncompressed) | ratio (.ctf/(D1+D2+D4-D5))
emacs-26.3.1 373678  |3794   |   219048   |  3842 | 
273910  | 0.46


DWARF is highly extensible format, what exactly is and is not emitted is
something that consumers can choose.
Yes, DWARF can be large, but mainly because it provides a lot of
information, the actual representation has been designed with size concerns
in mind and newer versions of the standard keep improving that too.

Jakub


Yes.

I started out to provide some numbers around the size impact of CTF vs DWARF
as it was a legitimate curiosity many of us have had. Comparing Compactness or
feature matrices is only one dimension of evaluating the utility of supporting
CTF in the toolchain (including GCC; Bintuils and GDB have already accepted
initial CTF support). The other dimension is a user friendly workflow which
supports current users and eases further adoption and growth.

Indu

Re: [PATCH] testsuite: ctf: make array in ctf-file-scope-1 fixed length

[Indu Bhagat]

On 3/1/24 11:01, David Faust wrote:

The array member of struct SFOO in the ctf-file-scope-1 caused the test
to fail for the BPF target, since BPF does not support dynamic stack
allocation. The array does not need to variable length for the sake of
the test, so make it fixed length instead to allow the test to run
successfully for the bpf-unknown-none target.

Tested on x86_64-linux-gnu, and on x86_64-linux-gnu host for
bpf-unknown-none target.



LGTM.
Thanks!


gcc/testsuite/

* gcc.dg/debug/ctf/ctf-file-scope-1.c (SFOO): Make array member
fixed-length.
---
  gcc/testsuite/gcc.dg/debug/ctf/ctf-file-scope-1.c | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/gcc/testsuite/gcc.dg/debug/ctf/ctf-file-scope-1.c 
b/gcc/testsuite/gcc.dg/debug/ctf/ctf-file-scope-1.c
index a683113e505..ddfb31da405 100644
--- a/gcc/testsuite/gcc.dg/debug/ctf/ctf-file-scope-1.c
+++ b/gcc/testsuite/gcc.dg/debug/ctf/ctf-file-scope-1.c
@@ -9,7 +9,7 @@
  
  int foo (int n)

  {
-  typedef struct { int a[n]; } SFOO;
+  typedef struct { int a[6]; } SFOO;
  
SFOO a;

__attribute__ ((noinline)) SFOO gfoo (void) { return a; }

[PATCH,V2] ctf: fix incorrect CTF for multi-dimensional array types

[Indu Bhagat]

From: Cupertino Miranda 

[Changes from V1]
  - Refactor the code a bit.
[End of changes from V1]

PR debug/114186

DWARF DIEs of type DW_TAG_subrange_type are linked together to represent
the information about the subsequent dimensions.  The CTF processing was
so far working through them in the opposite (incorrect) order.

While fixing the issue, refactor the code a bit for readability.

co-authored-By: Indu Bhagat 

gcc/
PR debug/114186
* dwarf2ctf.cc (gen_ctf_array_type): Invoke the ctf_add_array ()
in the correct order of the dimensions.
(gen_ctf_subrange_type): Refactor out handling of
DW_TAG_subrange_type DIE to here.

gcc/testsuite/
PR debug/114186
* gcc.dg/debug/ctf/ctf-array-6.c: Add test.
---

Testing notes:

Regression tested on x86_64-linux-gnu default target.
Regression tested for target bpf-unknown-none (btf.exp, ctf.exp, bpf.exp).

---
 gcc/dwarf2ctf.cc | 153 +--
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-6.c |  14 ++
 2 files changed, 84 insertions(+), 83 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-6.c

diff --git a/gcc/dwarf2ctf.cc b/gcc/dwarf2ctf.cc
index dca86edfffa9..3985de115a79 100644
--- a/gcc/dwarf2ctf.cc
+++ b/gcc/dwarf2ctf.cc
@@ -349,105 +349,92 @@ gen_ctf_pointer_type (ctf_container_ref ctfc, dw_die_ref 
ptr_type)
   return ptr_type_id;
 }
 
-/* Generate CTF for an array type.  */
+/* Recursively generate CTF for array dimensions starting at DIE C (of type
+   DW_TAG_subrange_type) until DIE LAST (of type DW_TAG_subrange_type) is
+   reached.  ARRAY_ELEMS_TYPE_ID is base type for the array.  */
 
 static ctf_id_t
-gen_ctf_array_type (ctf_container_ref ctfc, dw_die_ref array_type)
+gen_ctf_subrange_type (ctf_container_ref ctfc, ctf_id_t array_elems_type_id,
+  dw_die_ref c, dw_die_ref last)
 {
-  dw_die_ref c;
-  ctf_id_t array_elems_type_id = CTF_NULL_TYPEID;
+  ctf_arinfo_t arinfo;
+  ctf_id_t array_node_type_id = CTF_NULL_TYPEID;
+
+  dw_attr_node *upper_bound_at;
+  dw_die_ref array_index_type;
+  uint32_t array_num_elements;
+
+  /* When DW_AT_upper_bound is used to specify the size of an
+ array in DWARF, it is usually an unsigned constant
+ specifying the upper bound index of the array.  However,
+ for unsized arrays, such as foo[] or bar[0],
+ DW_AT_upper_bound is a signed integer constant
+ instead.  */
+
+  upper_bound_at = get_AT (c, DW_AT_upper_bound);
+  if (upper_bound_at
+  && AT_class (upper_bound_at) == dw_val_class_unsigned_const)
+/* This is the ound index.  */
+array_num_elements = get_AT_unsigned (c, DW_AT_upper_bound) + 1;
+  else if (get_AT (c, DW_AT_count))
+array_num_elements = get_AT_unsigned (c, DW_AT_count);
+  else
+{
+  /* This is a VLA of some kind.  */
+  array_num_elements = 0;
+}
 
-  int vector_type_p = get_AT_flag (array_type, DW_AT_GNU_vector);
-  if (vector_type_p)
-return array_elems_type_id;
+  /* Ok, mount and register the array type.  Note how the array
+ type we register here is the type of the elements in
+ subsequent "dimensions", if there are any.  */
+  arinfo.ctr_nelems = array_num_elements;
 
-  dw_die_ref array_elems_type = ctf_get_AT_type (array_type);
+  array_index_type = ctf_get_AT_type (c);
+  arinfo.ctr_index = gen_ctf_type (ctfc, array_index_type);
 
-  /* First, register the type of the array elements if needed.  */
-  array_elems_type_id = gen_ctf_type (ctfc, array_elems_type);
+  if (c == last)
+arinfo.ctr_contents = array_elems_type_id;
+  else
+arinfo.ctr_contents = gen_ctf_subrange_type (ctfc, array_elems_type_id,
+dw_get_die_sib (c), last);
 
-  /* DWARF array types pretend C supports multi-dimensional arrays.
- So for the type int[N][M], the array type DIE contains two
- subrange_type children, the first with upper bound N-1 and the
- second with upper bound M-1.
+  if (!ctf_type_exists (ctfc, c, &array_node_type_id))
+array_node_type_id = ctf_add_array (ctfc, CTF_ADD_ROOT, &arinfo, c);
 
- CTF, on the other hand, just encodes each array type in its own
- array type CTF struct.  Therefore we have to iterate on the
- children and create all the needed types.  */
+  return array_node_type_id;
+}
 
-  c = dw_get_die_child (array_type);
-  gcc_assert (c);
-  do
-{
-  ctf_arinfo_t arinfo;
-  dw_die_ref array_index_type;
-  uint32_t array_num_elements;
+/* Generate CTF for an ARRAY_TYPE.  */
 
-  c = dw_get_die_sib (c);
+static ctf_id_t
+gen_ctf_array_type (ctf_container_ref ctfc,
+   dw_die_ref array_type)
+{
+  dw_die_ref first, last, array_elems_type;
+  ctf_id_t array_elems_type_id = CTF_NULL_TYPEID;
+  ctf_id_t array_type_id = CTF_NULL_TYPEID;
 
-  if (dw_get_die_tag (c) == DW_TAG_subrange_type)
-   {
- dw_attr_node *upper_bound_at;
-

[PATCH, V3] ctf: fix incorrect CTF for multi-dimensional array types

[Indu Bhagat]

From: Cupertino Miranda 

[Changes from V2]
  - Fixed aarch64 new FAILs reported by Linaro CI.
  - Fixed typos and other nits pointed out in V2.
[End of changes from V2]

PR debug/114186

DWARF DIEs of type DW_TAG_subrange_type are linked together to represent
the information about the subsequent dimensions.  The CTF processing was
so far working through them in the opposite (incorrect) order.

While fixing the issue, refactor the code a bit for readability.

co-authored-By: Indu Bhagat 

gcc/
PR debug/114186
* dwarf2ctf.cc (gen_ctf_array_type): Invoke the ctf_add_array ()
in the correct order of the dimensions.
(gen_ctf_subrange_type): Refactor out handling of
DW_TAG_subrange_type DIE to here.

gcc/testsuite/
PR debug/114186
* gcc.dg/debug/ctf/ctf-array-6.c: Add test.
---

Testing notes:
 - Linaro CI reported three new FAILs introduced by ctf-array-6.c due to
   presence of char '#' on aarch64 where the ASM_COMMENT_START differs.
   Fixed and regression tested on aarch64.
 - Regression tested on x86_64-linux-gnu default target.
 - Regression tested for target bpf-unknown-none (btf.exp, ctf.exp, bpf.exp).
 - Kernel build with -gctf shows healthier CTF types for arrays.

---
 gcc/dwarf2ctf.cc | 158 +--
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-6.c |  14 ++
 2 files changed, 89 insertions(+), 83 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-6.c

diff --git a/gcc/dwarf2ctf.cc b/gcc/dwarf2ctf.cc
index dca86edfffa9..77d6bf896893 100644
--- a/gcc/dwarf2ctf.cc
+++ b/gcc/dwarf2ctf.cc
@@ -349,105 +349,97 @@ gen_ctf_pointer_type (ctf_container_ref ctfc, dw_die_ref 
ptr_type)
   return ptr_type_id;
 }
 
-/* Generate CTF for an array type.  */
+/* Recursively generate CTF for array dimensions starting at DIE C (of type
+   DW_TAG_subrange_type) until DIE LAST (of type DW_TAG_subrange_type) is
+   reached.  ARRAY_ELEMS_TYPE_ID is base type for the array.  */
 
 static ctf_id_t
-gen_ctf_array_type (ctf_container_ref ctfc, dw_die_ref array_type)
+gen_ctf_subrange_type (ctf_container_ref ctfc, ctf_id_t array_elems_type_id,
+  dw_die_ref c, dw_die_ref last)
 {
-  dw_die_ref c;
-  ctf_id_t array_elems_type_id = CTF_NULL_TYPEID;
+  ctf_arinfo_t arinfo;
+  ctf_id_t array_node_type_id = CTF_NULL_TYPEID;
 
-  int vector_type_p = get_AT_flag (array_type, DW_AT_GNU_vector);
-  if (vector_type_p)
-return array_elems_type_id;
+  dw_attr_node *upper_bound_at;
+  dw_die_ref array_index_type;
+  uint32_t array_num_elements;
 
-  dw_die_ref array_elems_type = ctf_get_AT_type (array_type);
+  if (dw_get_die_tag (c) == DW_TAG_subrange_type)
+{
+  /* When DW_AT_upper_bound is used to specify the size of an
+array in DWARF, it is usually an unsigned constant
+specifying the upper bound index of the array.  However,
+for unsized arrays, such as foo[] or bar[0],
+DW_AT_upper_bound is a signed integer constant
+instead.  */
+
+  upper_bound_at = get_AT (c, DW_AT_upper_bound);
+  if (upper_bound_at
+ && AT_class (upper_bound_at) == dw_val_class_unsigned_const)
+   /* This is the upper bound index.  */
+   array_num_elements = get_AT_unsigned (c, DW_AT_upper_bound) + 1;
+  else if (get_AT (c, DW_AT_count))
+   array_num_elements = get_AT_unsigned (c, DW_AT_count);
+  else
+   {
+ /* This is a VLA of some kind.  */
+ array_num_elements = 0;
+   }
+}
+  else
+gcc_unreachable ();
 
-  /* First, register the type of the array elements if needed.  */
-  array_elems_type_id = gen_ctf_type (ctfc, array_elems_type);
+  /* Ok, mount and register the array type.  Note how the array
+ type we register here is the type of the elements in
+ subsequent "dimensions", if there are any.  */
+  arinfo.ctr_nelems = array_num_elements;
 
-  /* DWARF array types pretend C supports multi-dimensional arrays.
- So for the type int[N][M], the array type DIE contains two
- subrange_type children, the first with upper bound N-1 and the
- second with upper bound M-1.
+  array_index_type = ctf_get_AT_type (c);
+  arinfo.ctr_index = gen_ctf_type (ctfc, array_index_type);
 
- CTF, on the other hand, just encodes each array type in its own
- array type CTF struct.  Therefore we have to iterate on the
- children and create all the needed types.  */
+  if (c == last)
+arinfo.ctr_contents = array_elems_type_id;
+  else
+arinfo.ctr_contents = gen_ctf_subrange_type (ctfc, array_elems_type_id,
+dw_get_die_sib (c), last);
 
-  c = dw_get_die_child (array_type);
-  gcc_assert (c);
-  do
-{
-  ctf_arinfo_t arinfo;
-  dw_die_ref array_index_type;
-  uint32_t array_num_elements;
+  if (!ctf_type_exists (ctfc, c, &array_node_type_id))
+array_node_type_id = ctf_add_array (ctfc, CTF_A

Re: [PATCH] btf: emit symbol refs in DATASEC entries only for BPF [PR114608]

[Indu Bhagat]

On 4/8/24 12:26, David Faust wrote:

The behavior introduced in
   fa60ac54964 btf: Emit labels in DATASEC bts_offset entries.

is only fully correct when compiling for the BPF target with BPF CO-RE
enabled.  In other cases, depending on optimizations, it can result in
an incorrect symbol reference in the entry bts_offset field for a symbol
which may not be emitted at all, causing link-time undefined symbol
reference errors like in PR114608.

The offending bts_offset field of BTF_KIND_DATASEC entries is in reality
only currently useful to consumers of BTF information for BPF programs
anyway.  Correct the regression by only emitting symbol references in
these entries when compiling for the BPF target.  For other targets, the
behavior returns to that prior to fa60ac54964.

The underlying cause is related to PR 113566 "btf: incorrect
BTF_KIND_DATASEC entries for variables which are optimized out." A
complete fix for 113566 is more involved and unsuitable for stage 4,
but will be addressed in the near future.

Tested on x86_64-linux-gnu and on x86_64-linux-gnu host for
bpf-unknown-none target.

OK?
Thanks.



LGTM.

I think adding a comment in the bugzilla for 114431 
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=114431 indicating a further 
patch would not hurt either.


Thanks


gcc/
PR debug/114608
* btfout.cc (btf_asm_datasec_entry): Only emit a symbol reference when
generating BTF for BPF CO-RE target.

gcc/testsuite/
PR debug/114608
* gcc.dg/debug/btf/btf-datasec-1.c: Check bts_offset symbol references
only for BPF target.
* gcc.dg/debug/btf/btf-datasec-2.c: Likewise.
* gcc.dg/debug/btf/btf-pr106773.c: Likewise.
---
  gcc/btfout.cc  |  2 +-
  gcc/testsuite/gcc.dg/debug/btf/btf-datasec-1.c | 10 ++
  gcc/testsuite/gcc.dg/debug/btf/btf-datasec-2.c |  7 ---
  gcc/testsuite/gcc.dg/debug/btf/btf-pr106773.c  |  3 ++-
  4 files changed, 13 insertions(+), 9 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 2e2b3524e83..4a8ec4d1ff0 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -1045,7 +1045,7 @@ btf_asm_datasec_entry (ctf_container_ref ctfc, struct 
btf_var_secinfo info)
  {
const char *symbol_name = get_name_for_datasec_entry (ctfc, info.type);
btf_asm_type_ref ("bts_type", ctfc, info.type);
-  if (symbol_name == NULL)
+  if (!btf_with_core_debuginfo_p () || symbol_name == NULL)
  dw2_asm_output_data (4, info.offset, "bts_offset");
else
  dw2_asm_output_offset (4, symbol_name, NULL, "bts_offset");
diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-1.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-1.c
index c8ebe5d07ca..15b76259218 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-1.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-1.c
@@ -19,10 +19,12 @@
  /* { dg-final { scan-assembler-times "0xf03\[\t \]+\[^\n\]*btt_info" 2 } 
} */
  /* { dg-final { scan-assembler-times "0xf01\[\t \]+\[^\n\]*btt_info" 1 } 
} */
  
-/* The offset entry for each variable in a DATSEC should contain a label.  */

-/* { dg-final { scan-assembler-times 
"(?:(?:\\.4byte|\\.long|data4\\.ua|\\.ualong|\\.uaword|\\.dword|long|dc\\.l|\\.word)\[\t
 \]|\\.vbyte\t4,\[\t \]?)_?\[a-e\]\[\t \]+\[^\n\]*bts_offset" 5 } } */
-/* { dg-final { scan-assembler-times "my_cstruct\[\t \]+\[^\n\]*bts_offset" 1 
} } */
-/* { dg-final { scan-assembler-times "bigarr\[\t \]+\[^\n\]*bts_offset" 1 } } 
*/
+/* { dg-final { scan-assembler-times "0\[\t \]+\[^\n\]*bts_offset" 7 { target 
{ ! bpf-*-* } } } } */
+
+/* For BPF target the offset entry for each variable in a DATSEC should 
contain a label.  */
+/* { dg-final { scan-assembler-times ".4byte\[\t \]\[a-e\]\[\t 
\]+\[^\n\]*bts_offset" 5 { target bpf-*-* } } } */
+/* { dg-final { scan-assembler-times "my_cstruct\[\t \]+\[^\n\]*bts_offset" 1 
{ target bpf-*-* } } } */
+/* { dg-final { scan-assembler-times "bigarr\[\t \]+\[^\n\]*bts_offset" 1 { 
target bpf-*-* } } } */
  
  /* Check that strings for each DATASEC have been added to the BTF string table.  */

  /* { dg-final { scan-assembler-times "ascii \".data.0\"\[\t 
\]+\[^\n\]*btf_aux_string" 1 } } */
diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-2.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-2.c
index 857d830e446..a89a239a504 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-2.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-2.c
@@ -10,9 +10,10 @@
  /* { dg-final { scan-assembler-times " BTF_KIND_DATASEC 
'.bar_sec'\[\\r\\n\]+\[^\\r\\n\]*0xf02\[\t \]+\[^\n\]*btt_info" 1 } } */
  
  /* Function entries should have offset with a label and size of 0 at compile time.  */

-/* { dg-final { scan-assembler-times "chacha\[\t \]+\[^\n\]*bts_offset" 1 } } 
*/
-/* { dg-final { scan-assembler-times "bar\[\t \]+\[^\n\]*bts_offset" 1 } } */
-/* { dg-final { scan-assembler-times "foo\[\t \]+\[^\n\]*bts_offset" 1 } } */
+/* { dg-final { scan-assembler-times "chacha\[\t \]+\[^\n\]*bts_off

Re: [PATCH] btf: improve btf-datasec-3.c test [PR 114642]

[Indu Bhagat]

On 4/8/24 2:01 PM, David Faust wrote:

This test failed on powerpc --target_board=unix'{-m32}' because two
variables were not placed in sections where the test silently (and
incorrectly) assumed they would be.

The important thing for the test is only that BTF_KIND_DATASEC entries
are NOT generated for the extern variable declarations without an
explicit section attribute.  Make the test more robust by placing the
non-extern variables in explicit sections, and invert the checks to
more accurately verify what we care about in this test.

Tested on x86_64-linux-gnu and x86_64-linux-gnu host for
powerpc64-linux-gnu and bpf-unkown-none targets.

OK?



Hi David,

LGTM.

I see that we have btf-datasec-1.c testcase for checking the case when 
vars are in sections .rodata/.bss/.data.


Thanks


gcc/testsuite/
PR testsuite/114642
* gcc.dg/debug/btf/btf-datasec-3.c: Make test more robust on different
architectures.
---
  gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c | 16 
  1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c
index 297340cabfa..6b127aa14da 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c
@@ -7,22 +7,22 @@
  
  extern int VERSION __attribute__((section (".version")));
  
-extern int test_bss1;

-extern int test_data1;
+extern int ext1;
+extern int ext2;
  
-int test_bss2;

-int test_data2 = 2;
+int var1 __attribute__((section (".sec_a")));
+int var2 __attribute__((section (".sec_b"))) = 2;
  
  int

  foo (void)
  {
-  test_bss2 = VERSION;
-  return test_bss1 + test_data1 + test_data2;
+  ext2 = VERSION;
+  return ext1 + var1 + var2;
  }
  
  /* There should be 3 DATASEC entries total.  Of the extern decls, only VERSION

 has a known section; entries are not created for the other two.  */
  /* { dg-final { scan-assembler-times "bts_type" 3 } } */
-/* { dg-final { scan-assembler-times "bts_type: \\(BTF_KIND_VAR 
'test_data2'\\)" 1 } } */
-/* { dg-final { scan-assembler-times "bts_type: \\(BTF_KIND_VAR 
'test_bss2'\\)" 1 } } */
  /* { dg-final { scan-assembler-times "bts_type: \\(BTF_KIND_VAR 'VERSION'\\)" 
1 } } */
+/* { dg-final { scan-assembler-not "bts_type: \\(BTF_KIND_VAR 'ext1'\\)" } } */
+/* { dg-final { scan-assembler-not "bts_type: \\(BTF_KIND_VAR 'ext2'\\)" } } */

[PATCH 2/2] btf: do not skip members of data type with type id BTF_VOID_TYPEID

[Indu Bhagat]

Testing the previous fix in gen_ctf_sou_type () reveals an issue in BTF
generation, however: BTF emission was currently decrementing the vlen
(indicating the number of members) to skip members of type CTF_K_UNKNOWN
altogether, but still emitting the BTF for the corresponding member (in
output_asm_btf_sou_fields ()).

One can see malformed BTF by executing the newly added CTF testcase
(gcc.dg/debug/ctf/ctf-bitfields-5.c) with -gbtf instead or even existing
btf-struct-2.c without this patch.

To fix the issue, it makes sense to rather _not_ skip members of data
type of type id BTF_VOID_TYPEID.

gcc/ChangeLog:
* btfout.cc (btf_asm_type): Do not skip emitting members of
unknown type.

gcc/testsuite/ChangeLog:
* btf-bitfields-4.c: Update the vlen check.
* btf-struct-2.c: Check that member named 'f' with void data
type is emitted.
---
 gcc/btfout.cc| 5 -
 gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c | 6 +++---
 gcc/testsuite/gcc.dg/debug/btf/btf-struct-2.c| 9 +
 3 files changed, 8 insertions(+), 12 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 4a8ec4d1ff0..ab491f0297f 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -820,11 +820,6 @@ btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
  /* Set kflag if this member is a representable bitfield.  */
  if (btf_dmd_representable_bitfield_p (ctfc, dmd))
btf_kflag = 1;
-
- /* Struct members that refer to unsupported types or bitfield formats
-shall be skipped. These are marked during preprocessing.  */
- else if (!btf_emit_id_p (dmd->dmd_type))
-   btf_vlen -= 1;
}
 }
 
diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
index c00c8b3d87f..d4a6ef6a1eb 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
@@ -6,14 +6,14 @@
In this test, we construct a structure such that the bitfield will have
an offset so large as to be unrepresentable in BTF. We expect that the
resulting BTF will describe the rest of the structure, ignoring the
-   non-representable bitfield.  */
+   non-representable bitfield by simply using void data type for the same.  */
 
 /* { dg-do compile } */
 /* { dg-options "-O0 -gbtf -dA" } */
 /* { dg-require-effective-target size32plus } */
 
-/* Struct with 3 members and no bitfield (kind_flag not set).  */
-/* { dg-final { scan-assembler-times "\[\t \]0x403\[\t 
\]+\[^\n\]*btt_info" 1 } } */
+/* Struct with 4 members and no bitfield (kind_flag not set).  */
+/* { dg-final { scan-assembler-times "\[\t \]0x404\[\t 
\]+\[^\n\]*btt_info" 1 } } */
 
 struct bigly
 {
diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-struct-2.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-struct-2.c
index e9ff06883db..fa7231be75c 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-struct-2.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-struct-2.c
@@ -2,14 +2,15 @@
unsupported type.
 
BTF does not support vector types (among other things). When
-   generating BTF for a struct (or union) type, members which refer to
-   unsupported types should be skipped.  */
+   generating BTF for a struct (or union) type.  Members which refer to
+   unsupported types should not be skipped, however.  */
 
 /* { dg-do compile } */
 /* { dg-options "-O0 -gbtf -dA" } */
 
-/* Expect a struct with only 2 members - 'f' should not be present.  */
-/* { dg-final { scan-assembler-times "\[\t \]0x402\[\t 
\]+\[^\n\]*btt_info" 1 } } */
+/* Expect a struct with 3 members - 'f' is present but is of data type void.  
*/
+/* { dg-final { scan-assembler-times "\[\t \]0x403\[\t 
\]+\[^\n\]*btt_info" 1 } } */
+/* { dg-final { scan-assembler-times " MEMBER 'f' 
idx=1\[\\r\\n\]+\[^\\r\\n\]*0\[\t \]+\[^\n\]*btm_type: void" 1 } } */
 
 struct with_float
 {
-- 
2.43.0

[PATCH 0/2] Fix PR debug/112878 and a BTF issue

[Indu Bhagat]

Hi,

The patch series includes two patches: first one is a fix for PR
debug/112878 and the second one is for an existing BTF generation issue.

Testing Notes:
 - Regression tested on x86_64-linux-gnu
 - Tested btf.exp, ctf.exp, bpf.exp for --target=bpf-unknown-none

Thanks,
Indu Bhagat (2):
  ctf: fix PR debug/112878
  btf: do not skip members of data type with type id BTF_VOID_TYPEID

 gcc/btfout.cc   |  5 -
 gcc/dwarf2ctf.cc| 15 ++-
 .../gcc.dg/debug/btf/btf-bitfields-4.c  |  6 +++---
 gcc/testsuite/gcc.dg/debug/btf/btf-struct-2.c   |  9 +
 .../gcc.dg/debug/ctf/ctf-bitfields-5.c  | 17 +
 5 files changed, 35 insertions(+), 17 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-5.c

-- 
2.43.0

[PATCH 1/2] ctf: fix PR debug/112878

[Indu Bhagat]

PR debug/112878: ICE: in ctf_add_slice, at ctfc.cc:499 with _BitInt > 255 in a 
struct and -gctf1

The CTF generation in GCC does not have a mechanism to roll-back an
already added type.  In this testcase presented in the PR, we hit a
representation limit in CTF slices (for a member of a struct) and ICE,
after the type for struct (CTF_K_STRUCT) has already been added to the
container.

To exit gracefully instead, we now check for both the offset and size of
the bitfield to be explicitly <= 255.  If the check fails, we emit the
member with type CTF_K_UNKNOWN.  Note that, the value 255 stems from the
existing binutils libctf checks which were motivated to guard against
malformed inputs.

Although it is not accurate to say that this is a CTF representation
limit, mark the code with TBD_CTF_REPRESENTATION_LIMIT for now so that
this can be taken care of with the next format version bump, when
libctf's checks for the slice data can be lifted as well.

gcc/ChangeLog:
PR debug/112878
* dwarf2ctf.cc (gen_ctf_sou_type): Check for conditions before
call to ctf_add_slice.  Use CTF_K_UNKNOWN type if fail.

gcc/testsuite/ChangeLog:
PR debug/112878
* gcc.dg/debug/ctf/ctf-bitfields-5.c: New test.
---
 gcc/dwarf2ctf.cc| 15 ++-
 .../gcc.dg/debug/ctf/ctf-bitfields-5.c  | 17 +
 2 files changed, 27 insertions(+), 5 deletions(-)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-5.c

diff --git a/gcc/dwarf2ctf.cc b/gcc/dwarf2ctf.cc
index 77d6bf89689..dc59569fe56 100644
--- a/gcc/dwarf2ctf.cc
+++ b/gcc/dwarf2ctf.cc
@@ -606,11 +606,16 @@ gen_ctf_sou_type (ctf_container_ref ctfc, dw_die_ref sou, 
uint32_t kind)
  if (attr)
bitpos += AT_unsigned (attr);
 
- field_type_id = ctf_add_slice (ctfc, CTF_ADD_NONROOT,
-field_type_id,
-bitpos - field_location,
-bitsize,
-c);
+ /* This is not precisely a TBD_CTF_REPRESENTATION_LIMIT, but
+surely something to look at for the next format version bump
+for CTF.  */
+ if (bitsize <= 255 && (bitpos - field_location) <= 255)
+   field_type_id = ctf_add_slice (ctfc, CTF_ADD_NONROOT,
+  field_type_id,
+  bitpos - field_location,
+  bitsize, c);
+ else
+   field_type_id = gen_ctf_unknown_type (ctfc);
}
 
  /* Add the field type to the struct or union type.  */
diff --git a/gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-5.c 
b/gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-5.c
new file mode 100644
index 000..fee8228647c
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-5.c
@@ -0,0 +1,17 @@
+/* Bitfield where the bit offset is > 255 is not allowed in CTF.
+
+   PR debug/112878.
+   This testcase is to ensure graceful handling. No slices are expected.  */
+
+/* { dg-do compile { target bitint } } */
+/* { dg-options "-O0 -gctf -dA" } */
+
+/* No slices are expected, but a struct with one member is expected.
+   CTF_K_UNKNOWN is also expected.  */
+/* { dg-final { scan-assembler-times "cts_type" 0 } } */
+/* { dg-final { scan-assembler-times "\[\t \]0x1a01\[\t 
\]+\[^\n\]*ctt_info" 1 } } */
+/* { dg-final { scan-assembler-times "ascii \"unknown.0\"\[\t 
\]+\[^\n\]*ctf_string" 1 } } */
+
+struct {
+  _BitInt(282) a : 280;
+} b;
-- 
2.43.0

Re: [PATCH] btf: emit non-representable bitfield as void

[Indu Bhagat]

On 4/11/24 11:53, David Faust wrote:

This patch fixes an issue with mangled BTF that could occur when
a struct type contains a bitfield member which cannot be represented
in BTF.  It is undefined what should happen in such cases, but we can
at least do something reasonable.

Commit

   936dd627cd9 "btf: do not skip members of data type with type id
   BTF_VOID_TYPEID"

made a similar change for un-representable non-bitfield members, but
had an unintended side-effect of mangling BTF for un-representable
bitfields: the struct (or union) would account for the offending
bitfield in its member count but the bitfield member itself was
not emitted, making the member count incorrect.

This change ensures that non-representable bitfield members of struct
and union types are always emitted with BTF_VOID_TYPEID.  This avoids
corrupting the BTF information for the entire struct or union type.

Tested on x86_64-linux-gnu and x86_64-linux-gnu host for
bpf-unknown-none target.



Hi David,

Thanks for fixing this.  One comment below.


gcc/
* btfout.cc (btf_asm_sou_member): Always emit non-representable
bitfield members as having 'void' type.  Refactor slightly.

gcc/testsuite/
* gcc.dg/debug/btf/btf-bitfields-4.c: Add two new checks.
---
  gcc/btfout.cc | 48 +--
  .../gcc.dg/debug/btf/btf-bitfields-4.c|  2 +
  2 files changed, 25 insertions(+), 25 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index ab491f0297f..e1ada41b1f4 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -922,41 +922,39 @@ static void
  btf_asm_sou_member (ctf_container_ref ctfc, ctf_dmdef_t * dmd, unsigned int 
idx)
  {
ctf_dtdef_ref ref_type = ctfc->ctfc_types_list[dmd->dmd_type];
+  ctf_id_t base_type = get_btf_id (dmd->dmd_type);
+  uint64_t sou_offset = dmd->dmd_offset;
+
+  dw2_asm_output_data (4, dmd->dmd_name_offset,
+  "MEMBER '%s' idx=%u",
+  dmd->dmd_name, idx);
  
/* Re-encode bitfields to BTF representation.  */

if (CTF_V2_INFO_KIND (ref_type->dtd_data.ctti_info) == CTF_K_SLICE)
  {
-  ctf_id_t base_type = ref_type->dtd_u.dtu_slice.cts_type;
unsigned short word_offset = ref_type->dtd_u.dtu_slice.cts_offset;
unsigned short bits = ref_type->dtd_u.dtu_slice.cts_bits;
-  uint64_t sou_offset = dmd->dmd_offset;
-
-  /* Pack the bit offset and bitfield size together.  */
-  sou_offset += word_offset;
  
-  /* If this bitfield cannot be represented, do not output anything.

-The parent struct/union 'vlen' field has already been updated.  */
if ((bits > 0xff) || (sou_offset > 0xff))


Why dont we reuse the btf_dmd_representable_bitfield_p () function here?

This one here checks for sou_off > 0xff, while that in 
btf_dmd_representable_bitfield_p checks for sou_offset + word_offset > 
0xff.  The latter is more precise.




-   return;
-
-  sou_offset &= 0x00ff;
-  sou_offset |= ((bits & 0xff) << 24);
+   {
+ /* Bitfield cannot be represented in BTF.  Emit the member as having
+'void' type.  */
+ base_type = BTF_VOID_TYPEID;
+   }
+  else
+   {
+ /* Pack the bit offset and bitfield size together.  */
+ sou_offset += word_offset;
+ sou_offset &= 0x00ff;
+ sou_offset |= ((bits & 0xff) << 24);
  
-  dw2_asm_output_data (4, dmd->dmd_name_offset,

-  "MEMBER '%s' idx=%u",
-  dmd->dmd_name, idx);
-  /* Refer to the base type of the slice.  */
-  btf_asm_type_ref ("btm_type", ctfc, get_btf_id (base_type));
-  dw2_asm_output_data (4, sou_offset, "btm_offset");
-}
-  else
-{
-  dw2_asm_output_data (4, dmd->dmd_name_offset,
-  "MEMBER '%s' idx=%u",
-  dmd->dmd_name, idx);
-  btf_asm_type_ref ("btm_type", ctfc, get_btf_id (dmd->dmd_type));
-  dw2_asm_output_data (4, dmd->dmd_offset, "btm_offset");
+ /* Refer to the base type of the slice.  */
+ base_type = get_btf_id (ref_type->dtd_u.dtu_slice.cts_type);
+   }
  }
+
+  btf_asm_type_ref ("btm_type", ctfc, base_type);
+  dw2_asm_output_data (4, sou_offset, "btm_offset");
  }
  
  /* Asm'out an enum constant following a BTF_KIND_ENUM{,64}.  */

diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
index d4a6ef6a1eb..20cdfaa057a 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
@@ -14,6 +14,8 @@
  
  /* Struct with 4 members and no bitfield (kind_flag not set).  */

  /* { dg-final { scan-assembler-times "\[\t \]0x404\[\t 
\]+\[^\n\]*btt_info" 1 } } */
+/* { dg-final { scan-assembler-times " MEMBER" 4 } } */
+/* { dg-final { scan-assembler-times " MEMBER 'unsup' 
idx=2\[\\r\\n\]+\[^\\r\\n\]*0\[\t \]+\[^\n\]*btm_type: void" 1 } } *

Re: [PATCH v2] btf: emit non-representable bitfield as void

[Indu Bhagat]

On 4/11/24 14:01, David Faust wrote:

[Changes from v1: use btf_dmd_representable_bitfield_p in
  btf_asm_sou_member, instead of a slightly incorrect equivalent check.]

This patch fixes an issue with mangled BTF that could occur when
a struct type contains a bitfield member which cannot be represented
in BTF.  It is undefined what should happen in such cases, but we can
at least do something reasonable.

Commit

   936dd627cd9 "btf: do not skip members of data type with type id
   BTF_VOID_TYPEID"

made a similar change for un-representable non-bitfield members, but
had an unintended side-effect of mangling BTF for un-representable
bitfields: the struct (or union) would account for the offending
bitfield in its member count but the bitfield member itself was
not emitted, making the member count incorrect.

This change ensures that non-representable bitfield members of struct
and union types are always emitted with BTF_VOID_TYPEID.  This avoids
corrupting the BTF information for the entire struct or union type.

Tested on x86_64-linux-gnu and x86_64-linux-gnu host for
bpf-unknown-none target.

gcc/
* btfout.cc (btf_asm_sou_member): Always emit non-representable
bitfield members as having 'void' type.  Refactor slightly.

gcc/testsuite/
* gcc.dg/debug/btf/btf-bitfields-4.c: Add two new checks.


LGTM.

Thanks


---
  gcc/btfout.cc | 54 +--
  .../gcc.dg/debug/btf/btf-bitfields-4.c|  2 +
  2 files changed, 28 insertions(+), 28 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index ab491f0297f..a1510574a93 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -922,41 +922,39 @@ static void
  btf_asm_sou_member (ctf_container_ref ctfc, ctf_dmdef_t * dmd, unsigned int 
idx)
  {
ctf_dtdef_ref ref_type = ctfc->ctfc_types_list[dmd->dmd_type];
+  ctf_id_t base_type = get_btf_id (dmd->dmd_type);
+  uint64_t sou_offset = dmd->dmd_offset;
+
+  dw2_asm_output_data (4, dmd->dmd_name_offset,
+  "MEMBER '%s' idx=%u",
+  dmd->dmd_name, idx);
  
/* Re-encode bitfields to BTF representation.  */

if (CTF_V2_INFO_KIND (ref_type->dtd_data.ctti_info) == CTF_K_SLICE)
  {
-  ctf_id_t base_type = ref_type->dtd_u.dtu_slice.cts_type;
-  unsigned short word_offset = ref_type->dtd_u.dtu_slice.cts_offset;
-  unsigned short bits = ref_type->dtd_u.dtu_slice.cts_bits;
-  uint64_t sou_offset = dmd->dmd_offset;
-
-  /* Pack the bit offset and bitfield size together.  */
-  sou_offset += word_offset;
-
-  /* If this bitfield cannot be represented, do not output anything.
-The parent struct/union 'vlen' field has already been updated.  */
-  if ((bits > 0xff) || (sou_offset > 0xff))
-   return;
+  if (btf_dmd_representable_bitfield_p (ctfc, dmd))
+   {
+ unsigned short word_offset = ref_type->dtd_u.dtu_slice.cts_offset;
+ unsigned short bits = ref_type->dtd_u.dtu_slice.cts_bits;
  
-  sou_offset &= 0x00ff;

-  sou_offset |= ((bits & 0xff) << 24);
+ /* Pack the bit offset and bitfield size together.  */
+ sou_offset += word_offset;
+ sou_offset &= 0x00ff;
+ sou_offset |= ((bits & 0xff) << 24);
  
-  dw2_asm_output_data (4, dmd->dmd_name_offset,

-  "MEMBER '%s' idx=%u",
-  dmd->dmd_name, idx);
-  /* Refer to the base type of the slice.  */
-  btf_asm_type_ref ("btm_type", ctfc, get_btf_id (base_type));
-  dw2_asm_output_data (4, sou_offset, "btm_offset");
-}
-  else
-{
-  dw2_asm_output_data (4, dmd->dmd_name_offset,
-  "MEMBER '%s' idx=%u",
-  dmd->dmd_name, idx);
-  btf_asm_type_ref ("btm_type", ctfc, get_btf_id (dmd->dmd_type));
-  dw2_asm_output_data (4, dmd->dmd_offset, "btm_offset");
+ /* Refer to the base type of the slice.  */
+ base_type = get_btf_id (ref_type->dtd_u.dtu_slice.cts_type);
+   }
+  else
+   {
+ /* Bitfield cannot be represented in BTF.  Emit the member as having
+'void' type.  */
+ base_type = BTF_VOID_TYPEID;
+   }
  }
+
+  btf_asm_type_ref ("btm_type", ctfc, base_type);
+  dw2_asm_output_data (4, sou_offset, "btm_offset");
  }
  
  /* Asm'out an enum constant following a BTF_KIND_ENUM{,64}.  */

diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
index d4a6ef6a1eb..20cdfaa057a 100644
--- a/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-bitfields-4.c
@@ -14,6 +14,8 @@
  
  /* Struct with 4 members and no bitfield (kind_flag not set).  */

  /* { dg-final { scan-assembler-times "\[\t \]0x404\[\t 
\]+\[^\n\]*btt_info" 1 } } */
+/* { dg-final { scan-assembler-times " MEMBER" 4 } } */
+/* { dg-final { scan-assembler-times " MEMBER 'unsup' 
idx=2\[\\r\\n\]+\[^\\r\\n\]*0

Re: [PATCH] btf: fix a possibly misleading asm debug comment

[Indu Bhagat]

On 4/11/24 14:02, David Faust wrote:

This patch fixes a small error that could occur in the debug comment
when emitting a type reference with btf_asm_type_ref.

While working on a previous patch, I noticed the following in the asm
output for the test btf-bitfields-4.c:

...
.long   0x39# MEMBER 'c' idx=3
.long   0x6 # btm_type: (BTF_KIND_UNKN '')
...
.long   0x34# TYPE 6 BTF_KIND_INT 'char'

The type for member 'c' is correct, but the comment for the member
incorrectly reads "BTF_KIND_UNKN ''".  This was caused by an
incorrect type lookup in btf_asm_type_ref that could happen if the
source file has types which can be represented in CTF but not in BTF.

This patch fixes the issue by changing btf_asm_type_ref to work fully
in the CTF ID space until writing out the final BTF ID.  That ensures
types are correctly identified when writing the asm debug comments,
like the following fixed comment for the above case.

...
.long   0x39# MEMBER 'c' idx=3
.long   0x6 # btm_type: (BTF_KIND_INT 'char')
...

Note that there was no problem with the actual BTF information, the
only error was in the comment.  This patch does not change the output
BTF information, and no tests were affected.

Tested on x86_64-linux-gnu and x86_64-linux-gnu host for
bpf-unknown-none target. Sanity checked by compiling kernel BPF
selftests.



Thanks.

The code is easier to follow now too.

LGTM.


gcc/
* btfout.cc (btf_asm_type_ref): Convert IDs to BTF internally and
fix potentially looking up wrong type for asm debug comment info.
Split into...
(btf_asm_datasec_type_ref): ... This. New.
(btf_asm_datasec_entry): Call it here, instead of btf_asm_type_ref.
(btf_asm_type, btf_asm_array, btf_asm_varent, btf_asm_sou_member)
(btf_asm_func_arg, btf_asm_func_type): Adapt btf_asm_type_ref call.
---
  gcc/btfout.cc | 84 ++-
  1 file changed, 50 insertions(+), 34 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index e1ada41b1f4..24bef3acfd1 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -738,36 +738,22 @@ btf_dmd_representable_bitfield_p (ctf_container_ref ctfc, 
ctf_dmdef_t *dmd)
  /* Asm'out a reference to another BTF type.  */
  
  static void

-btf_asm_type_ref (const char *prefix, ctf_container_ref ctfc, ctf_id_t ref_id)
+btf_asm_type_ref (const char *prefix, ctf_container_ref ctfc, ctf_id_t ctf_id)
  {
-  if (ref_id == BTF_VOID_TYPEID || ref_id == BTF_INVALID_TYPEID)
+  ctf_id_t btf_id = get_btf_id (ctf_id);
+  if (btf_id == BTF_VOID_TYPEID || btf_id == BTF_INVALID_TYPEID)
  {
/* There is no explicit void type.
 Also handle any invalid refs that made it this far, just in case.  */
-  dw2_asm_output_data (4, ref_id, "%s: void", prefix);
-}
-  else if (ref_id >= num_types_added + 1
-  && ref_id < num_types_added + num_vars_added + 1)
-{
-  /* Ref to a variable.  Should only appear in DATASEC entries.  */
-  ctf_id_t var_id = btf_relative_var_id (ref_id);
-  ctf_dvdef_ref dvd = ctfc->ctfc_vars_list[var_id];
-  dw2_asm_output_data (4, ref_id, "%s: (BTF_KIND_VAR '%s')",
-  prefix, dvd->dvd_name);
-
-}
-  else if (ref_id >= num_types_added + num_vars_added + 1)
-{
-  /* Ref to a FUNC record.  */
-  size_t func_id = btf_relative_func_id (ref_id);
-  ctf_dtdef_ref ref_type = (*funcs)[func_id];
-  dw2_asm_output_data (4, ref_id, "%s: (BTF_KIND_FUNC '%s')",
-  prefix, get_btf_type_name (ref_type));
+  dw2_asm_output_data (4, btf_id, "%s: void", prefix);
  }
else
  {
-  /* Ref to a standard type in the types list.  */
-  ctf_dtdef_ref ref_type = ctfc->ctfc_types_list[ref_id];
+  gcc_assert (btf_id <= num_types_added);
+
+  /* Ref to a standard type in the types list.  Note: take care that we
+must index the type list by the original CTF id, not the BTF id.  */
+  ctf_dtdef_ref ref_type = ctfc->ctfc_types_list[ctf_id];
uint32_t ref_kind
= get_btf_kind (CTF_V2_INFO_KIND (ref_type->dtd_data.ctti_info));
  
@@ -775,12 +761,43 @@ btf_asm_type_ref (const char *prefix, ctf_container_ref ctfc, ctf_id_t ref_id)

? btf_kind_name (BTF_KIND_ENUM)
: btf_kind_name (ref_kind);
  
-  dw2_asm_output_data (4, ref_id, "%s: (BTF_KIND_%s '%s')",

+  dw2_asm_output_data (4, btf_id, "%s: (BTF_KIND_%s '%s')",
   prefix, kind_name,
   get_btf_type_name (ref_type));
  }
  }
  
+/* Asm'out a reference to a BTF_KIND_VAR or BTF_KIND_FUNC type.  These type

+   kinds are BTF-specific, and should only be referred to by entries in
+   BTF_KIND_DATASEC records.  */
+
+static void
+btf_asm_datasec_type_ref (const char *prefix, ctf_container_ref ctfc,
+ ctf_id_t btf_id)
+{
+  if (btf_id >= num_types_adde

Re: [PATCH] btf: change encoding of forward-declared enums [PR111735]

[Indu Bhagat]

On 12/12/23 14:35, David Faust wrote:

The BTF specification does not formally define a representation for
forward-declared enum types such as:

   enum Foo;

Forward-declarations for struct and union types are represented by
BTF_KIND_FWD, which has a 1-bit flag distinguishing the two.

The de-facto standard format used by other tools like clang and pahole
is to represent forward-declared enums as BTF_KIND_ENUM with vlen=0,
i.e. as a regular enum type with no enumerators.  This patch changes
GCC to adopt that format, and makes a couple of minor cleanups in
btf_asm_type ().

Bootstrapped and tested on x86_64-linux-gnu.
Also tested on x86_64-linux-gnu host for bpf-unknown-none target.



LGTM.

Thanks


gcc/

PR debug/111735
* btfout.cc (btf_fwd_to_enum_p): New.
(btf_asm_type_ref): Special case references to enum forwards.
(btf_asm_type): Special case enum forwards. Rename btf_size_type to
btf_size, and change chained ifs switching on btf_kind into else ifs.

gcc/testsuite/

PR debug/111735
* gcc.dg/debug/btf/btf-forward-2.c: New test.
---
  gcc/btfout.cc | 46 ++-
  .../gcc.dg/debug/btf/btf-forward-2.c  | 18 
  2 files changed, 53 insertions(+), 11 deletions(-)
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-forward-2.c

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index db4f1084f85..3ec938874b6 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -268,6 +268,17 @@ btf_emit_id_p (ctf_id_t id)
  && (btf_id_map[id] <= BTF_MAX_TYPE));
  }
  
+/* Return true if DTD is a forward-declared enum.  The BTF representation

+   of forward declared enums is not formally defined.  */
+
+static bool
+btf_fwd_to_enum_p (ctf_dtdef_ref dtd)
+{
+  uint32_t btf_kind = get_btf_kind (CTF_V2_INFO_KIND 
(dtd->dtd_data.ctti_info));
+
+  return (btf_kind == BTF_KIND_FWD && dtd->dtd_data.ctti_type == CTF_K_ENUM);
+}
+
  /* Each BTF type can be followed additional, variable-length information
 completing the description of the type. Calculate the number of bytes
 of variable information required to encode a given type.  */
@@ -753,8 +764,12 @@ btf_asm_type_ref (const char *prefix, ctf_container_ref 
ctfc, ctf_id_t ref_id)
uint32_t ref_kind
= get_btf_kind (CTF_V2_INFO_KIND (ref_type->dtd_data.ctti_info));
  
+  const char *kind_name = btf_fwd_to_enum_p (ref_type)

+   ? btf_kind_name (BTF_KIND_ENUM)
+   : btf_kind_name (ref_kind);
+
dw2_asm_output_data (4, ref_id, "%s: (BTF_KIND_%s '%s')",
-  prefix, btf_kind_name (ref_kind),
+  prefix, kind_name,
   get_btf_type_name (ref_type));
  }
  }
@@ -765,11 +780,11 @@ btf_asm_type_ref (const char *prefix, ctf_container_ref 
ctfc, ctf_id_t ref_id)
  static void
  btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
  {
-  uint32_t btf_kind, btf_kflag, btf_vlen, btf_size_type;
+  uint32_t btf_kind, btf_kflag, btf_vlen, btf_size;
uint32_t ctf_info = dtd->dtd_data.ctti_info;
  
btf_kind = get_btf_kind (CTF_V2_INFO_KIND (ctf_info));

-  btf_size_type = dtd->dtd_data.ctti_type;
+  btf_size = dtd->dtd_data.ctti_size;
btf_vlen = CTF_V2_INFO_VLEN (ctf_info);
  
/* By now any unrepresentable types have been removed.  */

@@ -777,7 +792,7 @@ btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
  
/* Size 0 integers are redundant definitions of void. None should remain

   in the types list by this point.  */
-  gcc_assert (btf_kind != BTF_KIND_INT || btf_size_type >= 1);
+  gcc_assert (btf_kind != BTF_KIND_INT || btf_size >= 1);
  
/* Re-encode the ctti_info to BTF.  */

/* kflag is 1 for structs/unions with a bitfield member.
@@ -810,16 +825,26 @@ btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
   structs and forwards to unions. The dwarf2ctf conversion process stores
   the kind of the forward in ctti_type, but for BTF this must be 0 for
   forwards, with only the KIND_FLAG to distinguish.
- At time of writing, BTF forwards to enums are unspecified.  */
-  if (btf_kind == BTF_KIND_FWD)
+ Forwards to enum types are special-cased below.  */
+  else if (btf_kind == BTF_KIND_FWD)
  {
if (dtd->dtd_data.ctti_type == CTF_K_UNION)
btf_kflag = 1;
  
-  btf_size_type = 0;

+  /* PR debug/111735.  Encode foward-declared enums as BTF_KIND_ENUM
+with vlen=0.  A representation for these is not formally defined;
+this is the de-facto standard used by other tools like clang
+and pahole.  */
+  else if (dtd->dtd_data.ctti_type == CTF_K_ENUM)
+   {
+ btf_kind = BTF_KIND_ENUM;
+ btf_vlen = 0;
+   }
+
+  btf_size = 0;
  }
  
-  if (btf_kind == BTF_KIND_ENUM)

+  else if (btf_kind == BTF_KIND_ENUM)
  {
btf_kflag = dtd->dtd_enum_unsigned
? BTF_KF_ENUM_UNSIGNED
@@ -829,7 +854,7 @@ btf_asm_typ

[PATCH] btf: fix PR debug/112768

[Indu Bhagat]

PR debug/112768 - btf: fix asm comment output for BTF_KIND_FUNC* kinds

The patch adds a small function to abstract out the detail and return
the name of the type.  The patch also fixes the issue of BTF_KIND_FUNC
appearing in the comments with a 'null' string.

For btf-function-6.c testcase, after the patch:

.long   0   # TYPE 2 BTF_KIND_FUNC_PROTO ''
.long   0xd02   # btt_info: kind=13, kflag=0, vlen=2
.long   0x1 # btt_type: (BTF_KIND_INT 'int')
.long   0   # farg_name
.long   0x1 # farg_type: (BTF_KIND_INT 'int')
.long   0   # farg_name
.long   0x1 # farg_type: (BTF_KIND_INT 'int')
.long   0   # TYPE 3 BTF_KIND_FUNC_PROTO ''
.long   0xd01   # btt_info: kind=13, kflag=0, vlen=1
.long   0x1 # btt_type: (BTF_KIND_INT 'int')
.long   0x68# farg_name
.long   0x1 # farg_type: (BTF_KIND_INT 'int')
.long   0x5 # TYPE 4 BTF_KIND_FUNC 'extfunc'
.long   0xc02   # btt_info: kind=12, kflag=0, linkage=2
.long   0x2 # btt_type: (BTF_KIND_FUNC_PROTO '')
.long   0xd # TYPE 5 BTF_KIND_FUNC 'foo'
.long   0xc01   # btt_info: kind=12, kflag=0, linkage=1
.long   0x3 # btt_type: (BTF_KIND_FUNC_PROTO '')

gcc/ChangeLog:

PR debug/112768
* btfout.cc (get_btf_type_name): New definition.
(btf_collect_datasec): Update dtd_name to the original type name
string.
(btf_asm_type_ref): Use the new get_btf_type_name function
instead.
(btf_asm_type): Likewise.
(btf_asm_func_type): Likewise.

gcc/testsuite/ChangeLog:

PR debug/112768
* gcc.dg/debug/btf/btf-function-6.c: Empty string expected with
BTF_KIND_FUNC_PROTO.

Testing notes:
  - bootstrapped and reg tested on x86_64
  - No regressions in btf.exp on BPF target

---
 gcc/btfout.cc | 22 +++
 .../gcc.dg/debug/btf/btf-function-6.c |  4 ++--
 2 files changed, 20 insertions(+), 6 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 5f2e99ce472..1c25404b2c0 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -158,6 +158,19 @@ get_btf_kind (uint32_t ctf_kind)
   return BTF_KIND_UNKN;
 }
 
+/* Some BTF types, like BTF_KIND_FUNC_PROTO, are anonymous.  The machinery
+   in btfout to emit BTF, may reset dtd_data->ctti_name, but does not update
+   the name in the ctf_dtdef_ref type object (deliberate choice).  This
+   interface helps abstract out that state of affairs, while giving access to
+   the name of the type as intended.  */
+
+static const char *
+get_btf_type_name (ctf_dtdef_ref dtd)
+{
+  const char *anon = "";
+  return (dtd->dtd_data.ctti_name) ? dtd->dtd_name : anon;
+}
+
 /* Helper routines to map between 'relative' and 'absolute' IDs.
 
In BTF all records (including variables) are output in one long list, and 
all
@@ -425,6 +438,7 @@ btf_collect_datasec (ctf_container_ref ctfc)
   func_dtd->dtd_data = dtd->dtd_data;
   func_dtd->dtd_data.ctti_type = dtd->dtd_type;
   func_dtd->linkage = dtd->linkage;
+  func_dtd->dtd_name = dtd->dtd_name;
   func_dtd->dtd_type = num_types_added + num_types_created;
 
   /* Only the BTF_KIND_FUNC type actually references the name. The
@@ -722,7 +736,7 @@ btf_asm_type_ref (const char *prefix, ctf_container_ref 
ctfc, ctf_id_t ref_id)
   size_t func_id = btf_relative_func_id (ref_id);
   ctf_dtdef_ref ref_type = (*funcs)[func_id];
   dw2_asm_output_data (4, ref_id, "%s: (BTF_KIND_FUNC '%s')",
-  prefix, ref_type->dtd_name);
+  prefix, get_btf_type_name (ref_type));
 }
   else
 {
@@ -733,7 +747,7 @@ btf_asm_type_ref (const char *prefix, ctf_container_ref 
ctfc, ctf_id_t ref_id)
 
   dw2_asm_output_data (4, ref_id, "%s: (BTF_KIND_%s '%s')",
   prefix, btf_kind_name (ref_kind),
-  ref_type->dtd_name);
+  get_btf_type_name (ref_type));
 }
 }
 
@@ -809,7 +823,7 @@ btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
   dw2_asm_output_data (4, dtd->dtd_data.ctti_name,
   "TYPE %" PRIu64 " BTF_KIND_%s '%s'",
   get_btf_id (dtd->dtd_type), btf_kind_name (btf_kind),
-  dtd->dtd_name);
+  get_btf_type_name (dtd));
   dw2_asm_output_data (4, BTF_TYPE_INFO (btf_kind, btf_kflag, btf_vlen),
   "btt_info: kind=%u, kflag=%u, vlen=%u",
   btf_kind, btf_kflag, btf_vlen);
@@ -950,7 +964,7 @@ btf_asm_func_type (ctf_container_ref ctfc, ctf_dtdef_ref 
dtd, ctf_id_t id)
   ctf_id_t ref_id = dtd->dtd_data.ctti_type;
   dw2_asm_output_data (4, dtd->dtd_data.ctti_name,
   "TYPE %" PRIu64 " BTF_KIND_FUNC '%s'",
-  btf_absolute_func_id (id), dtd->dtd_name);
+  

[PATCH] btf: fix PR debug/112656

[Indu Bhagat]

PR debug/112656 - btf: function prototypes generated with name

With this patch, all BTF_KIND_FUNC_PROTO will appear anonymous in the
generated BTF section.

As noted in the discussion in the bugzilla, the number of
BTF_KIND_FUNC_PROTO types output varies across targets (BPF with -mco-re
vs non-BPF targets).  Hence the check in the test case merely checks
that all BTF_KIND_FUNC_PROTO appear anonymous.

gcc/ChangeLog:

PR debug/112656
* btfout.cc (btf_asm_type): Fixup ctti_name for all
BTF types of kind BTF_KIND_FUNC_PROTO.

gcc/testsuite/ChangeLog:

PR debug/112656
* gcc.dg/debug/btf/btf-function-7.c: New test.


Testing notes:
  - bootstrapped and reg tested on x86_64
  - No regressions in btf.exp on BPF target

---
 gcc/btfout.cc |  4 
 .../gcc.dg/debug/btf/btf-function-7.c | 19 +++
 2 files changed, 23 insertions(+)
 create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-function-7.c

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 1c25404b2c0..a5e0d640e19 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -820,6 +820,10 @@ btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
btf_kind = BTF_KIND_ENUM64;
}
 
+  /* PR debug/112656.  BTF_KIND_FUNC_PROTO is always anonymous.  */
+  if (btf_kind == BTF_KIND_FUNC_PROTO)
+dtd->dtd_data.ctti_name = 0;
+
   dw2_asm_output_data (4, dtd->dtd_data.ctti_name,
   "TYPE %" PRIu64 " BTF_KIND_%s '%s'",
   get_btf_id (dtd->dtd_type), btf_kind_name (btf_kind),
diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-function-7.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-function-7.c
new file mode 100644
index 000..b560dc75650
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-function-7.c
@@ -0,0 +1,19 @@
+/* Test BTF for inlined functions.
+
+   See PR/112656 - btf: function prototypes generated with name
+   BTF_KIND_FUNC_PROTO must be anonymous.  */
+
+/* { dg-do compile } */
+/* { dg-options "-O2 -gbtf -dA" } */
+
+/* { dg-final { scan-assembler-times "BTF_KIND_FUNC_PROTO 
''\\(\[0-9a-z\]*\\)'" 0 } } */
+
+static int log_event(const char *event_name, void *dev_ptr)
+{
+  return 666;
+}
+
+int foo ()
+{
+  return log_event ("foobar", ((void *)0));
+}
-- 
2.41.0

Re: [PATCH] btf: avoid wrong DATASEC entries for extern vars [PR112849]

[Indu Bhagat]

On 12/4/23 15:47, David Faust wrote:

The process of creating BTF_KIND_DATASEC records involves iterating
through variable declarations, determining which section they will be
placed in, and creating an entry in the appropriate DATASEC record
accordingly.

For variables without e.g. an explicit __attribute__((section)), we use
categorize_decl_for_section () to identify the appropriate named section
and corresponding BTF_KIND_DATASEC record.

This was incorrectly being done for 'extern' variable declarations as
well as non-extern ones, which meant that extern variable declarations
could result in BTF_KIND_DATASEC entries claiming the variable is
allocated in some section such as '.bss' without any knowledge whether
that is actually true. That resulted in errors building the Linux kernel
BPF selftests.

This patch corrects btf_collect_datasec () to avoid assuming a section
for extern variables, and only emit BTF_KIND_DATASEC entries for them if
they have a known section.

Bootstrapped + tested on x86_64-linux-gnu.
Tested on x86_64-linux-gnu host for bpf-unknown-none.



One comment below.

LGTM, otherwise.
Thanks


gcc/
PR debug/112849
* btfout.cc (btf_collect_datasec): Avoid incorrectly creating an
entry in a BTF_KIND_DATASEC record for extern variable decls without
a known section.

gcc/testsuite/
PR debug/112849
* gcc.dg/debug/btf/btf-datasec-3.c: New test.
---
  gcc/btfout.cc | 10 ++-
  .../gcc.dg/debug/btf/btf-datasec-3.c  | 27 +++
  2 files changed, 36 insertions(+), 1 deletion(-)
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index a5e0d640e19..db4f1084f85 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -486,7 +486,15 @@ btf_collect_datasec (ctf_container_ref ctfc)
  
/* Mark extern variables.  */

if (DECL_EXTERNAL (node->decl))
-   dvd->dvd_visibility = BTF_VAR_GLOBAL_EXTERN;
+   {
+ dvd->dvd_visibility = BTF_VAR_GLOBAL_EXTERN;
+
+ /* PR112849: avoid assuming a section for extern decls without
+an explicit section, which would result in incorrectly
+emitting a BTF_KIND_DATASEC entry for them.  */
+ if (node->get_section () == NULL)
+   continue;
+   }
  
const char *section_name = get_section_name (node);

if (section_name == NULL)
diff --git a/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c 
b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c
new file mode 100644
index 000..3c1c7a28c2a
--- /dev/null
+++ b/gcc/testsuite/gcc.dg/debug/btf/btf-datasec-3.c
@@ -0,0 +1,27 @@
+/* PR debug/112849
+   Test that we do not incorrectly create BTF_KIND_DATASEC entries for
+   extern decls with no known section.  */
+
+/* { dg-do compile } */
+/* { dg-options "-O0 -gbtf -dA" } */
+
+extern int VERSION __attribute__((section (".version")));
+
+extern int test_bss1;
+extern int test_data1;
+
+int test_bss2;
+int test_data2 = 2;
+
+int
+foo (void)
+{
+  test_bss2 = VERSION;
+  return test_bss1 + test_data1 + test_data2;
+}
+
+/* There should only be a DATASEC entries for VERSION out of the extern decls. 
 */


The statement is unclear as is. Perhaps you wanted to say "There should 
only be 3 DATASEC entries; including one for VERSION even though it is 
extern decl" ?



+/* { dg-final { scan-assembler-times "bts_type" 3 } } */
+/* { dg-final { scan-assembler-times "bts_type: \\(BTF_KIND_VAR 
'test_data2'\\)" 1 } } */
+/* { dg-final { scan-assembler-times "bts_type: \\(BTF_KIND_VAR 
'test_bss2'\\)" 1 } } */
+/* { dg-final { scan-assembler-times "bts_type: \\(BTF_KIND_VAR 'VERSION'\\)" 
1 } } */

[[PATCH][GCC13] 0/2] Fix combined tree build of GCC 13 with Binutils 2.41

[Indu Bhagat]

Hello,

To resolve the issue of combined Binutils (2.41) + GCC (13) failing to
install (https://sourceware.org/bugzilla/show_bug.cgi?id=31108), we will
need some backports.  This specific issue is with using --enable-shared
in the combined tree build; it arises due to missing install-*
dependencies in the top-level makefiles.

I think it makes sense to bring both of the following two commits (from
the trunk) to the GCC13 branch:

commit eff0e7a4ae31d1e4e64ae37bbc10d073d8579255
Author: Indu Bhagat 
Date:   Wed Jan 18 23:17:49 2023 -0800
toplevel: Makefile.def: add install-strip dependency on libsframe


commit dab58c93634bef06fd289f49109b5c370cd5c380
Author: Indu Bhagat 
Date:   Tue Nov 15 15:07:04 2022 -0800
bfd: linker: merge .sframe sections

This patch set cherry-picks the above two commits to GCC13 branch.  The
patches apply cleanly with no conflicts.

---
Testing notes:
 - Combined tree with GCC 13 (releases/gcc-13 branch) with binutils 2.41
   (binutils-2_41-release-point branch) with "--enable-shared
   --disable-bootstrap" builds and installs.
 - Bootstrapped and regression tested releases/gcc-13 branch (make
   check-gcc in a NOT combined tree build).
---

Thanks,
Indu Bhagat (2):
  bfd: linker: merge .sframe sections
  toplevel: Makefile.def: add install-strip dependency on libsframe

 Makefile.def |  7 +++
 Makefile.in  | 12 
 2 files changed, 19 insertions(+)

-- 
2.41.0

[[PATCH][GCC13] 1/2] bfd: linker: merge .sframe sections

[Indu Bhagat]

The linker merges all the input .sframe sections.  When merging, the
linker verifies that all the input .sframe sections have the same
abi/arch.

The linker uses libsframe library to perform key actions on the
.sframe sections - decode, read, and create output data.  This
implies buildsystem changes to make and install libsframe before
libbfd.

The linker places the output .sframe section in a new segment of its
own: PT_GNU_SFRAME.  A new segment is not added, however, if the
generated .sframe section is empty.

When a section is discarded from the final link, the corresponding
entries in the .sframe section for those functions are also deleted.

The linker sorts the SFrame FDEs on start address by default and sets
the SFRAME_F_FDE_SORTED flag in the .sframe section.

This patch also adds support for generation of SFrame unwind
information for the .plt* sections on x86_64.  SFrame unwind info is
generated for IBT enabled PLT, lazy/non-lazy PLT.

The existing linker option --no-ld-generated-unwind-info has been
adapted to include the control of whether .sframe unwind information
will be generated for the linker generated sections like PLT.

Changes to the linker script have been made as necessary.

ChangeLog:

* Makefile.def: Add install dependency on libsframe for libbfd.
* Makefile.in: Regenerated.

(cherry picked from commit dab58c93634bef06fd289f49109b5c370cd5c380)
---
 Makefile.def |  4 
 Makefile.in  | 11 +++
 2 files changed, 15 insertions(+)

diff --git a/Makefile.def b/Makefile.def
index 35e994eb77e..41512475042 100644
--- a/Makefile.def
+++ b/Makefile.def
@@ -457,11 +457,14 @@ dependencies = { module=all-gdbsupport; on=all-gnulib; };
 dependencies = { module=all-gdbsupport; on=all-intl; };
 
 // Host modules specific to binutils.
+// build libsframe before bfd for encoder/decoder support for linking
+// SFrame sections
 dependencies = { module=configure-bfd; on=configure-libiberty; hard=true; };
 dependencies = { module=configure-bfd; on=configure-intl; };
 dependencies = { module=all-bfd; on=all-libiberty; };
 dependencies = { module=all-bfd; on=all-intl; };
 dependencies = { module=all-bfd; on=all-zlib; };
+dependencies = { module=all-bfd; on=all-libsframe; };
 dependencies = { module=configure-opcodes; on=configure-libiberty; hard=true; 
};
 dependencies = { module=all-opcodes; on=all-libiberty; };
 
@@ -487,6 +490,7 @@ dependencies = { module=install-binutils; 
on=install-opcodes; };
 dependencies = { module=install-strip-binutils; on=install-strip-opcodes; };
 
 // Likewise for ld, libctf, and bfd.
+dependencies = { module=install-bfd; on=install-libsframe; };
 dependencies = { module=install-libctf; on=install-bfd; };
 dependencies = { module=install-ld; on=install-bfd; };
 dependencies = { module=install-ld; on=install-libctf; };
diff --git a/Makefile.in b/Makefile.in
index 06a9398e172..076a48944b8 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -65849,6 +65849,16 @@ all-stagetrain-bfd: maybe-all-stagetrain-zlib
 all-stagefeedback-bfd: maybe-all-stagefeedback-zlib
 all-stageautoprofile-bfd: maybe-all-stageautoprofile-zlib
 all-stageautofeedback-bfd: maybe-all-stageautofeedback-zlib
+all-bfd: maybe-all-libsframe
+all-stage1-bfd: maybe-all-stage1-libsframe
+all-stage2-bfd: maybe-all-stage2-libsframe
+all-stage3-bfd: maybe-all-stage3-libsframe
+all-stage4-bfd: maybe-all-stage4-libsframe
+all-stageprofile-bfd: maybe-all-stageprofile-libsframe
+all-stagetrain-bfd: maybe-all-stagetrain-libsframe
+all-stagefeedback-bfd: maybe-all-stagefeedback-libsframe
+all-stageautoprofile-bfd: maybe-all-stageautoprofile-libsframe
+all-stageautofeedback-bfd: maybe-all-stageautofeedback-libsframe
 configure-opcodes: configure-libiberty
 configure-stage1-opcodes: configure-stage1-libiberty
 configure-stage2-opcodes: configure-stage2-libiberty
@@ -65981,6 +65991,7 @@ all-stageautoprofile-binutils: 
maybe-all-stageautoprofile-libsframe
 all-stageautofeedback-binutils: maybe-all-stageautofeedback-libsframe
 install-binutils: maybe-install-opcodes
 install-strip-binutils: maybe-install-strip-opcodes
+install-bfd: maybe-install-libsframe
 install-libctf: maybe-install-bfd
 install-ld: maybe-install-bfd
 install-ld: maybe-install-libctf
-- 
2.41.0

[[PATCH][GCC13] 2/2] toplevel: Makefile.def: add install-strip dependency on libsframe

[Indu Bhagat]

As noted in PR libsframe/30014 - FTBFS: install-strip fails because
bfdlib relinks and fails to find libsframe, the install time
dependencies of libbfd need to be updated.

ChangeLog:

* Makefile.def: Reflect that libsframe needs to installed before
libbfd.  Reorder a bit to better track libsframe dependencies.
* Makefile.in: Regenerate.

(cherry picked from commit eff0e7a4ae31d1e4e64ae37bbc10d073d8579255)
---
 Makefile.def | 5 -
 Makefile.in  | 3 ++-
 2 files changed, 6 insertions(+), 2 deletions(-)

diff --git a/Makefile.def b/Makefile.def
index 41512475042..0c107cae128 100644
--- a/Makefile.def
+++ b/Makefile.def
@@ -490,7 +490,6 @@ dependencies = { module=install-binutils; 
on=install-opcodes; };
 dependencies = { module=install-strip-binutils; on=install-strip-opcodes; };
 
 // Likewise for ld, libctf, and bfd.
-dependencies = { module=install-bfd; on=install-libsframe; };
 dependencies = { module=install-libctf; on=install-bfd; };
 dependencies = { module=install-ld; on=install-bfd; };
 dependencies = { module=install-ld; on=install-libctf; };
@@ -498,6 +497,10 @@ dependencies = { module=install-strip-libctf; 
on=install-strip-bfd; };
 dependencies = { module=install-strip-ld; on=install-strip-bfd; };
 dependencies = { module=install-strip-ld; on=install-strip-libctf; };
 
+// libbfd depends on libsframe
+dependencies = { module=install-bfd; on=install-libsframe; };
+dependencies = { module=install-strip-bfd; on=install-strip-libsframe; };
+
 // libopcodes depends on libbfd
 dependencies = { module=configure-opcodes; on=configure-bfd; hard=true; };
 dependencies = { module=install-opcodes; on=install-bfd; };
diff --git a/Makefile.in b/Makefile.in
index 076a48944b8..c1a607ac564 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -65991,13 +65991,14 @@ all-stageautoprofile-binutils: 
maybe-all-stageautoprofile-libsframe
 all-stageautofeedback-binutils: maybe-all-stageautofeedback-libsframe
 install-binutils: maybe-install-opcodes
 install-strip-binutils: maybe-install-strip-opcodes
-install-bfd: maybe-install-libsframe
 install-libctf: maybe-install-bfd
 install-ld: maybe-install-bfd
 install-ld: maybe-install-libctf
 install-strip-libctf: maybe-install-strip-bfd
 install-strip-ld: maybe-install-strip-bfd
 install-strip-ld: maybe-install-strip-libctf
+install-bfd: maybe-install-libsframe
+install-strip-bfd: maybe-install-strip-libsframe
 configure-opcodes: configure-bfd
 configure-stage1-opcodes: configure-stage1-bfd
 configure-stage2-opcodes: configure-stage2-bfd
-- 
2.41.0

Re: [[PATCH][GCC13] 0/2] Fix combined tree build of GCC 13 with Binutils 2.41

[Indu Bhagat]

On 12/5/23 13:45, Jakub Jelinek wrote:

On Tue, Dec 05, 2023 at 01:36:30PM -0800, Indu Bhagat wrote:

To resolve the issue of combined Binutils (2.41) + GCC (13) failing to
install (https://sourceware.org/bugzilla/show_bug.cgi?id=31108), we will
need some backports.  This specific issue is with using --enable-shared
in the combined tree build; it arises due to missing install-*
dependencies in the top-level makefiles.

I think it makes sense to bring both of the following two commits (from
the trunk) to the GCC13 branch:

commit eff0e7a4ae31d1e4e64ae37bbc10d073d8579255
Author: Indu Bhagat 
Date:   Wed Jan 18 23:17:49 2023 -0800
toplevel: Makefile.def: add install-strip dependency on libsframe
 


commit dab58c93634bef06fd289f49109b5c370cd5c380
Author: Indu Bhagat 
Date:   Tue Nov 15 15:07:04 2022 -0800
bfd: linker: merge .sframe sections

This patch set cherry-picks the above two commits to GCC13 branch.  The
patches apply cleanly with no conflicts.


Won't this break building gcc 13 with in-tree older binutils which don't have
libsframe at all?  I think binutils 2.39 and older don't have it.



I tested with binutils-2_39-branch and releases/gcc-13 as well (with 
--enable-shared --disable-bootstrap). It builds and installs fine.


Indu

Re: [PATCH v3 1/6] ctf, btf: restructure CTF/BTF emission

[Indu Bhagat]

On 5/30/24 14:32, David Faust wrote:

This commit makes some structural changes to the CTF/BTF debug info
emission.  In particular:

  a) CTF is new always fully generated and emitted before any
 BTF-related procedures are run.  This means that BTF-related
 functions can change, even irreversibly, the shared in-memory
 representation used by the two formats without issue.

  b) BTF generation has fewer entry points, and is cleanly divided
 into early_finish and finish.

  c) BTF is now always emitted at finish (called from dwarf2out_finish),
 for all targets in non-LTO builds, rather than being emitted at
 early_finish for targets other than BPF CO-RE.  In LTO builds,
 BTF is emitted at early_finish as before.

 Note that this change alone does not alter the contents of BTF at
 all, regardless of whether it would have previously been emitted at
 early_finish or finish, because the calculation of the BTF to be
 emitted is not moved by this patch, only the write-out.

The changes are transparent to both CTF and BTF emission.



OK.

This will work to keep supporting -flto with BTF (for non-BPF targets) 
and of course -flto with CTF.


One question/nit below.


gcc/
* btfout.cc (btf_init_postprocess): Rename to...
(btf_early_finish): ...this.
(btf_output): Rename to...
(btf_finish): ...this.
* ctfc.h: Analogous changes.
* dwarf2ctf.cc (ctf_debug_early_finish): Conditionally call
btf_early_finish, or ctf_finalize as appropriate.  Emit BTF
here for LTO builds.
(ctf_debug_finish): Always call btf_finish here if generating
BTF info in non-LTO builds.
(ctf_debug_finalize, ctf_debug_init_postprocess): Delete.
* dwarf2out.cc (dwarf2out_early_finish): Remove call to
ctf_debug_init_postprocess.
---
  gcc/btfout.cc| 28 +
  gcc/ctfc.h   |  4 +--
  gcc/dwarf2ctf.cc | 65 +++-
  gcc/dwarf2out.cc |  2 --
  4 files changed, 50 insertions(+), 49 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 07f066a47068..1b6a9ed811f0 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -1491,6 +1491,34 @@ btf_finalize (void)
tu_ctfc = NULL;
  }
  
+/* Initial entry point of BTF generation, called at early_finish () after

+   CTF information has possibly been output.  Translate all CTF information
+   to BTF, and do any processing that must be done early, such as creating
+   BTF_KIND_FUNC records.  */
+
+void
+btf_early_finish (void)
+{
+  btf_init_postprocess ();
+}
+
+/* Late entry point for BTF generation, called from dwarf2out_finish ().
+   Complete and emit BTF information.  */
+
+void
+btf_finish (const char * filename)
+{
+  btf_output (filename);
+
+  /* If compiling for BPF with CO-RE info, we cannot deallocate until after
+ CO-RE information is created, which happens very late in BPF backend.


I am wondering if it is more precise to say that "until after contents 
for the .BTF.ext section are finalized" ? We have already called the 
btf_output () above, which means _some_ CO-RE information is already 
created (like the accessor strings for CO-RE relocs that go in the .BTF 
section).



+ Therefore, the deallocation (i.e. btf_finalize ()) is delayed until
+ TARGET_ASM_FILE_END for BPF CO-RE.  */
+  if (!btf_with_core_debuginfo_p ())
+btf_finalize ();
+}
+
+
  /* Traversal function for all BTF_KIND_FUNC type records.  */
  
  bool

diff --git a/gcc/ctfc.h b/gcc/ctfc.h
index fa188bf2f5a4..e7bd93901cfa 100644
--- a/gcc/ctfc.h
+++ b/gcc/ctfc.h
@@ -384,8 +384,8 @@ extern void ctf_init (void);
  extern void ctf_output (const char * filename);
  extern void ctf_finalize (void);
  
-extern void btf_output (const char * filename);

-extern void btf_init_postprocess (void);
+extern void btf_early_finish (void);
+extern void btf_finish (const char * filename);
  extern void btf_finalize (void);
  
  extern ctf_container_ref ctf_get_tu_ctfc (void);

diff --git a/gcc/dwarf2ctf.cc b/gcc/dwarf2ctf.cc
index dc59569fe560..8f9e2fada9e3 100644
--- a/gcc/dwarf2ctf.cc
+++ b/gcc/dwarf2ctf.cc
@@ -933,30 +933,6 @@ gen_ctf_type (ctf_container_ref ctfc, dw_die_ref die)
return type_id;
  }
  
-/* Prepare for output and write out the CTF debug information.  */

-
-static void
-ctf_debug_finalize (const char *filename, bool btf)
-{
-  if (btf)
-{
-  btf_output (filename);
-  /* btf_finalize when compiling BPF applciations gets deallocated by the
-BPF target in bpf_file_end.  */
-  if (btf_debuginfo_p () && !btf_with_core_debuginfo_p ())
-   btf_finalize ();
-}
-
-  else
-{
-  /* Emit the collected CTF information.  */
-  ctf_output (filename);
-
-  /* Reset the CTF state.  */
-  ctf_finalize ();
-}
-}
-
  bool
  ctf_do_die (dw_die_ref die)
  {
@@ -996,27 +972,27 @@ ctf_debug_init (void)
add_name_attribute (ctf_unknown_die, "unknown");
  }
  
-/* 

Re: [PATCH v3 2/6] ctf: use pointers instead of IDs internally

[Indu Bhagat]

On 5/30/24 14:32, David Faust wrote:

This patch replaces all inter-type references in the ctfc internal data
structures with pointers, rather than the references-by-ID which were
used previously.

A couple of small updates in the BPF backend are included to make it
compatible with the change.

This change is only to the in-memory representation of various CTF
structures to make them easier to work with in various cases.  It is
outwardly transparent; there is no change in emitted CTF.

gcc/
* btfout.cc (BTF_VOID_TYPEID, BTF_INIT_TYPEID): Move defines to
include/btf.h.
(btf_dvd_emit_preprocess_cb, btf_emit_preprocess)
(btf_dmd_representable_bitfield_p, btf_asm_array, btf_asm_varent)
(btf_asm_sou_member, btf_asm_func_arg, btf_init_postprocess):
Adapt to structural changes in ctf_* structs.
* ctfc.h (struct ctf_dtdef): Add forward declaration.
(ctf_dtdef_t, ctf_dtdef_ref): Move typedefs earlier.
(struct ctf_arinfo, struct ctf_funcinfo, struct ctf_sliceinfo)
(struct ctf_itype, struct ctf_dmdef, struct ctf_func_arg)
(struct ctf_dvdef): Use pointers instead of type IDs for
references to other types and use typedefs where appropriate.
(struct ctf_dtdef): Add ref_type member.
(ctf_type_exists): Use pointer instead of type ID.
(ctf_add_reftype, ctf_add_enum, ctf_add_slice, ctf_add_float)
(ctf_add_integer, ctf_add_unknown, ctf_add_pointer)
(ctf_add_array, ctf_add_forward, ctf_add_typedef)
(ctf_add_function, ctf_add_sou, ctf_add_enumerator)
(ctf_add_variable): Likewise. Return pointer instead of ID.
(ctf_lookup_tree_type): Return pointer to type instead of ID.
* ctfc.cc: Analogous changes.
* ctfout.cc (ctf_asm_type, ctf_asm_slice, ctf_asm_varent)
(ctf_asm_sou_lmember, ctf_asm_sou_member, ctf_asm_func_arg)
(output_ctf_objt_info): Adapt to changes.
* dwarf2ctf.cc (gen_ctf_type, gen_ctf_void_type)
(gen_ctf_unknown_type, gen_ctf_base_type, gen_ctf_pointer_type)
(gen_ctf_subrange_type, gen_ctf_array_type, gen_ctf_typedef)
(gen_ctf_modifier_type, gen_ctf_sou_type, gen_ctf_function_type)
(gen_ctf_enumeration_type, gen_ctf_variable, gen_ctf_function)
(gen_ctf_type, ctf_do_die): Likewise.
* config/bpf/btfext-out.cc (struct btf_ext_core_reloc): Use
pointer instead of type ID.
(bpf_core_reloc_add, bpf_core_get_sou_member_index)
(output_btfext_core_sections): Adapt to above changes.
* config/bpf/core-builtins.cc (process_type): Likewise.

include/
* btf.h (BTF_VOID_TYPEID, BTF_INIT_TYPEID): Move defines here,
from gcc/btfout.cc.


The patch looks OK overall. Few updates to code comments for function 
description or in the body of the function were missed.  I have pointed 
them out below.


OK with those addressed.

Thanks


---
  gcc/btfout.cc   |  40 +++---
  gcc/config/bpf/btfext-out.cc|  14 +-
  gcc/config/bpf/core-builtins.cc |   3 +-
  gcc/ctfc.cc | 139 +-
  gcc/ctfc.h  |  90 ++--
  gcc/ctfout.cc   |  22 +--
  gcc/dwarf2ctf.cc| 244 +++-
  include/btf.h   |   5 +
  8 files changed, 278 insertions(+), 279 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 1b6a9ed811f0..40e8d8c5c01b 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -61,11 +61,6 @@ static char btf_info_section_label[MAX_BTF_LABEL_BYTES];
  #define BTF_INFO_SECTION_LABEL  "Lbtf"
  #endif
  
-/* BTF encodes void as type id 0.  */

-
-#define BTF_VOID_TYPEID 0
-#define BTF_INIT_TYPEID 1
-
  #define BTF_INVALID_TYPEID 0x
  
  /* Mapping of CTF variables to the IDs they will be assigned when they are

@@ -626,7 +621,8 @@ btf_dvd_emit_preprocess_cb (ctf_dvdef_ref *slot, 
ctf_container_ref arg_ctfc)
  return 1;
  
/* Do not add variables which refer to unsupported types.  */

-  if (!voids.contains (var->dvd_type) && btf_removed_type_p (var->dvd_type))
+  if (!voids.contains (var->dvd_type->dtd_type)
+  && btf_removed_type_p (var->dvd_type->dtd_type))
  return 1;
  
arg_ctfc->ctfc_vars_list[num_vars_added] = var;

@@ -716,7 +712,7 @@ btf_emit_preprocess (ctf_container_ref ctfc)
  static bool
  btf_dmd_representable_bitfield_p (ctf_container_ref ctfc, ctf_dmdef_t *dmd)
  {
-  ctf_dtdef_ref ref_type = ctfc->ctfc_types_list[dmd->dmd_type];
+  ctf_dtdef_ref ref_type = ctfc->ctfc_types_list[dmd->dmd_type->dtd_type];
  
if (CTF_V2_INFO_KIND (ref_type->dtd_data.ctti_info) == CTF_K_SLICE)

  {
@@ -913,8 +909,8 @@ btf_asm_type (ctf_container_ref ctfc, ctf_dtdef_ref dtd)
  static void
  btf_asm_array (ctf_container_ref ctfc, ctf_arinfo_t arr)
  {
-  btf_asm_type_ref ("bta_elem_type", ctfc, arr.ctr_contents);
-  btf_asm_type_ref ("bta_index_type", ctfc, arr.ctr_index);
+  btf_asm_typ

Re: [PATCH v3 3/6] btf: refactor and simplify implementation

[Indu Bhagat]

On 5/30/24 14:32, David Faust wrote:

This patch heavily refactors btfout.cc to take advantage of the
structural changes in the prior commits.

Now that inter-type references are internally stored as simply pointers,
all the painful, brittle, confusing infrastructure that was used in the
process of converting CTF type IDs to BTF type IDs can be thrown out.
This greatly simplifies the entire process of converting from CTF to
BTF, making the code cleaner, easier to read, and easier to maintain.

In addition, we no longer need to worry about destructive changes in
internal data structures used commonly by CTF and BTF, which allows
deleting several ancillary data structures previously used in btfout.cc.

This is nearly transparent, but a few improvements have also been made:

  1) BTF_KIND_FUNC records are now _always_ constructed at early_finish,
 allowing us to construct records even for functions which are later
 inlined by optimizations. DATASEC entries for functions are only
 constructed at late_finish, to avoid incorrectly generating entries
 for functions which get inlined.



I find this aspect of the BTF specification "interesting": Neither 
BTF_KIND_VAR record, nor DATASEC entry must be emitted for optimized 
away variables.  But for functions, OTH,  BTF_KIND_FUNC (and 
BTF_KIND_FUNC_PROTO (albeit anon) ?) are expected for inlined functions, 
but no DATASEC.


Anayway, that aside, I have one comment below.  Other than that, this 
looks OK to me.


Thanks


  2) BTF_KIND_VAR records and DATASEC entries for them are now always
 constructed at (late) finish, which avoids cases where we could
 incorrectly create records for variables which were completely
 optimized away.  This fixes PR debug/113566 for non-LTO builds.
 In LTO builds, BTF must be emitted at early_finish, so some VAR
 records may be emitted for variables which are later optimized away.

  3) Some additional assembler comments have been added with more
 information for debugging.

gcc/
* btfout.cc (struct btf_datasec_entry): New.
(struct btf_datasec): Add `id' member.  Change `entries' to use
new struct btf_datasec_entry.
(func_map): New hash_map.
(max_translated_id): New.
(btf_var_ids, btf_id_map, holes, voids, num_vars_added)
(num_types_added, num_types_created): Delete.
(btf_absolute_var_id, btf_relative_var_id, btf_absolute_func_id)
(btf_relative_func_id, btf_absolute_datasec_id, init_btf_id_map)
(get_btf_id, set_btf_id, btf_emit_id_p): Delete.
(btf_removed_type_p): Delete.
(btf_dtd_kind, btf_emit_type_p): New helpers.
(btf_fwd_to_enum_p, btf_calc_num_vbytes): Use them.
(btf_collect_datasec): Delete.
(btf_dtd_postprocess_cb, btf_dvd_emit_preprocess_cb)
(btf_dtd_emit_preprocess_cb, btf_emit_preprocess): Delete.
(btf_dmd_representable_bitfield_p): Adapt to type reference changes
and delete now-unused ctfc argument.
(btf_asm_datasec_type_ref): Delete.
(btf_asm_type_ref): Adapt to type reference changes, simplify.
(btf_asm_type): Likewise. Mark struct/union types with bitfield
members.
(btf_asm_array): Adapt to data structure changes.
(btf_asm_varent): Likewise.
(btf_asm_sou_member): Likewise. Ensure non-bitfield members are
correctly re-encoded if struct or union contains any bitfield.
(btf_asm_func_arg, btf_asm_func_type, btf_asm_datasec_entry)
(btf_asm_datasec_type): Adapt to data structure changes.
(output_btf_header): Adapt to other changes, simplify type
length calculation, add info to assembler comments.
(output_btf_vars): Adapt to other changes.
(output_btf_strs): Fix overlong lines.
(output_asm_btf_sou_fields, output_asm_btf_enum_list)
(output_asm_btf_func_args_list, output_asm_btf_vlen_bytes)
(output_asm_btf_type, output_btf_types, output_btf_func_types)
(output_btf_datasec_types): Adapt to other changes.
(btf_init_postprocess): Delete.
(btf_output): Change to only perform output.
(btf_early_add_const_void, btf_early_add_func_records): New.
(btf_early_finish): Use them here. New.
(btf_datasec_push_entry): Adapt to data structure changes.
(btf_datasec_add_func, btf_datasec_add_var): New.
(btf_late_add_func_datasec_entries): New.
(btf_emit_variable_p): New helper.
(btf_late_add_vars): Use it here. New.
(btf_type_list_cb, btf_late_collect_translated_types): New.
(btf_late_assign_func_ids, btf_late_assign_var_ids)
(btf_late_assign_datasec_ids): New.
(btf_finish): Remove unused argument. Call new btf_late*
functions and btf_output.
(btf_finalize): Adapt to data structure changes.
* ctfc.h (struct ctf_dtdef): Convert existing boolean flags to
BOOL_BITFIELD and reorder.
(s

Re: [PATCH v3 4/6] btf: add -fprune-btf option

[Indu Bhagat]

On 5/30/24 14:32, David Faust wrote:

This patch adds a new option, -fprune-btf, to control BTF debug info
generation.

As the name implies, this option enables a kind of "pruning" of the BTF
information before it is emitted.  When enabled, rather than emitting
all type information translated from DWARF, only information for types
directly used in the source program is emitted.

The primary purpose of this pruning is to reduce the amount of
unnecessary BTF information emitted, especially for BPF programs.  It is
very common for BPF programs to incldue Linux kernel internal headers in


typo: incldue


order to have access to kernel data structures.  However, doing so often
has the side effect of also adding type definitions for a large number
of types which are not actually used by nor relevant to the program.
In these cases, -fprune-btf commonly reduces the size of the resulting
BTF information by 10x or more, as seen on average when compiling Linux
kernel BPF selftests.  This both slims down the size of the resulting
object and reduces the time required by the BPF loader to verify the
program and its BTF information.

Note that the pruning implemented in this patch follows the same rules
as the BTF pruning performed unconditionally by LLVM's BPF backend when
generating BTF.  In particular, the main sources of pruning are:

   1) Only generate BTF for types used by variables and functions at
  the file scope.  Note that with or without pruning, BTF_KIND_VAR
  entries are only generated for variables present in the final
  object - unused static variables or variables completely optimized
  away must not have VAR entries in BTF.



This needs adjusting. I think you mean to convey that the above is 
expected behavior of -fprune-btf (-gprune-btf) for BPF backend.  But the 
option as such is also available for non-BPF backends, where its 
behavior will not be the one stated here (due to BTF creation, pruning 
and output work being at the time of early_finish () when LTO is enabled)


Perhaps we clearly specify the behavior of -fprune-btf for BPF and 
non-BPF backends ? I wonder if the right approach is to instead disallow 
-fprune-btf  with -flto, however until it is clear what that effectively 
means.


One nit below.


   2) Avoid emitting full BTF for struct and union types which are only
  pointed-to by members of other struct/union types.  In these cases,
  the full BTF_KIND_STRUCT or BTF_KIND_UNION which would normally
  be emitted is replaced with a BTF_KIND_FWD, as though the
  underlying type was a forward-declared struct or union type.

gcc/
* btfout.cc (btf_used_types): New hash set.
(struct btf_fixup): New.
(fixups, forwards): New vecs.
(btf_output): Calculate num_types depending on flag_prune_btf.
(btf_early_finsih): New initialization for flag_prune_btf.
(btf_add_used_type): New function.
(btf_used_type_list_cb): Likewise.
(btf_late_collect_pruned_types): Likewise.
(btf_late_add_vars): Handle special case for variables in ".maps"
section when generating BTF for BPF CO-RE target.
(btf_late_finish): Use btf_late_collect_pruned_types when
flag_prune_btf in effect.  Move some initialization to btf_early_finish.
(btf_finalize): Additional deallocation for flag_prune_btf.
* common.opt (fprune-btf): New flag.
* ctfc.cc (init_ctf_strtable): Make non-static.
* ctfc.h (struct ctf_dtdef): Add visited_children_p boolean flag.
(init_ctf_strtable, ctfc_delete_strtab): Make extern.
* doc/invoke.texi (Debugging Options): Document -fprune-btf.

gcc/testsuite/
* gcc.dg/debug/btf/btf-prune-1.c: New test.
* gcc.dg/debug/btf/btf-prune-2.c: Likewise.
* gcc.dg/debug/btf/btf-prune-3.c: Likewise.
* gcc.dg/debug/btf/btf-prune-maps.c: Likewise.
---
  gcc/btfout.cc | 359 +-
  gcc/common.opt|   4 +
  gcc/ctfc.cc   |   2 +-
  gcc/ctfc.h|   3 +
  gcc/doc/invoke.texi   |  20 +
  gcc/testsuite/gcc.dg/debug/btf/btf-prune-1.c  |  25 ++
  gcc/testsuite/gcc.dg/debug/btf/btf-prune-2.c  |  33 ++
  gcc/testsuite/gcc.dg/debug/btf/btf-prune-3.c  |  35 ++
  .../gcc.dg/debug/btf/btf-prune-maps.c |  20 +
  9 files changed, 494 insertions(+), 7 deletions(-)
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-1.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-2.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-3.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-maps.c

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 32fda14f704b..a7da164f6b31 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -828,7 +828,10 @@ output_btf_types (ctf_container_ref ctfc)
  {
size_t i;
size_t num_types;
-  num_types = ct

Re: [PATCH v3 5/6] bpf,btf: enable BTF pruning by default for BPF

[Indu Bhagat]

On 5/30/24 14:32, David Faust wrote:

This patch enables -fprune-btf by default in the BPF backend when
generating BTF information, and fixes BPF CO-RE generation when using
-fprune-btf.

When generating BPF CO-RE information, we must ensure that types used
in CO-RE relocations always have sufficient BTF information emited so
that the CO-RE relocations can be processed by a BPF loader.  The BTF
pruning algorithm on its own does not have sufficient information to
determine which types are used in a BPF CO-RE relocation, so this
information must be supplied by the BPF backend, using a new
btf_mark_type_used function.

Co-authored-by: Cupertino Miranda 

gcc/
* btfout.cc (btf_mark_type_used): New.
* ctfc.h (btf_mark_type_used): Declare it here.
* config/bpf/bpf.cc (bpf_option_override): Enable -fprune-btf
by default if -gbtf is enabled.
* config/bpf/core-builtins.cc (extra_fn): New typedef.
(compute_field_expr): Add callback parameter, and call it if supplied.
Fix computation for MEM_REF.
(mark_component_type_as_used): New.
(bpf_mark_types_as_used): Likewise.
(bpf_expand_core_builtin): Call here.
* doc/invoke.texi (Debugging Options): Note that -fprune-btf is
enabled by default for BPF target when generating BTF.

gcc/testsuite/
* gcc.dg/debug/btf/btf-variables-5.c: Add -fno-prune-btf to dg-options.


OK for the CTF/BTF generic parts in btfout.cc/ctfc.h.

One minor comment below.


---
  gcc/btfout.cc | 22 ++
  gcc/config/bpf/bpf.cc |  5 ++
  gcc/config/bpf/core-builtins.cc   | 71 +--
  gcc/ctfc.h|  1 +
  gcc/doc/invoke.texi   |  3 +
  .../gcc.dg/debug/btf/btf-variables-5.c|  6 +-
  6 files changed, 100 insertions(+), 8 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index a7da164f6b31..35d2875e3f61 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -1501,6 +1501,28 @@ btf_late_assign_datasec_ids (ctf_container_ref ctfc)
  }
  }
  
+

+/* Manually mark that type T is used to ensure it will not be pruned.
+   Used by the BPF backend when generating BPF CO-RE to mark types used
+   in CO-RE relocations.  */
+
+void
+btf_mark_type_used (tree t)
+{
+  /* If we are not going to prune anyway, this is a no-op.  */
+  if (!flag_prune_btf)
+return;
+
+  gcc_assert (TYPE_P (t));
+  ctf_container_ref ctfc = ctf_get_tu_ctfc ();
+  ctf_dtdef_ref dtd = ctf_lookup_tree_type (ctfc, t);
+
+  if (!dtd)
+return;
+
+  btf_add_used_type (ctfc, dtd, false, false, true);
+}
+
  /* Callback used for assembling the only-used-types list.  Note that this is
 the same as btf_type_list_cb above, but the hash_set traverse requires a
 different function signature.  */
diff --git a/gcc/config/bpf/bpf.cc b/gcc/config/bpf/bpf.cc
index dd1bfe38d29b..775730700eba 100644
--- a/gcc/config/bpf/bpf.cc
+++ b/gcc/config/bpf/bpf.cc
@@ -221,6 +221,11 @@ bpf_option_override (void)
&& !(target_flags_explicit & MASK_BPF_CORE))
  target_flags |= MASK_BPF_CORE;
  
+  /* -gbtf implies -fprune-btf for BPF target.  */


-g also will imply -fprune-btf for BPF target now.  So perhaps "BTF, if 
generated, for BPF target is pruned by default" or something similar ?



+  if (btf_debuginfo_p ())
+SET_OPTION_IF_UNSET (&global_options, &global_options_set,
+flag_prune_btf, true);
+
/* Determine available features from ISA setting (-mcpu=).  */
if (bpf_has_jmpext == -1)
  bpf_has_jmpext = (bpf_isa >= ISA_V2);
diff --git a/gcc/config/bpf/core-builtins.cc b/gcc/config/bpf/core-builtins.cc
index 232bebcadbd5..86e2e9d6e39f 100644
--- a/gcc/config/bpf/core-builtins.cc
+++ b/gcc/config/bpf/core-builtins.cc
@@ -624,13 +624,20 @@ bpf_core_get_index (const tree node, bool *valid)
  
 ALLOW_ENTRY_CAST is an input arguments and specifies if the function should

 consider as valid expressions in which NODE entry is a cast expression (or
-   tree code nop_expr).  */
+   tree code nop_expr).
+
+   EXTRA_FN is a callback function to allow extra functionality with this
+   function traversal.  Currently used for marking used type during expand
+   pass.  */
+
+typedef void (*extra_fn) (tree);
  
  static unsigned char

  compute_field_expr (tree node, unsigned int *accessors,
bool *valid,
tree *access_node,
-   bool allow_entry_cast = true)
+   bool allow_entry_cast = true,
+   extra_fn callback = NULL)
  {
unsigned char n = 0;
unsigned int fake_accessors[MAX_NR_ACCESSORS];
@@ -647,6 +654,9 @@ compute_field_expr (tree node, unsigned int *accessors,
  
*access_node = node;
  
+  if (callback != NULL)

+callback (node);
+
switch (TREE_CODE (node))
  {
  case INDIRECT_REF:
@@ -664,17 +674,19 @@ compute_field_expr (tree node, unsig

Re: [PATCH v3 6/6] opts: allow any combination of DWARF, CTF, BTF

[Indu Bhagat]

On 5/30/24 14:32, David Faust wrote:

Previously it was not supported to generate both CTF and BTF debug info
in the same compiler run, as both formats made incompatible changes to
the same internal data structures.

With the structural change in the prior patches, in particular the
guarantee that CTF will always be fully emitted before any BTF
translation occurs, there is no longer anything preventing generation
of both CTF and BTF at the same time.  This patch changes option parsing
to allow any combination of -gdwarf, -gctf, and -gbtf at the same time.



I am not an approver for this change, but I have a comment below.


gcc/
* opts.cc (set_debug_level): Allow any combination of -gdwarf,
-gctf and -gbtf at the same time.
---
  gcc/opts.cc | 20 +++-
  1 file changed, 7 insertions(+), 13 deletions(-)

diff --git a/gcc/opts.cc b/gcc/opts.cc
index f80d5d4ba8f9..d58bea096a5f 100644
--- a/gcc/opts.cc
+++ b/gcc/opts.cc
@@ -3505,21 +3505,15 @@ set_debug_level (uint32_t dinfo, int extended, const 
char *arg,
  }
else
  {
-  /* Make and retain the choice if both CTF and DWARF debug info are to
-be generated.  */
-  if (((dinfo == DWARF2_DEBUG) || (dinfo == CTF_DEBUG))
+  /* Any combination of DWARF, CTF and BTF is allowed together.  */
+  if (((dinfo == DWARF2_DEBUG) || (dinfo == CTF_DEBUG)
+  || (dinfo == BTF_DEBUG))
  && ((opts->x_write_symbols == (DWARF2_DEBUG|CTF_DEBUG))
+ || (opts->x_write_symbols == (DWARF2_DEBUG|BTF_DEBUG))
+ || (opts->x_write_symbols == (CTF_DEBUG|BTF_DEBUG))
  || (opts->x_write_symbols == DWARF2_DEBUG)
- || (opts->x_write_symbols == CTF_DEBUG)))
-   {
- opts->x_write_symbols |= dinfo;
- opts_set->x_write_symbols |= dinfo;
-   }
-  /* However, CTF and BTF are not allowed together at this time.  */
-  else if (((dinfo == DWARF2_DEBUG) || (dinfo == BTF_DEBUG))
-  && ((opts->x_write_symbols == (DWARF2_DEBUG|BTF_DEBUG))
-  || (opts->x_write_symbols == DWARF2_DEBUG)
-  || (opts->x_write_symbols == BTF_DEBUG)))
+ || (opts->x_write_symbols == CTF_DEBUG)
+ || (opts->x_write_symbols == BTF_DEBUG)))


I realised that this check will cause failures on double occurrences of 
the command line flags:


$ gcc -c sort.c -g3 -gctf -flto -gbtf -gctf
gcc: error: debug format ‘ctf’ conflicts with prior selection

What do you think about the following check instead ?

if ((dinfo == DWARF2_DEBUG || dinfo == CTF_DEBUG || dinfo == BTF_DEBUG)
&& ((opts->x_write_symbols | (DWARF2_DEBUG|CTF_DEBUG|BTF_DEBUG))
== (DWARF2_DEBUG|CTF_DEBUG|BTF_DEBUG)))
  {

We have some testcases for checking -gctf -gdwarf (in 
debug/ctf/ctf-debug*), adding something for -gctf -gbtf will be great.



{
  opts->x_write_symbols |= dinfo;
  opts_set->x_write_symbols |= dinfo;

Re: [PATCH v4 4/6] btf: add -gprune-btf option

[Indu Bhagat]

On 6/24/24 09:11, David Faust wrote:

Ping.

Richard: I changed the option name as you asked but forgot to CC you on
the updated patch.  Is the new option OK?

Indu: You had some minor comments on the prior version which I have
addressed, not sure whether you meant the rest of the patch was OK or
not, or if you had a chance to review it.



Hi David,

Thanks for making the change in the commit message to clearly state the 
behavior of the option -gprune-btf with and without LTO build.  I did 
take a look at the V3 version of the patch, had tested it a bit too.


While there are still remain some gaps in my understanding of the 
algorithm, but overall I think this patch as such looks good and makes 
forward progress.


So, LGTM.

Thanks
Indu


Thanks!

archive:https://gcc.gnu.org/pipermail/gcc-patches/2024-June/654252.html

Re: [PATCH v2 1/6] ctf, btf: restructure CTF/BTF emission

[Indu Bhagat]

On 5/2/24 10:11, David Faust wrote:

This commit makes some structural changes to the CTF/BTF debug info
emission.  In particular:

  a) CTF is new always fully generated and emitted before any
 BTF-related procedures are run.  This means that BTF-related
 functions can change, even irreversibly, the shared in-memory
 representation used by the two formats without issue.

  b) BTF generation has fewer entry points, and is cleanly divided
 into early_finish and finish.

  c) BTF is now always emitted at finish (called from dwarf2out_finish),
 rather than being emitted at early_finish for targets other than
 BPF CO-RE.  Note that this change alone does not alter the contents
 of BTF at all, regardless of whether it would have previously been
 emitted at early_finish or finish.



This will necessitate that we disallow -gbtf with -flto for non-BPF 
targets.  Emitting BTF always at dwarf2out_finish will not work with LTO.



The changes are transparent to both CTF and BTF emission.

gcc/
* btfout.cc (btf_init_postprocess): Rename to...
(btf_early_finish): ...this.
(btf_output): Rename to...
(btf_finish): ...this.
* ctfc.h: Analogous changes.
* dwarf2ctf.cc (ctf_debug_early_finish): Conditionally call
btf_early_finish or ctf_finalize as appropriate.
(ctf_debug_finish): Always call btf_finish here if generating
BTF info.
(ctf_debug_finalize, ctf_debug_init_postprocess): Delete.
* dwarf2out.cc (dwarf2out_early_finish): Remove call to
ctf_debug_init_postprocess.
---
  gcc/btfout.cc| 28 +
  gcc/ctfc.h   |  4 ++--
  gcc/dwarf2ctf.cc | 54 +++-
  gcc/dwarf2out.cc |  2 --
  4 files changed, 42 insertions(+), 46 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 07f066a4706..1b6a9ed811f 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -1491,6 +1491,34 @@ btf_finalize (void)
tu_ctfc = NULL;
  }
  
+/* Initial entry point of BTF generation, called at early_finish () after

+   CTF information has possibly been output.  Translate all CTF information
+   to BTF, and do any processing that must be done early, such as creating
+   BTF_KIND_FUNC records.  */
+
+void
+btf_early_finish (void)
+{
+  btf_init_postprocess ();
+}
+
+/* Late entry point for BTF generation, called from dwarf2out_finish ().
+   Complete and emit BTF information.  */
+
+void
+btf_finish (const char * filename)
+{
+  btf_output (filename);
+
+  /* If compiling for BPF with CO-RE info, we cannot deallocate until after
+ CO-RE information is created, which happens very late in BPF backend.
+ Therefore, the deallocation (i.e. btf_finalize ()) is delayed until
+ TARGET_ASM_FILE_END for BPF CO-RE.  */
+  if (!btf_with_core_debuginfo_p ())
+btf_finalize ();
+}
+
+
  /* Traversal function for all BTF_KIND_FUNC type records.  */
  
  bool

diff --git a/gcc/ctfc.h b/gcc/ctfc.h
index fa188bf2f5a..e7bd93901cf 100644
--- a/gcc/ctfc.h
+++ b/gcc/ctfc.h
@@ -384,8 +384,8 @@ extern void ctf_init (void);
  extern void ctf_output (const char * filename);
  extern void ctf_finalize (void);
  
-extern void btf_output (const char * filename);

-extern void btf_init_postprocess (void);
+extern void btf_early_finish (void);
+extern void btf_finish (const char * filename);
  extern void btf_finalize (void);
  
  extern ctf_container_ref ctf_get_tu_ctfc (void);

diff --git a/gcc/dwarf2ctf.cc b/gcc/dwarf2ctf.cc
index dc59569fe56..ec94982e4b1 100644
--- a/gcc/dwarf2ctf.cc
+++ b/gcc/dwarf2ctf.cc
@@ -933,30 +933,6 @@ gen_ctf_type (ctf_container_ref ctfc, dw_die_ref die)
return type_id;
  }
  
-/* Prepare for output and write out the CTF debug information.  */

-
-static void
-ctf_debug_finalize (const char *filename, bool btf)
-{
-  if (btf)
-{
-  btf_output (filename);
-  /* btf_finalize when compiling BPF applciations gets deallocated by the
-BPF target in bpf_file_end.  */
-  if (btf_debuginfo_p () && !btf_with_core_debuginfo_p ())
-   btf_finalize ();
-}
-
-  else
-{
-  /* Emit the collected CTF information.  */
-  ctf_output (filename);
-
-  /* Reset the CTF state.  */
-  ctf_finalize ();
-}
-}
-
  bool
  ctf_do_die (dw_die_ref die)
  {
@@ -996,27 +972,21 @@ ctf_debug_init (void)
add_name_attribute (ctf_unknown_die, "unknown");
  }
  
-/* Preprocess the CTF debug information after initialization.  */

-
-void
-ctf_debug_init_postprocess (bool btf)
-{
-  /* Only BTF requires postprocessing right after init.  */
-  if (btf)
-btf_init_postprocess ();
-}
-
  /* Early finish CTF/BTF debug info.  */
  
  void

  ctf_debug_early_finish (const char * filename)
  {
-  /* Emit CTF debug info early always.  */
-  if (ctf_debug_info_level > CTFINFO_LEVEL_NONE
-  /* Emit BTF debug info early if CO-RE relocations are not
-required.  */
-  || (btf_debuginfo_p () && !btf_with_co

Re: [PATCH v2 3/6] ctf: use pointers instead of IDs internally

[Indu Bhagat]

On 5/2/24 10:11, David Faust wrote:

This patch replaces all inter-type references in the ctfc internal data
structures with pointers, rather than the references-by-ID which were
used previously.

A couple of small updates in the BPF backend are included to make it
compatible with the change.

This change is only to the in-memory representation of various CTF
structures to make them easier to work with in various cases.  It is
outwardly transparent; there is no change in emitted CTF.

gcc/
* btfout.cc (btf_dvd_emit_preprocess_cb, btf_emit_preprocess)
(btf_dmd_representable_bitfield_p, btf_asm_array, btf_asm_varent)
(btf_asm_sou_member, btf_asm_func_arg, btf_init_postprocess):
Adapt to structural changes in ctf_* structs.
* ctfc.h (struct ctf_dtdef): Add forward declaration.
(struct ctf_arinfo, struct ctf_funcinfo, struct ctf_sliceinfo)
(struct ctf_itype, struct ctf_dmdef, struct ctf_func_arg)
(struct ctf_dvdef): Use pointers instead of type IDs for
references to other types.
(struct ctf_dtdef): Add ref_type member.
(ctf_type_exists): Use pointer instead of type ID.
(ctf_add_reftype, ctf_add_enum, ctf_add_slice, ctf_add_float)
(ctf_add_integer, ctf_add_unknown, ctf_add_pointer)
(ctf_add_array, ctf_add_forward, ctf_add_typedef)
(ctf_add_function, ctf_add_sou, ctf_add_enumerator)
(ctf_add_variable): Likewise. Return pointer instead of ID.
(ctf_lookup_tree_type): Return pointer to type instead of ID.
* ctfc.cc: Analogous changes.
* ctfout.cc (ctf_asm_type, ctf_asm_slice, ctf_asm_varent)
(ctf_asm_sou_lmember, ctf_asm_sou_member, ctf_asm_func_arg)
(output_ctf_objt_info): Adapt to changes.
* dwarf2ctf.cc (gen_ctf_type, gen_ctf_void_type)
(gen_ctf_unknown_type, gen_ctf_base_type, gen_ctf_pointer_type)
(gen_ctf_subrange_type, gen_ctf_array_type, gen_ctf_typedef)
(gen_ctf_modifier_type, gen_ctf_sou_type, gen_ctf_function_type)
(gen_ctf_enumeration_type, gen_ctf_variable, gen_ctf_function)
(gen_ctf_type, ctf_do_die): Likewise.
* config/bpf/btfext-out.cc (struct btf_ext_core_reloc): Use
pointer instead of type ID.
(bpf_core_reloc_add, bpf_core_get_sou_member_index)
(output_btfext_core_sections): Adapt to above changes.
* config/bpf/core-builtins.cc (process_type): Likewise.
---
  gcc/btfout.cc   |  35 ++---
  gcc/config/bpf/btfext-out.cc|  12 +-
  gcc/config/bpf/core-builtins.cc |   3 +-
  gcc/ctfc.cc | 137 +-
  gcc/ctfc.h  |  61 
  gcc/ctfout.cc   |  19 +--
  gcc/dwarf2ctf.cc| 244 +++-
  7 files changed, 252 insertions(+), 259 deletions(-)

diff --git a/gcc/config/bpf/btfext-out.cc b/gcc/config/bpf/btfext-out.cc
index 7ec438fd1d1..ce596e33643 100644
--- a/gcc/config/bpf/btfext-out.cc
+++ b/gcc/config/bpf/btfext-out.cc
@@ -134,7 +134,7 @@ struct GTY ((chain_next ("%h.next"))) btf_ext_lineinfo
  
  /* Internal representation of a BPF CO-RE relocation record.  */

  struct GTY ((chain_next ("%h.next"))) btf_ext_core_reloc {
-  unsigned int bpfcr_type; /* BTF type ID of container.  */
+  ctf_dtdef_ref bpfcr_type;/* BTF type of container.  */


I find the comment "BTF type of container" associated with the member 
confusing.  May be we say " BTF type involved with the reloc " or 
something like that ?



unsigned int  bpfcr_astr_off;   /* Offset of access string in 
.BTF
   string table.  */
rtx_code_label * bpfcr_insn_label;  /* RTX label attached to instruction
@@ -296,13 +296,14 @@ bpf_core_reloc_add (const tree type, const char * 
section_name,
struct btf_ext_core_reloc *bpfcr = bpf_create_core_reloc (section_name, 
&sec);
  
ctf_container_ref ctfc = ctf_get_tu_ctfc ();

+  ctf_dtdef_ref dtd = ctf_lookup_tree_type (ctfc, type);
  
/* Buffer the access string in the auxiliary strtab.  */

bpfcr->bpfcr_astr_off = 0;
gcc_assert (accessor != NULL);
bpfcr->bpfcr_astr_off = btf_ext_add_string (accessor);
  
-  bpfcr->bpfcr_type = get_btf_id (ctf_lookup_tree_type (ctfc, type));

+  bpfcr->bpfcr_type = dtd;
bpfcr->bpfcr_insn_label = label;
bpfcr->bpfcr_kind = kind;
  
@@ -341,7 +342,7 @@ bpf_core_get_sou_member_index (ctf_container_ref ctfc, const tree node)

for (dmd = dtd->dtd_u.dtu_members;
 dmd != NULL; dmd = (ctf_dmdef_t *) ctf_dmd_list_next (dmd))
  {
- bool field_has_btf = get_btf_id (dmd->dmd_type) <= BTF_MAX_TYPE;
+ bool field_has_btf = (dmd->dmd_type && dmd->dmd_type->dtd_type <= 
BTF_MAX_TYPE);
  
  	  if (field == node)

return field_has_btf ? i : -1;
@@ -574,8 +575,9 @@ output_btfext_core_sections (void)
 false);
  

Re: [PATCH v2 4/6] btf: refactor and simplify implementation

[Indu Bhagat]

On 5/2/24 10:11, David Faust wrote:

This patch heavily refactors btfout.cc to take advantage of the
structural changes in the prior commits.

Now that inter-type references are internally stored as simply pointers,
all the painful, brittle, confusing infrastructure that was used in the
process of converting CTF type IDs to BTF type IDs can be thrown out.
This greatly simplifies the entire process of converting from CTF to
BTF, making the code cleaner, easier to read, and easier to maintain.

In addition, we no longer need to worry about destructive changes in
internal data structures used commonly by CTF and BTF, which allows
deleting several ancillary data structures previously used in btfout.cc.

This is nearly transparent, but a few improvements have also been made:

  1) BTF_KIND_FUNC records are now _always_ constructed at early_finish,
 allowing us to construct records even for functions which are later
 inlined by optimizations. DATASEC entries for functions are only
 constructed at late_finish, to avoid incorrectly generating entries
 for functions which get inlined.

  2) BTF_KIND_VAR records and DATASEC entries for them are now always
 constructed at (late) finish, which avoids cases where we could
 incorrectly create records for variables which were completely
 optimized away. This fixes PR debug/113566.



I am a bit unsure ATM, how this commit will look like once we revisit 
where the BTF is generated for BPF and non-BPF backends.  Since patch 
1/6 moved everything to (late)finish, but that will be problematic, I 
would like to defer reviewing this until the approach for 
afore-mentioned patch is pinned.



  3) Some additional assembler comments have been added with more
 information for debugging.

gcc/
* btfout.cc (struct btf_datasec_entry): New.
(struct btf_datasec): Add `id' member.  Change `entries' to use
new struct btf_datasec_entry.
(func_map): New hash_map.
(max_translated_id): New.
(btf_var_ids, btf_id_map, holes, voids, num_vars_added)
(num_types_added, num_types_created): Delete.
(btf_absolute_var_id, btf_relative_var_id, btf_absolute_func_id)
(btf_relative_func_id, btf_absolute_datasec_id, init_btf_id_map)
(get_btf_id, set_btf_id, btf_emit_id_p): Delete.
(btf_removed_type_p): Delete.
(btf_dtd_kind, btf_emit_type_p): New helpers.
(btf_fwd_to_enum_p, btf_calc_num_vbytes): Use them.
(btf_collect_datasec): Delete.
(btf_dtd_postprocess_cb, btf_dvd_emit_preprocess_cb)
(btf_dtd_emit_preprocess_cb, btf_emit_preprocess): Delete.
(btf_dmd_representable_bitfield_p): Adapt to type reference changes
and delete now-unused ctfc argument.
(btf_asm_datasec_type_ref): Delete.
(btf_asm_type_ref): Adapt to type reference changes, simplify.
(btf_asm_type): Likewise. Mark struct/union types with bitfield
members.
(btf_asm_array): Adapt to data structure changes.
(btf_asm_varent): Likewise.
(btf_asm_sou_member): Likewise. Ensure non-bitfield members are
correctly re-encoded if struct or union contains any bitfield.
(btf_asm_func_arg, btf_asm_func_type, btf_asm_datasec_entry)
(btf_asm_datasec_type): Adapt to data structure changes.
(output_btf_header): Adapt to other changes, simplify type
length calculation, add info to assembler comments.
(output_btf_vars): Adapt to other changes.
(output_btf_strs): Fix overlong lines.
(output_asm_btf_sou_fields, output_asm_btf_enum_list)
(output_asm_btf_func_args_list, output_asm_btf_vlen_bytes)
(output_asm_btf_type, output_btf_types, output_btf_func_types)
(output_btf_datasec_types): Adapt to other changes.
(btf_init_postprocess): Delete.
(btf_output): Change to only perform output.
(btf_early_add_const_void, btf_early_add_func_records): New.
(btf_early_finish): Use them here. New.
(btf_datasec_push_entry): Adapt to data structure changes.
(btf_datasec_add_func, btf_datasec_add_var): New.
(btf_late_add_func_datasec_entries): New.
(btf_emit_variable_p): New helper.
(btf_late_add_vars): Use it here. New.
(btf_type_list_cb, btf_late_collect_translated_types): New.
(btf_late_assign_func_ids, btf_late_assign_var_ids)
(btf_late_assign_datasec_ids): New.
(btf_finish): Remove unused argument. Call new btf_late*
functions and btf_output.
(btf_finalize): Adapt to data structure changes.
* ctfc.h (struct ctf_dtdef): Convert existing boolean flags to
BOOL_BITFIELD and reorder.
(struct ctf_dvdef): Add dvd_id member.
(btf_finish): Remove argument from prototype.
(get_btf_id): Delete prototype.
(funcs_traverse_callback, traverse_btf_func_types): Add an
explanatory comment.
* dwarf2ctf

Re: [PATCH v2 5/6] btf: add -fprune-btf option

[Indu Bhagat]

On 5/2/24 10:11, David Faust wrote:

This patch adds a new option, -fprune-btf, to control BTF debug info
generation.

As the name implies, this option enables a kind of "pruning" of the BTF
information before it is emitted.  When enabled, rather than emitting
all type information translated from DWARF, only information for types
directly used in the source program is emitted.

The primary purpose of this pruning is to reduce the amount of
unnecessary BTF information emitted, especially for BPF programs.  It is
very common for BPF programs to incldue Linux kernel internal headers in
order to have access to kernel data structures.  However, doing so often
has the side effect of also adding type definitions for a large number
of types which are not actually used by nor relevant to the program.
In these cases, -fprune-btf commonly reduces the size of the resulting
BTF information by approximately 10x.  This both slims down the size of
the resulting object and reduces the time required by the BPF loader to
verify the program and its BTF information.



The 10x reduction is substantial.  Do you think its is worthwhile to 
mention alongside that this data is the average observed for the kernel 
self-tests (I assume it is) ? Just useful info when parsing the commit 
logs, especially when some data is specified...



Note that the pruning implemented in this patch follows the same rules
as the BTF pruning performed unconditionally by LLVM's BPF backend when
generating BTF.  In particular, the main sources of pruning are:

   1) Only generate BTF for types used by variables and functions at
  the file scope.



I dont recollect anymore if BTF_KIND_VAR for unused static vars is also 
a correctness issue for BTF.  (With PR debug/113566, we know having 
BTF_KIND_DATASEC entries for optimized away vars is an issue).


It will be great to add some text here or elsewhere for posterity around 
this.



   2) Avoid emitting full BTF for struct and union types which are only
  pointed-to by members of other struct/union types.  In these cases,
  the full BTF_KIND_STRUCT or BTF_KIND_UNION which would normally
  be emitted is replaced with a BTF_KIND_FWD, as though the
  underlying type was a forward-declared struct or union type.

gcc/
* btfout.cc (btf_minimal_types): New hash set.
(struct btf_fixup): New.
(fixups, forwards): New vecs.
(btf_output): Calculate num_types depending on flag_prune_btf.
(btf_early_finsih): New initialization for flag_prune_btf.
(btf_mark_full_type_used): Likewise.
(btf_minimal_add_type): New function.
(btf_minimal_type_list_cb): Likewise.
(btf_late_collect_pruned_types): Likewise.
(btf_late_add_vars): Handle special case for variables in ".maps"
section when generating BTF for BPF CO-RE target.
(btf_late_finish): Use btf_late_collect_pruned_types when
flag_prune_btf in effect.  Move some initialization to btf_early_finish.
(btf_finalize): Additional deallocation for flag_prune_btf.
* common.opt (fprune-btf): New flag.
* ctfc.cc (init_ctf_strtable): Make non-static.
* ctfc.h (struct ctf_dtdef): Add visited_children_p boolean flag.
(init_ctf_strtable, ctfc_delete_strtab): Make extern.
* doc/invoke.texi (Debugging Options): Document -fprune-btf.

gcc/testsuite/
* gcc.dg/debug/btf/btf-prune-1.c: New test.
* gcc.dg/debug/btf/btf-prune-2.c: Likewise.
* gcc.dg/debug/btf/btf-prune-3.c: Likewise.
---
  gcc/btfout.cc| 394 ++-
  gcc/common.opt   |   4 +
  gcc/ctfc.cc  |   2 +-
  gcc/ctfc.h   |   5 +
  gcc/doc/invoke.texi  |  20 +
  gcc/testsuite/gcc.dg/debug/btf/btf-prune-1.c |  25 ++
  gcc/testsuite/gcc.dg/debug/btf/btf-prune-2.c |  33 ++
  gcc/testsuite/gcc.dg/debug/btf/btf-prune-3.c |  35 ++
  8 files changed, 511 insertions(+), 7 deletions(-)
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-1.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-2.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-prune-3.c

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 0af0bd39fc7..93d56492bbe 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -833,7 +833,10 @@ output_btf_types (ctf_container_ref ctfc)
  {
size_t i;
size_t num_types;
-  num_types = ctfc->ctfc_types->elements ();
+  if (flag_prune_btf)
+num_types = max_translated_id;
+  else
+num_types = ctfc->ctfc_types->elements ();
  
if (num_types)

  {
@@ -962,6 +965,211 @@ btf_early_add_func_records (ctf_container_ref ctfc)
  }
  }
  
+/* The set of types used directly in the source program, and any types manually

+   marked as used.  This is the set of types which will be emitted when
+   pruning (-fprune-btf) is enabled.  */


Nit: emitted when flag

Re: [PATCH v2 6/6] bpf,btf: enable BTF pruning by default for BPF

[Indu Bhagat]

On 5/2/24 10:11, David Faust wrote:

This patch enables -fprune-btf by default in the BPF backend when
generating BTF information, and fixes BPF CO-RE generation when using
-fprune-btf.

When generating BPF CO-RE information, we must ensure that types used
in CO-RE relocations always have sufficient BTF information emited so
that the CO-RE relocations can be processed by a BPF loader.  The BTF
pruning algorithm on its own does not have sufficient information to
determine which types are used in a BPF CO-RE relocation, so this
information must be supplied by the BPF backend, using a new
btf_mark_type_used function.

Co-authored-by: Cupertino Miranda 

gcc/
* btfout.cc (btf_mark_type_used): New.
* ctfc.h (btf_mark_type_used): Declare it here.
* config/bpf/bpf.cc (bpf_option_override): Enable -fprune-btf
by default if -gbtf is enabled.
* config/bpf/bcore-builtins.cc (extra_fn): New typedef.
(compute_field_expr): Add callback parameter, and call it if supplied.
Fix computation for MEM_REF.
(mark_component_type_as_used): New.
(bpf_mark_types_as_used): Likewise.
(bpf_expand_core_builtin): Call here.
* doc/invoke.texi (Debugging Options): Note that -fprune-btf is
enabled by default for BPF target when generating BTF.

gcc/testsuite/
* gcc.dg/debug/btf/btf-variables-5.c: Add -fno-prune-btf to dg-options.
---
  gcc/btfout.cc | 22 ++
  gcc/config/bpf/bpf.cc |  5 ++
  gcc/config/bpf/core-builtins.cc   | 70 +--
  gcc/ctfc.h|  1 +
  gcc/doc/invoke.texi   |  3 +
  .../gcc.dg/debug/btf/btf-variables-5.c|  2 +-
  6 files changed, 96 insertions(+), 7 deletions(-)

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 93d56492bbe..da2c9d35be9 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -1539,6 +1539,28 @@ btf_late_assign_datasec_ids (ctf_container_ref ctfc)
  }
  }
  
+

+/* Manually mark that type T is used to ensure it will not be pruned.
+   Used by the BPF backend when generating BPF CO-RE to mark types used
+   in CO-RE relocations.  */
+
+void
+btf_mark_type_used (tree t)
+{
+  /* If we are not going to prune anyway, this is a no-op.  */
+  if (!flag_prune_btf)
+return;
+
+  gcc_assert (TYPE_P (t));
+  ctf_container_ref ctfc = ctf_get_tu_ctfc ();
+  ctf_dtdef_ref dtd = ctf_lookup_tree_type (ctfc, t);
+
+  if (!dtd)
+return;
+
+  btf_minimal_add_type (ctfc, dtd, false, false);
+}
+
  /* Callback used for assembling the only-used-types list.  Note that this is
 the same as btf_type_list_cb above, but the hash_set traverse requires a
 different function signature.  */
diff --git a/gcc/config/bpf/bpf.cc b/gcc/config/bpf/bpf.cc
index e6ea211a2c6..75303ce8f46 100644
--- a/gcc/config/bpf/bpf.cc
+++ b/gcc/config/bpf/bpf.cc
@@ -221,6 +221,11 @@ bpf_option_override (void)
&& !(target_flags_explicit & MASK_BPF_CORE))
  target_flags |= MASK_BPF_CORE;
  
+  /* -gbtf implies -fprune-btf for BPF target.  */

+  if (btf_debuginfo_p ())
+SET_OPTION_IF_UNSET (&global_options, &global_options_set,
+flag_prune_btf, true);
+
/* Determine available features from ISA setting (-mcpu=).  */
if (bpf_has_jmpext == -1)
  bpf_has_jmpext = (bpf_isa >= ISA_V2);
diff --git a/gcc/config/bpf/core-builtins.cc b/gcc/config/bpf/core-builtins.cc
index d5a7de825ad..1b91b1c0d25 100644
--- a/gcc/config/bpf/core-builtins.cc
+++ b/gcc/config/bpf/core-builtins.cc
@@ -624,13 +624,20 @@ bpf_core_get_index (const tree node, bool *valid)
  
 ALLOW_ENTRY_CAST is an input arguments and specifies if the function should

 consider as valid expressions in which NODE entry is a cast expression (or
-   tree code nop_expr).  */
+   tree code nop_expr).
+
+   EXTRA_FN is a callback function to allow extra functionality with this
+   function traversal. Currently used for marking used type during expand
+   pass.  */
+
+typedef void (*extra_fn) (tree);
  
  static unsigned char

  compute_field_expr (tree node, unsigned int *accessors,
bool *valid,
tree *access_node,
-   bool allow_entry_cast = true)
+   bool allow_entry_cast = true,
+   extra_fn callback = NULL)
  {
unsigned char n = 0;
unsigned int fake_accessors[MAX_NR_ACCESSORS];
@@ -647,6 +654,9 @@ compute_field_expr (tree node, unsigned int *accessors,
  
*access_node = node;
  
+  if (callback != NULL)

+callback (node);
+
switch (TREE_CODE (node))
  {
  case INDIRECT_REF:
@@ -664,17 +674,19 @@ compute_field_expr (tree node, unsigned int *accessors,
  case COMPONENT_REF:
n = compute_field_expr (TREE_OPERAND (node, 0), accessors,
  valid,
- access_node, false);
+ access_node,

Type representation in CTF and DWARF

[Indu Bhagat]

   units are de-duplicated when using split-dwarf.

Thanks
Indu

Re: Type representation in CTF and DWARF

[Indu Bhagat]

On 10/07/2019 12:35 AM, Richard Biener wrote:

On Fri, Oct 4, 2019 at 9:12 PM Indu Bhagat  wrote:

Hello,

At GNU Tools Cauldron this year, some folks were curious to know more on how
the "type representation" in CTF compares vis-a-vis DWARF.

[...]

So, for the small C testcase with a union, enum, array, struct, typedef etc, I
see following sizes :

Compile with -fdebug-types-section -gdwarf-4 (size -A  excerpt):
  .debug_aranges 48 0
  .debug_info   150 0
  .debug_abbrev 314 0
  .debug_line73 0
  .debug_str455 0
  .debug_ranges  32 0
  .debug_types  578 0

Compile with -fdebug-types-section -gdwarf-5 (size -A  excerpt):
  .debug_aranges  48 0
  .debug_info732 0
  .debug_abbrev  309 0
  .debug_line 73 0
  .debug_str 455 0
  .debug_rnglists 23 0

Compile with -gt (size -A  excerpt):
  .ctf  966 0
  CTF strings sub-section size (ctf_strlen in disassmebly) = 374
  == > CTF section just for representing types = 966 - 374 = 592 bytes
  (The 592 bytes include the CTF header and other indexes etc.)

So, following points are what I would highlight. Hopefully this helps you see
that CTF has promise for the task of representing type debug info.

1. Type Information layout in sections:
 A .ctf section is self-sufficient to represent types in a program. All
 references within the CTF section are via either indexes or offsets into 
the
 CTF section. No relocations are necessary in CTF at this time. In contrast,
 DWARF type information is organized in multiple sections - .debug_info,
 .debug_abbrev and .debug_str sections in DWARF5; plus .debug_types in 
DWARF4.

2. Type Information encoding / compactness matters:
 Because the type information is organized across sections in DWARF (and
 contains some debug information like location etc.) , it is not feasible
 to put a distinct number to the size in bytes for representing type
 information in DWARF. But the size info of sections shown above should
 be helpful to show that CTF does show promise in compactly representing
 types.

 Lets see some size data. CTF string table (= 374 bytes) is left out of the
 discussion at hand because it will not be fair to compare with .debug_str
 section which contains other information than just names of types.

 The 592 bytes of the .ctf section are needed to represent types in CTF
 format. Now, when using DWARF5, the type information needs 732 bytes in
 .debug_info and 309 bytes in .debug_abbrev.

 In DWARF (when using -fdebug-types-section), the base types are duplicated
 across type units. So for the above example, the DWARF DIE representing
 'unsigned int' will appear in both the  DWARF trees for types - node and
 node_payload. In CTF, there is a single lone type 'unsigned int'.

It's not clear to me why you are using -fdebug-types-section for this
comparison?
With just -gdwarf-4 I get

.debug_info  292
.debug_abbrev 189
.debug_str   299

this contains all the info CTF provides (and more).  This sums to 780 bytes,
smaller than the CTF variant.  I skimmed over the info and there's not much
to strip to get to CTF levels, mainly locations.  The strings section also
has a quite large portion for GCC version and arguments, which is 93 bytes.
So overall the DWARF representation should clock in at less than 700 bytes,
more close to 650.

Richard.


It's not in favor of DWARF to go with just -gdwarf-4. Because the types
in the .debug_info section will not be de-duplicated. For more complicated code
bases with many compilation units, this will skew the results in favor of CTF
(once the CTF de-duplictor is ready :) ).

Now, one might argue that in this example, there is no role for de-duplicator.
Yes to that. But to all users of DWARF type debug information for _real
codebases_, -fdebug-types-section option is the best option. Isn't it ?

Keeping "the size of type debug information in the shipped artifact small" as
our target is meaningful for both CTF and DWARF.

De-duplication is a key contributor to reducing the size of the type debug
information; and both CTF and DWARF types can be de-duplicated. At this time, I
stuck to a simple example with one CU because it eases interpreting the CTF and
DWARF debug info in the binaries and because the CTF link-time de-duplication
is not fully ready.

(NickA suggested few days ago to compare how DWARF and CTF section sizes
 increase when a new member, or a new enum, or a new union etc are added. I can
 share some more data if there is interest in such a comparison. Few examples
 below :

1. Add a new member 'struct node_payload * a' to struct node_payload
   DWARF = 589 - 578 (.debug_ty

Re: Type representation in CTF and DWARF

[Indu Bhagat]

On 10/08/2019 08:37 AM, Pedro Alves wrote:

On 10/4/19 8:23 PM, Indu Bhagat wrote:

Hello,

At GNU Tools Cauldron this year, some folks were curious to know more on how
the "type representation" in CTF compares vis-a-vis DWARF.

I was one of those, and I brought this up to Jose, after your
presentation.  Glad to see the follow up!  Thanks much for this.

In your Cauldron presentation we saw CTF compared to full blown DWARF
as justification for CTF,


Hmm. And I thought I made the effort reqd to clarify my position that comparing
full-blown DWARF sizes to type-only CTF section sizes is not appropriate, let
alone to not use as a justification for CTF. My intention to show those numbers 
was
only to give some perspective to users curious to know the sizes of CTF debug
info (as generated by dwarf2ctf) because these sections will ideally be not
stripped out of shipped binaries.

The justification for CTF is and will remain - a compact, faster debug format
for type information and support some online debugging use-cases (like
backtraces) in future.


but I was more interested in a comparison between
CTF and a DWARF subset containing exactly only what you have available in
CTF.  Because if DWARF with everything-you-don't-need stripped out
is in the same ballpark, then I am puzzled on why add/maintain a new
Debug format, with all the duplication of effort that entails going
forward.


I shared some numbers on this in the previous emails in this thread. I thought
comparing DWARF's de-duplication-amenable offering (using
-fdebug-types-section) will be useful in this context.

For binaries compiled with -fdebug-types-section -gdwarf-4, here is some data.
The CTF sections are generated with dwarf2ctf because CTF link-time de-dup is
being worked on currently. The end result of link-time CTF de-dup is expected
to be at par with these .ctf section sizes.

The .ctf section sizes below include the CTF string table (.debug_str is
excluded from the calculations however):

(coreutils-0.22)
   .debug_info(D1) | .debug_abbrev(D2) | .debug_str | .debug_types(D3) | .ctf 
(uncompressed) | ratio (.ctf/(D1+D2+D3))
ls  109806 |  18876|  22042 |  12413   |   
26240 | 0.18
pwd 27902  |  7914 |  10851 |  5753|   
13929 | 0.33
groups 26920   |  8173 |  10674 |  5070|   
13378 | 0.33

(emacs-26.3)
   .debug_info(D1) | .debug_abbrev(D2) | .debug_str | .debug_types(D3) | .ctf 
(uncompressed) | ratio (.ctf/(D1+D2+D3))
emacs 3755083  |   202926  |  431926|   143462 |   
273910| 0.06


It is not easy to get an estimate of 'DWARF with everything-you-don't-need
stripped out'. At this time, I don't know of an easy way to make this comparison
more meaningful. Any suggestions ?


Also, it's my understanding that the current CTF format doesn't yet
support C++, Vector registers, etc., maybe other things, so if DWARF
was sufficient for your needs, then in the long run it sounds like
a better option to me, as then you wouldn't have to extend CTF _and_
DWARF whenever some feature is needed.


Yes, CTF does not support C++ at this time. To cover all of C (including
GNU C extensions), we need to add representation for things like Vector type,
non IEEE float etc. (somewhat infrequently occurring constructs)

The issue is not that DWARF cannot represent the required type information.
DWARF is voluminous and secondly, the current workflow to get to CTF from
source programs without direct toolchain support is tiresome and lengthy.

For current and future users of CTF, having the support for the format in the
toolchain is the best way to promote adoption and enhance community experience.


Maybe it would make sense to work on integrating CTF into the DWARF
standard itself, not sure?

I was also curious on your plans for adding unwinding support to CTF,
while the kernel (the main CTF user, IIUC), already has plans to
use its own unwinding format (ORC)?


Kernel's unwinding format (ORC) helps generate backtrace with function
identifiers. For some (ORCL) internal customers, the requirement is to go beyond
that and support input arg values. The requirement there is to generate
backtraces in a fast way, without relying on DWARF.


So with all those questions, I came out of the presentation
thinking that I could not really justify CTF if I were asked to.


Thanks for discussing this openly. I believe there are other GCC
maintainers who are undecided as well :)

I hope I have answered some of your concerns.


(Side note: the Cauldron page is missing slides for your
presentation, so I couldn't go and recheck some things
mentioned above.)

Thanks,
Pedro Alves


I mailed the organizers my slides. They should be online soon.

Thanks

Re: Type representation in CTF and DWARF

[Indu Bhagat]

On 10/09/2019 12:49 AM, Jakub Jelinek wrote:

On Wed, Oct 09, 2019 at 09:41:09AM +0200, Richard Biener wrote:

There's a mechanism to get type (and decl - I suppose CTF also
contains debug info
for function declarations not only its type?) info as part of early
debug generation.
The attached "hack" simply mangles dwarf2out to output this early info as the
only debug info (only verified on a small .c file).  We still have things like
file, line and column numbers for entities (not sure if CTF has those).

It should be possible to "hide" the hack behind a -gdwarf-like-ctf or similar.
I guess -g0.5 isn't desirable and we've taken both -g0 and -g1 already...
(and -g1 doesn't include types but just decls).

Yeah.  And if location info isn't in CTF, you can as well add an early
return in add_src_coords_attributes, like it has one for UNKNOWN_LOCATION
already.  Or if it is there, but just file/line and not column, you can use
-gno-column-info.  As has been mentioned earlier, you can use dwz utility
post-linking instead of -fdebug-types-section.

Jakub


Thanks for your pointers.

CTF does not encode location information. So, I used early exit in the
add_src_coords_attributes to avoid generation of location info (file, line,
column). To answer Richard's question, CTF does have type debug info
for function declarations and the argument types. So I think with these
changes, both CTF and DWARF generation will emit debug info for the same set of
types and decl.

Compile with -g -gdwarf-like-ctf and use dwz -o   (using
dwz compiled from the master branch) on the generated binaries:

(coreutils-0.22)
 .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | .ctf (uncompressed) 
| ratio (.ctf/(D1+D2+0.5*D4))
ls   30616   |1136   |21098   | 26240   
| 0.62
pwd  10734   |788|10433   | 13929   
| 0.83
groups 10706 |811|10249   | 13378   
| 0.80

(emacs-26.3)
 .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | .ctf (uncompressed) 
| ratio (.ctf/(D1+D2+0.5*D4))
emacs-26.3.1 674657  |6402   |   273963   |   273910
| 0.33

I chose to account for 50% of .debug_str because at this point, it will be
unfair to not account for them. Actually, one could even argue that upto 70%
of the .debug_str are names of entities. CTF section sizes do include the CTF
string tables.

Across coreutils, I see a geomean of 0.73 (ratio of
.ctf/(.debug_info + .debug_abbrev + 50% of .debug_str)). So, with the
"-gdwarf-like-ctf code stubs" and dwz, DWARF continues to have a larger
footprint than CTF (with 50% of .debug_str accounted for).

Indu

Re: Type representation in CTF and DWARF

[Indu Bhagat]

On 10/11/2019 04:23 AM, Richard Biener wrote:

Thanks for your pointers.

CTF does not encode location information. So, I used early exit in the
add_src_coords_attributes to avoid generation of location info (file, line,
column). To answer Richard's question, CTF does have type debug info
for function declarations and the argument types. So I think with these
changes, both CTF and DWARF generation will emit debug info for the same set of
types and decl.

Compile with -g -gdwarf-like-ctf and use dwz -o   (using
dwz compiled from the master branch) on the generated binaries:

(coreutils-0.22)
   .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | .ctf 
(uncompressed) | ratio (.ctf/(D1+D2+0.5*D4))
ls   30616   |1136   |21098   | 26240   
| 0.62
pwd  10734   |788|10433   | 13929   
| 0.83
groups 10706 |811|10249   | 13378   
| 0.80

(emacs-26.3)
   .debug_info(D1) | .debug_abbrev(D2) | .debug_str(D4) | .ctf 
(uncompressed) | ratio (.ctf/(D1+D2+0.5*D4))
emacs-26.3.1 674657  |6402   |   273963   |   273910
| 0.33

I chose to account for 50% of .debug_str because at this point, it will be
unfair to not account for them. Actually, one could even argue that upto 70%
of the .debug_str are names of entities. CTF section sizes do include the CTF
string tables.

Across coreutils, I see a geomean of 0.73 (ratio of
.ctf/(.debug_info + .debug_abbrev + 50% of .debug_str)). So, with the
"-gdwarf-like-ctf code stubs" and dwz, DWARF continues to have a larger
footprint than CTF (with 50% of .debug_str accounted for).

I'm not convinced this "improvement" in size is worth maintainig another
debug-info format much less since it lacks desirable features right now
and thus evaluation is tricky.

At least you can improve dwarf size considerably with a low amount of work.

I suspect another factor where dwarf is bigger compared to CTF is that dwarf
is recording typedef names as well as qualified type variants.  But maybe
CTF just has a more compact representation for the bits it actually implements.

Richard.


CTF represents typedefs and qualified type variants. They are included in the
the .ctf section sizes above.

Indu

[Patch] Minor GCC documentation correction for -Wformat-overflow

[Indu Bhagat]

In section "-Wformat-overflow=1", following is stated :

void f (int a, int b)
{
  char buf [12];
  sprintf (buf, "a = %i, b = %i\n", a, b);
}

" Increasing the size of the buffer by a single byte is sufficient to avoid
the warning,"

[size of an unknown int for the purpose of this warning is = 1 (to represent 0);
add 1 for newline, add 1 for null; add all the other chars in the format
string = 14]

The minimum increase however needs to be of 2 bytes. i.e., a buf of size 14 is
the minimum length for the warning in the example to go away.
So the correct statement should be -

" Increasing the size of the buffer by two bytes is sufficient to avoid the
warning,"

Alternatively, the size of buf can be bumped up to 13 in the sample code as done
in the patch below.

Thanks
 
--

gcc/ChangeLog:


* doc/invoke.texi: Correction in -Wformat-overflow code sample.

Index: gcc/doc/invoke.texi
===
--- gcc/doc/invoke.texi (revision 257646)
+++ gcc/doc/invoke.texi (working copy)
@@ -4184,7 +4184,7 @@
 @smallexample
 void f (int a, int b)
 @{
-  char buf [12];
+  char buf [13];
   sprintf (buf, "a = %i, b = %i\n", a, b);
 @}
 @end smallexample

[PATCH] PR86957

[Indu Bhagat]

Patch for PR 86957 " gcc should warn about missing profiles for a compilation unit 
or a new function with -fprofile-use".
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86957

The patch adds -Wmissing-profile warning flag to alert user about cases when 
there is no profile data for a compilation unit or a function when using 
-fprofile-use.

The flag is on by default when -fprofile-use is specified and generates errors
by default (like -Wcoverage-mismatch).

The attachment pr86957-missing-profile-diagnostic shows the behavior of GCC 
with the patch.

Thanks
Indu

--


gcc/ChangeLog:

2018-09-05  "Indu Bhagat"  <"indu.bha...@oracle.com">

* common.opt: New warning option -Wmissing-profile.
* coverage.c (get_coverage_counts): Add warning for missing .gcda file.
* doc/invoke.texi: Document -Wmissing-profile.
* profile.c (get_exec_counts): Add warning for missing feedback
  profile for a function.
* toplev.c (process_options): -Wmissing-profile warning as error.

diff --git a/gcc/common.opt b/gcc/common.opt
index ebc3ef4..d93ddca 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -811,6 +811,10 @@ Wcoverage-mismatch
 Common Var(warn_coverage_mismatch) Init(1) Warning
 Warn in case profiles in -fprofile-use do not match.
 
+Wmissing-profile
+Common Var(warn_missing_profile) Init(1) Warning
+Warn in case profiles in -fprofile-use do not exist.
+
 Wvector-operation-performance
 Common Var(warn_vector_operation_performance) Warning
 Warn when a vector operation is compiled outside the SIMD.
diff --git a/gcc/coverage.c b/gcc/coverage.c
index bae6f5c..df031df 100644
--- a/gcc/coverage.c
+++ b/gcc/coverage.c
@@ -341,16 +341,24 @@ get_coverage_counts (unsigned counter, unsigned expected,
 {
   static int warned = 0;
 
-  if (!warned++ && dump_enabled_p ())
+  if (!warned++)
{
- dump_user_location_t loc
-   = dump_user_location_t::from_location_t (input_location);
- dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+ warning (OPT_Wmissing_profile,
+  "%qs profile count data file not found,"
+  " regenerating profiles may help",
+  da_file_name);
+ if (dump_enabled_p ())
+   {
+ dump_user_location_t loc
+   = dump_user_location_t::from_location_t (input_location);
+ dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+  "file %s not found\n",
+  da_file_name);
+ dump_printf (MSG_OPTIMIZED_LOCATIONS,
   (flag_guess_branch_prob
-   ? "file %s not found, execution counts estimated\n"
-   : "file %s not found, execution counts assumed to "
-   "be zero\n"),
-  da_file_name);
+   ? "execution counts estimated\n"
+   : "execution counts assumed to be zero\n"));
+   }
}
   return NULL;
 }
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index ca92028..e62bcae 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -314,7 +314,7 @@ Objective-C and Objective-C++ Dialects}.
 -Wlogical-op  -Wlogical-not-parentheses  -Wlong-long @gol
 -Wmain  -Wmaybe-uninitialized  -Wmemset-elt-size  -Wmemset-transposed-args @gol
 -Wmisleading-indentation  -Wmissing-attributes -Wmissing-braces @gol
--Wmissing-field-initializers  -Wmissing-include-dirs @gol
+-Wmissing-field-initializers  -Wmissing-include-dirs  -Wmissing-profile @gol
 -Wno-multichar  -Wmultistatement-macros  -Wnonnull  -Wnonnull-compare @gol
 -Wnormalized=@r{[}none@r{|}id@r{|}nfc@r{|}nfkc@r{]} @gol
 -Wnull-dereference  -Wodr  -Wno-overflow  -Wopenmp-simd  @gol
@@ -4816,6 +4816,25 @@ This warning is enabled by @option{-Wall}.
 @opindex Wno-missing-include-dirs
 Warn if a user-supplied include directory does not exist.
 
+@item -Wmissing-profile
+@opindex Wmissing-profile
+@opindex Wno-missing-profile
+Warn if feedback profiles are missing when using the
+@option{-fprofile-use} option.
+If a new function or a new file is added to the user code between compiling
+with @option{-fprofile-gen} and with @option{-fprofile-use} without
+regenerating the profiles, the profile feedback data files do not contain any
+profile feedback information for the newly
+added function or file respectively.  Also, in the case when profile count data
+(.gcda) files are wiped out, GCC cannot use any profile feedback
+information.  In all these cases, warnings are issued to inform the user that a
+profile generation step is due.  By default, this warning is enabled and is
+treated as an error.  @option{-Wno-missing-profile} can be used to disable the
+warning or @opti

Re: [PATCH] PR86957

[Indu Bhagat]

Thanks for the reviews. I have incorporated all but one (See below; its the 
change in the warning's
brief summary in common.opt) in the patch.

In this patch,

1. -Wmissing-profile is a warning by default and is ON by default with
   -fprofile-use
2. Attached pr86957-missing-profile-diagnostic-2 shows the warning messages
3. Added a testcase for warning in the case of missing profile feedback data
   file for a compilation unit

Thanks

gcc/ChangeLog:

2018-09-14  "Indu Bhagat"

* common.opt: New warning option -Wmissing-profile.
* coverage.c (get_coverage_counts): Add warning for missing .gcda file.
* doc/invoke.texi: Document -Wmissing-profile.

gcc/testsuite/ChangeLog:

2018-09-14  "Indu Bhagat"

* gcc.dg/Wmissing-profile.c: New test.


On 09/11/2018 02:21 AM, Martin Liška wrote:

diff --git a/gcc/common.opt b/gcc/common.opt
index ebc3ef4..d93ddca 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -811,6 +811,10 @@ Wcoverage-mismatch
  Common Var(warn_coverage_mismatch) Init(1) Warning
  Warn in case profiles in -fprofile-use do not match.
  
+Wmissing-profile

+Common Var(warn_missing_profile) Init(1) Warning
+Warn in case profiles in -fprofile-use do not exist.

Maybe 'Want about missing profile for a function in -fprofile-use build.' ?



Since, Wmissing-profile also warns when feedback data file is missing for a 
compilation unit, the
suggested text above will be more restrictive. So I did not change.

[Testcase 1] Missing profile data file with -fprofile-use. The sort.c file 
contains two functions main() and stop()

gcc -c sort.c -fprofile-use -O1 
-fprofile-dir=/scratch/user/gcc-temp/fdo/profdata/
sort.c: In function ‘main’:
sort.c:29:1: warning: 
‘/scratch/user/gcc-temp/fdo/profdata//#scratch#user#gcc-temp#fdo#sort.gcda’ 
profile count data file not found [-Wmissing-profile]
29 | }
   | ^


[Testcase 2] bubble_sort.c has a non-empty function bubble_sort() for which 
profile data exists.
Now at profile-use phase, a user adds empty function before1() before an 
existing lone function bubble_sort() and empty function after1() after the lone 
existing function bubble_sort()

gcc -c bubble_sort.c -fprofile-use -O1 
-fprofile-dir=/scratch/user/gcc-temp/fdo/profdata/
bubble_sort.c:20:6: warning: profile for function ‘after1’ not found in profile 
data [-Wmissing-profile]
20 | void after1() { }
   |  ^~
bubble_sort.c: In function ‘bubble_sort’:
bubble_sort.c:20:1: error: source locations for function ‘bubble_sort’ have 
changed, the profile data may be out of date [-Werror=coverage-mismatch]
20 | void after1() { }
   | ^~~~
bubble_sort.c: In function ‘before1’:
bubble_sort.c:3:6: warning: profile for function ‘before1’ not found in profile 
data [-Wmissing-profile]
3 | void before1() { }
  |  ^~~
cc1: some warnings being treated as errors
make: *** [bubble_sort.o] Error 1


[Testcase 3] Use -Wno-missing-profile to disable warnings (A coverage-mismatch 
error remains here because of source file changes as mentioned in Testcase 2 
above; but no warnings are issued for before1() and after1())

gcc -c bubble_sort.c -fprofile-use -O1 -Wno-missing-profile 
-fprofile-dir=/scratch/user/gcc-temp/fdo/profdata/
bubble_sort.c: In function ‘bubble_sort’:
bubble_sort.c:20:1: error: source locations for function ‘bubble_sort’ have 
changed, the profile data may be out of date [-Werror=coverage-mismatch]
20 | void after1() { }
   | ^~~~
cc1: some warnings being treated as errors

diff --git a/gcc/common.opt b/gcc/common.opt
index ef6a630..53aac19 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -811,6 +811,10 @@ Wcoverage-mismatch
 Common Var(warn_coverage_mismatch) Init(1) Warning
 Warn in case profiles in -fprofile-use do not match.
 
+Wmissing-profile
+Common Var(warn_missing_profile) Init(1) Warning
+Warn in case profiles in -fprofile-use do not exist.
+
 Wvector-operation-performance
 Common Var(warn_vector_operation_performance) Warning
 Warn when a vector operation is compiled outside the SIMD.
diff --git a/gcc/coverage.c b/gcc/coverage.c
index bae6f5c..f9b54d8 100644
--- a/gcc/coverage.c
+++ b/gcc/coverage.c
@@ -341,16 +341,23 @@ get_coverage_counts (unsigned counter, unsigned expected,
 {
   static int warned = 0;
 
-  if (!warned++ && dump_enabled_p ())
+  if (!warned++)
 	{
-	  dump_user_location_t loc
-	= dump_user_location_t::from_location_t (input_location);
-	  dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+	  warning (OPT_Wmissing_profile,
+		   "%qs profile count data file not found",
+		   da_file_name);
+	  if (dump_enabled_p ())
+	{
+	  dump_user_location_t loc
+		= dump_user_location_t::from_location_t (input_location);
+	  dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+			   "file %s not found\n",
+			   da_file_name);
+	  dump_printf (MSG_OPTIMIZED_LOCATIONS,
 			   (flag_guess_branch_prob
-			? "file %s not found, executio

Re: [PATCH] PR86957

[Indu Bhagat]

On 09/17/2018 03:52 AM, Jan Hubicka wrote:

On 09/11/2018 02:21 AM, Martin Liška wrote:

--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -811,6 +811,10 @@ Wcoverage-mismatch
  Common Var(warn_coverage_mismatch) Init(1) Warning
  Warn in case profiles in -fprofile-use do not match.
  
+Wmissing-profile

+Common Var(warn_missing_profile) Init(1) Warning
+Warn in case profiles in -fprofile-use do not exist.

Maybe 'Want about missing profile for a function in -fprofile-use build.' ?


Since, it also warns when feedback file is missing for a compilation unit, the
suggested text above will be more restrictive. So I did not change.

Perhaps we want also to have reference from -fprofile-use documentation so 
users notice
this option.

Honza

Thanks

Yes, I had added that in the patch. Following additional text will 
appear at the end of

-fprofile-use :

+ Additionally, by default, GCC also emits a warning message if
+the feedback profiles do not exist (See @option{-Wmissing-profile}).

Re: [PATCH] PR86957

[Indu Bhagat]

Attached is the refreshed patch for trunk.

After commit 264462 (Remove arc profile histogram in non-LTO mode.), the API
of get_coverage_counts was changed a bit. So the main difference between the
current version of my patch from the previous one is that :

Now I use
+  if (counter == GCOV_COUNTER_ARCS)
+   warning_at (DECL_SOURCE_LOCATION (current_function_decl),
+   OPT_Wmissing_profile,
+   "profile for function %qD not found in profile data",
+   current_function_decl);

instead of
+  if (summary)
+   warning_at (DECL_SOURCE_LOCATION (current_function_decl),
+   OPT_Wmissing_profile,
+   "profile for function %qD not found in profile data",
+   current_function_decl);

to warn about missing profile of a function only once (get_coverage_counts is
called from two diff flows : getting exec counts for arc counter and computing
histogram for other other counters)

I am not sure of any better way to avoid superfluous warnings per function.

Is the patch OK ?

diff --git a/gcc/common.opt b/gcc/common.opt
index ef6a630..53aac19 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -811,6 +811,10 @@ Wcoverage-mismatch
 Common Var(warn_coverage_mismatch) Init(1) Warning
 Warn in case profiles in -fprofile-use do not match.
 
+Wmissing-profile
+Common Var(warn_missing_profile) Init(1) Warning
+Warn in case profiles in -fprofile-use do not exist.
+
 Wvector-operation-performance
 Common Var(warn_vector_operation_performance) Warning
 Warn when a vector operation is compiled outside the SIMD.
diff --git a/gcc/coverage.c b/gcc/coverage.c
index 26cce2b..4b6df8a 100644
--- a/gcc/coverage.c
+++ b/gcc/coverage.c
@@ -304,16 +304,23 @@ get_coverage_counts (unsigned counter, unsigned cfg_checksum,
 {
   static int warned = 0;
 
-  if (!warned++ && dump_enabled_p ())
+  if (!warned++)
 	{
-	  dump_user_location_t loc
-	= dump_user_location_t::from_location_t (input_location);
-	  dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+	  warning (OPT_Wmissing_profile,
+		   "%qs profile count data file not found",
+		   da_file_name);
+	  if (dump_enabled_p ())
+	{
+	  dump_user_location_t loc
+		= dump_user_location_t::from_location_t (input_location);
+	  dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+			   "file %s not found\n",
+			   da_file_name);
+	  dump_printf (MSG_OPTIMIZED_LOCATIONS,
 			   (flag_guess_branch_prob
-			? "file %s not found, execution counts estimated\n"
-			: "file %s not found, execution counts assumed to "
-			"be zero\n"),
-			   da_file_name);
+			? "execution counts estimated\n"
+			: "execution counts assumed to be zero\n"));
+	}
 	}
   return NULL;
 }
@@ -327,10 +334,17 @@ get_coverage_counts (unsigned counter, unsigned cfg_checksum,
   elt.ctr = counter;
   entry = counts_hash->find (&elt);
   if (!entry)
-/* The function was not emitted, or is weak and not chosen in the
-   final executable.  Silently fail, because there's nothing we
-   can do about it.  */
-return NULL;
+{
+  if (counter == GCOV_COUNTER_ARCS)
+	warning_at (DECL_SOURCE_LOCATION (current_function_decl),
+		OPT_Wmissing_profile,
+		"profile for function %qD not found in profile data",
+		current_function_decl);
+  /* The function was not emitted, or is weak and not chosen in the
+	 final executable.  Silently fail, because there's nothing we
+	 can do about it.  */
+  return NULL;
+}
   
   if (entry->cfg_checksum != cfg_checksum)
 {
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index abbd9ec..ed56af3 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -315,7 +315,7 @@ Objective-C and Objective-C++ Dialects}.
 -Wlogical-op  -Wlogical-not-parentheses  -Wlong-long @gol
 -Wmain  -Wmaybe-uninitialized  -Wmemset-elt-size  -Wmemset-transposed-args @gol
 -Wmisleading-indentation  -Wmissing-attributes -Wmissing-braces @gol
--Wmissing-field-initializers  -Wmissing-include-dirs @gol
+-Wmissing-field-initializers  -Wmissing-include-dirs  -Wmissing-profile @gol
 -Wno-multichar  -Wmultistatement-macros  -Wnonnull  -Wnonnull-compare @gol
 -Wnormalized=@r{[}none@r{|}id@r{|}nfc@r{|}nfkc@r{]} @gol
 -Wnull-dereference  -Wodr  -Wno-overflow  -Wopenmp-simd  @gol
@@ -4218,8 +4218,8 @@ Warn about an invalid memory access that is found by Pointer Bounds Checker
 @opindex Wcoverage-mismatch
 Warn if feedback profiles do not match when using the
 @option{-fprofile-use} option.
-If a source file is changed between compiling with @option{-fprofile-gen} and
-with @option{-fprofile-use}, the files with the profile feedback can fail
+If a source file is changed between compiling with @option{-fprofile-generate}
+and with @option{-fprofile-use}, the files with the profile feedback can fail
 to match the source file and GCC cannot use the profile feedback
 information.  By default, this warning is enabled a

Re: [PATCH] PR86957

[Indu Bhagat]

Done. Attached is updated patch.

Patch is tested on x86_64

Thanks


On 09/24/2018 09:37 AM, Martin Sebor wrote:

I would suggest to use the term "remove" or "delete" instead of
the informal "wipe out" when referring to removing files or their
contents.

Martin 


diff --git a/gcc/common.opt b/gcc/common.opt
index ef6a630..53aac19 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -811,6 +811,10 @@ Wcoverage-mismatch
 Common Var(warn_coverage_mismatch) Init(1) Warning
 Warn in case profiles in -fprofile-use do not match.
 
+Wmissing-profile
+Common Var(warn_missing_profile) Init(1) Warning
+Warn in case profiles in -fprofile-use do not exist.
+
 Wvector-operation-performance
 Common Var(warn_vector_operation_performance) Warning
 Warn when a vector operation is compiled outside the SIMD.
diff --git a/gcc/coverage.c b/gcc/coverage.c
index 8f12778..19c696c 100644
--- a/gcc/coverage.c
+++ b/gcc/coverage.c
@@ -304,16 +304,23 @@ get_coverage_counts (unsigned counter, unsigned cfg_checksum,
 {
   static int warned = 0;
 
-  if (!warned++ && dump_enabled_p ())
+  if (!warned++)
 	{
-	  dump_user_location_t loc
-	= dump_user_location_t::from_location_t (input_location);
-	  dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+	  warning (OPT_Wmissing_profile,
+		   "%qs profile count data file not found",
+		   da_file_name);
+	  if (dump_enabled_p ())
+	{
+	  dump_user_location_t loc
+		= dump_user_location_t::from_location_t (input_location);
+	  dump_printf_loc (MSG_OPTIMIZED_LOCATIONS, loc,
+			   "file %s not found\n",
+			   da_file_name);
+	  dump_printf (MSG_OPTIMIZED_LOCATIONS,
 			   (flag_guess_branch_prob
-			? "file %s not found, execution counts estimated\n"
-			: "file %s not found, execution counts assumed to "
-			"be zero\n"),
-			   da_file_name);
+			? "execution counts estimated\n"
+			: "execution counts assumed to be zero\n"));
+	}
 	}
   return NULL;
 }
@@ -327,10 +334,17 @@ get_coverage_counts (unsigned counter, unsigned cfg_checksum,
   elt.ctr = counter;
   entry = counts_hash->find (&elt);
   if (!entry)
-/* The function was not emitted, or is weak and not chosen in the
-   final executable.  Silently fail, because there's nothing we
-   can do about it.  */
-return NULL;
+{
+  if (counter == GCOV_COUNTER_ARCS)
+	warning_at (DECL_SOURCE_LOCATION (current_function_decl),
+		OPT_Wmissing_profile,
+		"profile for function %qD not found in profile data",
+		current_function_decl);
+  /* The function was not emitted, or is weak and not chosen in the
+	 final executable.  Silently fail, because there's nothing we
+	 can do about it.  */
+  return NULL;
+}
   
   if (entry->cfg_checksum != cfg_checksum)
 {
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index abbd9ec..1b6cd68 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -315,7 +315,7 @@ Objective-C and Objective-C++ Dialects}.
 -Wlogical-op  -Wlogical-not-parentheses  -Wlong-long @gol
 -Wmain  -Wmaybe-uninitialized  -Wmemset-elt-size  -Wmemset-transposed-args @gol
 -Wmisleading-indentation  -Wmissing-attributes -Wmissing-braces @gol
--Wmissing-field-initializers  -Wmissing-include-dirs @gol
+-Wmissing-field-initializers  -Wmissing-include-dirs  -Wmissing-profile @gol
 -Wno-multichar  -Wmultistatement-macros  -Wnonnull  -Wnonnull-compare @gol
 -Wnormalized=@r{[}none@r{|}id@r{|}nfc@r{|}nfkc@r{]} @gol
 -Wnull-dereference  -Wodr  -Wno-overflow  -Wopenmp-simd  @gol
@@ -4218,8 +4218,8 @@ Warn about an invalid memory access that is found by Pointer Bounds Checker
 @opindex Wcoverage-mismatch
 Warn if feedback profiles do not match when using the
 @option{-fprofile-use} option.
-If a source file is changed between compiling with @option{-fprofile-gen} and
-with @option{-fprofile-use}, the files with the profile feedback can fail
+If a source file is changed between compiling with @option{-fprofile-generate}
+and with @option{-fprofile-use}, the files with the profile feedback can fail
 to match the source file and GCC cannot use the profile feedback
 information.  By default, this warning is enabled and is treated as an
 error.  @option{-Wno-coverage-mismatch} can be used to disable the
@@ -4823,6 +4823,23 @@ This warning is enabled by @option{-Wall}.
 @opindex Wno-missing-include-dirs
 Warn if a user-supplied include directory does not exist.
 
+@item -Wmissing-profile
+@opindex Wmissing-profile
+@opindex Wno-missing-profile
+Warn if feedback profiles are missing when using the
+@option{-fprofile-use} option.
+This option diagnoses those cases where a new function or a new file is added
+to the user code between compiling with @option{-fprofile-generate} and with
+@option{-fprofile-use}, without regenerating the profiles.  In these cases, the
+profile feedback data files do not contain any profile feedback information for
+the newly added function or file respectively.  Also, in the case when profile
+c

[PATCH] Minor documentation correction in aarch64-simd.md

[Indu Bhagat]

In function minmax_replacement in tree-ssa-phiopt.c, MIN_EXPR/MAX_EXPR 
are substituted for when the following condition is false - (HONOR_NANS 
(type) || HONOR_SIGNED_ZEROS (type)). So for FP mode, this is false when 
_both_ of the following conditions are fulfilled : 1. flag_signed_zeros 
is zero and 2. flag_finite_math_only is set. So, the documentation in 
aarch64-simd.md is partially misleading. Here is a patch to correct 
that. Thanks


---

gcc/ChangeLog:

* config/aarch64/aarch64-simd.md: correct flags text for 
MIN_EXPR replacement




diff --git a/gcc/config/aarch64/aarch64-simd.md 
b/gcc/config/aarch64/aarch64-simd.md
index 1154fc3..7fd20fd 100644
--- a/gcc/config/aarch64/aarch64-simd.md
+++ b/gcc/config/aarch64/aarch64-simd.md
@@ -2211,8 +2211,9 @@
 ;; Max/Min are introduced by idiom recognition by GCC's mid-end.  An
 ;; expression like:
 ;;  a = (b < c) ? b : c;
-;; is idiom-matched as MIN_EXPR only if -ffinite-math-only is enabled
-;; either explicitly or indirectly via -ffast-math.
+;; is idiom-matched as MIN_EXPR only if -ffinite-math-only and
+;; -fno-signed-zeros are enabled either explicitly or indirectly via
+;; -ffast-math.
 ;;
 ;; MIN_EXPR and MAX_EXPR eventually map to 'smin' and 'smax' in RTL.
 ;; The 'smax' and 'smin' RTL standard pattern names do not specify which

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Thanks for your feedback. Comments inline.


On 05/21/2019 03:28 AM, Richard Biener wrote:

GCC RFC patch set :
Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
GIMPLE frontend.

I don't think you should need this - the GIMPLE "frontend" is intended for
unit testing only, I wouldn't like it to be exposed more.


When using -gt with -flto, I would still like the CTF hooks to be initialized
so that CTF can be generated when -flto is used. So the check in toplev.c is
done to allow only C and GNU GIMPLE.  I am fine with doing a string compare
with the language.hooks string if you suggest to go that way.




One of the main high-level design requirements that is relevant in the context
of the current GCC patch set is that - CTF and DWARF must be able to co-exist.
A user may want CTF debug information in isolation or with other debug formats.
A .ctf section is small and unlike other debug sections, ideally should not
need to be stripped out of the binary/executable.

High-level proposed plan (phase 1) :
In the next few patches, the functionality to generate contents of the CTF
section (.ctf) for a single compilation unit will be added.
Once CTF generation for a single compilation unit stabilizes, LTO and CTF
generation will be looked at.

Feedback and suggestions welcome.

You probably got asked this question multiple times already, but,
can CTF information be generated from DWARF instead?


Yes and No :) And that is indeed one of the motivation of the project - to
allow CTF generation where it's most suited aka the toolchain.

There do exist utilties for generation of CTF from DWARF. For example, one of
them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
to transform DWARF generated by the compiler into CTF.

A dependency of an external conversion utility for "post-processing" DWARF
offline poses several problems:

1. Deployment problems: the converter should be distributed and integrated in
   the build system of the program.  This, on occasions, can be intrusive.  For
   example, in terms of dependencies: the dwarf2ctf converter depends on
   libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
   compress the CTF information) and libctf (which both reads and writes the
   CTF data).

2. Performance problems: the conversion from DWARF to CTF can take a long time,
   especially in big programs such as the Linux kernel.

3. Maintainability problems: the converter should be maintained in order to
   reflect potential changes in the DWARF generated by the compiler.

4. Adoption problem: it is difficult for applications to adopt the usage of
   CTF, even if it happens to provide what they need, since it would require to
   write a conversion utility or integrate DTrace's.




The meaning of the CTF acronym suggests that there's nothing
like locations, registers, etc. but just a representation of the
types?


Yes. CTF is simply put Type information; no locations, registers etc.



Generally we are trying to walk away from supporting multiple
debug info formats because that gets in the way of being
more precise from the frontend side.  Since DWARF is the


With regard to whether the support for CTF imposes infeasible or distinct
requirements on the frontend - it does not appear to be the case (I have
been working on CTF generation in GCC for a SINGLE compilation unit; More see
below). I agree that CTF debug information generation should ideally not impose
additional requirements on the frontend.


defacto standard, extensible and with a rich feature set the
line of thinking is that other formats (like STABS) can be
generated by "post-processing" DWARF.  Such
post-processing could happen on the object files or
on the GCC internal DWARF data structures by
providing alternate output routines.  That is, the mid-term
design goal is to make DWARF generation the "API"
for GCC frontends to use when creating high-level
debug information rather than trying to abstract from
the debuginfo format via the current debug-hooks or
the other way around via language-hooks.


I am not sure if I understood the last part very well, so I will state how CTF
generation is intended to work. Does the following fit the design goal you
state ?

( Caveat : I have been working on the functionality to generate CTF for a SINGLE
  compilation unit. LTO bits remain. )

So far, there are no additional requirements on the frontend side. CTF hooks
are wrappers around DWARF debug hooks (much like go-dump hooks, and vms dbg
hooks).  We did notice that GCC does not have the infrastructure to register or
enlist multiple debug hooks; and now from your comments it is clear that this
is by design. Thanks for clarifying that.

Having said that, I use CTF hooks to go from TREE --> update CTF internal
structures or output CTF routines depending on the hook (e.g., type_decl or
finish respectively), rather than changing the dwarf* files with CTF APIs. The
CTF debug hooks relay control t

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

On 05/22/2019 02:04 AM, Richard Biener wrote:

The CTF debug information is kept in a CTF container distinct from the frontend
structures.  HashMaps are used to avoid generation of duplicate CTF and to
book-keep the generated CTF.

OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
data structures?  That is, in my view CTF should be emitted by
dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
is generated from).  It would be nice to avoid extra bookkeeping data structures
for CTF since those of DWARF should include all necessary information already.


CTF format has some characteristics which make it necessary to "pre-process"
the generated CTF data before asm'ing out into a section. E.g. few cases of why
"pre-processing" CTF is required before asm'ing out :
 1. CTF types do need to be emitted in "some" order :
CTF types can have references to other CTF types. This consequently implies
that the referenced type must appear BEFORE the referring type.
 2. CTF preamble holds offsets to the various subsections - function info,
variables, data types and CTF string table. To calculate the offsets, the
compiler needs to know the size in bytes of these sub-sections.  CTF
representation for some types like structures, functions, enums have
variable length of bytes trailing them (depending on the defintion of the
type).
 3. CTF variable entries need to be placed in the order of the names.

Because of some of these "features" of the CTF format, the compiler does need
to do a transition from runtime CTF generated data --> CTF binary data format
for a clean and readable code.

So, I think the needs are different enough to vouch for an implementation
segregated from dwarf* codebase.



Btw, do I read the CTF document posted to the binutils list (but not
cross-referenced
here :/) correctly in that you only want CTF debug for objects defined
in the file and
type information for the types refered to from that?  At


Yes. CTF is emitted for types at file-scope and global-scope only.  Types, vars
at function-scope should be skipped.


dwarf2out_early_finish time
it isn't fully known which symbols will end up being emitted (and with
LTO you only
would know at link time).


In nutshell, I am processing all decl at early_global_decl () time except
TYPE_DECL (Similar to DWARF, based on the thinking that if they are required
they will be reached at via other DECL).
In addition, I process all decl at type_decl () time except function-scope,
no-name decl, builtins.

Currently, it does look like CTF for possibly to-be-omitted symbols will be
generated... I assume even DWARF needs to handle this case. Can you point me to
how DWARF does this ?


It seems to me that linker support to garbage collect
unused entries would be the way to go forward (probably easy for the
declarations
but not so for the types)?


Hmm, garbage collecting unused types in linker - Let me get back to you on
this. It does not look easy. Decl should be doable though.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Hi Michael,

On 05/24/2019 06:04 AM, Michael Matz wrote:

Hello,

On Thu, 23 May 2019, Indu Bhagat wrote:


OK.  So I wonder how difficult it is to emit CTF by walking dwarf2outs own
data structures?  That is, in my view CTF should be emitted by
dwarf2out_early_finish ()  (which is also the point LTO type/decl debug
is generated from).  It would be nice to avoid extra bookkeeping data
structures
for CTF since those of DWARF should include all necessary information
already.

CTF format has some characteristics which make it necessary to "pre-process"
the generated CTF data before asm'ing out into a section. E.g. few cases of
why "pre-processing" CTF is required before asm'ing out :
  1. CTF types do need to be emitted in "some" order :
 CTF types can have references to other CTF types. This consequently
 implies
 that the referenced type must appear BEFORE the referring type.
  2. CTF preamble holds offsets to the various subsections - function info,
 variables, data types and CTF string table. To calculate the offsets, the
 compiler needs to know the size in bytes of these sub-sections.  CTF
 representation for some types like structures, functions, enums have
 variable length of bytes trailing them (depending on the defintion of the
 type).
  3. CTF variable entries need to be placed in the order of the names.

Because of some of these "features" of the CTF format, the compiler does
need to do a transition from runtime CTF generated data --> CTF binary
data format for a clean and readable code.

Sure, but this whole process could still be triggered from within
dwarf2out_early_finish, by walking the DWARF tree (instead of getting fed
types and decls via hooks) and generating the appropriate CTF data
structures.  (It's just a possibility, it might end up uglier that using
GCC trees)


I think not only is the code messier, but it's also wasted effort if user only
wants to generate CTF.


Imagine a world where debug hooks wouldn't exist (which is where we would
like to end up in a far away future), how would you then add CTF debug
info to the compiler (assuming it already emits DWARF)?  You would hook
yourself either into the DWARF routines that currently are fed the
entities or you would hook yourself into somewhere late in the pipeline
where the DWARF debug info is complete and you would generate CTF from
that.


So, I think the needs are different enough to vouch for an implementation
segregated from dwarf* codebase.

Of course.  We are merely discussing of where the triggering of processing
starts: debug hooks, or something like:

dwarf2out_early_finish() {
   ...
   if (ctf)
 ctf_emit();
}

(and then in addition if the current DWARF info would be the source of CTF
info, or if it'd be whatever the compiler gives you as trees)

The thing is, with debug hooks you'd have to invent a scheme of stacking
hooks on top of each other (because we want to generate DWARF and CTF from
the same compilation).  That seems like a wasted effort when our wish is
for the hooks to go away alltogether.


When the debug hooks go away, the functionality can be folded in. Much like
above, the ctf proposed implementation will do :

ctf_early_global_decl (tree decl)
{
  ctf_decl (decl);

  real_debug_hooks->early_global_decl (decl);
}

These ctf_* debug hooks wrappers are as lean as shown above.

I do understand now that if debug hooks are destined to go away, all the
implementation which wraps debug hooks (go dump hooks, vms debug hooks,
and now the proposed ctf debug hooks) will need some merging. But to generate
CTF, I think working on type or decl instead of DWARF dies to is a better
implementation because if user wants only CTF, no DWARF trees need to be
created.

This way we keep DWARF and CTF generation independent of each other (as the
user may want either one of these or both).


Ciao,
Michael.

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

On 05/24/2019 02:26 AM, Richard Biener wrote:

Currently, it does look like CTF for possibly to-be-omitted symbols will be
generated... I assume even DWARF needs to handle this case. Can you point me to
how DWARF does this ?

It emits the debug information.  DWARF outputs a representation of the source,
not only emitted objects.  We prune some "unused" bits if the user prefers us
to do that but we do not omit information on types or decls that are used in
the source but later eventually optimized away.


It seems to me that linker support to garbage collect
unused entries would be the way to go forward (probably easy for the
declarations
but not so for the types)?

Hmm, garbage collecting unused types in linker - Let me get back to you on
this. It does not look easy. Decl should be doable though.

For example DWARF has something like type units that can be refered
to via hashes.  GCC can output those into separate sections and I can
envision outputting separate debug (CTF) sections for each declaration.
The linker could then merge sections for declarations that survived
and pick up all referenced type sections.  Restrictions on ordering
for CTF may make this a bit difficult though, essentially forcing a
separate intermediate "unlinked" format and the linker regenerating
the final one.  OTOH CTF probably simply concatenates data from
different CUs?


Yes, I cannot see this happening with CTF easily without some format changes.

At link-time, there needs to be de-duplication of CTF types across CUs. This
linker component needs work at this time, although we do have a working
prototype.

Regarding the type units in DWARF, are the shared/common types duplicated
across the type units ? If not duplicated, how are the referenced types
maintained/denoted across type units ?

Thanks!
Indu

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

On 05/29/2019 12:15 AM, Richard Biener wrote:

Of course.  We are merely discussing of where the triggering of processing
starts: debug hooks, or something like:

dwarf2out_early_finish() {
...
if (ctf)
  ctf_emit();
}

(and then in addition if the current DWARF info would be the source of CTF
info, or if it'd be whatever the compiler gives you as trees)

The thing is, with debug hooks you'd have to invent a scheme of stacking
hooks on top of each other (because we want to generate DWARF and CTF from
the same compilation).  That seems like a wasted effort when our wish is
for the hooks to go away alltogether.


When the debug hooks go away, the functionality can be folded in. Much like
above, the ctf proposed implementation will do :

ctf_early_global_decl (tree decl)
{
ctf_decl (decl);

real_debug_hooks->early_global_decl (decl);
}

These ctf_* debug hooks wrappers are as lean as shown above.

I do understand now that if debug hooks are destined to go away, all the
implementation which wraps debug hooks (go dump hooks, vms debug hooks,
and now the proposed ctf debug hooks) will need some merging. But to generate
CTF, I think working on type or decl instead of DWARF dies to is a better
implementation because if user wants only CTF, no DWARF trees need to be
created.

This way we keep DWARF and CTF generation independent of each other (as the
user may want either one of these or both).

The user currently can't have both DWARF and STABS either.  That things like
godump uses debug hooks is just (convenient?) abuse.

In the end frontends will not call sth like dwarf2out_decl but maybe
gen_subroutine_die () or gen_template_die ().  So how do you expect
the "wrapping" to work there?

I understand you want CTF for "actually emitted" decls so I propose you
instead hook into the symtab code which would end up calling the
early_global_decl debug hook.  But please don't add new debug hook
users.


OK.

Will I need to tap both the callsites of the early_global_decl () debug hook ? :
  1. symbol_table::finalize_compilation_unit () in cgraphunit.c
  2. rest_of_decl_compilation () in passes.c
Or is the last one for something specific to godump debug hooks and C++ ?

I guess the above will take care of the CTF generation bit. For emission,
something similar should be done because DWARF hooks will not be initialized if
DWARF debuginfo is not requested by the user. So I cannot have the CTF emission
code in the dwarf2out*finish () debug hooks as suggested earlier.

Curious to know how the current debug hook users like dbx debug hooks will be
taken care of in the future design ? Is it just the wrapping/stacking of debug
hooks that's problematic and not the clean instances like dbx debug hooks ?

Re: [PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Hello,


On 05/24/2019 06:24 AM, Jakub Jelinek wrote:

On Tue, May 21, 2019 at 03:44:47PM -0700, Indu Bhagat wrote:

Yes and No :) And that is indeed one of the motivation of the project - to
allow CTF generation where it's most suited aka the toolchain.

There do exist utilties for generation of CTF from DWARF. For example, one of
them is the dwarf2ctf in the DTrace Linux distribution. dwarf2ctf works offline
to transform DWARF generated by the compiler into CTF.

So, if there is a conversion utility, why don't we just change gcc so that
if some new option is passed on the gcc driver link line, then that
post-processing utility will be invoked too?


Performing DWARF to CTF conversion at link-time is not recommended because for
any large project, it is time-consuming to generate, read in, and convert DWARF
to CTF in a post-processing step. These costs are prohibitive enough and affect
CTF adoption.

Data for some projects below.


A dependency of an external conversion utility for "post-processing" DWARF
offline poses several problems:

1. Deployment problems: the converter should be distributed and integrated in
the build system of the program.  This, on occasions, can be intrusive.  For
example, in terms of dependencies: the dwarf2ctf converter depends on
libdwarf from elfutils suite, glib2 (used for the GHashTable), zlib (used to
compress the CTF information) and libctf (which both reads and writes the
CTF data).

I don't see this as a problem.


2. Performance problems: the conversion from DWARF to CTF can take a long time,
especially in big programs such as the Linux kernel.

So optimize it?  Linux kernel certainly doesn't have extra large debug
information, compared to other projects.


DWARF generation only for it to be transformed into something compact like CTF
and eventually to be stripped away is not an efficient workflow; Further, only
a subset of this is amenable to "optimizations", i.e. compile-time generation
of DWARF is unavoidable in this flow.

For example, for the Linux kernel, the debuginfo in the object files is ~9GiB
(The object files without debuginfo is ~1.5GiB). That's a 500% increase in the
space requirements. Generating the DWARF at compile time adds 20% to the compile
time. Next, reading in this DWARF using libdwarf from elftuils takes ~1 min 10
seconds. Next, a conversion utility like dwarf2ctf will then need to perform
de-duplication, on this rather voluminous DWARF, as you see, for the purpose on
hand.

Not just the kernel, for another relatively large internal application we
measured a 19% and 23% increase in compile time for generating DWARF for
a randomly chosen subsets of C and C++ components respectively. For the entire
application, it already takes an order of 3-4 hours to perform a parallel build.
The space requirements for building the entire application with -g also increase
to about 5.5x.

These numbers I state above are on sufficiently beefy platforms. On the
"wimpier" platforms or single-threaded builds for that matter, the increase in
time costs to generate DWARF and then use an external utility (at link-time
or offline) just aggravates the pain for developers, who may need to do even
multiple builds a day.

So, in summary, this increase in build times and space costs is noticeable
(even prohibitive in some cases) and does impact a subset of kernel and
application developers who need to use CTF. Above all, effectively it limits
CTF adoption.




3. Maintainability problems: the converter should be maintained in order to
reflect potential changes in the DWARF generated by the compiler.

If you integrate the support into GCC, then it will need to be maintained
there as well, I bet it will be more work than on the conversion utility.


Yes, work is needed on all components of the toolchain to integrate CTF
generation. We are working on the compiler, linker and debugger (GDB) currently.

We are committing to maintain the CTF generation in GCC, just like we are doing
for Binutils, and will be doing for GDB.



Jakub

[PATCH,RFC,V2 1/3] Add new function lang_GNU_GIMPLE

[Indu Bhagat]

[Changes from V1]
  None

gcc/ChangeLog:

* langhooks.c (lang_GNU_GIMPLE): New Function.
* langhooks.h: New Prototype.

---
 gcc/ChangeLog   | 5 +
 gcc/langhooks.c | 9 +
 gcc/langhooks.h | 1 +
 3 files changed, 15 insertions(+)

diff --git a/gcc/langhooks.c b/gcc/langhooks.c
index 2df97f2..f3a64c1 100644
--- a/gcc/langhooks.c
+++ b/gcc/langhooks.c
@@ -825,3 +825,12 @@ lang_GNU_OBJC (void)
 {
   return strncmp (lang_hooks.name, "GNU Objective-C", 15) == 0;
 }
+
+/* Returns true if the current lang_hooks represents the GNU GIMPLE
+   frontend.  */
+
+bool
+lang_GNU_GIMPLE (void)
+{
+  return strncmp (lang_hooks.name, "GNU GIMPLE", 10) == 0;
+}
diff --git a/gcc/langhooks.h b/gcc/langhooks.h
index a45579b..0ac794e 100644
--- a/gcc/langhooks.h
+++ b/gcc/langhooks.h
@@ -570,5 +570,6 @@ extern bool lang_GNU_C (void);
 extern bool lang_GNU_CXX (void);
 extern bool lang_GNU_Fortran (void);
 extern bool lang_GNU_OBJC (void);
+extern bool lang_GNU_GIMPLE (void);
 
 #endif /* GCC_LANG_HOOKS_H */
-- 
1.8.3.1

[PATCH,RFC,V2 0/3] Support for CTF in GCC

[Indu Bhagat]

Hello,

Thanks for the feedback on the previous patch set.

This is the second posting of the RFC patch for CTF support in GCC. This patch
set does not rely on debug hooks, but it keeps CTF and DWARF debug info
generation separated in the compiler.

For CTF generation, callsites in symbol_table::finalize_compilation_unit and
rest_of_decl_compilation are used. For CTF emission, callsite in
symbol_table::finalize_compilation_unit is used.

Summary of the GCC RFC V2 patch set :
Patch 1 and Patch 2 have remain unchanged since V1.
Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
GIMPLE frontend.
Patch 2 and Patch 3 set up the framework for CTF support in GCC :
-- Patch 2 adds the new command line option for generating CTF. CTF generation
   is enabled in the compiler by specifying an explicit -gt or
   -gtLEVEL[LEVEL=1,2] :

-gtLEVEL

This is used to request CTF debug information and to specify how much CTF
debug information, LEVEL[=0,1,2] can be specified. If -gt is specified
(with no LEVEL), the default value of LEVEL is 2.

-gt0 (Level 0) produces no CTF debug information at all. Thus, -gt0
negates -gt.

-gt1 (Level 1) produces CTF information for tracebacks only. This includes
CTF callsite information, but does not include type information for other
entities.

-gt2 (Level 2) produces type information for entities (functions, variables
etc.) at file-scope or global-scope only. This level of information can be
used by dynamic tracers like DTrace.

--  Patch 3 initializes the CTF container if user-level option for CTF
generation is specified. CTF is generated for all to-be-emitted global
decls if gtLEVEL of 2 is specified. 

Tested on x86_64-linux and sparc64-linux.

Thanks

Indu Bhagat (3):
  Add new function lang_GNU_GIMPLE
  Add CTF command line options : -gtLEVEL
  Setup for CTF generation and emission

 gcc/ChangeLog   |  27 ++
 gcc/Makefile.in |   3 +
 gcc/cgraphunit.c|  12 +-
 gcc/common.opt  |   9 +
 gcc/ctfout.c| 163 
 gcc/ctfout.h|  52 +++
 gcc/doc/invoke.texi |  16 +
 gcc/flag-types.h|  13 +
 gcc/gengtype.c  |   4 +-
 gcc/langhooks.c |   9 +
 gcc/langhooks.h |   1 +
 gcc/opts.c  |  26 ++
 gcc/passes.c|   7 +-
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 ++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  18 +
 include/ChangeLog   |   4 +
 include/ctf.h   | 483 
 21 files changed, 913 insertions(+), 6 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

-- 
1.8.3.1

[PATCH,RFC,V2 2/3] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

-gtLEVEL is used to request CTF debug information and also to specify how much
CTF debug information.

[Changes from V1]
  None

gcc/ChangeLog:
 
* common.opt: Add CTF debug info options.
* doc/invoke.texi: Document the CTF debug info options.
* flag-types.h (enum ctf_debug_info_levels): New enum.
* opts.c (common_handle_option): New Function.
(set_ctf_debug_level): Handle the new CTF debug info options.

---
 gcc/ChangeLog   |  8 
 gcc/common.opt  |  9 +
 gcc/doc/invoke.texi | 16 
 gcc/flag-types.h| 13 +
 gcc/opts.c  | 26 ++
 5 files changed, 72 insertions(+)

diff --git a/gcc/common.opt b/gcc/common.opt
index e140416..499f27c 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -125,6 +125,11 @@ enum debug_info_levels debug_info_level = DINFO_LEVEL_NONE
 Variable
 bool use_gnu_debug_info_extensions
 
+; Level of CTF debugging information we are producing.  See flag-types.h
+; for the definitions of the different possible levels.
+Variable
+enum ctf_debug_info_levels ctf_debug_info_level = CTFINFO_LEVEL_NONE
+
 ; Original value of maximum field alignment in bytes, specified via
 ; -fpack-struct=.
 Variable
@@ -2995,6 +3000,10 @@ gcolumn-info
 Common Driver Var(debug_column_info,1) Init(1)
 Record DW_AT_decl_column and DW_AT_call_column in DWARF.
 
+gt
+Common Driver RejectNegative JoinedOrMissing
+Generate CTF debug information at default level.
+
 gdwarf
 Common Driver JoinedOrMissing Negative(gdwarf-)
 Generate debug information in default version of DWARF format.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 1520b2c..babf037 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -374,6 +374,7 @@ Objective-C and Objective-C++ Dialects}.
 @item Debugging Options
 @xref{Debugging Options,,Options for Debugging Your Program}.
 @gccoptlist{-g  -g@var{level}  -gdwarf  -gdwarf-@var{version} @gol
+-gt  -gt@var{level} @gol
 -ggdb  -grecord-gcc-switches  -gno-record-gcc-switches @gol
 -gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf @gol
 -gas-loc-support  -gno-as-loc-support @gol
@@ -7784,6 +7785,21 @@ other DWARF-related options such as
 @option{-fno-dwarf2-cfi-asm}) retain a reference to DWARF Version 2
 in their names, but apply to all currently-supported versions of DWARF.
 
+@item -gt
+@itemx -gt@var{level}
+@opindex gt
+Request CTF debug information and use level to specify how much CTF debug
+information should be produced.  If -gt is specified without a value for level,
+the default level of CTF debug information is 2.
+
+Level 0 produces no CTF debug information at all.  Thus, -gt0 negates -gt.
+
+Level 1 produces CTF information for tracebacks only.  This includes callsite
+information, but does not include type information.
+
+Level 2 produces type information for entities (functions, data objects etc.)
+at file-scope or global-scope only.
+
 @item -gstabs
 @opindex gstabs
 Produce debugging information in stabs format (if that is supported),
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index a210328..61a1432 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -105,6 +105,19 @@ enum dwarf_gnat_encodings
   Emit GNAT encodings for the rest.  */
 };
 
+/* CTF debug info levels.
+   CTF debug info levels are untied with DWARF debug info levels because CTF
+   may co-exist with DWARF.  */
+enum ctf_debug_info_levels
+{
+  CTFINFO_LEVEL_NONE = 0, /* Write no CTF debug info.  */
+  CTFINFO_LEVEL_TERSE = 1,/* Write CTF information to support tracebacks
+only.  Not Implemented.  */
+  CTFINFO_LEVEL_NORMAL = 2/* Write CTF type information for all entities
+(functions, data objects, variables etc.)
+at file-scope or global-scope only.  */
+};
+
 /* Enumerate Objective-c instance variable visibility settings. */
 
 enum ivar_visibility
diff --git a/gcc/opts.c b/gcc/opts.c
index 64f94ac..a471a76 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -195,6 +195,8 @@ static void set_debug_level (enum debug_info_type type, int 
extended,
 const char *arg, struct gcc_options *opts,
 struct gcc_options *opts_set,
 location_t loc);
+static void set_ctf_debug_level (const char *arg,
+struct gcc_options *opts, location_t loc);
 static void set_fast_math_flags (struct gcc_options *opts, int set);
 static void decode_d_option (const char *arg, struct gcc_options *opts,
 location_t loc, diagnostic_context *dc);
@@ -2683,6 +2685,10 @@ common_handle_option (struct gcc_options *opts,
   opts->x_flag_stack_usage_info = value != 0;
   break;
 
+case OPT_gt:
+  set_ctf_debug_level (arg, opts, loc);
+  break;
+
 case OPT_g:
   set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENS

[PATCH,RFC,V2 3/3] Setup for CTF generation and emission

[Indu Bhagat]

Initialize CTF container when -gtLEVEL is specified.  Generate CTF debug info
for global decls.  Import the CTF header from binutils.

[Changes from V1]
  - Instead of using the debug hooks infrastructure, the generation and
emission of CTF is done by exposing the CTF APIs to the rest of the 
compiler.
  - Calls to generate CTF are placed in symbol_table::finalize_compilation_unit
and rest_of_decl_compilation.
  - Call to emit CTF is placed in symbol_table::finalize_compilation_unit

gcc/ChangeLog:
 
* Makefile.in: Add ctfout.* files to GTFILES.
* cgraphunit.c (symbol_table::finalize_compilation_unit): Generate CTF
debug info for decl. Invoke CTF debug info emission.
* ctfout.c: New file.
* ctfout.h: Likewise.
* gengtype.c (open_base_files): Add ctfout.h to ifiles.
* passes.c (rest_of_decl_compilation): Generate CTF debug info for
decl.
* toplev.c (process_options): Warn and ignore -gtLEVEL if frontend is
not C.
(toplev::finalize): Finalize CTF containers.

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-1.c: New test.
* gcc.dg/debug/ctf/ctf-preamble-1.c: Likewise.
* gcc.dg/debug/ctf/ctf.exp: Add CTF testsuite.
* gcc.dg/debug/dwarf2-ctf-1.c: New test.

include/ChangeLog:

* ctf.h: Import from binutils.

---
 gcc/ChangeLog   |  14 +
 gcc/Makefile.in |   3 +
 gcc/cgraphunit.c|  12 +-
 gcc/ctfout.c| 163 
 gcc/ctfout.h|  52 +++
 gcc/gengtype.c  |   4 +-
 gcc/passes.c|   7 +-
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 ++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  18 +
 include/ChangeLog   |   4 +
 include/ctf.h   | 483 
 15 files changed, 826 insertions(+), 6 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index d9e0885..8ce2405 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1254,6 +1254,7 @@ OBJS = \
cfgloopanal.o \
cfgloopmanip.o \
cfgrtl.o \
+   ctfout.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
@@ -2532,6 +2533,8 @@ GTFILES = $(CPPLIB_H) $(srcdir)/input.h 
$(srcdir)/coretypes.h \
   $(srcdir)/dwarf2asm.c \
   $(srcdir)/dwarf2cfi.c \
   $(srcdir)/dwarf2out.c \
+  $(srcdir)/ctfout.h \
+  $(srcdir)/ctfout.c \
   $(srcdir)/tree-vect-generic.c \
   $(srcdir)/dojump.c $(srcdir)/emit-rtl.h \
   $(srcdir)/emit-rtl.c $(srcdir)/except.h $(srcdir)/explow.c $(srcdir)/expr.c \
diff --git a/gcc/cgraphunit.c b/gcc/cgraphunit.c
index f4d6688..2873d97 100644
--- a/gcc/cgraphunit.c
+++ b/gcc/cgraphunit.c
@@ -205,6 +205,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "lto-section-names.h"
 #include "stringpool.h"
 #include "attribs.h"
+#include "ctfout.h"
 
 /* Queue of cgraph nodes scheduled to be added into cgraph.  This is a
secondary queue used during optimization to accommodate passes that
@@ -2844,17 +2845,22 @@ symbol_table::finalize_compilation_unit (void)
 
   if (!seen_error ())
 {
-  /* Emit early debug for reachable functions, and by consequence,
-locally scoped symbols.  */
+  /* Emit early debug and ctf debug info for reachable functions, and by
+consequence, locally scoped symbols.  */
   struct cgraph_node *cnode;
   FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (cnode)
-   (*debug_hooks->early_global_decl) (cnode->decl);
+   {
+ (*debug_hooks->early_global_decl) (cnode->decl);
+ ctf_early_global_decl (cnode->decl);
+   }
 
   /* Clean up anything that needs cleaning up after initial debug
 generation.  */
   debuginfo_early_start ();
   (*debug_hooks->early_finish) (main_input_filename);
+  ctf_early_finish (main_input_filename);
   debuginfo_early_stop ();
+
 }
 
   /* Finally drive the pass manager.  */
diff --git a/gcc/ctfout.c b/gcc/ctfout.c
new file mode 100644
index 000..debb384
--- /dev/null
+++ b/gcc/ctfout.c
@@ -0,0 +1,163 @@
+/* Output ctf format from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistrib

Re: [PATCH,RFC,V2 0/3] Support for CTF in GCC

[Indu Bhagat]

Hello,

PING, In case this patch series slipped your attention.

Thanks
Indu

On Wed, Jun 12, 2019 at 10:50 AM Indu Bhagat  wrote:
>
> Hello,
>
> Thanks for the feedback on the previous patch set.
>
> This is the second posting of the RFC patch for CTF support in GCC. This patch
> set does not rely on debug hooks, but it keeps CTF and DWARF debug info
> generation separated in the compiler.
>
> For CTF generation, callsites in symbol_table::finalize_compilation_unit and
> rest_of_decl_compilation are used. For CTF emission, callsite in
> symbol_table::finalize_compilation_unit is used.
>
> Summary of the GCC RFC V2 patch set :
> Patch 1 and Patch 2 have remain unchanged since V1.
> Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
> GIMPLE frontend.
> Patch 2 and Patch 3 set up the framework for CTF support in GCC :
> -- Patch 2 adds the new command line option for generating CTF. CTF generation
>is enabled in the compiler by specifying an explicit -gt or
>-gtLEVEL[LEVEL=1,2] :
>
> -gtLEVEL
>
> This is used to request CTF debug information and to specify how much CTF
> debug information, LEVEL[=0,1,2] can be specified. If -gt is specified
> (with no LEVEL), the default value of LEVEL is 2.
>
> -gt0 (Level 0) produces no CTF debug information at all. Thus, -gt0
> negates -gt.
>
> -gt1 (Level 1) produces CTF information for tracebacks only. This includes
> CTF callsite information, but does not include type information for other
> entities.
>
> -gt2 (Level 2) produces type information for entities (functions, 
> variables
> etc.) at file-scope or global-scope only. This level of information can be
> used by dynamic tracers like DTrace.
>
> --  Patch 3 initializes the CTF container if user-level option for CTF
> generation is specified. CTF is generated for all to-be-emitted global
> decls if gtLEVEL of 2 is specified.
>
> Tested on x86_64-linux and sparc64-linux.
>
> Thanks
>
> Indu Bhagat (3):
>   Add new function lang_GNU_GIMPLE
>   Add CTF command line options : -gtLEVEL
>   Setup for CTF generation and emission
>
>  gcc/ChangeLog   |  27 ++
>  gcc/Makefile.in |   3 +
>  gcc/cgraphunit.c|  12 +-
>  gcc/common.opt  |   9 +
>  gcc/ctfout.c| 163 
>  gcc/ctfout.h|  52 +++
>  gcc/doc/invoke.texi |  16 +
>  gcc/flag-types.h|  13 +
>  gcc/gengtype.c  |   4 +-
>  gcc/langhooks.c |   9 +
>  gcc/langhooks.h |   1 +
>  gcc/opts.c  |  26 ++
>  gcc/passes.c|   7 +-
>  gcc/testsuite/ChangeLog |   7 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 ++
>  gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
>  gcc/toplev.c|  18 +
>  include/ChangeLog   |   4 +
>  include/ctf.h   | 483 
> 
>  21 files changed, 913 insertions(+), 6 deletions(-)
>  create mode 100644 gcc/ctfout.c
>  create mode 100644 gcc/ctfout.h
>  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
>  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
>  create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
>  create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
>  create mode 100644 include/ctf.h
>
> --
> 1.8.3.1
>

Re: [PATCH,RFC,V2 2/3] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

On 06/18/2019 12:24 PM, Bernhard Reutner-Fischer wrote:

On 12 June 2019 20:00:09 CEST, Indu Bhagat  wrote:

-gtLEVEL is used to request CTF debug information and also to specify
how much
CTF debug information.

The option name is way too generic IMO.
-gctfLEVEL or some such would at least  indicate its intended purpose, fwiw.

thanks


I thought about this too. The prime reason for -gtLEVEL is consistency with the
DWARF debug info command line options.

For DWARF, we have -gdwarf-VERSION and -gLEVEL. Although GCC, at this time, will
generate only the most recent CTF version (CTF_VERSION_3 in ); I don't
know for sure if the compiler will need to support the CTF_VERSION_3 and higher
simultaneously in the future. If we do, then -gctf-VERSION and -gtLEVEL will be
the most palatable choice for generating Type debug information (Type, hence
the "t" in gt) of various versions.

Thanks
Indu

Re: [committed] Trivial testsuite fix to cope with Indu's recent change

[Indu Bhagat]

This regression slipped in because my testing was a week old. The 
predict-22.c is a commit of Nov 28.


Thanks Jeff. I saw this regression yesterday and was about to send the 
patch shortly.


I should have kept the patch tested proactively.

Thanks

Indu


On 12/01/2018 08:56 AM, Jeff Law wrote:


Indu's change to clean up some profiling dumps clearly wasn't regression
tested as it fails on every target.

We have an edge to a block with a call to an explicitly marked cold
function in it.  We consider that a precise prediction.

Indu's change reflects that information into the dump file, changing it
from "count 0," to "count 0 (precise),".

This change fixes the test.  Indu, please do a regression test on your
changes.  That's standard procedure to avoid this kind of problem.

Installing on the trunk.

jeff

Re: [PATCH] PR86957

[Indu Bhagat]

On 12/05/2018 03:33 AM, Thomas Schwinge wrote:

Hi!

Sorry for my late follow-up; had a lot of catch up to do back then.

On Thu, 27 Sep 2018 11:47:31 +0200, Richard Biener  
wrote:

On Mon, Sep 24, 2018 at 9:14 PM Indu Bhagat  wrote:

Done. Attached is updated patch.

Patch is tested on x86_64

You obviously did_not_  properly test the patch since it causes a
bunch of new testsuite
failures: [...]

and more.  Please get up to speed with GCC development and testing requirements!

Also, with this commit, several test cases regressed from PASS to
UNSUPPORTED because of "{ dg-require-effective-target freorder }" running
into:

 fprofile_use_freorder16732.c: In function 'void foo()':
 fprofile_use_freorder16732.c:2:20: warning: 
'[...]/gcc/testsuite/g++/fprofile_use_freorder16732.gcda' profile count data 
file not found [-Wmissing-profile]

Iain had just mentioned that in<https://gcc.gnu.org/PR88310>, and fixed
in r266785,<https://gcc.gnu.org/ml/gcc-patches/2018-12/msg00209.html>.
(Back then, I had produced the same fix, but not yet posted.)  :-|

With "{ dg-require-effective-target freorder }" thusly restored, I then
also saw two other regressions/FAILs, back then:

 [-UNSUPPORTED: g++.dg/tree-prof/pr63581.C-]
 UNSUPPORTED: g++.dg/tree-prof/pr63581.C -fauto-profile
 {+PASS: g++.dg/tree-prof/pr63581.C compilation,  -fprofile-generate 
-D_PROFILE_GENERATE+}
 {+FAIL: g++.dg/tree-prof/pr63581.C compilation,  -fprofile-use 
-D_PROFILE_USE+}
 {+PASS: g++.dg/tree-prof/pr63581.C execution,-fprofile-generate 
-D_PROFILE_GENERATE+}
 {+UNRESOLVED: g++.dg/tree-prof/pr63581.C execution,-fprofile-use 
-D_PROFILE_USE+}
 
 [...]/gcc/testsuite/g++.dg/tree-prof/pr63581.C: In function 'int uptodate(page*)':

 [...]/gcc/testsuite/g++.dg/tree-prof/pr63581.C:28:19: warning: profile for 
function 'int uptodate(page*)' not found in profile data [-Wmissing-profile]

..., and:

 [-UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c-]
 UNSUPPORTED: gcc.dg/tree-prof/20041218-1.c -fauto-profile
 {+PASS: gcc.dg/tree-prof/20041218-1.c compilation,  -fprofile-generate 
-D_PROFILE_GENERATE+}
 {+FAIL: gcc.dg/tree-prof/20041218-1.c compilation,  -fprofile-use 
-D_PROFILE_USE+}
 {+PASS: gcc.dg/tree-prof/20041218-1.c execution,-fprofile-generate 
-D_PROFILE_GENERATE+}
 {+UNRESOLVED: gcc.dg/tree-prof/20041218-1.c execution,-fprofile-use 
-D_PROFILE_USE+}
 
 [...]/gcc/testsuite/gcc.dg/tree-prof/20041218-1.c: In function 'bar':

 [...]/gcc/testsuite/gcc.dg/tree-prof/20041218-1.c:58:1: warning: profile 
for function 'bar' not found in profile data [-Wmissing-profile]

..., for which I came up with the following patch.

But, this doesn't now seem to be necessary anymore, or am I confused?
Maybe this got fixed differently -- or is anybody still seeing these
FAILs?  (If not, then I'm not proposing to commit this, of course.)


I tested the trunk as of commit 85df98d7fbb6ef5c5b6e97190fff4028f4b70747
(Dec 5 PR c/87028) with and without the -Wmissing-profile on by default
in the compiler. I.e., with the following change :

Common Var(warn_missing_profile) Init(1) Warning
+Common Var(warn_missing_profile) Init(0) Warning

Two observations :
1. The two issues (tree-prof/pr63581.C and tree-prof/20041218-1.c) that you run
   into are not repeatable on my end. I am not sure why you run into them.

2. I do however see other tests (a total of 23) which are have regressed from
   PASS --> UNRESOLVED. A diff is attached.

   Each one of them is due to "Error/Warning threshold exceeded:  1 0 (max. 1 
3)"

E.g.,
ERROR: gcc.dg/tree-prof/20050826-2.c: error executing dg-final-use: couldn't open 
"20050826-2.c.116t.dom2 20050826-2.c.115t.dom2": no such file or directory
  Error/Warning threshold exceeded:  1 0 (max. 1 3)

(I have not been able to see the -Wmissing-profile warning explicitly in my logs
 yet. I was thinking a RUNTESTFLAGS="-v -v -v" should have helped, but it 
didnt.)

Any suggestions to resolve these tests are appreciated. I am not sure yet of the
most maintainable way to "allow the warning in all tests that it appears". I
need to look around a bit (Is Thomas' patch doing just what needs to be done 
here ?).

Thanks
Indu

UNRESOLVED: c-c++-common/unroll-1.c  -Wc++-compat : error executing dg-final: 
couldn't open "unroll-1.c.250r.loop2_unroll unroll-1.c.251r.loop2_unroll": no 
such file or directory
UNRESOLVED: gcc.dg/ipa/ipa-icf-38.c: error executing dg-final: couldn't open 
"ipa-icf-38.exe.wpa.069i.icf ipa-icf-38.exe.wpa.070i.icf": no such file or 
directory
UNRESOLVED: gcc.dg/tree-prof/20050826-2.c: Error executing dg-final-use: 
couldn't open "20050826-2.c.116t.dom2 20050826-2.c.115t.dom2": no such file or 
directory
UNRESOLVED: gcc.dg/tree-prof/cmpsf-1.c: Error exe

Re: [PATCH] PR86957

[Indu Bhagat]

On 12/06/2018 05:54 PM, Indu Bhagat wrote:
2. I do however see other tests (a total of 23) which are have 
regressed from

   PASS --> UNRESOLVED. A diff is attached.

   Each one of them is due to "Error/Warning threshold exceeded: 1 0 
(max. 1 3)"

False alarm.

Looks like there is some flakiness I ran into.

New testsuite runs (make check-gcc) on a fresh builds of the GCC (with 
and without missing-profile) enabled look OK.
One additional failure as expected (Wmissing-profile.c). No additional 
unresolved tests.


Indu

[PATCH,RFC,V5 1/6] Add new function lang_GNU_GIMPLE

[Indu Bhagat]

gcc/ChangeLog:

* langhooks.c (lang_GNU_GIMPLE): New Function.
* langhooks.h: New Prototype.

---
 gcc/ChangeLog   | 5 +
 gcc/langhooks.c | 9 +
 gcc/langhooks.h | 1 +
 3 files changed, 15 insertions(+)

diff --git a/gcc/langhooks.c b/gcc/langhooks.c
index 89fb5bc..66daa9a 100644
--- a/gcc/langhooks.c
+++ b/gcc/langhooks.c
@@ -821,3 +821,12 @@ lang_GNU_OBJC (void)
 {
   return strncmp (lang_hooks.name, "GNU Objective-C", 15) == 0;
 }
+
+/* Returns true if the current lang_hooks represents the GNU GIMPLE
+   frontend.  */
+
+bool
+lang_GNU_GIMPLE (void)
+{
+  return strncmp (lang_hooks.name, "GNU GIMPLE", 10) == 0;
+}
diff --git a/gcc/langhooks.h b/gcc/langhooks.h
index a45579b..0ac794e 100644
--- a/gcc/langhooks.h
+++ b/gcc/langhooks.h
@@ -570,5 +570,6 @@ extern bool lang_GNU_C (void);
 extern bool lang_GNU_CXX (void);
 extern bool lang_GNU_Fortran (void);
 extern bool lang_GNU_OBJC (void);
+extern bool lang_GNU_GIMPLE (void);
 
 #endif /* GCC_LANG_HOOKS_H */
-- 
1.8.3.1

[PATCH,RFC,V5 3/6] Setup for CTF generation and emission

[Indu Bhagat]

Initialize CTF container when -gtLEVEL is specified.  Generate CTF debug info
for global decls.  Import the CTF header from binutils.

gcc/ChangeLog:
 
* Makefile.in: Add ctfout.* files to GTFILES.
* cgraphunit.c (symbol_table::finalize_compilation_unit): Generate CTF
debug info for decl. Invoke CTF debug info emission.
* ctfout.c: New file.
* ctfout.h: Likewise.
* gengtype.c (open_base_files): Add ctfout.h to ifiles.
* passes.c (rest_of_decl_compilation): Generate CTF debug info for
decl.
* toplev.c (process_options): Inform the user and ignore -gtLEVEL if
frontend is not C.
(toplev::finalize): Finalize CTF containers.

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-1.c: New test.
* gcc.dg/debug/ctf/ctf-preamble-1.c: Likewise.
* gcc.dg/debug/ctf/ctf.exp: Add CTF testsuite.
* gcc.dg/debug/dwarf2-ctf-1.c: New test.

include/ChangeLog:

* ctf.h: Import from binutils.

---
 gcc/ChangeLog   |  14 +
 gcc/Makefile.in |   3 +
 gcc/cgraphunit.c|  12 +-
 gcc/ctfout.c| 175 +
 gcc/ctfout.h|  53 +++
 gcc/gengtype.c  |   4 +-
 gcc/passes.c|   7 +-
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 ++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  18 +
 include/ChangeLog   |   4 +
 include/ctf.h   | 483 
 15 files changed, 839 insertions(+), 6 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 99d88a4..62d9256 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1256,6 +1256,7 @@ OBJS = \
cfgloopanal.o \
cfgloopmanip.o \
cfgrtl.o \
+   ctfout.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
@@ -2534,6 +2535,8 @@ GTFILES = $(CPPLIB_H) $(srcdir)/input.h 
$(srcdir)/coretypes.h \
   $(srcdir)/dwarf2asm.c \
   $(srcdir)/dwarf2cfi.c \
   $(srcdir)/dwarf2out.c \
+  $(srcdir)/ctfout.h \
+  $(srcdir)/ctfout.c \
   $(srcdir)/tree-vect-generic.c \
   $(srcdir)/dojump.c $(srcdir)/emit-rtl.h \
   $(srcdir)/emit-rtl.c $(srcdir)/except.h $(srcdir)/explow.c $(srcdir)/expr.c \
diff --git a/gcc/cgraphunit.c b/gcc/cgraphunit.c
index cb08efe..e8130a1 100644
--- a/gcc/cgraphunit.c
+++ b/gcc/cgraphunit.c
@@ -205,6 +205,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "lto-section-names.h"
 #include "stringpool.h"
 #include "attribs.h"
+#include "ctfout.h"
 
 /* Queue of cgraph nodes scheduled to be added into cgraph.  This is a
secondary queue used during optimization to accommodate passes that
@@ -2851,17 +2852,22 @@ symbol_table::finalize_compilation_unit (void)
 
   if (!seen_error ())
 {
-  /* Emit early debug for reachable functions, and by consequence,
-locally scoped symbols.  */
+  /* Emit early debug and CTF debug info for reachable functions, and by
+consequence, locally scoped symbols.  */
   struct cgraph_node *cnode;
   FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (cnode)
-   (*debug_hooks->early_global_decl) (cnode->decl);
+   {
+ (*debug_hooks->early_global_decl) (cnode->decl);
+ ctf_early_global_decl (cnode->decl);
+   }
 
   /* Clean up anything that needs cleaning up after initial debug
 generation.  */
   debuginfo_early_start ();
   (*debug_hooks->early_finish) (main_input_filename);
+  ctf_early_finish (main_input_filename);
   debuginfo_early_stop ();
+
 }
 
   /* Finally drive the pass manager.  */
diff --git a/gcc/ctfout.c b/gcc/ctfout.c
new file mode 100644
index 000..471cf80
--- /dev/null
+++ b/gcc/ctfout.c
@@ -0,0 +1,175 @@
+/* Output CTF format from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PUR

[PATCH,RFC,V5 2/6] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

-gtLEVEL is used to request CTF debug information and also to specify how much
CTF debug information.

gcc/ChangeLog:
 
* common.opt: Add CTF debug info options.
* doc/invoke.texi: Document the CTF debug info options.
* flag-types.h (enum ctf_debug_info_levels): New enum.
* opts.c (common_handle_option): New Function.
(set_ctf_debug_level): Handle the new CTF debug info options.

---
 gcc/ChangeLog   |  8 
 gcc/common.opt  |  9 +
 gcc/doc/invoke.texi | 16 
 gcc/flag-types.h| 13 +
 gcc/opts.c  | 26 ++
 5 files changed, 72 insertions(+)

diff --git a/gcc/common.opt b/gcc/common.opt
index f2214ed..283a959 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -125,6 +125,11 @@ enum debug_info_levels debug_info_level = DINFO_LEVEL_NONE
 Variable
 bool use_gnu_debug_info_extensions
 
+; Level of CTF debugging information we are producing.  See flag-types.h
+; for the definitions of the different possible levels.
+Variable
+enum ctf_debug_info_levels ctf_debug_info_level = CTFINFO_LEVEL_NONE
+
 ; Original value of maximum field alignment in bytes, specified via
 ; -fpack-struct=.
 Variable
@@ -3011,6 +3016,10 @@ gcolumn-info
 Common Driver Var(debug_column_info,1) Init(1)
 Record DW_AT_decl_column and DW_AT_call_column in DWARF.
 
+gt
+Common Driver RejectNegative JoinedOrMissing
+Generate CTF debug information at default level.
+
 gdwarf
 Common Driver JoinedOrMissing Negative(gdwarf-)
 Generate debug information in default version of DWARF format.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index bfcd76e..af42da9 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -378,6 +378,7 @@ Objective-C and Objective-C++ Dialects}.
 @item Debugging Options
 @xref{Debugging Options,,Options for Debugging Your Program}.
 @gccoptlist{-g  -g@var{level}  -gdwarf  -gdwarf-@var{version} @gol
+-gt  -gt@var{level} @gol
 -ggdb  -grecord-gcc-switches  -gno-record-gcc-switches @gol
 -gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf @gol
 -gas-loc-support  -gno-as-loc-support @gol
@@ -7823,6 +7824,21 @@ other DWARF-related options such as
 @option{-fno-dwarf2-cfi-asm}) retain a reference to DWARF Version 2
 in their names, but apply to all currently-supported versions of DWARF.
 
+@item -gt
+@itemx -gt@var{level}
+@opindex gt
+Request CTF debug information and use level to specify how much CTF debug
+information should be produced.  If -gt is specified without a value for level,
+the default level of CTF debug information is 2.
+
+Level 0 produces no CTF debug information at all.  Thus, -gt0 negates -gt.
+
+Level 1 produces CTF information for tracebacks only.  This includes callsite
+information, but does not include type information.
+
+Level 2 produces type information for entities (functions, data objects etc.)
+at file-scope or global-scope only.
+
 @item -gstabs
 @opindex gstabs
 Produce debugging information in stabs format (if that is supported),
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index a210328..61a1432 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -105,6 +105,19 @@ enum dwarf_gnat_encodings
   Emit GNAT encodings for the rest.  */
 };
 
+/* CTF debug info levels.
+   CTF debug info levels are untied with DWARF debug info levels because CTF
+   may co-exist with DWARF.  */
+enum ctf_debug_info_levels
+{
+  CTFINFO_LEVEL_NONE = 0, /* Write no CTF debug info.  */
+  CTFINFO_LEVEL_TERSE = 1,/* Write CTF information to support tracebacks
+only.  Not Implemented.  */
+  CTFINFO_LEVEL_NORMAL = 2/* Write CTF type information for all entities
+(functions, data objects, variables etc.)
+at file-scope or global-scope only.  */
+};
+
 /* Enumerate Objective-c instance variable visibility settings. */
 
 enum ivar_visibility
diff --git a/gcc/opts.c b/gcc/opts.c
index 07f701c..db3711c 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -196,6 +196,8 @@ static void set_debug_level (enum debug_info_type type, int 
extended,
 const char *arg, struct gcc_options *opts,
 struct gcc_options *opts_set,
 location_t loc);
+static void set_ctf_debug_level (const char *arg,
+struct gcc_options *opts, location_t loc);
 static void set_fast_math_flags (struct gcc_options *opts, int set);
 static void decode_d_option (const char *arg, struct gcc_options *opts,
 location_t loc, diagnostic_context *dc);
@@ -2759,6 +2761,10 @@ common_handle_option (struct gcc_options *opts,
   opts->x_flag_stack_usage_info = value != 0;
   break;
 
+case OPT_gt:
+  set_ctf_debug_level (arg, opts, loc);
+  break;
+
 case OPT_g:
   set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENSIONS, arg, opts, opts_set,

[PATCH,RFC,V5 0/6] Support for CTF in GCC

[Indu Bhagat]

Hello,

This patch series adds support for CTF generation in GCC.

[Changes from V4]

1. Patch 4 brings in a number of bug fixes and enhancements. Few of them are:
   - Use DECL_UID () instead of htab_hash_pointer () for _DECL
   - Handle boolean and zero-sized bitfields
   - Do not inform () when skipping types.  Arguably this was intended for the
 GCC developer and not for the end user.
   - Other bugfixes around de-dupilication at the time of CTF Array or CTF
 function generation.

2. Patch 5 has added testcases.

3. Patch 6 is a new patch added for handling CTF section when LTO is enabled.

Work on CTF spec document is also in progress; will share something very soon.
In the interim, https://sourceware.org/ml/binutils/2019-04/msg00277.html may be
helpful.

For more context, please also see previous posting
https://gcc.gnu.org/ml/gcc-patches/2019-07/msg01209.html

NickA recently posted a changeset to the binutils mailing list that adds initial
support for linking CTF sections. Please see
https://sourceware.org/ml/binutils/2019-09/msg00045.html

( This current GCC  patch set has the ctf.h in sync with the above-mentioned
   binutils patch set. )

I will be connecting online to talk about the CTF support in GNU toolchain at
Cauldron 2019.  Posting these patches so that there is added context.

Testing :
Apart from the usual bootstrap and regression testing on x86_64/linux,
sparc64/linux, I have now compiled more codebases with -gt.

Thanks

Indu Bhagat (6):
  Add new function lang_GNU_GIMPLE
  Add CTF command line options : -gtLEVEL
  Setup for CTF generation and emission
  CTF generation for a single compilation unit
  Update CTF testsuite
  Handle CTF sections when LTO is enabled

 gcc/ChangeLog  |   97 +
 gcc/Makefile.in|5 +
 gcc/cgraphunit.c   |   12 +-
 gcc/common.opt |9 +
 gcc/ctfcreate.c|  557 ++
 gcc/ctfout.c   | 1941 
 gcc/ctfout.h   |  364 
 gcc/ctfutils.c |  198 ++
 gcc/doc/invoke.texi|   16 +
 gcc/flag-types.h   |   13 +
 gcc/gengtype.c |4 +-
 gcc/langhooks.c|9 +
 gcc/langhooks.h|1 +
 gcc/opts.c |   26 +
 gcc/passes.c   |7 +-
 gcc/testsuite/ChangeLog|   46 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c |6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c |   10 +
 .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  |   23 +
 .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   |   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   |   31 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-2.c   |   39 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   |   30 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   |   39 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-3.c   |   16 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-4.c   |   22 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-complex-1.c |   22 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   |   44 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   |   30 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c   |   41 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-4.c   |   21 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c|   21 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-2.c|   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c   |   16 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-1.c |   36 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-2.c |   16 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-3.c |   21 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-func-index-1.c  |   25 +
 .../gcc.dg/debug/ctf/ctf-function-pointers-1.c |   24 +
 .../gcc.dg/debug/ctf/ctf-function-pointers-2.c |   18 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-functions-1.c   |   34 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-int-1.c |   17 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-objt-index-1.c  |   29 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-pointers-1.c|   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c|   11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-skip-types-1.c  |   33 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-str-table-1.c   |   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-1.c  |   25 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-2.c  |   30 +
 .../gcc.dg/debug/ctf/ctf-struct-array-1.c  |   36 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-typedef-1.c |   23 +
 .../gcc.dg/debug/ctf/ctf-typedef-struct-1.c|   12 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-union-1.c   |   14 +
 gcc

[PATCH,RFC,V5 5/6] Update CTF testsuite

[Indu Bhagat]

[Changes from V4]
Added new testcases
 - gcc.dg/debug/ctf/ctf-bitfields-3.c
 - gcc.dg/debug/ctf/ctf-bitfields-4.c
 - gcc.dg/debug/ctf/ctf-cvr-quals-4.c
 - gcc.dg/debug/ctf/ctf-forward-3.c
 - gcc.dg/debug/ctf/ctf-function-pointers-2.c

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-2.c: New test.
* gcc.dg/debug/ctf/ctf-anonymous-struct-1.c: New test.
* gcc.dg/debug/ctf/ctf-anonymous-union-1.c: New test.
* gcc.dg/debug/ctf/ctf-array-1.c: New test.
* gcc.dg/debug/ctf/ctf-array-2.c: New test.
* gcc.dg/debug/ctf/ctf-bitfields-1.c: New test.
* gcc.dg/debug/ctf/ctf-bitfields-2.c: New test.
* gcc.dg/debug/ctf/ctf-bitfields-3.c: New test.
* gcc.dg/debug/ctf/ctf-bitfields-4.c: New test.
* gcc.dg/debug/ctf/ctf-complex-1.c: New test.
* gcc.dg/debug/ctf/ctf-cvr-quals-1.c: New test.
* gcc.dg/debug/ctf/ctf-cvr-quals-2.c: New test.
* gcc.dg/debug/ctf/ctf-cvr-quals-3.c: New test.
* gcc.dg/debug/ctf/ctf-cvr-quals-4.c: New test.
* gcc.dg/debug/ctf/ctf-enum-1.c: New test.
* gcc.dg/debug/ctf/ctf-enum-2.c: New test.
* gcc.dg/debug/ctf/ctf-float-1.c: New test.
* gcc.dg/debug/ctf/ctf-forward-1.c: New test.
* gcc.dg/debug/ctf/ctf-forward-2.c: New test.
* gcc.dg/debug/ctf/ctf-forward-3.c: New test.
* gcc.dg/debug/ctf/ctf-func-index-1.c: New test.
* gcc.dg/debug/ctf/ctf-function-pointers-1.c: New test.
* gcc.dg/debug/ctf/ctf-function-pointers-2.c: New test.
* gcc.dg/debug/ctf/ctf-functions-1.c: New test.
* gcc.dg/debug/ctf/ctf-int-1.c: New test.
* gcc.dg/debug/ctf/ctf-objt-index-1.c: New test.
* gcc.dg/debug/ctf/ctf-pointers-1.c: New test.
* gcc.dg/debug/ctf/ctf-skip-types-1.c: New test.
* gcc.dg/debug/ctf/ctf-str-table-1.c: New test.
* gcc.dg/debug/ctf/ctf-struct-1.c: New test.
* gcc.dg/debug/ctf/ctf-struct-2.c: New test.
* gcc.dg/debug/ctf/ctf-struct-array-1.c: New test.
* gcc.dg/debug/ctf/ctf-typedef-1.c: New test.
* gcc.dg/debug/ctf/ctf-typedef-struct-1.c: New test.
* gcc.dg/debug/ctf/ctf-union-1.c: New test.
* gcc.dg/debug/ctf/ctf-variables-1.c: New test.

---
 gcc/testsuite/ChangeLog| 39 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c | 10 +
 .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  | 23 +++
 .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   | 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   | 31 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-2.c   | 39 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   | 30 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   | 39 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-3.c   | 16 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-4.c   | 22 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-complex-1.c | 22 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   | 44 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   | 30 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c   | 41 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-4.c   | 21 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c| 21 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-2.c| 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c   | 16 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-1.c | 36 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-2.c | 16 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-3.c | 21 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-func-index-1.c  | 25 
 .../gcc.dg/debug/ctf/ctf-function-pointers-1.c | 24 
 .../gcc.dg/debug/ctf/ctf-function-pointers-2.c | 18 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-functions-1.c   | 34 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-int-1.c | 17 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-objt-index-1.c  | 29 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-pointers-1.c| 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-skip-types-1.c  | 33 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-str-table-1.c   | 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-1.c  | 25 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-2.c  | 30 +++
 .../gcc.dg/debug/ctf/ctf-struct-array-1.c  | 36 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-typedef-1.c | 23 +++
 .../gcc.dg/debug/ctf/ctf-typedef-struct-1.c| 12 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-union-1.c   | 14 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-variables-1.c   | 25 
 37 files changed, 966 insertions(+)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c
 create mode 10

[PATCH,RFC,V5 6/6] Handle CTF sections when LTO is enabled

[Indu Bhagat]

libiberty/ChangeLog:

* simple-object.c (handle_lto_debug_sections): Copy CTF section.

---
 libiberty/ChangeLog   | 5 +
 libiberty/simple-object.c | 3 +++
 2 files changed, 8 insertions(+)

diff --git a/libiberty/simple-object.c b/libiberty/simple-object.c
index b00c265..96bdcf8 100644
--- a/libiberty/simple-object.c
+++ b/libiberty/simple-object.c
@@ -298,6 +298,9 @@ handle_lto_debug_sections (const char *name, int rename)
  COMDAT sections in objects produced by GCC.  */
   else if (strcmp (name, ".comment") == 0)
 return strcpy (newname, name);
+  /* Copy over .ctf section under the same name if present.  */
+  else if (strcmp (name, ".ctf") == 0)
+return strcpy (newname, name);
   free (newname);
   return NULL;
 }
-- 
1.8.3.1

[PATCH,RFC,V5 4/6] CTF generation for a single compilation unit

[Indu Bhagat]

For each translation unit, a CTF container (ctf_container_t) is used to
keep the CTF debug info.

- ctfout.c hosts the compiler facing routines for CTF generation and emission.
- ctfcreate.c contains the CTF format specific CTF creation routines.
- ctfutils.c contains helper routines for CTF creation.

[Changes from V4]
 Bugfixes and Enhancements:
  - Use DECL_UID () instead of htab_hash_pointer () for _DECL
  - Handle boolean and zero-sized bitfields
  - Do not inform () when skipping types.  Arguably this was intended for the
GCC developer and not for the end user.
  - Other bugfixes around de-dupilication at the time of CTF Array or CTF
function generation.

gcc/ChangeLog
 
* Makefile.in: Add new object files.
* ctfcreate.c: New file.
* ctfout.c (ctf_dtu_d_union_selector): New helper function for garbage
collection of dtd_u union in ctf_dtdef_t.
(ctfc_add_cuname): New function to add compilation unit name to CTF
container.
(ctf_dtdef_hash::hash): New function to generate hashkey for a CTF type
record.
(hash_dtd_tree_decl): New function.
(ctf_dtdef_hash::equal): Likewise.
(is_ctf_base_type): Likewise.
(get_cvr_quals_for_type): Likewise.
(get_type_name_string): Likewise.
(get_decl_name_string): Likewise.
(ctf_type_exists): Likewise.
(init_ctf_string_table): Likewise.
(new_ctf_container): Allocate contents of CTF container.
(delete_ctf_container): Cleanup contents of CTF container.
(init_ctf_sections): Update code comments regarding LTO.
(gen_ctf_base_type): New function.
(gen_ctf_pointer_type): Likewise.
(gen_ctf_array_type): Likewise.
(gen_ctf_forward_type): Likewise.
(gen_ctf_enum_const_list): Likewise.
(gen_ctf_enum_type): Likewise.
(gen_ctf_function_type): Likewise.
(gen_ctf_cvrquals): Likewise.
(gen_ctf_sou_type): Likewise.
(gen_ctf_typedef): Likewise.
(gen_ctf_variable): Likewise.
(gen_ctf_function): Likewise.
(gen_ctf_type): Likewise.
(gen_ctf_bitfield_type_for_decl): Likewise.
(gen_ctf_type_for_decl): Likewise.
(ctf_preprocess_var): Likewise.
(ctf_dvd_preprocess_cb): Likewise.
(ctf_dtd_preprocess_cb): Likewise.
(ctf_preprocess): Likewise.
(ctf_asm_preamble): Likewise.
(ctf_asm_stype): Likewise.
(ctf_asm_type): Likewise.
(ctf_asm_slice): Likewise.
(ctf_asm_array): Likewise.
(ctf_asm_varent): Likewise.
(ctf_asm_sou_lmember): Likewise.
(ctf_asm_sou_member): Likewise.
(ctf_asm_enum_const): Likewise.
(output_ctf_header): Output the CTF section if the CTF container is not
empty.
(output_ctf_obj_info): New function.
(output_ctf_func_info): Likewise.
(output_ctf_objtidx): Likewise.
(output_ctf_funcidx): Likewise.
(output_ctf_vars): Likewise.
(output_ctf_strs): Likewise.
(output_asm_ctf_sou_fields): Likewise.
(output_asm_ctf_enum_list): Likewise.
(output_asm_ctf_vlen_bytes): Likewise.
(output_asm_ctf_type): Likewise.
(output_ctf_types): Likewise.
(ctf_decl): Likewise.
(ctf_early_finish): Trigger CTF emission.
(ctf_early_global_decl): Invoke CTF generation function.
(ctfout_c_finalize): Add cleanup of CTF container.
* ctfout.h (typedef struct GTY): New data structures.
(struct ctf_dtdef_hash): CTF type structure hasher.
* ctfutils.c: New file.

include/ChangeLog:
 
* ctf.h: Sync with binutils.  Keep ctf_slice_t aligned.  Add CTF obj
index and func index section.

---
 gcc/ChangeLog |   70 ++
 gcc/Makefile.in   |2 +
 gcc/ctfcreate.c   |  557 
 gcc/ctfout.c  | 1828 -
 gcc/ctfout.h  |  317 +-
 gcc/ctfutils.c|  198 ++
 include/ChangeLog |5 +
 include/ctf.h |   59 +-
 8 files changed, 2986 insertions(+), 50 deletions(-)
 create mode 100644 gcc/ctfcreate.c
 create mode 100644 gcc/ctfutils.c

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 62d9256..38ff0be 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1257,6 +1257,8 @@ OBJS = \
cfgloopmanip.o \
cfgrtl.o \
ctfout.o \
+   ctfutils.o \
+   ctfcreate.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
diff --git a/gcc/ctfcreate.c b/gcc/ctfcreate.c
new file mode 100644
index 000..52c9821
--- /dev/null
+++ b/gcc/ctfcreate.c
@@ -0,0 +1,557 @@
+/* Functions to create and update CTF from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or

[PATCH] minor FDO profile related fixes

[Indu Bhagat]

I have been looking at -fdump-ipa-profile dump with an intention to sanitize
bits of information so that one may use it to judge the "quality of a profile"
in FDO.

The overall question I want to address is - are there ways to know which
functions were not run in the training run, i.e. have ZERO profile ?
(This patch corrects some dumped info; in a subsequent patch I would like to add
some more explicit information/diagnostics.)

Towards that end, I noticed that there are a couple of misleading bits of
information (so I think) in the symbol table dump listing all functions in the
compilation unit :
   --- "globally 0" appears even when profile data has not been fed by feedback
profile (not the intent as the documentation of profile_guessed_global0
 in profile-count.h suggests).
   --- "unlikely_executed" should appear only when there is profile feedback or
   a function attribute is specified (as per documentation of
   node_frequency in coretypes.h). "unlikely_executed" in case of STALE or
   NO profile is misleading in my opinion.

Summary of changes :

1. This patch makes some adjustments around how x_profile_status of a function
is set - x_profile_status should be set to PROFILE_READ only when there is a
profile for a function read from the .gcda file. So, instead of relying on
profile_info (set whenever the gcda feedback file is present, even if the
function does not have a profile available in the file), use exec_counts
(non null when function has a profile (the latter may or may not be zero)). In
essence, x_profile_status and profile_count::m_quality
are set consistent to the stated intent (in code comments.)

2. A minor change in coverage.c is for more precise location of the message

Following -fdump-ipa-profile dump excerpts show the effect :


 -O1, -O2, -O3


0. APPLICABLE PROFILE
Trunk : Function flags: count:224114269 body hot
After Patch : Function flags: count:224114269 (precise) body hot

1. STALE PROFILE
(i.e., those cases covered by Wcoverage-mismatch; when control flow changes
 between profile-generate and profile-use)
Trunk : Function flags: count:224114269 body hot
After Patch : Function flags: count:224114269 (precise) body hot

2. NO PROFILE
(i.e., those cases covered by Wmissing-profile; when function has no profile
 available in the .gcda file)
Trunk (missing .gcda file) : Function flags: count:1073741824 (estimated 
locally) body
Trunk (missing function) : Function flags: count: 1073741824 (estimated 
locally, globally 0) body unlikely_executed
After Patch (missing .gcda file) : Function flags: count:1073741824 (estimated 
locally) body
After Patch (missing function) : Function flags: count:1073741824 (estimated 
locally) body

3. ZERO PROFILE (functions not run in training run)
Trunk : Function flags: count: 1073741824 (estimated locally, globally 0) body 
unlikely_executed
After Patch (remains the same) : count: 1073741824 (estimated locally, globally 
0) body unlikely_executed

--
O0
--
In O0, flag_guess_branch_prob is not set. This makes the profile_quality set to
(precise) for most of the above cases.

0. APPLICABLE PROFILE
Trunk : Function flags: count:224114269 body hot
After Patch : Function flags: count:224114269 (precise) body hot

1. STALE PROFILE
(i.e., those cases covered by Wcoverage-mismatch; when control flow changes
 between profile-generate and profile-use)
Trunk : Function flags: count:224114269 body hot
After Patch : Function flags: count:224114269 (precise) body hot

2. NO PROFILE
(i.e., those cases covered by Wmissing-profile; when function has no profile
 available in the .gcda file)
Trunk (missing file) : Function flags: body
Trunk (missing function) : Function flags: count:0 body unlikely_executed
After Patch (missing file) :  Function flags: body
*** After Patch (missing function) : Function flags: count:0 (precise) body
(*** This remains misleading, and I do not have a solution for this; as use of 
heuristics
 to guess branch probability is not allowed in O0)

3. ZERO PROFILE (functions not run in training run)
Trunk : Function flags: count:0 body unlikely_executed
After Patch : Function flags: count:0 (precise) body

--

make check-gcc on x86_64 shows no new failures.

(A related PR was https://gcc.gnu.org/bugzilla/show_bug.cgi?id=86957 where we 
added diagnostics for the NO PROFILE case.)

diff --git a/gcc/coverage.c b/gcc/coverage.c
index 599a3bb..7595e6c 100644
--- a/gcc/coverage.c
+++ b/gcc/coverage.c
@@ -358,7 +358,7 @@ get_coverage_counts (unsigned counter, unsigned 
cfg_checksum,
   if (warning_printed && dump_enabled_p ())
{
  dump_user_location_t loc
-   = dump_user_location_t::from_location_t (input_location);
+   = dump_user_location_

Re: [PATCH] minor FDO profile related fixes

[Indu Bhagat]

Changelog entry attached. Sorry about that.

Comments inline.

Thanks

On 11/09/2018 04:23 PM, Jeff Law wrote:

On 11/7/18 5:49 PM, Indu Bhagat wrote:

diff --git a/gcc/coverage.c b/gcc/coverage.c
index 599a3bb..7595e6c 100644
--- a/gcc/coverage.c
+++ b/gcc/coverage.c
@@ -358,7 +358,7 @@ get_coverage_counts (unsigned counter, unsigned 
cfg_checksum,
if (warning_printed && dump_enabled_p ())
{
  dump_user_location_t loc
-   = dump_user_location_t::from_location_t (input_location);
+   = dump_user_location_t::from_function_decl (current_function_decl);
dump_printf_loc (MSG_MISSED_OPTIMIZATION, loc,
 "use -Wno-error=coverage-mismatch to tolerate "
 "the mismatch but performance may drop if the "

Patches should include a ChangeLog entry.  Because your patch didn't
include one, it's unclear if this hunk is something you really intended
to submit or not.  What's the point behind going from using
input_location to the function decl?


This is intended to be a part of the patch. The patch intends to make opt-info 
and dumps related to ipa-profile more precise.
Using from_function_decl gives the precise location of the function. 
from_location_t gives the same incorrect location for each function
with the coverage-mismatch issue in a compilation unit (location appears to be 
the end of file)
 


diff --git a/gcc/profile.c b/gcc/profile.c
index 2130319..57e3f3c 100644
--- a/gcc/profile.c
+++ b/gcc/profile.c
@@ -698,6 +698,11 @@ compute_branch_probabilities (unsigned cfg_checksum, 
unsigned lineno_checksum)
}
  }
  
+  if (exec_counts)

+{
+  profile_status_for_fn (cfun) = PROFILE_READ;
+}
+

Extraneous curly braces.  In general if you have a single statement,
don't bother with curly braces.


I will correct this.


@@ -705,7 +710,7 @@ compute_branch_probabilities (unsigned cfg_checksum, 
unsigned lineno_checksum)
bb->count = profile_count::from_gcov_type (bb_gcov_count (bb));
/* If function was not trained, preserve local estimates including 
statically
   determined zero counts.  */
-  else
+  else if (profile_status_for_fn (cfun) == PROFILE_READ)
  FOR_ALL_BB_FN (bb, cfun)
if (!(bb->count == profile_count::zero ()))
  bb->count = bb->count.global0 ();

So I'm guessing the point here is to only set the count to global0 when
we read in profile data for the function and the profile data didn't
have any hits for the function.  Right?


Yes, correct.
(Longer answer : Without this more selective conditional, count is set 
to global0 even for functions which were added

 between profile-generate and profile-use.)


@@ -1317,12 +1327,12 @@ branch_prob (void)
values.release ();
free_edge_list (el);
coverage_end_function (lineno_checksum, cfg_checksum);
-  if (flag_branch_probabilities && profile_info)
+  if (flag_branch_probabilities
+  && (profile_status_for_fn (cfun) == PROFILE_READ))

So what's the point of this change?  Under what circumstances do the two
conditionals do something different.  I don't know this code well, but
ISTM that if profile_info is nonzero, then the profile status is going
to be PROFILE_READ.  Is that not the case?


On Trunk, there is a direct relationship between the three :

1. (.gcda file) is present  --implies--> 2. profile_info set 
--implies--> 3. profile_status set to PROFILE_READ


But for functions added between profile-generate and profile-use, 3. 
should not be done based on 2.
For such functions, profile_info may be set (because there is a .gcda 
file), but profile_status should not be set

to PROFILE_READ, because such functions have no feedback data.


Jeff


gcc/ChangeLog:

2018-11-07  "Indu Bhagat"  <"indu.bha...@oracle.com">

* coverage.c (get_coverage_counts): Use from_function_decl for precise
function location.
* profile-count.c (profile_count::dump): Add handling for precise
profile quality.
* profile.c (compute_branch_probabilities): Rely on exec_counts instead
of profile_info to set x_profile_status of function.
(branch_prob): Do not set x_profile_status of function based on
profile_info. Done above based on exec_counts.

Re: [PATCH] minor FDO profile related fixes

[Indu Bhagat]

On 11/12/2018 01:48 AM, Martin Liška wrote:

make check-gcc on x86_64 shows no new failures.

(A related PR washttps://gcc.gnu.org/bugzilla/show_bug.cgi?id=86957  where we 
added diagnostics for the NO PROFILE case.)

Hi.

Thanks for the patch. I'm not a maintainer, but the idea of the patch looks 
correct to me.
One question about adding "(precise)", have you verified that the patch can 
survive regression
tests?

Thanks,
Martin



make check-gcc (configured with --enable-languages=c,c++,fortran,go,lto) 
shows no new failures.

make -k check also looks ok.

[PATCH,RFC,V3 2/5] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

-gtLEVEL is used to request CTF debug information and also to specify how much
CTF debug information.

gcc/ChangeLog :
 
* common.opt: Add CTF debug info options.
* doc/invoke.texi: Document the CTF debug info options.
* flag-types.h (enum ctf_debug_info_levels): New enum.
* opts.c (common_handle_option): New Function.
(set_ctf_debug_level): Handle the new CTF debug info options.

---
 gcc/ChangeLog   |  8 
 gcc/common.opt  |  9 +
 gcc/doc/invoke.texi | 16 
 gcc/flag-types.h| 13 +
 gcc/opts.c  | 26 ++
 5 files changed, 72 insertions(+)

diff --git a/gcc/common.opt b/gcc/common.opt
index a1544d0..52cd7b2 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -125,6 +125,11 @@ enum debug_info_levels debug_info_level = DINFO_LEVEL_NONE
 Variable
 bool use_gnu_debug_info_extensions
 
+; Level of CTF debugging information we are producing.  See flag-types.h
+; for the definitions of the different possible levels.
+Variable
+enum ctf_debug_info_levels ctf_debug_info_level = CTFINFO_LEVEL_NONE
+
 ; Original value of maximum field alignment in bytes, specified via
 ; -fpack-struct=.
 Variable
@@ -2999,6 +3004,10 @@ gcolumn-info
 Common Driver Var(debug_column_info,1) Init(1)
 Record DW_AT_decl_column and DW_AT_call_column in DWARF.
 
+gt
+Common Driver RejectNegative JoinedOrMissing
+Generate CTF debug information at default level.
+
 gdwarf
 Common Driver JoinedOrMissing Negative(gdwarf-)
 Generate debug information in default version of DWARF format.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 03d2895..74d20db 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -374,6 +374,7 @@ Objective-C and Objective-C++ Dialects}.
 @item Debugging Options
 @xref{Debugging Options,,Options for Debugging Your Program}.
 @gccoptlist{-g  -g@var{level}  -gdwarf  -gdwarf-@var{version} @gol
+-gt  -gt@var{level} @gol
 -ggdb  -grecord-gcc-switches  -gno-record-gcc-switches @gol
 -gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf @gol
 -gas-loc-support  -gno-as-loc-support @gol
@@ -7798,6 +7799,21 @@ other DWARF-related options such as
 @option{-fno-dwarf2-cfi-asm}) retain a reference to DWARF Version 2
 in their names, but apply to all currently-supported versions of DWARF.
 
+@item -gt
+@itemx -gt@var{level}
+@opindex gt
+Request CTF debug information and use level to specify how much CTF debug
+information should be produced.  If -gt is specified without a value for level,
+the default level of CTF debug information is 2.
+
+Level 0 produces no CTF debug information at all.  Thus, -gt0 negates -gt.
+
+Level 1 produces CTF information for tracebacks only.  This includes callsite
+information, but does not include type information.
+
+Level 2 produces type information for entities (functions, data objects etc.)
+at file-scope or global-scope only.
+
 @item -gstabs
 @opindex gstabs
 Produce debugging information in stabs format (if that is supported),
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index a210328..61a1432 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -105,6 +105,19 @@ enum dwarf_gnat_encodings
   Emit GNAT encodings for the rest.  */
 };
 
+/* CTF debug info levels.
+   CTF debug info levels are untied with DWARF debug info levels because CTF
+   may co-exist with DWARF.  */
+enum ctf_debug_info_levels
+{
+  CTFINFO_LEVEL_NONE = 0, /* Write no CTF debug info.  */
+  CTFINFO_LEVEL_TERSE = 1,/* Write CTF information to support tracebacks
+only.  Not Implemented.  */
+  CTFINFO_LEVEL_NORMAL = 2/* Write CTF type information for all entities
+(functions, data objects, variables etc.)
+at file-scope or global-scope only.  */
+};
+
 /* Enumerate Objective-c instance variable visibility settings. */
 
 enum ivar_visibility
diff --git a/gcc/opts.c b/gcc/opts.c
index b38bfb1..03346ec 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -195,6 +195,8 @@ static void set_debug_level (enum debug_info_type type, int 
extended,
 const char *arg, struct gcc_options *opts,
 struct gcc_options *opts_set,
 location_t loc);
+static void set_ctf_debug_level (const char *arg,
+struct gcc_options *opts, location_t loc);
 static void set_fast_math_flags (struct gcc_options *opts, int set);
 static void decode_d_option (const char *arg, struct gcc_options *opts,
 location_t loc, diagnostic_context *dc);
@@ -2684,6 +2686,10 @@ common_handle_option (struct gcc_options *opts,
   opts->x_flag_stack_usage_info = value != 0;
   break;
 
+case OPT_gt:
+  set_ctf_debug_level (arg, opts, loc);
+  break;
+
 case OPT_g:
   set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENSIONS, arg, opts, opts_set

[PATCH,RFC,V3 1/5] Add new function lang_GNU_GIMPLE

[Indu Bhagat]

gcc/ChangeLog:

* langhooks.c (lang_GNU_GIMPLE): New Function.
* langhooks.h: New Prototype.
 
---
 gcc/ChangeLog   | 5 +
 gcc/langhooks.c | 9 +
 gcc/langhooks.h | 1 +
 3 files changed, 15 insertions(+)

diff --git a/gcc/langhooks.c b/gcc/langhooks.c
index 2df97f2..f3a64c1 100644
--- a/gcc/langhooks.c
+++ b/gcc/langhooks.c
@@ -825,3 +825,12 @@ lang_GNU_OBJC (void)
 {
   return strncmp (lang_hooks.name, "GNU Objective-C", 15) == 0;
 }
+
+/* Returns true if the current lang_hooks represents the GNU GIMPLE
+   frontend.  */
+
+bool
+lang_GNU_GIMPLE (void)
+{
+  return strncmp (lang_hooks.name, "GNU GIMPLE", 10) == 0;
+}
diff --git a/gcc/langhooks.h b/gcc/langhooks.h
index a45579b..0ac794e 100644
--- a/gcc/langhooks.h
+++ b/gcc/langhooks.h
@@ -570,5 +570,6 @@ extern bool lang_GNU_C (void);
 extern bool lang_GNU_CXX (void);
 extern bool lang_GNU_Fortran (void);
 extern bool lang_GNU_OBJC (void);
+extern bool lang_GNU_GIMPLE (void);
 
 #endif /* GCC_LANG_HOOKS_H */
-- 
1.8.3.1

[PATCH,RFC,V3 3/5] Setup for CTF generation and emission

[Indu Bhagat]

Initialize CTF container when -gtLEVEL is specified.  Generate CTF debug info
for global decls.  Import the CTF header from binutils.

gcc/ChangeLog :
 
* Makefile.in: Add ctfout.* files to GTFILES.
* cgraphunit.c (symbol_table::finalize_compilation_unit): Generate CTF
debug info for decl. Invoke CTF debug info emission.
* ctfout.c: New file.
* ctfout.h: Likewise.
* gengtype.c (open_base_files): Add ctfout.h to ifiles.
* passes.c (rest_of_decl_compilation): Generate CTF debug info for
decl.
* toplev.c (process_options): Warn and ignore -gtLEVEL if frontend is
not C.
(toplev::finalize): Finalize CTF containers.

gcc/testsuite/ChangeLog :

* gcc.dg/debug/ctf/ctf-1.c: New test.
* gcc.dg/debug/ctf/ctf-preamble-1.c: Likewise.
* gcc.dg/debug/ctf/ctf.exp: Add CTF testsuite.
* gcc.dg/debug/dwarf2-ctf-1.c: New test.

include/ChangeLog :

* ctf.h: Import from binutils.

---
 gcc/ChangeLog   |  14 +
 gcc/Makefile.in |   3 +
 gcc/cgraphunit.c|  12 +-
 gcc/ctfout.c| 175 +
 gcc/ctfout.h|  53 +++
 gcc/gengtype.c  |   4 +-
 gcc/passes.c|   7 +-
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 ++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  18 +
 include/ChangeLog   |   4 +
 include/ctf.h   | 483 
 15 files changed, 839 insertions(+), 6 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index d9e0885..8ce2405 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1254,6 +1254,7 @@ OBJS = \
cfgloopanal.o \
cfgloopmanip.o \
cfgrtl.o \
+   ctfout.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
@@ -2532,6 +2533,8 @@ GTFILES = $(CPPLIB_H) $(srcdir)/input.h 
$(srcdir)/coretypes.h \
   $(srcdir)/dwarf2asm.c \
   $(srcdir)/dwarf2cfi.c \
   $(srcdir)/dwarf2out.c \
+  $(srcdir)/ctfout.h \
+  $(srcdir)/ctfout.c \
   $(srcdir)/tree-vect-generic.c \
   $(srcdir)/dojump.c $(srcdir)/emit-rtl.h \
   $(srcdir)/emit-rtl.c $(srcdir)/except.h $(srcdir)/explow.c $(srcdir)/expr.c \
diff --git a/gcc/cgraphunit.c b/gcc/cgraphunit.c
index f4d6688..9cf6252 100644
--- a/gcc/cgraphunit.c
+++ b/gcc/cgraphunit.c
@@ -205,6 +205,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "lto-section-names.h"
 #include "stringpool.h"
 #include "attribs.h"
+#include "ctfout.h"
 
 /* Queue of cgraph nodes scheduled to be added into cgraph.  This is a
secondary queue used during optimization to accommodate passes that
@@ -2844,17 +2845,22 @@ symbol_table::finalize_compilation_unit (void)
 
   if (!seen_error ())
 {
-  /* Emit early debug for reachable functions, and by consequence,
-locally scoped symbols.  */
+  /* Emit early debug and CTF debug info for reachable functions, and by
+consequence, locally scoped symbols.  */
   struct cgraph_node *cnode;
   FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (cnode)
-   (*debug_hooks->early_global_decl) (cnode->decl);
+   {
+ (*debug_hooks->early_global_decl) (cnode->decl);
+ ctf_early_global_decl (cnode->decl);
+   }
 
   /* Clean up anything that needs cleaning up after initial debug
 generation.  */
   debuginfo_early_start ();
   (*debug_hooks->early_finish) (main_input_filename);
+  ctf_early_finish (main_input_filename);
   debuginfo_early_stop ();
+
 }
 
   /* Finally drive the pass manager.  */
diff --git a/gcc/ctfout.c b/gcc/ctfout.c
new file mode 100644
index 000..1ce9829
--- /dev/null
+++ b/gcc/ctfout.c
@@ -0,0 +1,175 @@
+/* Output CTF format from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  S

[PATCH,RFC,V3 0/5] Support for CTF in GCC

[Indu Bhagat]

Hello,

This patch series adds support for CTF generation in GCC.

[Changes from V2]
 - Patch 1, 2, and 3 have minor edits if any.
 - Patch 4 is a new addition.
 - Patch 5 is a new addition.

Summary of the GCC RFC V3 patch set :
Patch 1, 2, and 3 do the preparatory work of adding the CTF command line options
and setting up the framework for CTF generation and emission.  More details on
these patches can be seen in the previous posting
https://gcc.gnu.org/ml/gcc-patches/2019-06/msg00718.html

With Patch 4 in the current set, the compiler can generate a .ctf section for a
single compilation unit if -gt (when unspecified, LEVEL defaults to 2) or -gt2
is specified.  Recall that -gt2 produces type information for entities
(functions, variables etc.) at file-scope or global-scope.

For each translation unit, a CTF container (ctf_container_t) is used to
keep the generated CTF.  Two hash_map structures are kept to hold the generated
CTF for type and variables.  CTF does need pre-processing before emission into
a section; there are code comments in ctfout.c to help understand this.

There are a couple of TBDs and FIXMEs in Patch 4 which will be resolved as I
progress further; Inputs on some of which will be very helpful :

- ctf_dtdef_hash : The compiler uses a hashing scheme to keep track of whether
  CTF has been generated for a type of decl.  For a type, the hashing scheme
  uses TYPE_UID, but for a decl it uses htab_hash_pointer (decl).  Is there a
  better way to do this ? (See hash_dtd_tree_decl in ctfout.c)

- delete_ctf_container routine in ctfout.c : I have used the GTY (()) tags in
  the CTF container structs.  Does this ensure that if I set the CTF container
  global variable (ctfc) to NULL, the garbage collection machinery will take
  care of cleaning up the the internals of the container (including hash_map).
  Haven't been able to get a definitive answer looking at the code in
  hash-map.h and the generated code in gtype-desc.c.

Testing :
- Bootstrapped and regression tested on x86_64/linux and aarch64/linux.
  Also bootstrapped on SPARC64/linux with some testing.
- Parsed .ctf sections of libdtrace-ctf files via a CTF dumping utility on
  x86_64/linux.  This simply ensures that the CTF sections are well-formed.
- Interaction with an internally available GDB looks promising.  Basic whatis
  and ptype tests work.  GDB patches to uptake CTF debug info are in the works
  and will be upstreamed soon.

In the subsequent patches, I intend to close some open ends in the current
patch and add LTO support.

Thanks,

Indu Bhagat (5):
  Add new function lang_GNU_GIMPLE
  Add CTF command line options : -gtLEVEL
  Setup for CTF generation and emission
  CTF generation for a single compilation unit
  Update CTF testsuite

 gcc/ChangeLog  |   91 +
 gcc/Makefile.in|5 +
 gcc/cgraphunit.c   |   12 +-
 gcc/common.opt |9 +
 gcc/ctfcreate.c|  526 ++
 gcc/ctfout.c   | 1739 
 gcc/ctfout.h   |  359 
 gcc/ctfutils.c |  198 +++
 gcc/doc/invoke.texi|   16 +
 gcc/flag-types.h   |   13 +
 gcc/gengtype.c |4 +-
 gcc/langhooks.c|9 +
 gcc/langhooks.h|1 +
 gcc/opts.c |   26 +
 gcc/passes.c   |7 +-
 gcc/testsuite/ChangeLog|   35 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c |6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c |   10 +
 .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  |   23 +
 .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   |   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   |   31 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   |   30 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   |   39 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   |   44 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   |   30 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c   |   41 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c|   21 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c   |   16 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-1.c |   36 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-2.c |   16 +
 .../gcc.dg/debug/ctf/ctf-function-pointers-1.c |   24 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-functions-1.c   |   34 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-int-1.c |   17 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-pointers-1.c|   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c|   11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-str-table-1.c   |   26 +

[PATCH,RFC,V3 5/5] Update CTF testsuite

[Indu Bhagat]

gcc/testsuite/ChangeLog : 
 
* gcc.dg/debug/ctf/ctf-2.c: New test.
* gcc.dg/debug/ctf/ctf-anonymous-struct-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-anonymous-union-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-array-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-bitfields-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-bitfields-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-cvr-quals-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-cvr-quals-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-cvr-quals-3.c: Likewise.
* gcc.dg/debug/ctf/ctf-enum-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-forward-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-forward-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-function-pointers-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-functions-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-pointers-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-str-table-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-struct-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-struct-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-struct-array-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-typedef-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-typedef-struct-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-union-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-variables-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-float-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-int-1.c: Likewise.

---
 gcc/testsuite/ChangeLog| 28 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c | 10 +
 .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  | 23 +++
 .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   | 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   | 31 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   | 30 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   | 39 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   | 44 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   | 30 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c   | 41 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c| 21 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c   | 16 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-1.c | 36 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-2.c | 16 
 .../gcc.dg/debug/ctf/ctf-function-pointers-1.c | 24 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-functions-1.c   | 34 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-int-1.c | 17 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-pointers-1.c| 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-str-table-1.c   | 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-1.c  | 25 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-2.c  | 30 +++
 .../gcc.dg/debug/ctf/ctf-struct-array-1.c  | 36 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-typedef-1.c | 23 +++
 .../gcc.dg/debug/ctf/ctf-typedef-struct-1.c| 12 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-union-1.c   | 14 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-variables-1.c   | 25 
 26 files changed, 683 insertions(+)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-anonymous-struct-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-anonymous-union-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-function-pointers-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-functions-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-int-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-pointers-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-str-table-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-array-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-typedef-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-typedef-struct-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-union-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-vari

[PATCH,RFC,V3 4/5] CTF generation for a single compilation unit

[Indu Bhagat]

For each translation unit, a CTF container (ctf_container_t) is used to
keep the CTF debug info.

- ctfout.c hosts the compiler facing routines for CTF generation and emission.
- ctfcreate.c contains the CTF format specific CTF creation routines.
- ctfutils.c contains helper routines for CTF creation.

gcc/ChangeLog :
 
* Makefile.in: Add new object files.
* ctfcreate.c: New file.
* ctfout.c (ctf_dtu_d_union_selector): New helper function for garbage
collection of dtd_u union in ctf_dtdef_t.
(ctf_dtdef_hash::hash): New function to generate hashkey for a CTF Type
record.
(hash_dtd_tree_decl): New function.
(ctf_dtdef_hash::equal): Likewise.
(is_ctf_base_type): Likewise.
(get_cvr_quals_for_type): Likewise.
(get_type_name_string): Likewise.
(get_decl_name_string): Likewise.
(ctf_type_exists): Likewise.
(init_ctf_string_table): Likewise.
(new_ctf_container): Allocate contents of CTF container.
(delete_ctf_container): Cleanup contents of CTF container.
(init_ctf_sections): Prepare for LTO.
(gen_ctf_base_type): New function.
(gen_ctf_pointer_type): Likewise.
(gen_ctf_array_type): Likewise.
(gen_ctf_forward_type): Likewise.
(gen_ctf_enum_const_list): Likewise.
(gen_ctf_enum_type): Likewise.
(gen_ctf_function_type): Likewise.
(gen_ctf_cvrquals): Likewise.
(gen_ctf_sou_type): Likewise.
(gen_ctf_typedef): Likewise.
(gen_ctf_variable): Likewise.
(gen_ctf_function): Likewise.
(gen_ctf_type): Likewise.
(gen_ctf_bitfield_type_for_decl): Likewise.
(gen_ctf_type_for_decl): Likewise.
(ctf_preprocess_var): Likewise.
(ctf_dvd_preprocess_cb): Likewise.
(ctf_dtd_preprocess_cb): Likewise.
(ctf_preprocess): Likewise.
(ctf_asm_stype): Likewise.
(ctf_asm_type): Likewise.
(ctf_asm_slice): Likewise.
(ctf_asm_array): Likewise.
(ctf_asm_varent): Likewise.
(ctf_asm_sou_lmember): Likewise.
(ctf_asm_sou_member): Likewise.
(ctf_asm_enum_const): Likewise.
(output_ctf_header): Output the CTF section if the CTF container is not
empty.
(output_ctf_func_info): New function.
(output_ctf_vars): Likewise.
(output_ctf_strs): Likewise.
(output_asm_ctf_sou_fields): Likewise.
(output_asm_ctf_enum_list): Likewise.
(output_asm_ctf_vlen_bytes): Likewise.
(output_asm_ctf_type): Likewise.
(output_ctf_types): Likewise.
(ctf_decl): Likewise.
(ctf_early_finish): Trigger CTF emission.
(ctf_early_global_decl): Invoke CTF generation function.
(ctfout_c_finalize): Add cleanup of CTF container.
* ctfout.h (typedef struct GTY): New Data Structures.
(struct ctf_dtdef_hash): CTF Type structure hasher.
* ctfutils.c: New file.

include/ChangeLog :
 
* ctf.h: Sync with binutils.  Keep ctf_slice_t aligned.

---
 gcc/ChangeLog |   64 +++
 gcc/Makefile.in   |2 +
 gcc/ctfcreate.c   |  526 ++
 gcc/ctfout.c  | 1582 -
 gcc/ctfout.h  |  312 ++-
 gcc/ctfutils.c|  198 +++
 include/ChangeLog |4 +
 include/ctf.h |   10 +-
 8 files changed, 2683 insertions(+), 15 deletions(-)
 create mode 100644 gcc/ctfcreate.c
 create mode 100644 gcc/ctfutils.c

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 8ce2405..1851403 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1255,6 +1255,8 @@ OBJS = \
cfgloopmanip.o \
cfgrtl.o \
ctfout.o \
+   ctfutils.o \
+   ctfcreate.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
diff --git a/gcc/ctfcreate.c b/gcc/ctfcreate.c
new file mode 100644
index 000..0ee700f
--- /dev/null
+++ b/gcc/ctfcreate.c
@@ -0,0 +1,526 @@
+/* Functions to create and update CTF from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+.  */
+
+/* Create CTF types.  The code is mostly adapted from libctf.
+
+   These functions perform the task of adding CTF types to the CTF container.
+   No de-duplication is done by them; the onus is on the calling function to do
+   so.  The caller must first do a 

Re: [PATCH,RFC,V3 0/5] Support for CTF in GCC

[Indu Bhagat]

Ping.
Can someone please review these patches ? We would like to get the
support for CTF integrated soon.
Thanks
Indu

On Wed, Jun 26, 2019 at 11:38 PM Indu Bhagat  wrote:
>
> Hello,
>
> This patch series adds support for CTF generation in GCC.
>
> [Changes from V2]
>  - Patch 1, 2, and 3 have minor edits if any.
>  - Patch 4 is a new addition.
>  - Patch 5 is a new addition.
>
> Summary of the GCC RFC V3 patch set :
> Patch 1, 2, and 3 do the preparatory work of adding the CTF command line 
> options
> and setting up the framework for CTF generation and emission.  More details on
> these patches can be seen in the previous posting
> https://gcc.gnu.org/ml/gcc-patches/2019-06/msg00718.html
>
> With Patch 4 in the current set, the compiler can generate a .ctf section for 
> a
> single compilation unit if -gt (when unspecified, LEVEL defaults to 2) or -gt2
> is specified.  Recall that -gt2 produces type information for entities
> (functions, variables etc.) at file-scope or global-scope.
>
> For each translation unit, a CTF container (ctf_container_t) is used to
> keep the generated CTF.  Two hash_map structures are kept to hold the 
> generated
> CTF for type and variables.  CTF does need pre-processing before emission into
> a section; there are code comments in ctfout.c to help understand this.
>
> There are a couple of TBDs and FIXMEs in Patch 4 which will be resolved as I
> progress further; Inputs on some of which will be very helpful :
>
> - ctf_dtdef_hash : The compiler uses a hashing scheme to keep track of whether
>   CTF has been generated for a type of decl.  For a type, the hashing scheme
>   uses TYPE_UID, but for a decl it uses htab_hash_pointer (decl).  Is there a
>   better way to do this ? (See hash_dtd_tree_decl in ctfout.c)
>
> - delete_ctf_container routine in ctfout.c : I have used the GTY (()) tags in
>   the CTF container structs.  Does this ensure that if I set the CTF container
>   global variable (ctfc) to NULL, the garbage collection machinery will take
>   care of cleaning up the the internals of the container (including hash_map).
>   Haven't been able to get a definitive answer looking at the code in
>   hash-map.h and the generated code in gtype-desc.c.
>
> Testing :
> - Bootstrapped and regression tested on x86_64/linux and aarch64/linux.
>   Also bootstrapped on SPARC64/linux with some testing.
> - Parsed .ctf sections of libdtrace-ctf files via a CTF dumping utility on
>   x86_64/linux.  This simply ensures that the CTF sections are well-formed.
> - Interaction with an internally available GDB looks promising.  Basic whatis
>   and ptype tests work.  GDB patches to uptake CTF debug info are in the works
>   and will be upstreamed soon.
>
> In the subsequent patches, I intend to close some open ends in the current
> patch and add LTO support.
>
> Thanks,
>
> Indu Bhagat (5):
>   Add new function lang_GNU_GIMPLE
>   Add CTF command line options : -gtLEVEL
>   Setup for CTF generation and emission
>   CTF generation for a single compilation unit
>   Update CTF testsuite
>
>  gcc/ChangeLog  |   91 +
>  gcc/Makefile.in|5 +
>  gcc/cgraphunit.c   |   12 +-
>  gcc/common.opt |9 +
>  gcc/ctfcreate.c|  526 ++
>  gcc/ctfout.c   | 1739 
> 
>  gcc/ctfout.h   |  359 
>  gcc/ctfutils.c |  198 +++
>  gcc/doc/invoke.texi|   16 +
>  gcc/flag-types.h   |   13 +
>  gcc/gengtype.c |4 +-
>  gcc/langhooks.c|9 +
>  gcc/langhooks.h|1 +
>  gcc/opts.c |   26 +
>  gcc/passes.c   |7 +-
>  gcc/testsuite/ChangeLog|   35 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c |6 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c |   10 +
>  .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  |   23 +
>  .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   |   26 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   |   31 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   |   30 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   |   39 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   |   44 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   |   30 +
>  gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals

Re: [PATCH,RFC,V3 0/5] Support for CTF in GCC

[Indu Bhagat]

On 07/02/2019 08:18 PM, Jeff Law wrote:

On 7/2/19 11:54 AM, Indu Bhagat wrote:

Ping.
Can someone please review these patches ? We would like to get the
support for CTF integrated soon.

I'm not sure there's really even consensus that we want CTF support in
GCC.  Though I think that the changes you've made in the last several
weeks do make it somewhat more palatable.  But ultimately the first step
is to get that consensus.


Thanks for your message.  Absolutely, consensus is the first step.  We are
happy to take all the constructive feedback and answer all the concerns to make
certain that CTF support in toolchain will be a useful and worthwhile
contribution.



I'd hazard a guess that Jakub in particular isn't on board as he's been
pushing to some degree for post-processing or perhaps doing it via a
plug in.

Richi has been guiding you a bit through how to make the changes easier
to integrate, but I haven't seen him state one way or the other his
preference on whether or not CTF support is something we want.

I'm hesitant to add CTF support in GCC, but can understand how it might
be useful given the kernel's aversion to everything dwarf.  But if the
kernel is the primary consumer than I'd lean towards post-processing.


Kernel is just *one* of the consumers. There are other applications, external
and internal to Oracle, that have shown interest. Not just that, a couple of
distro and package maintainers have shown interest in enabling CTF by default.

Post-processing in kernel and other internally available large applications has
been a deterrent for adoption because of high space and compile-time costs. I
answered some of Jakub's concerns in the post here
https://gcc.gnu.org/ml/gcc-patches/2019-06/msg00131.html.

I would even argue that the usecases will only grow if CTF is properly
supported in the toolchain.

Thanks

Re: [PATCH,RFC,V3 0/5] Support for CTF in GCC

[Indu Bhagat]

On 07/03/2019 05:31 AM, Richard Biener wrote:

On Wed, Jul 3, 2019 at 5:18 AM Jeff Law  wrote:

On 7/2/19 11:54 AM, Indu Bhagat wrote:

Ping.
Can someone please review these patches ? We would like to get the
support for CTF integrated soon.

I'm not sure there's really even consensus that we want CTF support in
GCC.  Though I think that the changes you've made in the last several
weeks do make it somewhat more palatable.  But ultimately the first step
is to get that consensus.

I'd hazard a guess that Jakub in particular isn't on board as he's been
pushing to some degree for post-processing or perhaps doing it via a
plug in.

Richi has been guiding you a bit through how to make the changes easier
to integrate, but I haven't seen him state one way or the other his
preference on whether or not CTF support is something we want.

I'm mostly worried about the lack of a specification and the appearant
restriction on a subset of C (the patches have gcc_unreachable ()
in paths that can be reached by VECTOR_TYPE or COMPLEX_TYPE
not to mention FIXED_POINT_TYPE, etc...).


RE lack of specification : I cannot agree more; This does need to absolutely 
exist
if we envision CTF support in toolchain to be useful to the community.
We plan on getting to this task once the Linker changes are scoped and closer
to done (~ a couple of weeks from now). Will this work ?

RE subset of C : It is true that CTF format currently does leave out a very
small subset of C like FIXED_POINT as you noted ( CTF does have representation
for COMPLEX_TYPE, if my code paths culminate to gcc_unreachable () for that, I
should fix them ).  The end goal is to make it support all of C, and not just a
subset.

Meanwhile, I intend to make the compiler skip types when a C construct is not
supported instead of crashing because of gcc_unreachable (). (You may have also
noted stubs with "TBD WARN instead" notes in the patch series I sent.)




While CTF might be easy and fast to parse and small I fear it will
go the STABS way of being not extensible and bitrotten.


FWIW, I can understand this. We will maintain it. And I hope it will also be a
community effort thereafter with active consumers, so there is a positive
feedback loop.



Given it appears to generate only debug info for symbols and no locations
or whatnot it should be sufficient to introspect the compilation to generate
the CTF info on the side and then merge it in at link-time.  Which makes
me wonder if this shouldn't be a plugin for now until it is more complete
and can be evaluated better (comments in the patches indicate even the
on-disk format is in flux?).  Adding plugin hook invocations to the three
places the CTF info generation hooks off should be easy.


Yes, some bits of the on-disk format are being adapted to make it easier to
adopt the CTF format across the board. E.g., we recently added CU name in the
CTF header. As another example, we added CTF_K_SLICE type because there existed
no way in CTF to represent enum bitfields. For the most part though, CTF format
has stayed as is.

Hmm...a GCC plugin for CTF generation at compile-time may work out for a single
compilation unit.  But I am not sure how will LTO be supported in that case.
Basically, for LTO and -gtLEVEL to work together, I need the lto-wrapper to be
aware of the presence of .ctf sections (so I think). I will need to combine the
.ctf sections from multiple compilation units into a CTF archive, which the
linker can then de-duplicate.

Even if I assume that the technical hurdle in the above paragraph is solvable
within the purview of a plugin, I fear worse problems of adoption, maintenance
and distribution in the long run, if CTF support unfortunately ever remains to 
be
done via a plugin for reasons unforeseen.

Going the plugin route for the short term, will continue to suffer similar
problems of distribution and support.

- Is the plugin infrastructure supported on most platforms ? Also, I see that
  the plugin infrastructure supports all gcc versions from 4.5 onwards.
  Can someone confirm ? ( We minimally want the toolchain support with
  GCC 4.8.5 and GCC 8 and later, for now. )

- How will the plugin be distributed for a variety of platforms and
  architectures outside of what Oracle Linux commits to support ?

  Unless you are suggesting that the GCC plugin be distributed within GCC,
  meanwhile ? Well, that may be acceptable in the short term, depending on how
  I resolve some points raised above.




That said, the patch series isn't ready for integration since it will
crash left and right -- did you bootstrap and run the testsuite
with -gt?



Bootstrap and Testsuite : Yes, I have.  On x86_64/linux, sparc64/linux,
  aarch64/linux.
Run testsuite with -gt : Not yet. Believe me, it's on my plate. And I already
 regret not having done it sooner :)
Bootstrap with -gt : Not yet. I should try s

Re: [PATCH,RFC,V3 0/5] Support for CTF in GCC

[Indu Bhagat]

On 07/04/2019 03:43 AM, Richard Biener wrote:

On Thu, Jul 4, 2019 at 2:36 AM Indu Bhagat  wrote:

[...]
RE subset of C : It is true that CTF format currently does leave out a very
small subset of C like FIXED_POINT as you noted ( CTF does have representation
for COMPLEX_TYPE, if my code paths culminate to gcc_unreachable () for that, I
should fix them ).  The end goal is to make it support all of C, and not just a
subset.

What about other languages?  GCC supports C++, Ada, Objective-C, Go, D,
Fortran, Modula-2, BRIG (this list is not necessarily complete and may change
in the future).


The format supports C only at this time. Other languages are not on the radar
yet. However, we have no intrinsic objection to them. Although, languages
that already have fully-fledged type introspection and interpreted/
managed languages are probably out of scope, since they already have
what CTF provides.






Given it appears to generate only debug info for symbols and no locations
or whatnot it should be sufficient to introspect the compilation to generate
the CTF info on the side and then merge it in at link-time.  Which makes
me wonder if this shouldn't be a plugin for now until it is more complete
and can be evaluated better (comments in the patches indicate even the
on-disk format is in flux?).  Adding plugin hook invocations to the three
places the CTF info generation hooks off should be easy.

Yes, some bits of the on-disk format are being adapted to make it easier to
adopt the CTF format across the board. E.g., we recently added CU name in the
CTF header. As another example, we added CTF_K_SLICE type because there existed
no way in CTF to represent enum bitfields. For the most part though, CTF format
has stayed as is.

I hope the format is versioned at least.


Yes, the format is versioned. The current version is CTF_VERSION_3.  All these
format changes I talked about above are a part of CTF_VERSION_3.

libctf handles backward compatibility for users of CTF in the toolchain; all
transparently to the user. This means that, in future, when CTF version needs
to be bumped, libctf will either support older version and/or transparently
upgrade to the new version for further consumers.

It also means that the compiler does not always need to change merely because
the format has changed: (depending on the change) the linker can transparently
adjust, as will all consumers if they try to read unlinked object files.




That said, the patch series isn't ready for integration since it will
crash left and right -- did you bootstrap and run the testsuite
with -gt?



Bootstrap and Testsuite : Yes, I have.  On x86_64/linux, sparc64/linux,
aarch64/linux.
Run testsuite with -gt : Not yet. Believe me, it's on my plate. And I already
   regret not having done it sooner :)
Bootstrap with -gt : Not yet. I should try soon.

(I have compiled libdtrace-ctf with -gt and parsed the .ctf sections with the
patch set.)

About the patch being not ready for integration : Yes, you're right.
That's why I chose to retain 'RFC' for this patch series as well. I am working
on issues, testing the compiler, and closing on the open ends in the
implementation.

I will refresh the patch series when I have made a meaningful stride ahead. Any
further suggestions on functional/performance testing will be helpful too.

What's the functional use of CTF?  Print nice backtraces (without showing
function argument values)?


CTF, at this time, is type information for entities at global or file scope.
This can be used by online debuggers, program tracers (dynamic tracing); More
generally, it provides type introspection for C programs, with an optional
library API to allow them to get at their own types quite more easily than
DWARF. So, the umbrella usecases are - all C programs that want to introspect
their own types quickly; and applications that want to introspect other
programs's types quickly.

(Even with the exception of its embedded string table, it is already small
 enough to  be kept around in stripped binaries so that it can be relied upon
 to be present.)

We are also extending the format so it is useful for other on-line debugging
tools, such as backtracers.

Indu

Re: [PATCH,RFC,V3 0/5] Support for CTF in GCC

[Indu Bhagat]

On 07/04/2019 03:43 AM, Richard Biener wrote:

Hmm...a GCC plugin for CTF generation at compile-time may work out for a single
compilation unit.  But I am not sure how will LTO be supported in that case.
Basically, for LTO and -gtLEVEL to work together, I need the lto-wrapper to be
aware of the presence of .ctf sections (so I think). I will need to combine the
.ctf sections from multiple compilation units into a CTF archive, which the
linker can then de-duplicate.

True.  lto-wrapper does this kind of dancing for the much more complex set of
DWARF sections already.


Even if I assume that the technical hurdle in the above paragraph is solvable
within the purview of a plugin, I fear worse problems of adoption, maintenance
and distribution in the long run, if CTF support unfortunately ever remains to 
be
done via a plugin for reasons unforeseen.

Going the plugin route for the short term, will continue to suffer similar
problems of distribution and support.

- Is the plugin infrastructure supported on most platforms ? Also, I see that
the plugin infrastructure supports all gcc versions from 4.5 onwards.
Can someone confirm ? ( We minimally want the toolchain support with
GCC 4.8.5 and GCC 8 and later, for now. )

The infrastructure is quite old but you'd need new invocation hooks so this
won't help.



OK then.  I will continue to focus on my current implementation without
exploring the plugin option at this time.  Thanks for confirming.

Indu

[PATCH,RFC,V4 3/5] Setup for CTF generation and emission

[Indu Bhagat]

Initialize CTF container when -gtLEVEL is specified.  Generate CTF debug info
for global decls.  Import the CTF header from binutils.

[Changes from V3]
Inform the user instead of warning if -gtLEVEL is used and the frontend is not
C.

gcc/ChangeLog:
 
* Makefile.in: Add ctfout.* files to GTFILES.
* cgraphunit.c (symbol_table::finalize_compilation_unit): Generate CTF
debug info for decl. Invoke CTF debug info emission.
* ctfout.c: New file.
* ctfout.h: Likewise.
* gengtype.c (open_base_files): Add ctfout.h to ifiles.
* passes.c (rest_of_decl_compilation): Generate CTF debug info for
decl.
* toplev.c (process_options): Inform the user and ignore -gtLEVEL if
frontend is not C.
(toplev::finalize): Finalize CTF containers.

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-1.c: New test.
* gcc.dg/debug/ctf/ctf-preamble-1.c: Likewise.
* gcc.dg/debug/ctf/ctf.exp: Add CTF testsuite.
* gcc.dg/debug/dwarf2-ctf-1.c: New test.

include/ChangeLog:

* ctf.h: Import from binutils.

---
 gcc/ChangeLog   |  14 +
 gcc/Makefile.in |   3 +
 gcc/cgraphunit.c|  12 +-
 gcc/ctfout.c| 175 +
 gcc/ctfout.h|  53 +++
 gcc/gengtype.c  |   4 +-
 gcc/passes.c|   7 +-
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 ++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  18 +
 include/ChangeLog   |   4 +
 include/ctf.h   | 483 
 15 files changed, 839 insertions(+), 6 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 597dc01..5487377 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1256,6 +1256,7 @@ OBJS = \
cfgloopanal.o \
cfgloopmanip.o \
cfgrtl.o \
+   ctfout.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
@@ -2534,6 +2535,8 @@ GTFILES = $(CPPLIB_H) $(srcdir)/input.h 
$(srcdir)/coretypes.h \
   $(srcdir)/dwarf2asm.c \
   $(srcdir)/dwarf2cfi.c \
   $(srcdir)/dwarf2out.c \
+  $(srcdir)/ctfout.h \
+  $(srcdir)/ctfout.c \
   $(srcdir)/tree-vect-generic.c \
   $(srcdir)/dojump.c $(srcdir)/emit-rtl.h \
   $(srcdir)/emit-rtl.c $(srcdir)/except.h $(srcdir)/explow.c $(srcdir)/expr.c \
diff --git a/gcc/cgraphunit.c b/gcc/cgraphunit.c
index 5999b9e..05d54f5 100644
--- a/gcc/cgraphunit.c
+++ b/gcc/cgraphunit.c
@@ -205,6 +205,7 @@ along with GCC; see the file COPYING3.  If not see
 #include "lto-section-names.h"
 #include "stringpool.h"
 #include "attribs.h"
+#include "ctfout.h"
 
 /* Queue of cgraph nodes scheduled to be added into cgraph.  This is a
secondary queue used during optimization to accommodate passes that
@@ -2844,17 +2845,22 @@ symbol_table::finalize_compilation_unit (void)
 
   if (!seen_error ())
 {
-  /* Emit early debug for reachable functions, and by consequence,
-locally scoped symbols.  */
+  /* Emit early debug and CTF debug info for reachable functions, and by
+consequence, locally scoped symbols.  */
   struct cgraph_node *cnode;
   FOR_EACH_FUNCTION_WITH_GIMPLE_BODY (cnode)
-   (*debug_hooks->early_global_decl) (cnode->decl);
+   {
+ (*debug_hooks->early_global_decl) (cnode->decl);
+ ctf_early_global_decl (cnode->decl);
+   }
 
   /* Clean up anything that needs cleaning up after initial debug
 generation.  */
   debuginfo_early_start ();
   (*debug_hooks->early_finish) (main_input_filename);
+  ctf_early_finish (main_input_filename);
   debuginfo_early_stop ();
+
 }
 
   /* Finally drive the pass manager.  */
diff --git a/gcc/ctfout.c b/gcc/ctfout.c
new file mode 100644
index 000..471cf80
--- /dev/null
+++ b/gcc/ctfout.c
@@ -0,0 +1,175 @@
+/* Output CTF format from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT

[PATCH,RFC,V4 0/5] Support for CTF in GCC

[Indu Bhagat]

Hello,

I have made some progress on the patch set that adds support for CTF generation
in GCC. I am sharing the current status so that those interested can review
and/or try it out.

For a brief summary of the individual patches in this set, please see previous
posting https://gcc.gnu.org/ml/gcc-patches/2019-06/msg01710.html

[Changes from V3]

1. Patch 3 has inform () instead of warning () when frontend in not C.

2. Patch 4 has evolved. Apart from some bug fixes, some key features that have
   been implemented are :

   - CTF function and object index sub-sections
 A CTF section has a close association with the symbol table in that the
 entries in the func-info and obj-info need to be in the same order as
 entries in the symbol table.
 Since the compiler does not know the order of the entries in the symbol
 table, these new sub-sections allow a clean handshake between the compiler
 and the rest of the toolchain. The compiler only makes sure the entries in 
 the func-info and the func-info-index are in sync; and similarly that the
 entries in the obj-info and obj-info-index are in sync.

 A CTF consumer will use these indexes if present.

 Here is the binutils posting for adding these sub-sections to the CTF
 section https://sourceware.org/ml/binutils/2019-07/msg00157.html
   
   - Skip types when CTF lacks representation for them.
 CTF does not have representation for VECTOR_TYPE, Complex Integer type,
 Non IEEE float type, FIXED_POINT type and finally those enum constants
 that are not representable with a signed 32-bit integer.

 All these cases are skipped for CTF generation.  A type with type ID = 0
 is used for such declarations.  All these cases are tagged with a
 TBD_CTF_REPRESENTATION_LIMIT in the source files at this time.

 A new testcase testsuite/gcc.dg/debug/ctf/ctf-skip-types-1.c has been
 added that non-exhaustively lists specimens of types that are currently
 being skipped by the compiler for CTF generation.

 Having noted these skipped types for now, we will be working on a
 representation for them. Also, a careful perusal of the GNU C extensions
 in context of CTF representation is needed.

   - CTF CU name
 Each CTF section now holds a reference to the CU name in the CTF string
 table.

Testing :

Apart from the usual bootstrap and regression testing on x86_64/linux and
sparc64/linux, I have now compiled more codebases with -gt.  With this patch
set, I was able to use -gt for compiling gcc libraries, and also run dg.exp
suite with -gt.

( PS : Linker support for CTF is being actively worked on as well
 https://sourceware.org/ml/binutils/2019-07/msg00159.html. This current GCC
 patch set has the ctf.h in sync with the afore-mentioned binutils patch set. )

Thanks

Indu Bhagat (5):
  Add new function lang_GNU_GIMPLE
  Add CTF command line options : -gtLEVEL
  Setup for CTF generation and emission
  CTF generation for a single compilation unit
  Update CTF testsuite

 gcc/ChangeLog  |   97 +
 gcc/Makefile.in|5 +
 gcc/cgraphunit.c   |   12 +-
 gcc/common.opt |9 +
 gcc/ctfcreate.c|  531 ++
 gcc/ctfout.c   | 1924 
 gcc/ctfout.h   |  364 
 gcc/ctfutils.c |  198 ++
 gcc/doc/invoke.texi|   16 +
 gcc/flag-types.h   |   13 +
 gcc/gengtype.c |4 +-
 gcc/langhooks.c|9 +
 gcc/langhooks.h|1 +
 gcc/opts.c |   26 +
 gcc/passes.c   |7 +-
 gcc/testsuite/ChangeLog|   41 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c |6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c |   10 +
 .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  |   23 +
 .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   |   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   |   31 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-2.c   |   39 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   |   30 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   |   39 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-complex-1.c |   22 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   |   44 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   |   30 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c   |   41 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c|   21 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-2.c|   26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c   |   16 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf

[PATCH,RFC,V4 1/5] Add new function lang_GNU_GIMPLE

[Indu Bhagat]

gcc/ChangeLog:

* langhooks.c (lang_GNU_GIMPLE): New Function.
* langhooks.h: New Prototype.

---
 gcc/ChangeLog   | 5 +
 gcc/langhooks.c | 9 +
 gcc/langhooks.h | 1 +
 3 files changed, 15 insertions(+)

diff --git a/gcc/langhooks.c b/gcc/langhooks.c
index 2df97f2..f3a64c1 100644
--- a/gcc/langhooks.c
+++ b/gcc/langhooks.c
@@ -825,3 +825,12 @@ lang_GNU_OBJC (void)
 {
   return strncmp (lang_hooks.name, "GNU Objective-C", 15) == 0;
 }
+
+/* Returns true if the current lang_hooks represents the GNU GIMPLE
+   frontend.  */
+
+bool
+lang_GNU_GIMPLE (void)
+{
+  return strncmp (lang_hooks.name, "GNU GIMPLE", 10) == 0;
+}
diff --git a/gcc/langhooks.h b/gcc/langhooks.h
index a45579b..0ac794e 100644
--- a/gcc/langhooks.h
+++ b/gcc/langhooks.h
@@ -570,5 +570,6 @@ extern bool lang_GNU_C (void);
 extern bool lang_GNU_CXX (void);
 extern bool lang_GNU_Fortran (void);
 extern bool lang_GNU_OBJC (void);
+extern bool lang_GNU_GIMPLE (void);
 
 #endif /* GCC_LANG_HOOKS_H */
-- 
1.8.3.1

[PATCH,RFC,V4 2/5] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

-gtLEVEL is used to request CTF debug information and also to specify how much
CTF debug information.

gcc/ChangeLog:
 
* common.opt: Add CTF debug info options.
* doc/invoke.texi: Document the CTF debug info options.
* flag-types.h (enum ctf_debug_info_levels): New enum.
* opts.c (common_handle_option): New Function.
(set_ctf_debug_level): Handle the new CTF debug info options.

---
 gcc/ChangeLog   |  8 
 gcc/common.opt  |  9 +
 gcc/doc/invoke.texi | 16 
 gcc/flag-types.h| 13 +
 gcc/opts.c  | 26 ++
 5 files changed, 72 insertions(+)

diff --git a/gcc/common.opt b/gcc/common.opt
index b998b25..cfa7d5c 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -125,6 +125,11 @@ enum debug_info_levels debug_info_level = DINFO_LEVEL_NONE
 Variable
 bool use_gnu_debug_info_extensions
 
+; Level of CTF debugging information we are producing.  See flag-types.h
+; for the definitions of the different possible levels.
+Variable
+enum ctf_debug_info_levels ctf_debug_info_level = CTFINFO_LEVEL_NONE
+
 ; Original value of maximum field alignment in bytes, specified via
 ; -fpack-struct=.
 Variable
@@ -3007,6 +3012,10 @@ gcolumn-info
 Common Driver Var(debug_column_info,1) Init(1)
 Record DW_AT_decl_column and DW_AT_call_column in DWARF.
 
+gt
+Common Driver RejectNegative JoinedOrMissing
+Generate CTF debug information at default level.
+
 gdwarf
 Common Driver JoinedOrMissing Negative(gdwarf-)
 Generate debug information in default version of DWARF format.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 758aef3..70ab5b4 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -375,6 +375,7 @@ Objective-C and Objective-C++ Dialects}.
 @item Debugging Options
 @xref{Debugging Options,,Options for Debugging Your Program}.
 @gccoptlist{-g  -g@var{level}  -gdwarf  -gdwarf-@var{version} @gol
+-gt  -gt@var{level} @gol
 -ggdb  -grecord-gcc-switches  -gno-record-gcc-switches @gol
 -gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf @gol
 -gas-loc-support  -gno-as-loc-support @gol
@@ -7812,6 +7813,21 @@ other DWARF-related options such as
 @option{-fno-dwarf2-cfi-asm}) retain a reference to DWARF Version 2
 in their names, but apply to all currently-supported versions of DWARF.
 
+@item -gt
+@itemx -gt@var{level}
+@opindex gt
+Request CTF debug information and use level to specify how much CTF debug
+information should be produced.  If -gt is specified without a value for level,
+the default level of CTF debug information is 2.
+
+Level 0 produces no CTF debug information at all.  Thus, -gt0 negates -gt.
+
+Level 1 produces CTF information for tracebacks only.  This includes callsite
+information, but does not include type information.
+
+Level 2 produces type information for entities (functions, data objects etc.)
+at file-scope or global-scope only.
+
 @item -gstabs
 @opindex gstabs
 Produce debugging information in stabs format (if that is supported),
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index a210328..61a1432 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -105,6 +105,19 @@ enum dwarf_gnat_encodings
   Emit GNAT encodings for the rest.  */
 };
 
+/* CTF debug info levels.
+   CTF debug info levels are untied with DWARF debug info levels because CTF
+   may co-exist with DWARF.  */
+enum ctf_debug_info_levels
+{
+  CTFINFO_LEVEL_NONE = 0, /* Write no CTF debug info.  */
+  CTFINFO_LEVEL_TERSE = 1,/* Write CTF information to support tracebacks
+only.  Not Implemented.  */
+  CTFINFO_LEVEL_NORMAL = 2/* Write CTF type information for all entities
+(functions, data objects, variables etc.)
+at file-scope or global-scope only.  */
+};
+
 /* Enumerate Objective-c instance variable visibility settings. */
 
 enum ivar_visibility
diff --git a/gcc/opts.c b/gcc/opts.c
index 46a19a2..3b617c8 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -195,6 +195,8 @@ static void set_debug_level (enum debug_info_type type, int 
extended,
 const char *arg, struct gcc_options *opts,
 struct gcc_options *opts_set,
 location_t loc);
+static void set_ctf_debug_level (const char *arg,
+struct gcc_options *opts, location_t loc);
 static void set_fast_math_flags (struct gcc_options *opts, int set);
 static void decode_d_option (const char *arg, struct gcc_options *opts,
 location_t loc, diagnostic_context *dc);
@@ -2686,6 +2688,10 @@ common_handle_option (struct gcc_options *opts,
   opts->x_flag_stack_usage_info = value != 0;
   break;
 
+case OPT_gt:
+  set_ctf_debug_level (arg, opts, loc);
+  break;
+
 case OPT_g:
   set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENSIONS, arg, opts, opts_set,

[PATCH,RFC,V4 5/5] Update CTF testsuite

[Indu Bhagat]

[Changes from V3]
Added new testcases
 - gcc.dg/debug/ctf/ctf-array-2.c
 - gcc.dg/debug/ctf/ctf-complex-1.c
 - gcc.dg/debug/ctf/ctf-enum-2.c
 - gcc.dg/debug/ctf/ctf-func-index-1.c
 - gcc.dg/debug/ctf/ctf-objt-index-1.c
 - gcc.dg/debug/ctf/ctf-skip-types-1.c

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-2.c: New test.
* gcc.dg/debug/ctf/ctf-anonymous-struct-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-anonymous-union-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-array-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-array-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-bitfields-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-bitfields-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-complex-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-cvr-quals-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-cvr-quals-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-cvr-quals-3.c: Likewise.
* gcc.dg/debug/ctf/ctf-enum-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-enum-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-float-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-forward-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-forward-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-func-index-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-function-pointers-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-functions-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-int-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-objt-index-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-pointers-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-skip-types-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-str-table-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-struct-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-struct-2.c: Likewise.
* gcc.dg/debug/ctf/ctf-struct-array-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-typedef-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-typedef-struct-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-union-1.c: Likewise.
* gcc.dg/debug/ctf/ctf-variables-1.c: Likewise.


---
 gcc/testsuite/ChangeLog| 34 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c | 10 +
 .../gcc.dg/debug/ctf/ctf-anonymous-struct-1.c  | 23 +++
 .../gcc.dg/debug/ctf/ctf-anonymous-union-1.c   | 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c   | 31 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-2.c   | 39 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c   | 30 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c   | 39 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-complex-1.c | 22 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c   | 44 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-2.c   | 30 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-3.c   | 41 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-1.c| 21 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-enum-2.c| 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-float-1.c   | 16 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-1.c | 36 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-forward-2.c | 16 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-func-index-1.c  | 25 
 .../gcc.dg/debug/ctf/ctf-function-pointers-1.c | 24 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-functions-1.c   | 34 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-int-1.c | 17 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-objt-index-1.c  | 29 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-pointers-1.c| 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-skip-types-1.c  | 33 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-str-table-1.c   | 26 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-1.c  | 25 
 gcc/testsuite/gcc.dg/debug/ctf/ctf-struct-2.c  | 30 +++
 .../gcc.dg/debug/ctf/ctf-struct-array-1.c  | 36 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-typedef-1.c | 23 +++
 .../gcc.dg/debug/ctf/ctf-typedef-struct-1.c| 12 ++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-union-1.c   | 14 +++
 gcc/testsuite/gcc.dg/debug/ctf/ctf-variables-1.c   | 25 
 32 files changed, 863 insertions(+)
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-anonymous-struct-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-anonymous-union-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-array-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-bitfields-2.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-complex-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-quals-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-cvr-qual

[PATCH,RFC,V4 4/5] CTF generation for a single compilation unit

[Indu Bhagat]

For each translation unit, a CTF container (ctf_container_t) is used to
keep the CTF debug info.

- ctfout.c hosts the compiler facing routines for CTF generation and emission.
- ctfcreate.c contains the CTF format specific CTF creation routines.
- ctfutils.c contains helper routines for CTF creation.

[Changes from V3]
   - Bugfixes
   - Implementation for CTF function and object index sub-sections.
   - Skip types when CTF lacks representation for them.
   - CTF Compilation Unit name support (CU name).


gcc/ChangeLog:

* Makefile.in: Add new object files.
* ctfcreate.c: New file.
* ctfout.c (ctf_dtu_d_union_selector): New helper function for garbage
collection of dtd_u union in ctf_dtdef_t.
(ctfc_add_cuname): New function to add compilation unit name to CTF
container.
(ctf_dtdef_hash::hash): New function to generate hashkey for a CTF type
record.
(hash_dtd_tree_decl): New function.
(ctf_dtdef_hash::equal): Likewise.
(is_ctf_base_type): Likewise.
(get_cvr_quals_for_type): Likewise.
(get_type_name_string): Likewise.
(get_decl_name_string): Likewise.
(ctf_type_exists): Likewise.
(init_ctf_string_table): Likewise.
(new_ctf_container): Allocate contents of CTF container.
(delete_ctf_container): Cleanup contents of CTF container.
(init_ctf_sections): Update code comments regarding LTO.
(gen_ctf_base_type): New function.
(gen_ctf_pointer_type): Likewise.
(gen_ctf_array_type): Likewise.
(gen_ctf_forward_type): Likewise.
(gen_ctf_enum_const_list): Likewise.
(gen_ctf_enum_type): Likewise.
(gen_ctf_function_type): Likewise.
(gen_ctf_cvrquals): Likewise.
(gen_ctf_sou_type): Likewise.
(gen_ctf_typedef): Likewise.
(gen_ctf_variable): Likewise.
(gen_ctf_function): Likewise.
(gen_ctf_type): Likewise.
(gen_ctf_bitfield_type_for_decl): Likewise.
(gen_ctf_type_for_decl): Likewise.
(ctf_preprocess_var): Likewise.
(ctf_dvd_preprocess_cb): Likewise.
(ctf_dtd_preprocess_cb): Likewise.
(ctf_preprocess): Likewise.
(ctf_asm_preamble): Likewise.
(ctf_asm_stype): Likewise.
(ctf_asm_type): Likewise.
(ctf_asm_slice): Likewise.
(ctf_asm_array): Likewise.
(ctf_asm_varent): Likewise.
(ctf_asm_sou_lmember): Likewise.
(ctf_asm_sou_member): Likewise.
(ctf_asm_enum_const): Likewise.
(output_ctf_header): Output the CTF section if the CTF container is not
empty.
(output_ctf_obj_info): New function.
(output_ctf_func_info): Likewise.
(output_ctf_objtidx): Likewise.
(output_ctf_funcidx): Likewise.
(output_ctf_vars): Likewise.
(output_ctf_strs): Likewise.
(output_asm_ctf_sou_fields): Likewise.
(output_asm_ctf_enum_list): Likewise.
(output_asm_ctf_vlen_bytes): Likewise.
(output_asm_ctf_type): Likewise.
(output_ctf_types): Likewise.
(ctf_decl): Likewise.
(ctf_early_finish): Trigger CTF emission.
(ctf_early_global_decl): Invoke CTF generation function.
(ctfout_c_finalize): Add cleanup of CTF container.
* ctfout.h (typedef struct GTY): New data structures.
(struct ctf_dtdef_hash): CTF type structure hasher.
* ctfutils.c: New file.

include/ChangeLog:
 
* ctf.h: Sync with binutils.  Keep ctf_slice_t aligned.  Add CTF obj
index and func index section.

---
 gcc/ChangeLog |   70 +++
 gcc/Makefile.in   |2 +
 gcc/ctfcreate.c   |  531 
 gcc/ctfout.c  | 1811 -
 gcc/ctfout.h  |  317 +-
 gcc/ctfutils.c|  198 ++
 include/ChangeLog |5 +
 include/ctf.h |   58 +-
 8 files changed, 2942 insertions(+), 50 deletions(-)
 create mode 100644 gcc/ctfcreate.c
 create mode 100644 gcc/ctfutils.c

diff --git a/gcc/ctfcreate.c b/gcc/ctfcreate.c
new file mode 100644
index 000..f14ee69
--- /dev/null
+++ b/gcc/ctfcreate.c
@@ -0,0 +1,531 @@
+/* Functions to create and update CTF from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+.  */
+
+/* Create CTF types.  The code is mostly adapted from l

Re: [PATCH,RFC,V4 4/5] CTF generation for a single compilation unit

[Indu Bhagat]

This patch was missing a stub from Makefile.in

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 5487377..0994f3b 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1257,6 +1257,8 @@ OBJS = \
cfgloopmanip.o \
cfgrtl.o \
ctfout.o \
+   ctfutils.o \
+   ctfcreate.o \
symtab.o \
cgraph.o \
cgraphbuild.o \

Sorry for inconvenience,
Indu

On Wed, Jul 17, 2019 at 4:58 PM Indu Bhagat  wrote:
>
> For each translation unit, a CTF container (ctf_container_t) is used to
> keep the CTF debug info.
>
> - ctfout.c hosts the compiler facing routines for CTF generation and emission.
> - ctfcreate.c contains the CTF format specific CTF creation routines.
> - ctfutils.c contains helper routines for CTF creation.
>
> [Changes from V3]
>- Bugfixes
>- Implementation for CTF function and object index sub-sections.
>- Skip types when CTF lacks representation for them.
>- CTF Compilation Unit name support (CU name).
>
>
> gcc/ChangeLog:
>
> * Makefile.in: Add new object files.
> * ctfcreate.c: New file.
> * ctfout.c (ctf_dtu_d_union_selector): New helper function for garbage
> collection of dtd_u union in ctf_dtdef_t.
> (ctfc_add_cuname): New function to add compilation unit name to CTF
> container.
> (ctf_dtdef_hash::hash): New function to generate hashkey for a CTF 
> type
> record.
> (hash_dtd_tree_decl): New function.
> (ctf_dtdef_hash::equal): Likewise.
> (is_ctf_base_type): Likewise.
> (get_cvr_quals_for_type): Likewise.
> (get_type_name_string): Likewise.
> (get_decl_name_string): Likewise.
> (ctf_type_exists): Likewise.
> (init_ctf_string_table): Likewise.
> (new_ctf_container): Allocate contents of CTF container.
> (delete_ctf_container): Cleanup contents of CTF container.
> (init_ctf_sections): Update code comments regarding LTO.
> (gen_ctf_base_type): New function.
> (gen_ctf_pointer_type): Likewise.
> (gen_ctf_array_type): Likewise.
> (gen_ctf_forward_type): Likewise.
> (gen_ctf_enum_const_list): Likewise.
> (gen_ctf_enum_type): Likewise.
> (gen_ctf_function_type): Likewise.
> (gen_ctf_cvrquals): Likewise.
> (gen_ctf_sou_type): Likewise.
> (gen_ctf_typedef): Likewise.
> (gen_ctf_variable): Likewise.
> (gen_ctf_function): Likewise.
> (gen_ctf_type): Likewise.
> (gen_ctf_bitfield_type_for_decl): Likewise.
> (gen_ctf_type_for_decl): Likewise.
> (ctf_preprocess_var): Likewise.
> (ctf_dvd_preprocess_cb): Likewise.
> (ctf_dtd_preprocess_cb): Likewise.
> (ctf_preprocess): Likewise.
> (ctf_asm_preamble): Likewise.
> (ctf_asm_stype): Likewise.
> (ctf_asm_type): Likewise.
> (ctf_asm_slice): Likewise.
> (ctf_asm_array): Likewise.
> (ctf_asm_varent): Likewise.
> (ctf_asm_sou_lmember): Likewise.
> (ctf_asm_sou_member): Likewise.
> (ctf_asm_enum_const): Likewise.
> (output_ctf_header): Output the CTF section if the CTF container is 
> not
> empty.
> (output_ctf_obj_info): New function.
> (output_ctf_func_info): Likewise.
> (output_ctf_objtidx): Likewise.
> (output_ctf_funcidx): Likewise.
> (output_ctf_vars): Likewise.
> (output_ctf_strs): Likewise.
> (output_asm_ctf_sou_fields): Likewise.
> (output_asm_ctf_enum_list): Likewise.
> (output_asm_ctf_vlen_bytes): Likewise.
> (output_asm_ctf_type): Likewise.
> (output_ctf_types): Likewise.
> (ctf_decl): Likewise.
> (ctf_early_finish): Trigger CTF emission.
> (ctf_early_global_decl): Invoke CTF generation function.
> (ctfout_c_finalize): Add cleanup of CTF container.
> * ctfout.h (typedef struct GTY): New data structures.
> (struct ctf_dtdef_hash): CTF type structure hasher.
> * ctfutils.c: New file.
>
> include/ChangeLog:
>
> * ctf.h: Sync with binutils.  Keep ctf_slice_t aligned.  Add CTF obj
> index and func index section.
>
> ---
>  gcc/ChangeLog |   70 +++
>  gcc/Makefile.in   |2 +
>  gcc/ctfcreate.c   |  531 
>  gcc/ctfout.c  | 1811 
> -
>  gcc/ctfout.h  |  317 +-
>  gcc/ctfutils.c|  198 ++
>  include/ChangeLog |5 +
>  include/ctf.h |   58 +-
>  8 files changed, 2942 insertions(+), 50 deletions(-)
>  create mode 100644 gcc/ctfcreate.c
>  create mo

[PATCH,RFC 1/3] Add new function lang_GNU_GIMPLE

[Indu Bhagat]

gcc/ChangeLog:

* langhooks.c (lang_GNU_GIMPLE): New Function.
* langhooks.h (lang_GNU_GIMPLE): New Prototype.

---
 gcc/ChangeLog   | 5 +
 gcc/langhooks.c | 9 +
 gcc/langhooks.h | 1 +
 3 files changed, 15 insertions(+)

diff --git a/gcc/langhooks.c b/gcc/langhooks.c
index 2df97f2..f3a64c1 100644
--- a/gcc/langhooks.c
+++ b/gcc/langhooks.c
@@ -825,3 +825,12 @@ lang_GNU_OBJC (void)
 {
   return strncmp (lang_hooks.name, "GNU Objective-C", 15) == 0;
 }
+
+/* Returns true if the current lang_hooks represents the GNU GIMPLE
+   frontend.  */
+
+bool
+lang_GNU_GIMPLE (void)
+{
+  return strncmp (lang_hooks.name, "GNU GIMPLE", 10) == 0;
+}
diff --git a/gcc/langhooks.h b/gcc/langhooks.h
index a45579b..0ac794e 100644
--- a/gcc/langhooks.h
+++ b/gcc/langhooks.h
@@ -570,5 +570,6 @@ extern bool lang_GNU_C (void);
 extern bool lang_GNU_CXX (void);
 extern bool lang_GNU_Fortran (void);
 extern bool lang_GNU_OBJC (void);
+extern bool lang_GNU_GIMPLE (void);
 
 #endif /* GCC_LANG_HOOKS_H */
-- 
1.8.3.1

[PATCH,RFC 2/3] Add CTF command line options : -gtLEVEL

[Indu Bhagat]

-gtLEVEL is used to request CTF debug information and also to specify how much
CTF debug information.

gcc/ChangeLog:

* common.opt: Add CTF debug info options.
* doc/invoke.texi: Document the CTF debug info options.
* flag-types.h (enum ctf_debug_info_levels): New enum.
* opts.c (set_ctf_debug_level): New Function.
(common_handle_option):  Handle the new CTF debug info options.

---
 gcc/ChangeLog   |  8 
 gcc/common.opt  |  9 +
 gcc/doc/invoke.texi | 16 
 gcc/flag-types.h| 13 +
 gcc/opts.c  | 26 ++
 5 files changed, 72 insertions(+)

diff --git a/gcc/common.opt b/gcc/common.opt
index d342c4f..2b5cfb8 100644
--- a/gcc/common.opt
+++ b/gcc/common.opt
@@ -125,6 +125,11 @@ enum debug_info_levels debug_info_level = DINFO_LEVEL_NONE
 Variable
 bool use_gnu_debug_info_extensions
 
+; Level of CTF debugging information we are producing.  See flag-types.h
+; for the definitions of the different possible levels.
+Variable
+enum ctf_debug_info_levels ctf_debug_info_level = CTFINFO_LEVEL_NONE
+
 ; Original value of maximum field alignment in bytes, specified via
 ; -fpack-struct=.
 Variable
@@ -2995,6 +3000,10 @@ gcolumn-info
 Common Driver Var(debug_column_info,1) Init(1)
 Record DW_AT_decl_column and DW_AT_call_column in DWARF.
 
+gt
+Common Driver RejectNegative JoinedOrMissing
+Generate CTF debug information at default level.
+
 gdwarf
 Common Driver JoinedOrMissing Negative(gdwarf-)
 Generate debug information in default version of DWARF format.
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 5e3e887..2ed39c0 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -373,6 +373,7 @@ Objective-C and Objective-C++ Dialects}.
 @item Debugging Options
 @xref{Debugging Options,,Options for Debugging Your Program}.
 @gccoptlist{-g  -g@var{level}  -gdwarf  -gdwarf-@var{version} @gol
+-gt  -gt@var{level} @gol
 -ggdb  -grecord-gcc-switches  -gno-record-gcc-switches @gol
 -gstabs  -gstabs+  -gstrict-dwarf  -gno-strict-dwarf @gol
 -gas-loc-support  -gno-as-loc-support @gol
@@ -7766,6 +7767,21 @@ other DWARF-related options such as
 @option{-fno-dwarf2-cfi-asm}) retain a reference to DWARF Version 2
 in their names, but apply to all currently-supported versions of DWARF.
 
+@item -gt
+@itemx -gt@var{level}
+@opindex gt
+Request CTF debug information and use level to specify how much CTF debug
+information should be produced.  If -gt is specified without a value for level,
+the default level of CTF debug information is 2.
+
+Level 0 produces no CTF debug information at all.  Thus, -gt0 negates -gt.
+
+Level 1 produces CTF information for tracebacks only.  This includes callsite
+information, but does not include type information.
+
+Level 2 produces type information for entities (functions, data objects etc.)
+at file-scope or global-scope only.
+
 @item -gstabs
 @opindex gstabs
 Produce debugging information in stabs format (if that is supported),
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index a210328..61a1432 100644
--- a/gcc/flag-types.h
+++ b/gcc/flag-types.h
@@ -105,6 +105,19 @@ enum dwarf_gnat_encodings
   Emit GNAT encodings for the rest.  */
 };
 
+/* CTF debug info levels.
+   CTF debug info levels are untied with DWARF debug info levels because CTF
+   may co-exist with DWARF.  */
+enum ctf_debug_info_levels
+{
+  CTFINFO_LEVEL_NONE = 0, /* Write no CTF debug info.  */
+  CTFINFO_LEVEL_TERSE = 1,/* Write CTF information to support tracebacks
+only.  Not Implemented.  */
+  CTFINFO_LEVEL_NORMAL = 2/* Write CTF type information for all entities
+(functions, data objects, variables etc.)
+at file-scope or global-scope only.  */
+};
+
 /* Enumerate Objective-c instance variable visibility settings. */
 
 enum ivar_visibility
diff --git a/gcc/opts.c b/gcc/opts.c
index 879ab17..dfd646b 100644
--- a/gcc/opts.c
+++ b/gcc/opts.c
@@ -195,6 +195,8 @@ static void set_debug_level (enum debug_info_type type, int 
extended,
 const char *arg, struct gcc_options *opts,
 struct gcc_options *opts_set,
 location_t loc);
+static void set_ctf_debug_level (const char *arg,
+struct gcc_options *opts, location_t loc);
 static void set_fast_math_flags (struct gcc_options *opts, int set);
 static void decode_d_option (const char *arg, struct gcc_options *opts,
 location_t loc, diagnostic_context *dc);
@@ -2689,6 +2691,10 @@ common_handle_option (struct gcc_options *opts,
   opts->x_flag_stack_usage_info = value != 0;
   break;
 
+case OPT_gt:
+  set_ctf_debug_level (arg, opts, loc);
+  break;
+
 case OPT_g:
   set_debug_level (NO_DEBUG, DEFAULT_GDB_EXTENSIONS, arg, opts, opts_set,

[PATCH,RFC 0/3] Support for CTF in GCC

[Indu Bhagat]

Background :
CTF is the Compact Ansi-C Type Format. It is a format designed to express some
characteristics (specifically Type information) of the data types in a C
program. CTF format is compact and fast; It was originally designed for
use-cases like dynamic tracing, online in-application debugging among others.

A patch to the binutils mailing list to add libctf is currently under
review (https://sourceware.org/ml/binutils/2019-05/msg00154.html and
https://sourceware.org/ml/binutils/2019-05/msg00212.html).  libctf
provides means to create, update, read and manipulate CTF information.

This GCC patch set is preliminary work and the purpose is to gather comments and
feedback about CTF support in GCC.

(For technical introduction into the CTF format, the CTF header or
https://sourceware.org/ml/binutils/2019-04/msg00277.html will be useful.)

Project Details :
The project aims to add the support for CTF in the GNU toolchain. Adding CTF
support in the GNU toolchain will help the community in developing and 
converging the tools and use-cases where a light-weight debug format is needed.

De-duplication is a key aspect of the CTF format which ensures its compactness.
A parallel effort is ongoing to support de-duplication of CTF types at the
link-time.

In phase 1, we are making the compiler, linker and the debugger (GDB) capable
of handling the CTF format.

CTF format, in its present form, does not have callsite information.  We are
working on this as well. Once the CTF format extensions are agreed upon, the 
-gt1 option (see below) will begin to take form, in phase 2 of the project.

GCC RFC patch set :
Patch 1 is a simple addition of a new function lang_GNU_GIMPLE to check for
GIMPLE frontend.
Patch 2 and Patch 3 set up the framework for CTF support in GCC :
-- Patch 2 adds the new command line option for generating CTF. CTF generation
   is enabled in the compiler by specifying an explicit -gt or
   -gtLEVEL[LEVEL=1,2] :

-gtLEVEL

This is used to request CTF debug information and to specify how much CTF
debug information, LEVEL[=0,1,2] can be specified. If -gt is specified
(with no LEVEL), the default value of LEVEL is 2.

-gt0 (Level 0) produces no CTF debug information at all. Thus, -gt0
negates -gt.

-gt1 (Level 1) produces CTF information for tracebacks only. This includes
CTF callsite information, but does not include type information for other
entities.

-gt2 (Level 2) produces type information for entities (functions, variables
etc.) at file-scope or global-scope only. This level of information can be
used by dynamic tracers like DTrace.

--  Patch 3 adds the CTF debug hooks and initializes them if the required 
user-level options are specified. 
CTF debug hooks are wrappers around the DWARF debug hooks.

One of the main high-level design requirements that is relevant in the context
of the current GCC patch set is that - CTF and DWARF must be able to co-exist.
A user may want CTF debug information in isolation or with other debug formats.
A .ctf section is small and unlike other debug sections, ideally should not
need to be stripped out of the binary/executable.

High-level proposed plan (phase 1) :
In the next few patches, the functionality to generate contents of the CTF
section (.ctf) for a single compilation unit will be added.
Once CTF generation for a single compilation unit stabilizes, LTO and CTF
generation will be looked at.

Feedback and suggestions welcome.

Thanks

Indu Bhagat (3):
  Add new function lang_GNU_GIMPLE
  Add CTF command line options : -gtLEVEL
  Create CTF debug hooks

 gcc/ChangeLog   |  24 ++
 gcc/Makefile.in |   3 +
 gcc/common.opt  |   9 +
 gcc/ctfout.c| 171 +
 gcc/ctfout.h|  41 +++
 gcc/debug.h |   4 +
 gcc/doc/invoke.texi |  16 +
 gcc/flag-types.h|  13 +
 gcc/gengtype.c  |   4 +-
 gcc/langhooks.c |   9 +
 gcc/langhooks.h |   1 +
 gcc/opts.c  |  26 ++
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 +++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  24 ++
 include/ChangeLog   |   4 +
 include/ctf.h   | 471 
 20 files changed, 890 insertions(+), 2 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create

[PATCH,RFC 3/3] Create CTF debug hooks

[Indu Bhagat]

Initialize CTF debug hooks when -gtLEVEL is specified.  Import the CTF header
from binutils.

gcc/ChangeLog:

* Makefile.in: Add ctfout.* files to GTFILES.
* ctfout.c: New file.
* ctfout.h: Likewise.
* debug.h (ctf_debug_init): New function.
* gengtype.c (open_base_files): Add ctfout.h to ifiles.
* toplev.c (process_options): Warn and ignore -gtLEVEL if frontend is
not C.
(toplev::finalize): Finalize CTF containers.

gcc/testsuite/ChangeLog:

* gcc.dg/debug/ctf/ctf-1.c: New test.
* gcc.dg/debug/ctf/ctf-preamble-1.c: Likewise.
* gcc.dg/debug/ctf/ctf.exp: Add CTF testsuite.
* gcc.dg/debug/dwarf2-ctf-1.c: New test.

include/ChangeLog:

* ctf.h: Import from binutils.

---
 gcc/ChangeLog   |  11 +
 gcc/Makefile.in |   3 +
 gcc/ctfout.c| 171 +
 gcc/ctfout.h|  41 +++
 gcc/debug.h |   4 +
 gcc/gengtype.c  |   4 +-
 gcc/testsuite/ChangeLog |   7 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c  |   6 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c |  11 +
 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp  |  41 +++
 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c   |   7 +
 gcc/toplev.c|  24 ++
 include/ChangeLog   |   4 +
 include/ctf.h   | 471 
 14 files changed, 803 insertions(+), 2 deletions(-)
 create mode 100644 gcc/ctfout.c
 create mode 100644 gcc/ctfout.h
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf-preamble-1.c
 create mode 100644 gcc/testsuite/gcc.dg/debug/ctf/ctf.exp
 create mode 100644 gcc/testsuite/gcc.dg/debug/dwarf2-ctf-1.c
 create mode 100644 include/ctf.h

diff --git a/gcc/Makefile.in b/gcc/Makefile.in
index 6677f77..44d9078 100644
--- a/gcc/Makefile.in
+++ b/gcc/Makefile.in
@@ -1254,6 +1254,7 @@ OBJS = \
cfgloopanal.o \
cfgloopmanip.o \
cfgrtl.o \
+   ctfout.o \
symtab.o \
cgraph.o \
cgraphbuild.o \
@@ -2532,6 +2533,8 @@ GTFILES = $(CPPLIB_H) $(srcdir)/input.h 
$(srcdir)/coretypes.h \
   $(srcdir)/dwarf2asm.c \
   $(srcdir)/dwarf2cfi.c \
   $(srcdir)/dwarf2out.c \
+  $(srcdir)/ctfout.h \
+  $(srcdir)/ctfout.c \
   $(srcdir)/tree-vect-generic.c \
   $(srcdir)/dojump.c $(srcdir)/emit-rtl.h \
   $(srcdir)/emit-rtl.c $(srcdir)/except.h $(srcdir)/explow.c $(srcdir)/expr.c \
diff --git a/gcc/ctfout.c b/gcc/ctfout.c
new file mode 100644
index 000..9e9c48f
--- /dev/null
+++ b/gcc/ctfout.c
@@ -0,0 +1,171 @@
+/* Output ctf format from GCC.
+   Copyright (C) 2019 Free Software Foundation, Inc.
+
+This file is part of GCC.
+
+GCC is free software; you can redistribute it and/or modify it under
+the terms of the GNU General Public License as published by the Free
+Software Foundation; either version 3, or (at your option) any later
+version.
+
+GCC is distributed in the hope that it will be useful, but WITHOUT ANY
+WARRANTY; without even the implied warranty of MERCHANTABILITY or
+FITNESS FOR A PARTICULAR PURPOSE.  See the GNU General Public License
+for more details.
+
+You should have received a copy of the GNU General Public License
+along with GCC; see the file COPYING3.  If not see
+.  */
+
+#include "config.h"
+#include "system.h"
+#include "coretypes.h"
+#include "target.h"
+#include "rtl.h"
+#include "tree.h"
+#include "memmodel.h"
+#include "tm_p.h"
+#include "varasm.h"
+#include "output.h"
+#include "dwarf2asm.h"
+#include "debug.h"
+#include "ctfout.h"
+
+/* CTF debug hooks.  This is initialized by ctf_debug_init.  */
+static struct gcc_debug_hooks ctf_debug_hooks;
+
+/* The real debug hooks.  */
+static const struct gcc_debug_hooks *real_debug_hooks;
+
+/* A CTF container object - one per translation unit.  */
+static GTY(()) ctf_container_ref ctf_container;
+
+static int ctf_label_num;
+
+/* Pointers to various ctf sections.  */
+static GTY(()) section *ctf_info_section;
+
+/* Section names used to hold CTF debugging information.  */
+#ifndef CTF_INFO_SECTION_NAME
+#define CTF_INFO_SECTION_NAME  ".ctf"
+#endif
+
+/* Section flags for .ctf section.  */
+
+/* CTF debug info section.  */
+#define CTF_INFO_SECTION_FLAGS (SECTION_DEBUG)
+
+/* Maximum size (in bytes) of an artificially generated ctf label.  */
+#define MAX_CTF_LABEL_BYTES 40
+
+static char ctf_info_section_label[MAX_CTF_LABEL_BYTES];
+
+#ifndef CTF_INFO_SECTION_LABEL
+#define CTF_INFO_SECTION_LABEL "Lctf"
+#endif
+
+/* Forward declarations for functions defined in this file.  */
+static inline ctf_container_ref
+new_ctf_container (unsigned char ctp_flags);
+
+static void output_ctf_preamble (void);
+static void output

Re: [PATCH 5/5] doc: document btf_type_tag and btf_decl_tag attributes

[Indu Bhagat]

On 10/30/24 11:31 AM, David Faust wrote:

gcc/

* doc/extend.texi (Common Variable Attributes): Document new
btf_decl_tag attribute.
(Common Type Attributes): Document new btf_type_tag attribute.
---
  gcc/doc/extend.texi | 68 +
  1 file changed, 68 insertions(+)

diff --git a/gcc/doc/extend.texi b/gcc/doc/extend.texi
index 42bd567119d..fd8f2425947 100644
--- a/gcc/doc/extend.texi
+++ b/gcc/doc/extend.texi
@@ -7920,6 +7920,41 @@ align them on any target.
  The @code{aligned} attribute can also be used for functions
  (@pxref{Common Function Attributes}.)
  
+@cindex @code{btf_decl_tag} variable attribute

+@item btf_decl_tag (@var{argument})
+The @code{btf_decl_tag} attribute may be used to associate variable
+declarations, struct or union member declarations, function
+declarations, or function parameter declarations with arbitrary strings.


Should the "or function parameter..." instead be "and function 
parameter..." ?



+These strings are not interpreted by the compiler in any way, and have
+no effect on code generation.  Instead, these user-provided strings
+are recorded in DWARF (via @code{DW_AT_GNU_annotation} and
+@code{DW_TAG_GNU_annotation} extensions) and BTF information (via
+@code{BTF_KIND_DECL_TAG} records), and associated to the attributed
+declaration.  If neither DWARF nor BTF information is generated, the
+attribute has no effect.
+
+The argument is treated as an ordinary string in the source language
+with no additional special rules.
+
+The attribute may be supplied multiple times for a single declaration,
+in which case each distinct argument string will be recorded in a
+separate DIE or BTF record, each associated to the declaration.  For
+a single declaration with multiple @code{btf_decl_tag} attributes,
+the order of the @code{DW_TAG_GNU_annotation} DIEs produced is not
+guaranteed to maintain the order of attributes in the source code.
+
+For example:
+
+@smallexample
+int * foo __attribute__ ((btf_decl_tag ("__percpu")));


No space between * and foo.


+@end smallexample
+
+@noindent
+when compiled with @code{-gbtf} results in an additional
+@code{BTF_KIND_DECL_TAG} BTF record to be emitted in the BTF info,
+associating the string ``__rcu'' with the normal @code{BTF_KIND_VAR}


"__rcu" -> "__percpu"


+record for the variable ``foo''.
+
  @cindex @code{counted_by} variable attribute
  @item counted_by (@var{count})
  The @code{counted_by} attribute may be attached to the C99 flexible array
@@ -9109,6 +9144,39 @@ is given by the product of arguments 1 and 2, and that
  @code{malloc_type}, like the standard C function @code{malloc},
  returns an object whose size is given by argument 1 to the function.
  
+@cindex @code{btf_type_tag} type attribute

+@item btf_type_tag (@var{argument})
+The @code{btf_type_tag} attribute may be used to associate (to ``tag'')
+particular types with arbitrary string annotations.  These annotations
+are recorded in debugging info by supported debug formats, currently
+DWARF (via @code{DW_AT_GNU_annotation} and @code{DW_TAG_GNU_annotation}
+extensions) and BTF (via @code{BTF_KIND_TYPE_TAG} records).  These
+annotation string are not interpreted by the compiler in any way, and


"The annotation string is not" ?


+have no effect on code generation.  If neither DWARF nor BTF
+information is generated, the attribute has no effect.
+
+The argument is treated as an ordinary string in the source language
+with no additional special rules.
+
+The attribute may be supplied multiple times for a single declaration,
+in which case each distinct argument string will be recorded in a
+separate DIE or BTF record, each associated to the type.  For a single
+type with multiple @code{btf_decl_tag} attributes, the order of the


btf_decl_tag -> btf_type_tag


+@code{DW_TAG_GNU_annotation} DIEs produced is not guaranteed to
+maintain the order of attributes in the source code.
+
+For example
+


Perhaps "For example, the following code:" for better continuity with 
what follows ("associates the string...")



+@smallexample
+int * __attribute__ ((btf_type_tag ("__user"))) foo;
+@end smallexample
+
+@noindent
+associates the string ``__user'' to the pointer-to-integer type of
+the declaration.  This string will be recorded in DWARF and/or BTF
+information associated with the appropriate pointer type DIE or
+@code{BTF_KIND_PTR} record.


"and/or BTF record of kind BTF_KIND_PTR respectively" ?

But I think the sentence still reads a bit funny - may be reword it to 
say things in a similar prose as done for btf_decl_tag documentation above.



+
  @cindex @code{copy} type attribute
  @item copy
  @itemx copy (@var{expression})

Re: [PATCH 4/5] btf: generate and output DECL_TAG and TYPE_TAG records

[Indu Bhagat]

On 10/30/24 11:31 AM, David Faust wrote:

Support the btf_decl_tag and btf_type_tag attributes in BTF by creating
and emitting BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG records,
respectively, for them.

Some care is required when -gprune-btf is in effect to avoid emitting
decl or type tags for declarations or types which have been pruned and
will not be emitted in BTF.

gcc/
* btfout.cc (get_btf_kind): Handle DECL_TAG and TYPE_TAG kinds.
(btf_calc_num_vbytes): Likewise.
(btf_asm_type): Likewise.
(output_asm_btf_vlen_bytes): Likewise.
(output_btf_tags): New.
(btf_output): Call it here.
(btf_add_used_type): Replace with simple wrapper around...
(btf_add_used_type_1): ...the implementation.  Handle
BTF_KIND_DECL_TAG and BTF_KIND_TYPE_TAG.
(btf_add_vars): Update btf_add_used_type call.
(btf_assign_tag_ids): New.
(btf_mark_type_used): Update btf_add_used_type call.
(btf_collect_pruned_types): Likewise.  Handle type and decl tags.
(btf_finish): Call btf_assign_tag_ids.

gcc/testsuite/
* gcc.dg/debug/btf/btf-decl-tag-1.c: New test.
* gcc.dg/debug/btf/btf-decl-tag-2.c: New test.
* gcc.dg/debug/btf/btf-decl-tag-3.c: New test.
* gcc.dg/debug/btf/btf-decl-tag-4.c: New test.
* gcc.dg/debug/btf/btf-type-tag-1.c: New test.
* gcc.dg/debug/btf/btf-type-tag-2.c: New test.
* gcc.dg/debug/btf/btf-type-tag-3.c: New test.
* gcc.dg/debug/btf/btf-type-tag-4.c: New test.
* gcc.dg/debug/btf/btf-type-tag-c2x-1.c: New test.

include/
* btf.h (BTF_KIND_DECL_TAG, BTF_KIND_TYPE_TAG) New defines.
(struct btf_decl_tag): New.
---
  gcc/btfout.cc | 176 +++---
  .../gcc.dg/debug/btf/btf-decl-tag-1.c |  14 ++
  .../gcc.dg/debug/btf/btf-decl-tag-2.c |  22 +++
  .../gcc.dg/debug/btf/btf-decl-tag-3.c |  22 +++
  .../gcc.dg/debug/btf/btf-decl-tag-4.c |  34 
  .../gcc.dg/debug/btf/btf-type-tag-1.c |  27 +++
  .../gcc.dg/debug/btf/btf-type-tag-2.c |  17 ++
  .../gcc.dg/debug/btf/btf-type-tag-3.c |  21 +++
  .../gcc.dg/debug/btf/btf-type-tag-4.c |  25 +++
  .../gcc.dg/debug/btf/btf-type-tag-c2x-1.c |  23 +++
  include/btf.h |  14 ++
  11 files changed, 371 insertions(+), 24 deletions(-)
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decl-tag-1.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decl-tag-2.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decl-tag-3.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-decl-tag-4.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-type-tag-1.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-type-tag-2.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-type-tag-3.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-type-tag-4.c
  create mode 100644 gcc/testsuite/gcc.dg/debug/btf/btf-type-tag-c2x-1.c

diff --git a/gcc/btfout.cc b/gcc/btfout.cc
index 083ca48d627..e8190f685f9 100644
--- a/gcc/btfout.cc
+++ b/gcc/btfout.cc
@@ -141,6 +141,8 @@ get_btf_kind (uint32_t ctf_kind)
  case CTF_K_VOLATILE: return BTF_KIND_VOLATILE;
  case CTF_K_CONST:return BTF_KIND_CONST;
  case CTF_K_RESTRICT: return BTF_KIND_RESTRICT;
+case CTF_K_DECL_TAG: return BTF_KIND_DECL_TAG;
+case CTF_K_TYPE_TAG: return BTF_KIND_TYPE_TAG;
  default:;
  }
return BTF_KIND_UNKN;
@@ -217,6 +219,7 @@ btf_calc_num_vbytes (ctf_dtdef_ref dtd)
  case BTF_KIND_CONST:
  case BTF_KIND_RESTRICT:
  case BTF_KIND_FUNC:
+case BTF_KIND_TYPE_TAG:
  /* These kinds have no vlen data.  */
break;
  
@@ -256,6 +259,10 @@ btf_calc_num_vbytes (ctf_dtdef_ref dtd)

vlen_bytes += vlen * sizeof (struct btf_var_secinfo);
break;
  
+case BTF_KIND_DECL_TAG:

+  vlen_bytes += sizeof (struct btf_decl_tag);
+  break;
+
  default:
break;
  }
@@ -452,6 +459,20 @@ btf_asm_type (ctf_dtdef_ref dtd)
 and should write 0.  */
dw2_asm_output_data (4, 0, "(unused)");
return;
+case BTF_KIND_DECL_TAG:
+  {
+   if (dtd->ref_type)
+ break;
+   else if (dtd->dtd_u.dtu_tag.ref_var)
+ {
+   /* ref_type is NULL for decl tag attached to a variable.  */
+   ctf_dvdef_ref dvd = dtd->dtd_u.dtu_tag.ref_var;
+   dw2_asm_output_data (4, dvd->dvd_id,
+"btt_type: (BTF_KIND_VAR '%s')",
+dvd->dvd_name);
+   return;
+ }
+  }
  default:
break;
  }
@@ -801,6 +822,12 @@ output_asm_btf_vlen_bytes (ctf_container_ref ctfc, 
ctf_dtdef_ref dtd)
 at this point.  */
gcc_unreachable ();
  
+case BTF_KIND_DECL_TAG:

+  dw2_asm_output_data (4, dtd->dtd_u.dtu_tag.component_idx,
+  "component_

[RFC 2/9] aarch64: add new define_insn for subg

[Indu Bhagat]

subg (Subtract with Tag) is an Armv8.5-A memory tagging (MTE)
instruction.  It can be used to subtract an immediate value scaled by
the tag granule from the address in the source register.

gcc/ChangeLog:

* config/aarch64/aarch64.md (subg): New definition.
---
 gcc/config/aarch64/aarch64.md | 17 +
 1 file changed, 17 insertions(+)

diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
index 8d10197c9e8d..1ec872afef71 100644
--- a/gcc/config/aarch64/aarch64.md
+++ b/gcc/config/aarch64/aarch64.md
@@ -8193,6 +8193,23 @@
   [(set_attr "type" "memtag")]
 )
 
+(define_insn "subg"
+  [(set (match_operand:DI 0 "register_operand" "=rk")
+   (ior:DI
+(and:DI (minus:DI (match_operand:DI 1 "register_operand" "rk")
+ (match_operand:DI 2 "aarch64_granule16_uimm6" "i"))
+(const_int -1080863910568919041)) ;; 0xf0ff...
+(ashift:DI
+ (unspec:QI
+  [(and:QI (lshiftrt:DI (match_dup 1) (const_int 56)) (const_int 15))
+   (match_operand:QI 3 "aarch64_memtag_tag_offset" "i")]
+  UNSPEC_GEN_TAG)
+ (const_int 56]
+  "TARGET_MEMTAG"
+  "subg\\t%0, %1, #%2, #%3"
+  [(set_attr "type" "memtag")]
+)
+
 (define_insn "subp"
   [(set (match_operand:DI 0 "register_operand" "=r")
(minus:DI
-- 
2.43.0

[RFC 3/9] aarch64: add new insn definition for st2g

[Indu Bhagat]

Store Allocation Tags (st2g) is an Armv8.5-A memory tagging (MTE)
instruction. It stores an allocation tag to two tag granules of memory.

TBD:
  - Not too sure what is the best way to generate the st2g yet; A
subsequent patch will emit them in one of the target hooks.
  - the current define_insn may need fixing.  The construct comparing
the two offsets should rather be defined as a new predicate?

gcc/ChangeLog:

* config/aarch64/aarch64.md (st2g): New definition.
---
 gcc/config/aarch64/aarch64.md | 20 
 1 file changed, 20 insertions(+)

diff --git a/gcc/config/aarch64/aarch64.md b/gcc/config/aarch64/aarch64.md
index 1ec872afef71..a2a69a9c0d3e 100644
--- a/gcc/config/aarch64/aarch64.md
+++ b/gcc/config/aarch64/aarch64.md
@@ -8252,6 +8252,26 @@
   [(set_attr "type" "memtag")]
 )
 
+;; ST2G updates allocation tags for two memory granules (i.e. 32 bytes) at
+;; once, without zero initialization.
+(define_insn "st2g"
+  [(set (mem:QI (unspec:DI
+[(plus:DI (match_operand:DI 1 "register_operand" "rk")
+  (match_operand:DI 2 "aarch64_granule16_simm9" "i"))]
+UNSPEC_TAG_SPACE))
+   (and:QI (lshiftrt:DI (match_operand:DI 0 "register_operand" "rk")
+(const_int 56)) (const_int 15)))
+   (set (mem:QI (unspec:DI
+[(plus:DI (match_dup 1)
+  (match_operand:DI 3 "aarch64_granule16_simm9" "i"))]
+UNSPEC_TAG_SPACE))
+   (and:QI (lshiftrt:DI (match_dup 0)
+(const_int 56)) (const_int 15)))]
+  "TARGET_MEMTAG && (INTVAL (operands[2]) - 16 == INTVAL (operands[3]))"
+  "st2g\\t%0, [%1, #%2]"
+  [(set_attr "type" "memtag")]
+)
+
 ;; Load/Store 64-bit (LS64) instructions.
 (define_insn "ld64b"
   [(set (match_operand:V8DI 0 "register_operand" "=r")
-- 
2.43.0

[RFC 4/9] opts: doc: aarch64: add new memtag sanitizer

[Indu Bhagat]

Add new command line option -fsanitize=memtag with the following
new params:
 --param memtag-instrument-stack [0,1] (default 1) to use MTE
insns for enabling dynamic checking of stack variables.
 --param memtag-instrument-alloca [0,1] (default 1) to use MTE
insns for enabling dynamic checking of stack allocas.

Add errors to convey that memtag sanitizer does not work with
hwaddress and address sanitizers.  Also error out if memtag ISA
extension is not enabled.

MEMTAG sanitizer will use the HWASAN machinery, but with a few
differences:
  - The tags are always generated at runtime by the hardware, so
-fsanitize=memtag enforces a --param hwasan-random-frame-tag=1

Add documentation in gcc/doc/invoke.texi.

TBD:
  - Add new command line option -fsanitize-memtag-mode=str, where str
can be sync, async or asymm (see
https://docs.kernel.org/arch/arm64/memory-tagging-extension.html).  This
option will not affect code generation; the information will eventually
be need to be passed to perhaps the linker.  This is contingent on
what ABI we define between userspace applications and the kernel for
communicating that the stack be PROT_MTE.

gcc/
* builtins.def: Adjust the macro to include the new
SANTIZIE_MEMTAG.
* flag-types.h (enum sanitize_code): Add new enumerator for
ANITIZE_MEMTAG.
* opts.cc (finish_options): MEMTAG conflicts with hwaddress and
address sanitizers.
(common_handle_option): SANITIZE_MEMTAG tags are always
generated (randomly) by the hardware.
* params.opt: Add new params for MEMTAG sanitizer.
doc/
* invoke.texi: Update documentation.

gcc/config/
* aarch64/aarch64.cc (aarch64_override_options_internal): Error
out if MTE is not available.
---
 gcc/builtins.def  |  1 +
 gcc/config/aarch64/aarch64.cc |  4 
 gcc/doc/invoke.texi   | 11 +--
 gcc/flag-types.h  |  2 ++
 gcc/opts.cc   | 15 +++
 gcc/params.opt|  8 
 6 files changed, 39 insertions(+), 2 deletions(-)

diff --git a/gcc/builtins.def b/gcc/builtins.def
index 0c76ebc5e31a..659c7c2b5c13 100644
--- a/gcc/builtins.def
+++ b/gcc/builtins.def
@@ -245,6 +245,7 @@ along with GCC; see the file COPYING3.  If not see
   true, true, true, ATTRS, true, \
  (flag_sanitize & (SANITIZE_ADDRESS | SANITIZE_THREAD \
| SANITIZE_HWADDRESS \
+   | SANITIZE_MEMTAG \
| SANITIZE_UNDEFINED \
| SANITIZE_UNDEFINED_NONDEFAULT) \
   || flag_sanitize_coverage))
diff --git a/gcc/config/aarch64/aarch64.cc b/gcc/config/aarch64/aarch64.cc
index f2b53475adbe..1ef2dbcf9030 100644
--- a/gcc/config/aarch64/aarch64.cc
+++ b/gcc/config/aarch64/aarch64.cc
@@ -18518,6 +18518,10 @@ aarch64_override_options_internal (struct gcc_options 
*opts)
   && !fixed_regs[R18_REGNUM])
 error ("%<-fsanitize=shadow-call-stack%> requires %<-ffixed-x18%>");
 
+  if (flag_sanitize & SANITIZE_MEMTAG && !TARGET_MEMTAG)
+error ("%<-fsanitize=memtag%> requires the ISA extension %qs",
+  "memtag");
+
   aarch64_feature_flags isa_flags = aarch64_get_isa_flags (opts);
   if ((isa_flags & (AARCH64_FL_SM_ON | AARCH64_FL_ZA_ON))
   && !(isa_flags & AARCH64_FL_SME))
diff --git a/gcc/doc/invoke.texi b/gcc/doc/invoke.texi
index 7146163d66d0..f8bd273b07ad 100644
--- a/gcc/doc/invoke.texi
+++ b/gcc/doc/invoke.texi
@@ -17626,8 +17626,9 @@ the available options are shown at startup of the 
instrumented program.  See
 
@url{https://github.com/google/sanitizers/wiki/AddressSanitizerFlags#run-time-flags}
 for a list of supported options.
 The option cannot be combined with @option{-fsanitize=thread} or
-@option{-fsanitize=hwaddress}.  Note that the only target
-@option{-fsanitize=hwaddress} is currently supported on is AArch64.
+@option{-fsanitize=hwaddress} or @option{-fsanitize=memtag}.  Note that the
+only target @option{-fsanitize=hwaddress} and @option{-fsanitize=memtag} are
+currently supported on is AArch64.
 
 To get more accurate stack traces, it is possible to use options such as
 @option{-O0}, @option{-O1}, or @option{-Og} (which, for instance, prevent
@@ -17676,6 +17677,12 @@ possible by specifying the command-line options
 @option{--param hwasan-instrument-allocas=1} respectively. Using a random frame
 tag is not implemented for kernel instrumentation.
 
+@opindex fsanitize=memtag
+@item -fsanitize=memtag
+Use Memory Tagging Extension instructions instead of instrumentation to allow
+the detection of memory errors.  This option is available only on those AArch64
+architectures that support Memory Tagging Extensions.
+
 @opindex fsanitize=pointer-compare
 @item -fsanitize=pointer-compare
 Instrument comparison operation (<, <=, >, >=) with pointer operands.
diff --git a/gcc/flag-types.h b/gcc/flag-types.h
index df56337

[RFC 7/9] hwasan: add support for generating MTE instructions for memory tagging

[Indu Bhagat]

Memory tagging is used for detecting memory safety bugs.  On AArch64, the
memory tagging extension (MTE) helps in reducing the overheads of memory
tagging:
 - CPU: MTE instructions for efficiently tagging and untagging memory.
 - Memory: New memory type, Normal Tagged Memory, added to the Arm
   Architecture.

The MEMory TAGging (MEMTAG) sanitizer uses the same infrastructure as
HWASAN.  MEMTAG and HWASAN are both hardware-assisted solutions, and
rely on the same sanitizer machinery in parts.  So, define new
constructs that allow MEMTAG and HWASAN to share the infrastructure:

  - hwassist_sanitize_p () is true when either SANITIZE_MEMTAG or
SANITIZE_HWASAN is true.
  - hwassist_sanitize_stack_p () is when hwassist_sanitize_p () and
stack variables are to be sanitized.

MEMTAG and HWASAN do have differences, however, and hence, the need to
conditionalize using memtag_sanitize_p () in the relevant places. E.g.,

  - Instead of generating the libcall __hwasan_tag_memory, MEMTAG needs
to invoke the target-specific hook TARGET_MEMTAG_TAG_MEMORY to tag
memory.  Similar approach can be seen for handling
handle_builtin_alloca, where instead of doing the gimple
transformations, target hooks are used.

  - Add a new internal function HWASAN_ALLOCA_POISON to handle
dynamically allocated stack when MEMTAG sanitizer is enabled. At
expansion, this allows to, in turn, invoke target-hooks to increment
tag, and use the generated tag to finally tag the dynamically allocated
memory.

TBD:
 - Not sure if we really need param_memtag_instrument_mem_intrinsics
   explicitly.
 - Conditionalizing using hwassist_sanitize_p (), memtag_sanitize_p ()
   etc looks unappetizing in some cases.  Not sure if there is a better
   way.  Is this generally the right thing to do, or is there some
   desirable refactorings.
 - adding decl to hwasan_stack_var. double check if this is necessary.
   See how we update the RTL for decl at expand_one_stack_var_at. And
   then use the RTL for decl in hwasan_emit_prologue
 - In hwasan_frame_base (), see if checking for memtag_sanitize_p () for
   force_reg etc is really necessary.  Revisit to see what gives, fix or
   add documentation.
 - Error out if user specifies stack alloc alignment not a factor of 16 ?

gcc/ChangeLog:

* asan.cc (struct hwasan_stack_var):
(handle_builtin_stack_restore): Accommodate MEMTAG sanitizer.
(handle_builtin_alloca): Expand differently if MEMTAG sanitizer.
(get_mem_refs_of_builtin_call): Include MEMTAG along with
HWASAN.
(memtag_sanitize_stack_p): New definition.
(memtag_sanitize_allocas_p): Likewise.
(memtag_memintrin): Likewise.
(hwassist_sanitize_p): Likewise.
(hwassist_sanitize_stack_p): Likewise.
(report_error_func): Include MEMTAG along with HWASAN.
(build_check_stmt): Likewise.
(instrument_derefs): MEMTAG too does not deal with globals yet.
(instrument_builtin_call):
(maybe_instrument_call): Include MEMTAG along with HWASAN.
(asan_expand_mark_ifn): Likewise.
(asan_expand_check_ifn): Likewise.
(asan_expand_poison_ifn): Expand differently if MEMTAG sanitizer.
(asan_instrument):
(hwasan_frame_base):
(hwasan_record_stack_var):
(hwasan_emit_prologue): Expand differently if MEMTAG sanitizer.
(hwasan_emit_untag_frame): Likewise.
* asan.h (hwasan_record_stack_var):
(memtag_sanitize_stack_p): New declaration.
(memtag_sanitize_allocas_p): Likewise.
(hwassist_sanitize_p): Likewise.
(hwassist_sanitize_stack_p): Likewise.
(asan_sanitize_use_after_scope): Include MEMTAG along with
HWASAN.
* cfgexpand.cc (align_local_variable): Likewise.
(expand_one_stack_var_at): Likewise.
(expand_stack_vars): Likewise.
(expand_one_stack_var_1): Likewise.
(init_vars_expansion): Likewise.
(expand_used_vars): Likewise.
(pass_expand::execute): Likewise.
* gimplify.cc (asan_poison_variable): Likewise.
* internal-fn.cc (expand_HWASAN_ALLOCA_POISON): New definition.
(expand_HWASAN_ALLOCA_UNPOISON): Expand differently if MEMTAG
sanitizer.
(expand_HWASAN_MARK): Likewise.
* internal-fn.def (HWASAN_ALLOCA_POISON): Define new.
* params.opt: Document new param. FIXME.
* sanopt.cc (pass_sanopt::execute): Include MEMTAG along with
HWASAN.
---
 gcc/asan.cc | 236 +---
 gcc/asan.h  |   9 +-
 gcc/cfgexpand.cc|  37 +++
 gcc/gimplify.cc |   5 +-
 gcc/internal-fn.cc  |  69 +++--
 gcc/internal-fn.def |   1 +
 gcc/params.opt  |   4 +
 gcc/sanopt.cc   |   2 +-
 8 files changed, 275 insertions(+), 88 deletions(-)

diff --git a/gcc/asan.cc b/gcc/asan.cc
index 95a9009011f7..92c16c67c7e5 100644
--- a/gcc/asan.cc
+++ b/gcc/asan.cc
@@ -299,6 

[RFC 8/9] asan: memtag: enable pass_asan for memtag sanitizer

[Indu Bhagat]

Check for SANITIZER_MEMTAG in the gate function for pass_asan gimple
pass; enable it.

TBD:
  - This commit was initially carved out in order to ensure each patch
works in isolation.  Need to revisit and double check this.

gcc/ChangeLog:

* asan.cc (memtag_sanitize_p): Fix definition.
(gate): Add gate_memtag ().
(gate_memtag): New definition.
* asan.h (gate_memtag): New declaration.
---
 gcc/asan.cc | 12 +---
 gcc/asan.h  |  1 +
 2 files changed, 10 insertions(+), 3 deletions(-)

diff --git a/gcc/asan.cc b/gcc/asan.cc
index 92c16c67c7e5..762c83ce5e5f 100644
--- a/gcc/asan.cc
+++ b/gcc/asan.cc
@@ -1889,7 +1889,7 @@ hwasan_memintrin (void)
 bool
 memtag_sanitize_p ()
 {
-  return false;
+  return sanitize_flags_p (SANITIZE_MEMTAG);
 }
 
 /* Are we tagging the stack?  */
@@ -4414,7 +4414,7 @@ public:
   opt_pass * clone () final override { return new pass_asan (m_ctxt); }
   bool gate (function *) final override
   {
-return gate_asan () || gate_hwasan ();
+return gate_asan () || gate_hwasan () || gate_memtag ();
   }
   unsigned int execute (function *) final override
   {
@@ -4456,7 +4456,7 @@ public:
   /* opt_pass methods: */
   bool gate (function *) final override
 {
-  return !optimize && (gate_asan () || gate_hwasan ());
+  return !optimize && (gate_asan () || gate_hwasan () || gate_memtag ());
 }
   unsigned int execute (function *) final override
   {
@@ -4990,4 +4990,10 @@ gate_hwasan ()
   return hwasan_sanitize_p ();
 }
 
+bool
+gate_memtag ()
+{
+  return memtag_sanitize_p ();
+}
+
 #include "gt-asan.h"
diff --git a/gcc/asan.h b/gcc/asan.h
index d169a769f780..8aa16c9931ed 100644
--- a/gcc/asan.h
+++ b/gcc/asan.h
@@ -57,6 +57,7 @@ extern bool gate_hwasan (void);
 extern bool memtag_sanitize_p (void);
 extern bool memtag_sanitize_stack_p (void);
 extern bool memtag_sanitize_allocas_p (void);
+extern bool gate_memtag (void);
 
 bool hwassist_sanitize_p (void);
 bool hwassist_sanitize_stack_p (void);
-- 
2.43.0

[RFC 1/9] opts: use unsigned HOST_WIDE_INT for sanitizer flags

[Indu Bhagat]

Currently, the data type of sanitizer flags is unsigned int, with
SANITIZE_SHADOW_CALL_STACK (1UL << 31) being highest individual
enumerator for enum sanitize_code.  Use 'unsigned HOST_WIDE_INT' data
type to allow for more distinct instrumentation modes be added when
needed.

FIXME:
1. Is using d_ulong_type for build_int_cst in gcc/d/d-attribs.cc, and
   uint64_type_node in gcc/c-family/c-attribs.cc OK ? To get type
   associated with unsigned HOST_WIDE_INT ?

gcc/ChangeLog:

* asan.h (sanitize_flags_p): Use 'unsigned HOST_WIDE_INT'
instead of 'unsigned int'.
* common.opt: Likewise.
* dwarf2asm.cc (dw2_output_indirect_constant_1): Likewise.
* opts.cc (find_sanitizer_argument): Likewise.
(report_conflicting_sanitizer_options): Likewise.
(parse_sanitizer_options): Likewise.
(parse_no_sanitize_attribute): Likewise.
* opts.h (parse_sanitizer_options): Likewise.
(parse_no_sanitize_attribute): Likewise.
* tree-cfg.cc (print_no_sanitize_attr_value): Likewise.

gcc/c-family/ChangeLog:

* c-attribs.cc (add_no_sanitize_value): Likewise.
(handle_no_sanitize_attribute): Likewise.
(handle_no_sanitize_address_attribute): Likewise.
(handle_no_sanitize_thread_attribute): Likewise.
(handle_no_address_safety_analysis_attribute): Likewise.
* c-common.h (add_no_sanitize_value): Likewise.

gcc/c/ChangeLog:

* c-parser.cc (c_parser_declaration_or_fndef): Likewise.

gcc/cp/ChangeLog:

* typeck.cc (get_member_function_from_ptrfunc): Likewise.

gcc/d/ChangeLog:

* d-attribs.cc (d_handle_no_sanitize_attribute): Likewise.
---
 gcc/asan.h|  5 +++--
 gcc/c-family/c-attribs.cc | 17 +
 gcc/c-family/c-common.h   |  2 +-
 gcc/c/c-parser.cc |  4 ++--
 gcc/common.opt|  6 +++---
 gcc/cp/typeck.cc  |  2 +-
 gcc/d/d-attribs.cc|  9 +
 gcc/dwarf2asm.cc  |  2 +-
 gcc/opts.cc   | 21 -
 gcc/opts.h|  9 +
 gcc/tree-cfg.cc   |  2 +-
 11 files changed, 43 insertions(+), 36 deletions(-)

diff --git a/gcc/asan.h b/gcc/asan.h
index d1bf8b1e701b..751ead187e35 100644
--- a/gcc/asan.h
+++ b/gcc/asan.h
@@ -239,9 +239,10 @@ asan_protect_stack_decl (tree decl)
remove all flags mentioned in "no_sanitize" of DECL_ATTRIBUTES.  */
 
 inline bool
-sanitize_flags_p (unsigned int flag, const_tree fn = current_function_decl)
+sanitize_flags_p (unsigned HOST_WIDE_INT flag,
+ const_tree fn = current_function_decl)
 {
-  unsigned int result_flags = flag_sanitize & flag;
+  unsigned HOST_WIDE_INT result_flags = flag_sanitize & flag;
   if (result_flags == 0)
 return false;
 
diff --git a/gcc/c-family/c-attribs.cc b/gcc/c-family/c-attribs.cc
index 7fd480e6d41b..66e66ba5d575 100644
--- a/gcc/c-family/c-attribs.cc
+++ b/gcc/c-family/c-attribs.cc
@@ -1401,23 +1401,24 @@ handle_cold_attribute (tree *node, tree name, tree 
ARG_UNUSED (args),
 /* Add FLAGS for a function NODE to no_sanitize_flags in DECL_ATTRIBUTES.  */
 
 void
-add_no_sanitize_value (tree node, unsigned int flags)
+add_no_sanitize_value (tree node, unsigned HOST_WIDE_INT flags)
 {
   tree attr = lookup_attribute ("no_sanitize", DECL_ATTRIBUTES (node));
   if (attr)
 {
-  unsigned int old_value = tree_to_uhwi (TREE_VALUE (attr));
+  unsigned HOST_WIDE_INT old_value = tree_to_uhwi (TREE_VALUE (attr));
   flags |= old_value;
 
   if (flags == old_value)
return;
 
-  TREE_VALUE (attr) = build_int_cst (unsigned_type_node, flags);
+  TREE_VALUE (attr) = build_int_cst (TREE_TYPE (attr), flags);
 }
   else
 DECL_ATTRIBUTES (node)
   = tree_cons (get_identifier ("no_sanitize"),
-  build_int_cst (unsigned_type_node, flags),
+  // FIXME
+  build_int_cst (uint64_type_node, flags),
   DECL_ATTRIBUTES (node));
 }
 
@@ -1428,7 +1429,7 @@ static tree
 handle_no_sanitize_attribute (tree *node, tree name, tree args, int,
  bool *no_add_attrs)
 {
-  unsigned int flags = 0;
+  unsigned HOST_WIDE_INT flags = 0;
   *no_add_attrs = true;
   if (TREE_CODE (*node) != FUNCTION_DECL)
 {
@@ -1465,7 +1466,7 @@ handle_no_sanitize_address_attribute (tree *node, tree 
name, tree, int,
   if (TREE_CODE (*node) != FUNCTION_DECL)
 warning (OPT_Wattributes, "%qE attribute ignored", name);
   else
-add_no_sanitize_value (*node, SANITIZE_ADDRESS);
+add_no_sanitize_value (*node, (HOST_WIDE_INT) SANITIZE_ADDRESS);
 
   return NULL_TREE;
 }
@@ -1481,7 +1482,7 @@ handle_no_sanitize_thread_attribute (tree *node, tree 
name, tree, int,
   if (TREE_CODE (*node) != FUNCTION_DECL)
 warning (OPT_Wattributes, "%qE attribute ignored", name);
   else
-add_no_sanitize_value (*node, SANITIZE_THREAD);
+add_no_sanitize_value (*node, (HOST_WIDE_INT) SANITIZE_THREAD);
 
   retu