[PATCH] x86-64: Expand bounded memset and memcpy like memmove

[H.J. Lu]

commit 401199377c50045ede560daf3f6e8b51749c2a87
Author: H.J. Lu <hjl.to...@gmail.com>
Date:   Tue Jun 17 10:17:17 2025 +0800

    x86: Improve vector_loop/unrolled_loop for memset/memcpy

uses move_by_pieces and store_by_pieces for memcpy and memset epilogues
with the fixed epilogue size.  Since move_by_pieces and store_by_pieces
don't use the maximum size info, they generate extra branches and moves
for bounded memcpy and memset.  Commit

commit b41f96465190751561f6909e858604ceab00595b
Author: H.J. Lu <hjl.to...@gmail.com>
Date:   Mon Oct 20 16:14:34 2025 +0800

x86-64: Inline memmove with overlapping unaligned loads and stores.

inlines memmove with overlapping unaligned and stores which reduces the
numbers of branches as well as moves when the maximum size is known.
Rename ix86_expand_movmem to ix86_expand_set_or_movmem and extend it to
inline bounded memcpy and memset.  Update ix86_expand_set_or_cpymem to
call ix86_expand_set_or_movmem for TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM.

In addition to reduce 727.cppcheck_r O2 code size by ~9%, there're
another ~8 benchmarks whose code sizes are reduced >2% across spec2026
and spec2017 with march=x86-64-v3 O2, no big code size impact for Ofast,
performance impact is negligible(slightly better, but all under noise
range) for both O2 and Ofast.

gcc/

PR target/125856
PR target/125865
* config/i386/i386-expand.cc (promote_duplicated_reg): Also
duplicate non-const0_rtx and non-constm1_rtx integer constants to
integer vector.
(ix86_expand_set_or_cpymem): Call ix86_expand_unroll_movmem for
TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM.
(ix86_expand_memset_val): New.
(ix86_expand_n_move_movmem): Renamed to ...
(ix86_expand_n_move_set_or_movmem): This.  Add a pointer to rtx
argument for memset.  Also expand memset.
(ix86_expand_n_overlapping_move_movmem): Renamed to ...
(ix86_expand_n_overlapping_move_set_or_movmem): This.  Add a
pointer to rtx argument for memset.  Also expand memset.
(ix86_expand_less_move_movmem): Renamed to ...
(ix86_expand_less_move_set_or_movmem): This.  Add a pointer to
rtx argument for memset.  Also expand memset.
(ix86_expand_movmem): Renamed to ...
(ix86_expand_set_or_movmem): This.  Add bool arguments for memcpy
and memset.  Also expand memcpy and memset.
* config/i386/i386-protos.h (ix86_expand_movmem): Renamed to ...
(ix86_expand_set_or_movmem): This.  Add bool arguments for memcpy
and memset.
* config/i386/i386.h (TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM):
New.
* config/i386/i386.md (movmem<mode>): Replace ix86_expand_movmem
with ix86_expand_set_or_movmem.
* config/i386/x86-tune.def
(X86_TUNE_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM): New.

gcc/testsuite/

PR target/125856
PR target/125865
* gcc.target/i386/builtin-memcpy-1a.c: New test.
* gcc.target/i386/builtin-memcpy-1b.c: Likewise.
* gcc.target/i386/builtin-memcpy-2a.c: Likewise.
* gcc.target/i386/builtin-memcpy-2b.c: Likewise.
* gcc.target/i386/builtin-memcpy-2c.c: Likewise.
* gcc.target/i386/builtin-memcpy-3a.c: Likewise.
* gcc.target/i386/builtin-memcpy-3b.c: Likewise.
* gcc.target/i386/builtin-memcpy-3c.c: Likewise.
* gcc.target/i386/builtin-memcpy-4a.c: Likewise.
* gcc.target/i386/builtin-memcpy-4b.c: Likewise.
* gcc.target/i386/builtin-memcpy-4c.c: Likewise.
* gcc.target/i386/builtin-memcpy-5a.c: Likewise.
* gcc.target/i386/builtin-memcpy-5b.c: Likewise.
* gcc.target/i386/builtin-memcpy-5c.c: Likewise.
* gcc.target/i386/builtin-memcpy-6a.c: Likewise.
* gcc.target/i386/builtin-memcpy-6b.c: Likewise.
* gcc.target/i386/builtin-memcpy-6c.c: Likewise.
* gcc.target/i386/builtin-memcpy-7a.c: Likewise.
* gcc.target/i386/builtin-memcpy-7b.c: Likewise.
* gcc.target/i386/builtin-memcpy-7c.c: Likewise.
* gcc.target/i386/builtin-memset-1a.c: Likewise.
* gcc.target/i386/builtin-memset-1b.c: Likewise.
* gcc.target/i386/builtin-memset-2a.c: Likewise.
* gcc.target/i386/builtin-memset-2b.c: Likewise.
* gcc.target/i386/builtin-memset-2c.c: Likewise.
* gcc.target/i386/builtin-memset-3a.c: Likewise.
* gcc.target/i386/builtin-memset-3b.c: Likewise.
* gcc.target/i386/builtin-memset-3c.c: Likewise.
* gcc.target/i386/builtin-memset-4a.c: Likewise.
* gcc.target/i386/builtin-memset-4b.c: Likewise.
* gcc.target/i386/builtin-memset-4c.c: Likewise.
* gcc.target/i386/builtin-memset-5a.c: Likewise.
* gcc.target/i386/builtin-memset-5b.c: Likewise.
* gcc.target/i386/builtin-memset-5c.c: Likewise.
* gcc.target/i386/builtin-memset-6a.c: Likewise.
* gcc.target/i386/builtin-memset-6b.c: Likewise.
* gcc.target/i386/builtin-memset-6c.c: Likewise.
* gcc.target/i386/builtin-memset-7a.c: Likewise.
* gcc.target/i386/builtin-memset-7b.c: Likewise.
* gcc.target/i386/builtin-memset-7c.c: Likewise.


--
H.J.

From 245613de2bd774a1db9011186d21c28f0c137bdd Mon Sep 17 00:00:00 2001
From: "H.J. Lu" <hjl.tools@gmail.com>
Date: Wed, 17 Jun 2026 17:32:21 +0800
Subject: [PATCH] x86-64: Expand bounded memset and memcpy like memmove

commit 401199377c50045ede560daf3f6e8b51749c2a87
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Tue Jun 17 10:17:17 2025 +0800

    x86: Improve vector_loop/unrolled_loop for memset/memcpy

uses move_by_pieces and store_by_pieces for memcpy and memset epilogues
with the fixed epilogue size.  Since move_by_pieces and store_by_pieces
don't use the maximum size info, they generate extra branches and moves
for bounded memcpy and memset.  Commit

commit b41f96465190751561f6909e858604ceab00595b
Author: H.J. Lu <hjl.tools@gmail.com>
Date:   Mon Oct 20 16:14:34 2025 +0800

x86-64: Inline memmove with overlapping unaligned loads and stores.

inlines memmove with overlapping unaligned and stores which reduces the
numbers of branches as well as moves when the maximum size is known.
Rename ix86_expand_movmem to ix86_expand_set_or_movmem and extend it to
inline bounded memcpy and memset.  Update ix86_expand_set_or_cpymem to
call ix86_expand_set_or_movmem for TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM.

In addition to reduce 727.cppcheck_r O2 code size by ~9%, there're
another ~8 benchmarks whose code sizes are reduced >2% across spec2026
and spec2017 with march=x86-64-v3 O2, no big code size impact for Ofast,
performance impact is negligible(slightly better, but all under noise
range) for both O2 and Ofast.

gcc/

	PR target/125856
	PR target/125865
	* config/i386/i386-expand.cc (promote_duplicated_reg): Also
	duplicate non-const0_rtx and non-constm1_rtx integer constants to
	integer vector.
	(ix86_expand_set_or_cpymem): Call ix86_expand_unroll_movmem for
	TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM.
	(ix86_expand_memset_val): New.
	(ix86_expand_n_move_movmem): Renamed to ...
	(ix86_expand_n_move_set_or_movmem): This.  Add a pointer to rtx
	argument for memset.  Also expand memset.
	(ix86_expand_n_overlapping_move_movmem): Renamed to ...
	(ix86_expand_n_overlapping_move_set_or_movmem): This.  Add a
	pointer to rtx argument for memset.  Also expand memset.
	(ix86_expand_less_move_movmem): Renamed to ...
	(ix86_expand_less_move_set_or_movmem): This.  Add a pointer to
	rtx argument for memset.  Also expand memset.
	(ix86_expand_movmem): Renamed to ...
	(ix86_expand_set_or_movmem): This.  Add bool arguments for memcpy
	and memset.  Also expand memcpy and memset.
	* config/i386/i386-protos.h (ix86_expand_movmem): Renamed to ...
	(ix86_expand_set_or_movmem): This.  Add bool arguments for memcpy
	and memset.
	* config/i386/i386.h (TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM):
	New.
	* config/i386/i386.md (movmem<mode>): Replace ix86_expand_movmem
	with ix86_expand_set_or_movmem.
	* config/i386/x86-tune.def
	(X86_TUNE_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM): New.

gcc/testsuite/

	PR target/125856
	PR target/125865
	* gcc.target/i386/builtin-memcpy-1a.c: New test.
	* gcc.target/i386/builtin-memcpy-1b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-2a.c: Likewise.
	* gcc.target/i386/builtin-memcpy-2b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-2c.c: Likewise.
	* gcc.target/i386/builtin-memcpy-3a.c: Likewise.
	* gcc.target/i386/builtin-memcpy-3b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-3c.c: Likewise.
	* gcc.target/i386/builtin-memcpy-4a.c: Likewise.
	* gcc.target/i386/builtin-memcpy-4b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-4c.c: Likewise.
	* gcc.target/i386/builtin-memcpy-5a.c: Likewise.
	* gcc.target/i386/builtin-memcpy-5b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-5c.c: Likewise.
	* gcc.target/i386/builtin-memcpy-6a.c: Likewise.
	* gcc.target/i386/builtin-memcpy-6b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-6c.c: Likewise.
	* gcc.target/i386/builtin-memcpy-7a.c: Likewise.
	* gcc.target/i386/builtin-memcpy-7b.c: Likewise.
	* gcc.target/i386/builtin-memcpy-7c.c: Likewise.
	* gcc.target/i386/builtin-memset-1a.c: Likewise.
	* gcc.target/i386/builtin-memset-1b.c: Likewise.
	* gcc.target/i386/builtin-memset-2a.c: Likewise.
	* gcc.target/i386/builtin-memset-2b.c: Likewise.
	* gcc.target/i386/builtin-memset-2c.c: Likewise.
	* gcc.target/i386/builtin-memset-3a.c: Likewise.
	* gcc.target/i386/builtin-memset-3b.c: Likewise.
	* gcc.target/i386/builtin-memset-3c.c: Likewise.
	* gcc.target/i386/builtin-memset-4a.c: Likewise.
	* gcc.target/i386/builtin-memset-4b.c: Likewise.
	* gcc.target/i386/builtin-memset-4c.c: Likewise.
	* gcc.target/i386/builtin-memset-5a.c: Likewise.
	* gcc.target/i386/builtin-memset-5b.c: Likewise.
	* gcc.target/i386/builtin-memset-5c.c: Likewise.
	* gcc.target/i386/builtin-memset-6a.c: Likewise.
	* gcc.target/i386/builtin-memset-6b.c: Likewise.
	* gcc.target/i386/builtin-memset-6c.c: Likewise.
	* gcc.target/i386/builtin-memset-7a.c: Likewise.
	* gcc.target/i386/builtin-memset-7b.c: Likewise.
	* gcc.target/i386/builtin-memset-7c.c: Likewise.

Signed-off-by: H.J. Lu <hjl.tools@gmail.com>
---
 gcc/config/i386/i386-expand.cc                | 564 +++++++++++++-----
 gcc/config/i386/i386-protos.h                 |   2 +-
 gcc/config/i386/i386.h                        |   2 +
 gcc/config/i386/i386.md                       |   2 +-
 gcc/config/i386/x86-tune.def                  |   9 +
 .../gcc.target/i386/builtin-memcpy-1a.c       |  37 ++
 .../gcc.target/i386/builtin-memcpy-1b.c       |  27 +
 .../gcc.target/i386/builtin-memcpy-2a.c       |  63 ++
 .../gcc.target/i386/builtin-memcpy-2b.c       |  58 ++
 .../gcc.target/i386/builtin-memcpy-2c.c       |  58 ++
 .../gcc.target/i386/builtin-memcpy-3a.c       |  74 +++
 .../gcc.target/i386/builtin-memcpy-3b.c       |  69 +++
 .../gcc.target/i386/builtin-memcpy-3c.c       |  69 +++
 .../gcc.target/i386/builtin-memcpy-4a.c       |  90 +++
 .../gcc.target/i386/builtin-memcpy-4b.c       |  80 +++
 .../gcc.target/i386/builtin-memcpy-4c.c       |  80 +++
 .../gcc.target/i386/builtin-memcpy-5a.c       | 114 ++++
 .../gcc.target/i386/builtin-memcpy-5b.c       |  97 +++
 .../gcc.target/i386/builtin-memcpy-5c.c       |  91 +++
 .../gcc.target/i386/builtin-memset-1a.c       |  32 +
 .../gcc.target/i386/builtin-memset-1b.c       |  27 +
 .../gcc.target/i386/builtin-memset-2a.c       |  59 ++
 .../gcc.target/i386/builtin-memset-2b.c       |  54 ++
 .../gcc.target/i386/builtin-memset-2c.c       |  54 ++
 .../gcc.target/i386/builtin-memset-3a.c       |  70 +++
 .../gcc.target/i386/builtin-memset-3b.c       |  65 ++
 .../gcc.target/i386/builtin-memset-3c.c       |  11 +
 .../gcc.target/i386/builtin-memset-4a.c       |  80 +++
 .../gcc.target/i386/builtin-memset-4b.c       |  82 +++
 .../gcc.target/i386/builtin-memset-4c.c       |  81 +++
 .../gcc.target/i386/builtin-memset-5a.c       |  93 +++
 .../gcc.target/i386/builtin-memset-5b.c       |  93 +++
 .../gcc.target/i386/builtin-memset-5c.c       |  89 +++
 .../gcc.target/i386/builtin-memset-6a.c       | 109 ++++
 .../gcc.target/i386/builtin-memset-6b.c       |  97 +++
 .../gcc.target/i386/builtin-memset-6c.c       |  91 +++
 .../gcc.target/i386/builtin-memset-7a.c       | 108 ++++
 .../gcc.target/i386/builtin-memset-7b.c       | 103 ++++
 .../gcc.target/i386/builtin-memset-7c.c       |  99 +++
 39 files changed, 2928 insertions(+), 155 deletions(-)
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-1a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-1b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-2a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-2b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-2c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-3a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-3b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-3c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-4a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-4b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-4c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-5a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-5b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memcpy-5c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-1a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-1b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-2a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-2b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-2c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-3a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-3b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-3c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-4a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-4b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-4c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-5a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-5b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-5c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-6a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-6b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-6c.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-7a.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-7b.c
 create mode 100644 gcc/testsuite/gcc.target/i386/builtin-memset-7c.c

diff --git a/gcc/config/i386/i386-expand.cc b/gcc/config/i386/i386-expand.cc
index 160541bc1e0..349fef02ced 100644
--- a/gcc/config/i386/i386-expand.cc
+++ b/gcc/config/i386/i386-expand.cc
@@ -9540,6 +9540,33 @@ ix86_expand_set_or_cpymem (rtx dst, rtx src, rtx count_exp, rtx val_exp,
 			   rtx max_size_exp, rtx probable_max_size_exp,
 			   bool issetmem)
 {
+  if (TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM)
+    {
+      /* Expand memset and memcpy as memmove if preferred.  Since
+
+	 commit b41f96465190751561f6909e858604ceab00595b
+	 Author: H.J. Lu <hjl.tools@gmail.com>
+	 Date:   Mon Oct 20 16:14:34 2025 +0800
+
+	 x86-64: Inline memmove with overlapping unaligned loads and stores.
+
+	 inlines memmove with overlapping unaligned and stores, which
+	 reduces the numbers of branches and memory moves, comparing
+	 against the regular memset and memcpy inlining.  */
+      rtx operands[9];
+      operands[0] = dst;
+      operands[1] = issetmem ? val_exp : src;
+      operands[2] = count_exp;
+      operands[3] = align_exp;
+      operands[4] = expected_align_exp;
+      operands[5] = expected_size_exp;
+      operands[6] = min_size_exp;
+      operands[7] = max_size_exp;
+      operands[8] = probable_max_size_exp;
+      if (ix86_expand_set_or_movmem (operands, !issetmem, issetmem))
+	return true;
+    }
+
   rtx destreg;
   rtx srcreg = NULL;
   rtx_code_label *label = NULL;
@@ -10123,34 +10150,99 @@ ix86_expand_unroll_movmem (rtx dst, rtx src, rtx destreg, rtx srcreg,
   return true;
 }
 
+/* Value kind in MEMSET_VALS:
+
+   memset_val_byte:   The value rtx in QImode.
+   memset_val_word:   The value rtx in word_mode.
+   memset_val_vector: The value rtx in QI vector mode.
+
+ */
+enum memset_val_kind
+{
+  memset_val_byte = 0,
+  memset_val_word = 1,
+  memset_val_vector = 2,
+  memset_val_max = 3
+};
+
+/* Return a value rtx in MODE for memset from MEMSET_VALS.  */
+
+static rtx
+ix86_expand_memset_val (rtx *memset_vals, machine_mode mode)
+{
+  rtx val;
+  if (mode == QImode)
+    val = memset_vals[memset_val_byte];
+  else if (mode == word_mode)
+    val = memset_vals[memset_val_word];
+  else if (GET_MODE_CLASS (mode) == MODE_VECTOR_INT)
+    {
+      if (GET_MODE (memset_vals[memset_val_vector]) == mode)
+	val = memset_vals[memset_val_vector];
+      else
+	{
+	  machine_mode vmode = GET_MODE (memset_vals[memset_val_vector]);
+	if (memset_vals[memset_val_vector] == CONST0_RTX (vmode))
+	  val = CONST0_RTX (mode);
+	else if (memset_vals[memset_val_vector] == CONSTM1_RTX (vmode))
+	  val = CONSTM1_RTX (mode);
+	else
+	  val = gen_rtx_SUBREG (mode, memset_vals[memset_val_vector], 0);
+	}
+    }
+  else
+    {
+      gcc_assert (mode == HImode || mode == SImode);
+      if (memset_vals[memset_val_word] == const0_rtx
+	  || memset_vals[memset_val_word] == constm1_rtx)
+	val = memset_vals[memset_val_word];
+      else
+	val = gen_rtx_SUBREG (mode, memset_vals[memset_val_word], 0);
+    }
+
+  return val;
+}
+
 /* Expand memmove of size with MOVES * mode size and MOVES <= 4.  If
    FORWARD is true, copy forward.  Otherwise copy backward.  */
 
 static void
-ix86_expand_n_move_movmem (rtx destmem, rtx srcmem, machine_mode mode,
-			   unsigned int moves, bool forward)
+ix86_expand_n_move_set_or_movmem (rtx destmem, rtx srcmem,
+				  rtx *memset_vals, machine_mode mode,
+				  unsigned int moves, bool forward)
 {
   gcc_assert (moves <= 4);
 
   unsigned int i;
   rtx tmp[8];
 
-  for (i = 0; i < moves; i++)
-    tmp[i] = gen_reg_rtx (mode);
-
   rtx step;
   if (forward)
     step = GEN_INT (GET_MODE_SIZE (mode));
   else
     step = GEN_INT (-GET_MODE_SIZE (mode));
 
-  /* Load MOVES.  */
-  for (i = 0; i < moves - 1; i++)
+  if (memset_vals)
+    {
+      /* Expand memset.  */
+      rtx val = ix86_expand_memset_val (memset_vals, mode);
+      for (i = 0; i < moves; i++)
+	tmp[i] = val;
+    }
+  else
     {
+      /* Expand memmove.  */
+      for (i = 0; i < moves; i++)
+	tmp[i] = gen_reg_rtx (mode);
+
+      /* Load MOVES.  */
+      for (i = 0; i < moves - 1; i++)
+	{
+	  emit_move_insn (tmp[i], srcmem);
+	  srcmem = offset_address (srcmem, step, GET_MODE_SIZE (mode));
+	}
       emit_move_insn (tmp[i], srcmem);
-      srcmem = offset_address (srcmem, step, GET_MODE_SIZE (mode));
     }
-  emit_move_insn (tmp[i], srcmem);
 
   /* Store MOVES.  */
   for (i = 0; i < moves - 1; i++)
@@ -10227,10 +10319,12 @@ ix86_expand_store_movmem (rtx dst, rtx destreg, rtx count_exp,
    MOVES >= 2 and MOVES <= 8.  */
 
 static void
-ix86_expand_n_overlapping_move_movmem (rtx dst, rtx src, rtx destreg,
-				       rtx srcreg, rtx count_exp,
-				       machine_mode mode,
-				       unsigned int moves)
+ix86_expand_n_overlapping_move_set_or_movmem (rtx dst, rtx src,
+					      rtx *memset_vals,
+					      rtx destreg, rtx srcreg,
+					      rtx count_exp,
+					      machine_mode mode,
+					      unsigned int moves)
 {
   gcc_assert (moves >= 2 && moves <= 8 && (moves & 1) == 0);
 
@@ -10238,35 +10332,46 @@ ix86_expand_n_overlapping_move_movmem (rtx dst, rtx src, rtx destreg,
   unsigned int i, j;
   rtx tmp[8];
 
-  for (i = 0; i < moves; i++)
-    tmp[i] = gen_reg_rtx (mode);
+  if (memset_vals)
+    {
+      /* Expand memset.  */
+      rtx val = ix86_expand_memset_val (memset_vals, mode);
+      for (i = 0; i < moves; i++)
+	tmp[i] = val;
+    }
+  else
+    {
+      /* Expand memmove.  */
+      for (i = 0; i < moves; i++)
+	tmp[i] = gen_reg_rtx (mode);
 
-  rtx base_srcmem = change_address (src, mode, srcreg);
+      rtx base_srcmem = change_address (src, mode, srcreg);
 
-  /* Load the first half.  */
-  rtx srcmem = base_srcmem;
-  for (i = 0; i < half_moves - 1; i++)
-    {
+      /* Load the first half.  */
+      rtx srcmem = base_srcmem;
+      for (i = 0; i < half_moves - 1; i++)
+	{
+	  emit_move_insn (tmp[i], srcmem);
+	  srcmem = offset_address (srcmem,
+				   GEN_INT (GET_MODE_SIZE (mode)),
+				   GET_MODE_SIZE (mode));
+	}
       emit_move_insn (tmp[i], srcmem);
-      srcmem = offset_address (srcmem,
-			       GEN_INT (GET_MODE_SIZE (mode)),
-			       GET_MODE_SIZE (mode));
-    }
-  emit_move_insn (tmp[i], srcmem);
 
-  /* Load the second half.  */
-  srcmem = offset_address (base_srcmem, count_exp, 1);
-  srcmem = offset_address (srcmem,
-			   GEN_INT (-GET_MODE_SIZE (mode)),
-			   GET_MODE_SIZE (mode));
-  for (j = half_moves, i = 0; i < half_moves - 1; i++, j++)
-    {
-      emit_move_insn (tmp[j], srcmem);
+      /* Load the second half.  */
+      srcmem = offset_address (base_srcmem, count_exp, 1);
       srcmem = offset_address (srcmem,
 			       GEN_INT (-GET_MODE_SIZE (mode)),
 			       GET_MODE_SIZE (mode));
+      for (j = half_moves, i = 0; i < half_moves - 1; i++, j++)
+	{
+	  emit_move_insn (tmp[j], srcmem);
+	  srcmem = offset_address (srcmem,
+				   GEN_INT (-GET_MODE_SIZE (mode)),
+				   GET_MODE_SIZE (mode));
+	}
+      emit_move_insn (tmp[j], srcmem);
     }
-  emit_move_insn (tmp[j], srcmem);
 
   rtx base_destmem = change_address (dst, mode, destreg);
 
@@ -10297,11 +10402,12 @@ ix86_expand_n_overlapping_move_movmem (rtx dst, rtx src, rtx destreg,
 /* Expand memmove of size < mode size which is <= 64.  */
 
 static void
-ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
-			      rtx srcreg, rtx count_exp,
-			      unsigned HOST_WIDE_INT min_size,
-			      machine_mode mode,
-			      rtx_code_label *done_label)
+ix86_expand_less_move_set_or_movmem (rtx dst, rtx src, rtx *memset_vals,
+				     rtx destreg, rtx srcreg,
+				     rtx count_exp,
+				     unsigned HOST_WIDE_INT min_size,
+				     machine_mode mode,
+				     rtx_code_label *done_label)
 {
   bool skip = false;
   machine_mode count_mode = counter_mode (count_exp);
@@ -10403,9 +10509,16 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
 				 profile_probability::unlikely ());
 
       /* Move 1 byte.  */
-      rtx tmp0 = gen_reg_rtx (QImode);
-      rtx srcmem = change_address (src, QImode, srcreg);
-      emit_move_insn (tmp0, srcmem);
+      rtx tmp0;
+      /* Use the value rtx in QImode for memset.  */
+      if (memset_vals)
+	tmp0 = memset_vals[memset_val_byte];
+      else
+	{
+	  tmp0 = gen_reg_rtx (QImode);
+	  rtx srcmem = change_address (src, QImode, srcreg);
+	  emit_move_insn (tmp0, srcmem);
+	}
       rtx destmem = change_address (dst, QImode, destreg);
       emit_move_insn (destmem, tmp0);
 
@@ -10416,11 +10529,16 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
       emit_barrier ();
     }
 
+  machine_mode vector_mode;
   if (between_32_63_label)
     {
       emit_label (between_32_63_label);
-      ix86_expand_n_overlapping_move_movmem (dst, src, destreg, srcreg,
-					     count_exp, OImode, 2);
+      vector_mode = memset_vals ? V32QImode : OImode;
+      ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+						    memset_vals,
+						    destreg, srcreg,
+						    count_exp,
+						    vector_mode, 2);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10428,8 +10546,12 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
   if (between_16_31_label)
     {
       emit_label (between_16_31_label);
-      ix86_expand_n_overlapping_move_movmem (dst, src, destreg, srcreg,
-					     count_exp, TImode, 2);
+      vector_mode = memset_vals ? V16QImode : TImode;
+      ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+						    memset_vals,
+						    destreg, srcreg,
+						    count_exp,
+						    vector_mode, 2);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10437,8 +10559,11 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
   if (between_8_15_label)
     {
       emit_label (between_8_15_label);
-      ix86_expand_n_overlapping_move_movmem (dst, src, destreg, srcreg,
-					     count_exp, DImode, 2);
+      ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+						    memset_vals,
+						    destreg, srcreg,
+						    count_exp, DImode,
+						    2);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10446,8 +10571,11 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
   if (between_4_7_label)
     {
       emit_label (between_4_7_label);
-      ix86_expand_n_overlapping_move_movmem (dst, src, destreg, srcreg,
-					     count_exp, SImode, 2);
+      ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+						    memset_vals,
+						    destreg, srcreg,
+						    count_exp, SImode,
+						    2);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10455,8 +10583,11 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
   if (between_2_3_label)
     {
       emit_label (between_2_3_label);
-      ix86_expand_n_overlapping_move_movmem (dst, src, destreg, srcreg,
-					     count_exp, HImode, 2);
+      ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+						    memset_vals,
+						    destreg, srcreg,
+						    count_exp, HImode,
+						    2);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10484,7 +10615,7 @@ ix86_expand_less_move_movmem (rtx dst, rtx src, rtx destreg,
  */
 
 bool
-ix86_expand_movmem (rtx operands[])
+ix86_expand_set_or_movmem (rtx operands[], bool iscpymem, bool issetmem)
 {
   /* Since there are much less registers available in 32-bit mode, don't
      inline movmem in 32-bit mode.  */
@@ -10492,14 +10623,26 @@ ix86_expand_movmem (rtx operands[])
     return false;
 
   rtx dst = operands[0];
-  rtx src = operands[1];
+  rtx src, memset_val_exp;
+  if (issetmem)
+    {
+      src = nullptr;
+      memset_val_exp = operands[1];
+    }
+  else
+    {
+      src = operands[1];
+      memset_val_exp = nullptr;
+    }
   rtx count_exp = operands[2];
   rtx expected_size_exp = operands[5];
   rtx min_size_exp = operands[6];
+  rtx max_size_exp = operands[7];
   rtx probable_max_size_exp = operands[8];
   unsigned HOST_WIDE_INT count = HOST_WIDE_INT_0U;
   HOST_WIDE_INT expected_size = HOST_WIDE_INT_M1U;
   unsigned HOST_WIDE_INT min_size = HOST_WIDE_INT_0U;
+  unsigned HOST_WIDE_INT max_size = HOST_WIDE_INT_M1U;
   unsigned HOST_WIDE_INT probable_max_size = HOST_WIDE_INT_M1U;
 
   if (CONST_INT_P (count_exp))
@@ -10514,6 +10657,8 @@ ix86_expand_movmem (rtx operands[])
     {
       if (min_size_exp)
 	min_size = INTVAL (min_size_exp);
+      if (max_size_exp)
+	max_size = INTVAL (max_size_exp);
       if (probable_max_size_exp)
 	probable_max_size = INTVAL (probable_max_size_exp);
       if (CONST_INT_P (expected_size_exp))
@@ -10525,18 +10670,31 @@ ix86_expand_movmem (rtx operands[])
     return false;
 
   addr_space_t dst_as = MEM_ADDR_SPACE (dst);
-  addr_space_t src_as = MEM_ADDR_SPACE (src);
+  addr_space_t src_as = (issetmem
+			 ? ADDR_SPACE_GENERIC
+			 : MEM_ADDR_SPACE (src));
   int dynamic_check;
   bool noalign;
   enum stringop_alg alg = decide_alg (count, expected_size, min_size,
-				      probable_max_size, false, false,
+				      probable_max_size, issetmem,
+				      (issetmem
+				       && memset_val_exp == const0_rtx),
 				      dst_as, src_as, &dynamic_check,
 				      &noalign, false);
   if (alg == libcall)
     return false;
 
+  /* Expand memcpy and memset like memmove only for bounded size.  */
+  if ((iscpymem || issetmem)
+      && (count != 0				/* Fixed size.  */
+	  || max_size == 0			/* Unbounded size.  */
+	  || max_size == HOST_WIDE_INT_M1U))	/* Unbounded size.  */
+    return false;
+
   rtx destreg = ix86_copy_addr_to_reg (XEXP (dst, 0));
-  rtx srcreg = ix86_copy_addr_to_reg (XEXP (src, 0));
+  rtx srcreg = (issetmem
+		? nullptr
+		: ix86_copy_addr_to_reg (XEXP (src, 0)));
 
   unsigned int move_max = MOVE_MAX;
   machine_mode mode = smallest_int_mode_for_size
@@ -10559,6 +10717,53 @@ ix86_expand_movmem (rtx operands[])
 				    mode))
     return true;
 
+  rtx memset_vals[memset_val_max];
+  rtx *memset_vals_p;
+  if (issetmem)
+    {
+      /* Use vector mode if MODE size > word size.  */
+      unsigned int size = GET_MODE_SIZE (mode);
+      poly_uint64 nunits;
+      if (size > UNITS_PER_WORD)
+	{
+	  nunits = size / GET_MODE_SIZE (QImode);
+	  mode = mode_for_vector (QImode, nunits).require ();
+	}
+
+      /* Populate MEMSET_VALS to expand memset.  */
+      rtx val_word;
+      memset_vals[memset_val_byte] = memset_val_exp;
+      if (memset_val_exp == const0_rtx || memset_val_exp == constm1_rtx)
+	val_word = memset_val_exp;
+      else
+	val_word = promote_duplicated_reg (word_mode, memset_val_exp);
+      memset_vals[memset_val_word] = val_word;
+      if (GET_MODE_SIZE (mode) > UNITS_PER_WORD)
+	{
+	  if (memset_val_exp == const0_rtx)
+	    memset_vals[memset_val_vector] = CONST0_RTX (mode);
+	  else if (memset_val_exp == constm1_rtx)
+	    memset_vals[memset_val_vector] = CONSTM1_RTX (mode);
+	  else
+	    {
+	      /* Use the vector mode based on WORD_MODE to avoid extra
+		 GPR moves.  */
+	      nunits = size / GET_MODE_SIZE (word_mode);
+	      machine_mode vector_mode
+		= mode_for_vector (word_mode, nunits).require ();
+	      rtx vector = promote_duplicated_reg (vector_mode,
+						   val_word);
+	      memset_vals[memset_val_vector]
+		= convert_to_mode (mode, vector, 1);
+	    }
+	}
+      else
+	memset_vals[memset_val_vector] = nullptr;
+      memset_vals_p = memset_vals;
+    }
+  else
+    memset_vals_p = nullptr;
+
   rtx_code_label *done_label = gen_label_rtx ();
 
   rtx_code_label *less_vec_label = nullptr;
@@ -10586,8 +10791,10 @@ ix86_expand_movmem (rtx operands[])
   if (min_size == 0 || min_size <= 2 * move_max)
     {
       /* Size >= MOVE_MAX and size <= 2 * MOVE_MAX.  */
-      ix86_expand_n_overlapping_move_movmem (dst, src, destreg, srcreg,
-					     count_exp, mode, 2);
+      ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+						    memset_vals_p,
+						    destreg, srcreg,
+						    count_exp, mode, 2);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10596,9 +10803,9 @@ ix86_expand_movmem (rtx operands[])
     {
       /* Size < MOVE_MAX.  */
       emit_label (less_vec_label);
-      ix86_expand_less_move_movmem (dst, src, destreg, srcreg,
-				    count_exp, min_size, mode,
-				    done_label);
+      ix86_expand_less_move_set_or_movmem (dst, src, memset_vals_p,
+					   destreg, srcreg, count_exp,
+					   min_size, mode, done_label);
       emit_jump_insn (gen_jump (done_label));
       emit_barrier ();
     }
@@ -10631,9 +10838,11 @@ ix86_expand_movmem (rtx operands[])
       if (probable_max_size == 0 || probable_max_size > 4 * move_max)
 	{
 	  /* Size > 4 * MOVE_MAX and size <= 8 * MOVE_MAX.  */
-	  ix86_expand_n_overlapping_move_movmem (dst, src, destreg,
-						 srcreg, count_exp,
-						 mode, 8);
+	  ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+							memset_vals_p,
+							destreg, srcreg,
+							count_exp, mode,
+							8);
 	  emit_jump_insn (gen_jump (done_label));
 	  emit_barrier ();
 	}
@@ -10642,9 +10851,11 @@ ix86_expand_movmem (rtx operands[])
 	{
 	  /* Size > 2 * MOVE_MAX and size <= 4 * MOVE_MAX.  */
 	  emit_label (last_4x_vec_label);
-	  ix86_expand_n_overlapping_move_movmem (dst, src, destreg,
-						 srcreg, count_exp,
-						 mode, 4);
+	  ix86_expand_n_overlapping_move_set_or_movmem (dst, src,
+							memset_vals_p,
+							destreg, srcreg,
+							count_exp, mode,
+							4);
 	  emit_jump_insn (gen_jump (done_label));
 	  emit_barrier ();
 	}
@@ -10657,34 +10868,66 @@ ix86_expand_movmem (rtx operands[])
 	  rtx loop_count = gen_reg_rtx (count_mode);
 	  emit_move_insn (loop_count, count_exp);
 
-	  /* Jump to MORE_8X_VEC_BACKWARD_LABEL if source address is
-	     lower than destination address.  */
-	  rtx_code_label *more_8x_vec_backward_label = gen_label_rtx ();
-	  emit_cmp_and_jump_insns (srcreg, destreg, LTU, nullptr,
-				   GET_MODE (destreg), 1,
-				   more_8x_vec_backward_label);
-
-	  /* Skip if source == destination which is less common.  */
-	  emit_cmp_and_jump_insns (srcreg, destreg, EQ, nullptr,
-				   GET_MODE (destreg), 1, done_label,
-				   profile_probability::unlikely ());
-
-	  rtx base_destreg = gen_reg_rtx (GET_MODE (destreg));
-	  emit_move_insn (base_destreg, destreg);
-
-	  /* Load the last 4 * MOVE_MAX.  */
+	  rtx_code_label *more_8x_vec_backward_label;
+	  rtx base_destreg;
+	  rtx srcmem;
 	  rtx regs[4];
-	  ix86_expand_load_movmem (src, srcreg, count_exp, mode,
-				   ARRAY_SIZE (regs), regs, true);
+	  if (iscpymem || issetmem)
+	    {
+	      /* Always store forward for memcpy and memset.  */
+	      more_8x_vec_backward_label = nullptr;
+	      if (iscpymem)
+		{
+		  /* Load the last 4 * MOVE_MAX for memcpy.  */
+		  ix86_expand_load_movmem (src, srcreg, count_exp, mode,
+					   ARRAY_SIZE (regs), regs,
+					   true);
+		  srcmem = change_address (src, mode, srcreg);
+		}
+	      else
+		{
+		  /* Fill REGS with MEMSET_VALS for memset.  */
+		  rtx val = ix86_expand_memset_val (memset_vals, mode);
+		  for (unsigned int i = 0; i < 4; i++)
+		    regs[i] = val;
+		  srcmem = nullptr;
+		}
+	      base_destreg = gen_reg_rtx (GET_MODE (destreg));
+	      emit_move_insn (base_destreg, destreg);
+	    }
+	  else
+	    {
+	      /* Jump to MORE_8X_VEC_BACKWARD_LABEL if source address is
+		 lower than destination address.  */
+	      more_8x_vec_backward_label = gen_label_rtx ();
+	      emit_cmp_and_jump_insns (srcreg, destreg, LTU, nullptr,
+				       GET_MODE (destreg), 1,
+				       more_8x_vec_backward_label);
+
+	      /* Skip if source == destination which is less common.  */
+	      emit_cmp_and_jump_insns (srcreg, destreg, EQ, nullptr,
+				       GET_MODE (destreg), 1, done_label,
+				       profile_probability::unlikely ());
+
+	      base_destreg = gen_reg_rtx (GET_MODE (destreg));
+	      emit_move_insn (base_destreg, destreg);
+
+	      /* Load the last 4 * MOVE_MAX.  */
+	      ix86_expand_load_movmem (src, srcreg, count_exp, mode,
+				       ARRAY_SIZE (regs), regs, true);
+
+	      srcmem = change_address (src, mode, srcreg);
+	    }
 
-	  rtx srcmem = change_address (src, mode, srcreg);
 	  rtx destmem = change_address (dst, mode, destreg);
 
 	  /* Copy forward with a 4 * MOVE_MAX loop.  */
 	  rtx_code_label *loop_4x_vec_forward_label = gen_label_rtx ();
 	  emit_label (loop_4x_vec_forward_label);
 
-	  ix86_expand_n_move_movmem (destmem, srcmem, mode, 4, true);
+	  ix86_expand_n_move_set_or_movmem (destmem, srcmem,
+					    memset_vals_p, mode, 4,
+					    true);
 
 	  rtx tmp;
 	  rtx delta = GEN_INT (4 * MOVE_MAX);
@@ -10702,10 +10945,14 @@ ix86_expand_movmem (rtx operands[])
 				     OPTAB_DIRECT);
 	  if (tmp != destreg)
 	    emit_move_insn (destreg, tmp);
-	  tmp = expand_simple_binop (GET_MODE (srcreg), PLUS, srcreg,
-				     delta, nullptr, 1, OPTAB_DIRECT);
-	  if (tmp != srcreg)
-	    emit_move_insn (srcreg, tmp);
+	  if (!issetmem)
+	    {
+	      tmp = expand_simple_binop (GET_MODE (srcreg), PLUS,
+					 srcreg, delta, nullptr, 1,
+					 OPTAB_DIRECT);
+	      if (tmp != srcreg)
+		emit_move_insn (srcreg, tmp);
+	    }
 
 	  /* Stop if LOOP_EXP <= 4 * MOVE_MAX.  */
 	  emit_cmp_and_jump_insns (loop_count, delta, GTU, nullptr,
@@ -10719,67 +10966,78 @@ ix86_expand_movmem (rtx operands[])
 	  emit_jump_insn (gen_jump (done_label));
 	  emit_barrier ();
 
-	  /* Copy backward with a 4 * MOVE_MAX loop.  */
-	  emit_label (more_8x_vec_backward_label);
-
-	  base_destreg = gen_reg_rtx (GET_MODE (destreg));
-	  emit_move_insn (base_destreg, destreg);
-
-	  /* Load the first 4 * MOVE_MAX.  */
-	  ix86_expand_load_movmem (src, srcreg, count_exp, mode,
-				   ARRAY_SIZE (regs), regs, false);
-
-	  /* Increment DESTREG and SRCREG by COUNT_EXP.  */
-	  tmp = expand_simple_binop (GET_MODE (destreg), PLUS,
-				     destreg, count_exp, nullptr, 1,
-				     OPTAB_DIRECT);
-	  if (tmp != destreg)
-	    emit_move_insn (destreg, tmp);
-	  tmp = expand_simple_binop (GET_MODE (srcreg), PLUS, srcreg,
-				     count_exp, nullptr, 1, OPTAB_DIRECT);
-	  if (tmp != srcreg)
-	    emit_move_insn (srcreg, tmp);
-
-	  srcmem = change_address (src, mode, srcreg);
-	  destmem = change_address (dst, mode, destreg);
-	  rtx step = GEN_INT (-GET_MODE_SIZE (mode));
-	  srcmem = offset_address (srcmem, step, GET_MODE_SIZE (mode));
-	  destmem = offset_address (destmem, step, GET_MODE_SIZE (mode));
-
-	  rtx_code_label *loop_4x_vec_backward_label = gen_label_rtx ();
-	  emit_label (loop_4x_vec_backward_label);
-
-	  ix86_expand_n_move_movmem (destmem, srcmem, mode, 4, false);
-
-	  /* Decrement LOOP_COUNT by 4 * MOVE_MAX.  */
-	  tmp = expand_simple_binop (GET_MODE (loop_count), MINUS,
-				     loop_count, delta, nullptr, 1,
-				     OPTAB_DIRECT);
-	  if (tmp != loop_count)
-	    emit_move_insn (loop_count, tmp);
-
-	  /* Decrement DESTREG and SRCREG by 4 * MOVE_MAX.  */
-	  tmp = expand_simple_binop (GET_MODE (destreg), MINUS,
-				     destreg, delta, nullptr, 1,
-				     OPTAB_DIRECT);
-	  if (tmp != destreg)
-	    emit_move_insn (destreg, tmp);
-	  tmp = expand_simple_binop (GET_MODE (srcreg), MINUS, srcreg,
-				     delta, nullptr, 1, OPTAB_DIRECT);
-	  if (tmp != srcreg)
-	    emit_move_insn (srcreg, tmp);
-
-	  /* Stop if LOOP_EXP <= 4 * MOVE_MAX.  */
-	  emit_cmp_and_jump_insns (loop_count, delta, GTU, nullptr,
-				   GET_MODE (loop_count), 1,
-				   loop_4x_vec_backward_label);
-
-	  /* Store the first 4 * MOVE_MAX.  */
-	  ix86_expand_store_movmem (dst, base_destreg, count_exp, mode,
-				    ARRAY_SIZE (regs), regs, false);
-
-	  emit_jump_insn (gen_jump (done_label));
-	  emit_barrier ();
+	  if (more_8x_vec_backward_label)
+	    {
+	      /* Copy backward with a 4 * MOVE_MAX loop.  */
+	      emit_label (more_8x_vec_backward_label);
+
+	      base_destreg = gen_reg_rtx (GET_MODE (destreg));
+	      emit_move_insn (base_destreg, destreg);
+
+	      /* Load the first 4 * MOVE_MAX.  */
+	      ix86_expand_load_movmem (src, srcreg, count_exp, mode,
+				       ARRAY_SIZE (regs), regs, false);
+
+	      /* Increment DESTREG and SRCREG by COUNT_EXP.  */
+	      tmp = expand_simple_binop (GET_MODE (destreg), PLUS,
+					 destreg, count_exp, nullptr, 1,
+					 OPTAB_DIRECT);
+	      if (tmp != destreg)
+		emit_move_insn (destreg, tmp);
+	      tmp = expand_simple_binop (GET_MODE (srcreg), PLUS, srcreg,
+					 count_exp, nullptr, 1,
+					 OPTAB_DIRECT);
+	      if (tmp != srcreg)
+		emit_move_insn (srcreg, tmp);
+
+	      srcmem = change_address (src, mode, srcreg);
+	      destmem = change_address (dst, mode, destreg);
+	      rtx step = GEN_INT (-GET_MODE_SIZE (mode));
+	      srcmem = offset_address (srcmem, step,
+				       GET_MODE_SIZE (mode));
+	      destmem = offset_address (destmem, step,
+					GET_MODE_SIZE (mode));
+
+	      rtx_code_label *loop_4x_vec_backward_label
+		= gen_label_rtx ();
+	      emit_label (loop_4x_vec_backward_label);
+
+	      ix86_expand_n_move_set_or_movmem (destmem, srcmem,
+						memset_vals_p, mode, 4,
+						false);
+
+	      /* Decrement LOOP_COUNT by 4 * MOVE_MAX.  */
+	      tmp = expand_simple_binop (GET_MODE (loop_count), MINUS,
+					 loop_count, delta, nullptr, 1,
+					 OPTAB_DIRECT);
+	      if (tmp != loop_count)
+		emit_move_insn (loop_count, tmp);
+
+	      /* Decrement DESTREG and SRCREG by 4 * MOVE_MAX.  */
+	      tmp = expand_simple_binop (GET_MODE (destreg), MINUS,
+					 destreg, delta, nullptr, 1,
+					 OPTAB_DIRECT);
+	      if (tmp != destreg)
+		emit_move_insn (destreg, tmp);
+	      tmp = expand_simple_binop (GET_MODE (srcreg), MINUS,
+					 srcreg, delta, nullptr, 1,
+					 OPTAB_DIRECT);
+	      if (tmp != srcreg)
+		emit_move_insn (srcreg, tmp);
+
+	      /* Stop if LOOP_EXP <= 4 * MOVE_MAX.  */
+	      emit_cmp_and_jump_insns (loop_count, delta, GTU, nullptr,
+				       GET_MODE (loop_count), 1,
+				       loop_4x_vec_backward_label);
+
+	      /* Store the first 4 * MOVE_MAX.  */
+	      ix86_expand_store_movmem (dst, base_destreg, count_exp,
+					mode, ARRAY_SIZE (regs), regs,
+					false);
+
+	      emit_jump_insn (gen_jump (done_label));
+	      emit_barrier ();
+	    }
 	}
     }
 
diff --git a/gcc/config/i386/i386-protos.h b/gcc/config/i386/i386-protos.h
index ea23f75804a..54f5f42e4be 100644
--- a/gcc/config/i386/i386-protos.h
+++ b/gcc/config/i386/i386-protos.h
@@ -80,7 +80,7 @@ extern void substitute_vpternlog_operands (rtx[]);
 extern bool ix86_expand_strlen (rtx, rtx, rtx, rtx);
 extern bool ix86_expand_set_or_cpymem (rtx, rtx, rtx, rtx, rtx, rtx,
 				       rtx, rtx, rtx, rtx, bool);
-extern bool ix86_expand_movmem (rtx[]);
+extern bool ix86_expand_set_or_movmem (rtx[], bool, bool);
 extern bool ix86_expand_cmpstrn_or_cmpmem (rtx, rtx, rtx, rtx, rtx, bool);
 
 extern enum reg_class ix86_insn_base_reg_class (rtx_insn *);
diff --git a/gcc/config/i386/i386.h b/gcc/config/i386/i386.h
index c8dd17f24ce..e8bcadd2a6c 100644
--- a/gcc/config/i386/i386.h
+++ b/gcc/config/i386/i386.h
@@ -367,6 +367,8 @@ extern unsigned char ix86_tune_features[X86_TUNE_LAST];
   ix86_tune_features[X86_TUNE_PREFER_KNOWN_REP_MOVSB_STOSB]
 #define TARGET_MISALIGNED_MOVE_STRING_PRO_EPILOGUES \
 	ix86_tune_features[X86_TUNE_MISALIGNED_MOVE_STRING_PRO_EPILOGUES]
+#define TARGET_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM \
+	ix86_tune_features[X86_TUNE_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM]
 #define TARGET_QIMODE_MATH	ix86_tune_features[X86_TUNE_QIMODE_MATH]
 #define TARGET_HIMODE_MATH	ix86_tune_features[X86_TUNE_HIMODE_MATH]
 #define TARGET_PROMOTE_QI_REGS	ix86_tune_features[X86_TUNE_PROMOTE_QI_REGS]
diff --git a/gcc/config/i386/i386.md b/gcc/config/i386/i386.md
index 05d6f6d62f0..ca659518aab 100644
--- a/gcc/config/i386/i386.md
+++ b/gcc/config/i386/i386.md
@@ -26571,7 +26571,7 @@ (define_expand "movmem<mode>"
    (use (match_operand:SI 8 ""))]
   ""
 {
-  if (ix86_expand_movmem (operands))
+  if (ix86_expand_set_or_movmem (operands, false, false))
     DONE;
   FAIL;
 })
diff --git a/gcc/config/i386/x86-tune.def b/gcc/config/i386/x86-tune.def
index d4c27351ad7..329d8825d20 100644
--- a/gcc/config/i386/x86-tune.def
+++ b/gcc/config/i386/x86-tune.def
@@ -344,6 +344,15 @@ DEF_TUNE (X86_TUNE_MISALIGNED_MOVE_STRING_PRO_EPILOGUES,
 	  m_386 | m_486 | m_CORE_ALL | m_AMD_MULTIPLE | m_ZHAOXIN | m_TREMONT
 	  | m_CORE_HYBRID | m_CORE_ATOM | m_C86_4G | m_GENERIC)
 
+/* X86_TUNE_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM: Enable inlining bounded
+   memset and memcpy like memmove with overlapping unaligned moves.  This
+   requires target to handle misaligned moves and partial memory stalls
+   reasonably well.  */
+DEF_TUNE (X86_TUNE_INLINE_SET_OR_CPYMEM_LIKE_MOVMEM,
+	  "inline_set_or_cpymem_like_movmem",
+	  m_386 | m_486 | m_CORE_ALL | m_AMD_MULTIPLE | m_ZHAOXIN | m_TREMONT
+	  | m_CORE_HYBRID | m_CORE_ATOM | m_C86_4G | m_GENERIC)
+
 /* X86_TUNE_USE_SAHF: Controls use of SAHF.  */
 DEF_TUNE (X86_TUNE_USE_SAHF, "use_sahf",
 	  m_PPRO | m_P4_NOCONA | m_CORE_ALL | m_BONNELL | m_SILVERMONT
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-1a.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-1a.c
new file mode 100644
index 00000000000..c4710b41139
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-1a.c
@@ -0,0 +1,37 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	movdqu	\(%rsi\), %xmm0
+**	movups	%xmm0, \(%rdi\)
+**	movdqu	16\(%rsi\), %xmm0
+**	movups	%xmm0, 16\(%rdi\)
+**	movdqu	32\(%rsi\), %xmm0
+**	movups	%xmm0, 32\(%rdi\)
+**	movdqu	48\(%rsi\), %xmm0
+**	movups	%xmm0, 48\(%rdi\)
+**	movdqu	64\(%rsi\), %xmm0
+**	movups	%xmm0, 64\(%rdi\)
+**	movdqu	80\(%rsi\), %xmm0
+**	movups	%xmm0, 80\(%rdi\)
+**	movdqu	96\(%rsi\), %xmm0
+**	movups	%xmm0, 96\(%rdi\)
+**	movdqu	112\(%rsi\), %xmm0
+**	movups	%xmm0, 112\(%rdi\)
+**	movzbl	128\(%rsi\), %eax
+**	movb	%al, 128\(%rdi\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dest, const char *src)
+{
+  __builtin_memcpy (dest, src, 129);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-1b.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-1b.c
new file mode 100644
index 00000000000..7c27f8c0c2e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-1b.c
@@ -0,0 +1,27 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -mtune=znver3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	movl	\(%rsi\), %eax
+**	movq	%rdi, %rcx
+**	leaq	4\(%rdi\), %rdi
+**	movl	%eax, -4\(%rdi\)
+**	movl	125\(%rsi\), %eax
+**	movl	%eax, 121\(%rdi\)
+**	andq	\$-4, %rdi
+**	subq	%rdi, %rcx
+**	subq	%rcx, %rsi
+**	addl	\$129, %ecx
+**	shrl	\$2, %ecx
+**	rep movsl
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memcpy-1a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-2a.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-2a.c
new file mode 100644
index 00000000000..6fe5102fe91
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-2a.c
@@ -0,0 +1,63 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$15, %rdx
+**	jbe	.L9
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	cmpl	\$8, %edx
+**	jnb	.L10
+**	cmpl	\$4, %edx
+**	jnb	.L5
+**	cmpl	\$1, %edx
+**	ja	.L6
+**	jb	.L1
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+gcc_memcpy (void *a, void *b, __SIZE_TYPE__ n)
+{
+  if (n < 16)
+    __builtin_memcpy (a, b, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-2b.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-2b.c
new file mode 100644
index 00000000000..1b0a64e5ea5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-2b.c
@@ -0,0 +1,58 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$15, %rdx
+**	jbe	.L10
+**.L8:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	cmpl	\$8, %edx
+**	jnb	.L11
+**	cmpl	\$4, %edx
+**	jnb	.L5
+**	cmpl	\$1, %edx
+**	ja	.L6
+**	jb	.L8
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memcpy-2a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-2c.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-2c.c
new file mode 100644
index 00000000000..761c377557b
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-2c.c
@@ -0,0 +1,58 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$15, %rdx
+**	jbe	.L10
+**.L8:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	cmpl	\$8, %edx
+**	jnb	.L11
+**	cmpl	\$4, %edx
+**	jnb	.L5
+**	cmpl	\$1, %edx
+**	ja	.L6
+**	jb	.L8
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memcpy-2a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-3a.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-3a.c
new file mode 100644
index 00000000000..707424163ea
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-3a.c
@@ -0,0 +1,74 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$31, %rdx
+**	jbe	.L10
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	cmpl	\$16, %edx
+**	jnb	.L11
+**	cmpl	\$8, %edx
+**	jnb	.L5
+**	cmpl	\$4, %edx
+**	jnb	.L6
+**	cmpl	\$1, %edx
+**	ja	.L7
+**	jb	.L1
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%edx, %edx
+**	movdqu	\(%rsi\), %xmm1
+**	movdqu	-16\(%rsi,%rdx\), %xmm0
+**	movups	%xmm1, \(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+gcc_memcpy (void *a, void *b, __SIZE_TYPE__ n)
+{
+  if (n < 32)
+    __builtin_memcpy (a, b, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-3b.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-3b.c
new file mode 100644
index 00000000000..65eb7d4b3d3
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-3b.c
@@ -0,0 +1,69 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$31, %rdx
+**	jbe	.L11
+**.L9:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	cmpl	\$16, %edx
+**	jnb	.L12
+**	cmpl	\$8, %edx
+**	jnb	.L5
+**	cmpl	\$4, %edx
+**	jnb	.L6
+**	cmpl	\$1, %edx
+**	ja	.L7
+**	jb	.L9
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %xmm1
+**	vmovdqu	-16\(%rsi,%rdx\), %xmm0
+**	vmovdqu	%xmm1, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memcpy-3a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-3c.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-3c.c
new file mode 100644
index 00000000000..b4df4399bfe
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-3c.c
@@ -0,0 +1,69 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$31, %rdx
+**	jbe	.L11
+**.L9:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	cmpl	\$16, %edx
+**	jnb	.L12
+**	cmpl	\$8, %edx
+**	jnb	.L5
+**	cmpl	\$4, %edx
+**	jnb	.L6
+**	cmpl	\$1, %edx
+**	ja	.L7
+**	jb	.L9
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %xmm1
+**	vmovdqu	-16\(%rsi,%rdx\), %xmm0
+**	vmovdqu	%xmm1, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memcpy-3a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-4a.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-4a.c
new file mode 100644
index 00000000000..7c349807db6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-4a.c
@@ -0,0 +1,90 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$63, %rdx
+**	jbe	.L12
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	cmpl	\$16, %edx
+**	jnb	.L13
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L1
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	cmpl	\$32, %edx
+**	ja	.L5
+**	movl	%edx, %edx
+**	movdqu	\(%rsi\), %xmm1
+**	movdqu	-16\(%rsi,%rdx\), %xmm0
+**	movups	%xmm1, \(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movdqu	\(%rsi\), %xmm3
+**	movdqu	16\(%rsi\), %xmm2
+**	addq	%rdx, %rsi
+**	movdqu	-16\(%rsi\), %xmm1
+**	movdqu	-32\(%rsi\), %xmm0
+**	movups	%xmm3, \(%rdi\)
+**	movups	%xmm2, 16\(%rdi\)
+**	movups	%xmm1, -16\(%rdi,%rdx\)
+**	movups	%xmm0, -32\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+gcc_memcpy (void *a, void *b, __SIZE_TYPE__ n)
+{
+  if (n < 64)
+    __builtin_memcpy (a, b, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-4b.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-4b.c
new file mode 100644
index 00000000000..0c95c7017c7
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-4b.c
@@ -0,0 +1,80 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$63, %rdx
+**	jbe	.L12
+**.L10:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	cmpl	\$32, %edx
+**	jnb	.L13
+**	cmpl	\$16, %edx
+**	jnb	.L5
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L10
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %ymm1
+**	vmovdqu	-32\(%rsi,%rdx\), %ymm0
+**	vmovdqu	%ymm1, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %xmm1
+**	vmovdqu	-16\(%rsi,%rdx\), %xmm0
+**	vmovdqu	%xmm1, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**...
+*/
+
+#include "builtin-memcpy-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-4c.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-4c.c
new file mode 100644
index 00000000000..d7d771a5507
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-4c.c
@@ -0,0 +1,80 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$63, %rdx
+**	jbe	.L12
+**.L10:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	cmpl	\$32, %edx
+**	jnb	.L13
+**	cmpl	\$16, %edx
+**	jnb	.L5
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L10
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %ymm1
+**	vmovdqu	-32\(%rsi,%rdx\), %ymm0
+**	vmovdqu	%ymm1, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %xmm1
+**	vmovdqu	-16\(%rsi,%rdx\), %xmm0
+**	vmovdqu	%xmm1, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**...
+*/
+
+#include "builtin-memcpy-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-5a.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-5a.c
new file mode 100644
index 00000000000..780015f6782
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-5a.c
@@ -0,0 +1,114 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$127, %rdx
+**	jbe	.L12
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	cmpl	\$16, %edx
+**	jnb	.L13
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L1
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	cmpl	\$32, %edx
+**	ja	.L5
+**	movl	%edx, %edx
+**	movdqu	\(%rsi\), %xmm1
+**	movdqu	-16\(%rsi,%rdx\), %xmm0
+**	movups	%xmm1, \(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	cmpl	\$64, %edx
+**	ja	.L14
+**	movl	%edx, %edx
+**	movdqu	\(%rsi\), %xmm3
+**	movdqu	16\(%rsi\), %xmm2
+**	addq	%rdx, %rsi
+**	movdqu	-16\(%rsi\), %xmm1
+**	movdqu	-32\(%rsi\), %xmm0
+**	movups	%xmm3, \(%rdi\)
+**	movups	%xmm2, 16\(%rdi\)
+**	movups	%xmm1, -16\(%rdi,%rdx\)
+**	movups	%xmm0, -32\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	movl	%edx, %edx
+**	movdqu	\(%rsi\), %xmm7
+**	movdqu	16\(%rsi\), %xmm6
+**	movdqu	32\(%rsi\), %xmm5
+**	movdqu	48\(%rsi\), %xmm4
+**	addq	%rdx, %rsi
+**	movdqu	-16\(%rsi\), %xmm3
+**	movdqu	-32\(%rsi\), %xmm2
+**	movdqu	-48\(%rsi\), %xmm1
+**	movdqu	-64\(%rsi\), %xmm0
+**	movups	%xmm7, \(%rdi\)
+**	movups	%xmm6, 16\(%rdi\)
+**	movups	%xmm5, 32\(%rdi\)
+**	movups	%xmm4, 48\(%rdi\)
+**	movups	%xmm3, -16\(%rdi,%rdx\)
+**	movups	%xmm2, -32\(%rdi,%rdx\)
+**	movups	%xmm1, -48\(%rdi,%rdx\)
+**	movups	%xmm0, -64\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+gcc_memcpy (void *a, void *b, __SIZE_TYPE__ n)
+{
+  if (n < 128)
+    __builtin_memcpy (a, b, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-5b.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-5b.c
new file mode 100644
index 00000000000..683740c2f7a
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-5b.c
@@ -0,0 +1,97 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$127, %rdx
+**	jbe	.L14
+**.L12:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	cmpl	\$32, %edx
+**	jnb	.L15
+**	cmpl	\$16, %edx
+**	jnb	.L6
+**	cmpl	\$8, %edx
+**	jnb	.L7
+**	cmpl	\$4, %edx
+**	jnb	.L8
+**	cmpl	\$1, %edx
+**	ja	.L9
+**	jb	.L12
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L15:
+**	cmpl	\$64, %edx
+**	ja	.L5
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %ymm1
+**	vmovdqu	-32\(%rsi,%rdx\), %ymm0
+**	vmovdqu	%ymm1, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %ymm3
+**	vmovdqu	32\(%rsi\), %ymm2
+**	addq	%rdx, %rsi
+**	vmovdqu	-32\(%rsi\), %ymm1
+**	vmovdqu	-64\(%rsi\), %ymm0
+**	vmovdqu	%ymm3, \(%rdi\)
+**	vmovdqu	%ymm2, 32\(%rdi\)
+**	vmovdqu	%ymm1, -32\(%rdi,%rdx\)
+**	vmovdqu	%ymm0, -64\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %xmm1
+**	vmovdqu	-16\(%rsi,%rdx\), %xmm0
+**	vmovdqu	%xmm1, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**...
+*/
+
+#include "builtin-memcpy-5a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memcpy-5c.c b/gcc/testsuite/gcc.target/i386/builtin-memcpy-5c.c
new file mode 100644
index 00000000000..31540fdd101
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memcpy-5c.c
@@ -0,0 +1,91 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target { lp64 } } {^\t?\.} } } */
+
+/*
+**gcc_memcpy:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$127, %rdx
+**	jbe	.L13
+**.L11:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	cmpl	\$64, %edx
+**	jnb	.L14
+**	cmpl	\$32, %edx
+**	jnb	.L5
+**	cmpl	\$16, %edx
+**	jnb	.L6
+**	cmpl	\$8, %edx
+**	jnb	.L7
+**	cmpl	\$4, %edx
+**	jnb	.L8
+**	cmpl	\$1, %edx
+**	ja	.L9
+**	jb	.L11
+**	movzbl	\(%rsi\), %eax
+**	movb	%al, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	movl	%edx, %edx
+**	vmovdqu64	\(%rsi\), %zmm1
+**	vmovdqu64	-64\(%rsi,%rdx\), %zmm0
+**	vmovdqu64	%zmm1, \(%rdi\)
+**	vmovdqu64	%zmm0, -64\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %ymm1
+**	vmovdqu	-32\(%rsi,%rdx\), %ymm0
+**	vmovdqu	%ymm1, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	vmovdqu	\(%rsi\), %xmm1
+**	vmovdqu	-16\(%rsi,%rdx\), %xmm0
+**	vmovdqu	%xmm1, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movq	\(%rsi\), %rcx
+**	movq	-8\(%rsi,%rdx\), %rax
+**	movq	%rcx, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movl	\(%rsi\), %ecx
+**	movl	-4\(%rsi,%rdx\), %eax
+**	movl	%ecx, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**.L9:
+**	movl	%edx, %edx
+**	movzwl	\(%rsi\), %ecx
+**	movzwl	-2\(%rsi,%rdx\), %eax
+**	movw	%cx, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memcpy-5a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-1a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-1a.c
new file mode 100644
index 00000000000..33f813cdbcf
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-1a.c
@@ -0,0 +1,32 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	movd	%esi, %xmm0
+**	movb	%sil, 128\(%rdi\)
+**	punpcklbw	%xmm0, %xmm0
+**	punpcklwd	%xmm0, %xmm0
+**	pshufd	\$0, %xmm0, %xmm0
+**	movups	%xmm0, \(%rdi\)
+**	movups	%xmm0, 16\(%rdi\)
+**	movups	%xmm0, 32\(%rdi\)
+**	movups	%xmm0, 48\(%rdi\)
+**	movups	%xmm0, 64\(%rdi\)
+**	movups	%xmm0, 80\(%rdi\)
+**	movups	%xmm0, 96\(%rdi\)
+**	movups	%xmm0, 112\(%rdi\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dest, int c)
+{
+  __builtin_memset (dest, c, 129);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-1b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-1b.c
new file mode 100644
index 00000000000..0e1edf200d6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-1b.c
@@ -0,0 +1,27 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -mtune=znver3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	movzbl	%sil, %eax
+**	movabsq	\$72340172838076673, %rdx
+**	movq	%rdi, %rcx
+**	leaq	8\(%rdi\), %rdi
+**	imulq	%rdx, %rax
+**	movq	%rax, -8\(%rdi\)
+**	movq	%rax, 113\(%rdi\)
+**	andq	\$-8, %rdi
+**	subq	%rdi, %rcx
+**	addl	\$129, %ecx
+**	shrl	\$3, %ecx
+**	rep stosq
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-1a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-2a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-2a.c
new file mode 100644
index 00000000000..9c72189a40d
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-2a.c
@@ -0,0 +1,59 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$15, %rdx
+**	jbe	.L9
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	cmpl	\$8, %edx
+**	jnb	.L10
+**	cmpl	\$4, %edx
+**	jnb	.L5
+**	cmpl	\$1, %edx
+**	ja	.L6
+**	jb	.L1
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dest, int c, __SIZE_TYPE__ n)
+{
+  if (n < 16)
+    __builtin_memset (dest, c, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-2b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-2b.c
new file mode 100644
index 00000000000..49a7e49dc4f
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-2b.c
@@ -0,0 +1,54 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$15, %rdx
+**	jbe	.L10
+**.L8:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	cmpl	\$8, %edx
+**	jnb	.L11
+**	cmpl	\$4, %edx
+**	jnb	.L5
+**	cmpl	\$1, %edx
+**	ja	.L6
+**	jb	.L8
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-2a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-2c.c b/gcc/testsuite/gcc.target/i386/builtin-memset-2c.c
new file mode 100644
index 00000000000..f3dfb845918
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-2c.c
@@ -0,0 +1,54 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$15, %rdx
+**	jbe	.L10
+**.L8:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	cmpl	\$8, %edx
+**	jnb	.L11
+**	cmpl	\$4, %edx
+**	jnb	.L5
+**	cmpl	\$1, %edx
+**	ja	.L6
+**	jb	.L8
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-2a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-3a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-3a.c
new file mode 100644
index 00000000000..83601dd34a5
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-3a.c
@@ -0,0 +1,70 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$31, %rdx
+**	jbe	.L10
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	movq	%rax, %xmm0
+**	punpcklqdq	%xmm0, %xmm0
+**	cmpl	\$16, %edx
+**	jnb	.L11
+**	cmpl	\$8, %edx
+**	jnb	.L5
+**	cmpl	\$4, %edx
+**	jnb	.L6
+**	cmpl	\$1, %edx
+**	ja	.L7
+**	jb	.L1
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%edx, %edx
+**	movups	%xmm0, \(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dest, int c, __SIZE_TYPE__ n)
+{
+  if (n < 32)
+    __builtin_memset (dest, c, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-3b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-3b.c
new file mode 100644
index 00000000000..25f96ada02c
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-3b.c
@@ -0,0 +1,65 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$31, %rdx
+**	jbe	.L11
+**.L9:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	vmovq	%rax, %xmm1
+**	vpunpcklqdq	%xmm1, %xmm1, %xmm0
+**	cmpl	\$16, %edx
+**	jnb	.L12
+**	cmpl	\$8, %edx
+**	jnb	.L5
+**	cmpl	\$4, %edx
+**	jnb	.L6
+**	cmpl	\$1, %edx
+**	ja	.L7
+**	jb	.L9
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	movl	%edx, %edx
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-3a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-3c.c b/gcc/testsuite/gcc.target/i386/builtin-memset-3c.c
new file mode 100644
index 00000000000..cce2f5062dd
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-3c.c
@@ -0,0 +1,11 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**...
+*/
+
+#include "builtin-memset-3a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-4a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-4a.c
new file mode 100644
index 00000000000..e80d31dc71e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-4a.c
@@ -0,0 +1,80 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$63, %rdx
+**	jbe	.L12
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	movq	%rax, %xmm0
+**	punpcklqdq	%xmm0, %xmm0
+**	cmpl	\$16, %edx
+**	jnb	.L13
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L1
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	movups	%xmm0, \(%rdi\)
+**	cmpl	\$32, %edx
+**	ja	.L5
+**	movl	%edx, %edx
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	movups	%xmm0, 16\(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	movups	%xmm0, -32\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dest, int c, __SIZE_TYPE__ n)
+{
+  if (n < 64)
+    __builtin_memset (dest, c, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-4b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-4b.c
new file mode 100644
index 00000000000..df566f25a35
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-4b.c
@@ -0,0 +1,82 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$63, %rdx
+**	jbe	.L13
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	vmovq	%rax, %xmm1
+**	vpbroadcastq	%xmm1, %ymm0
+**	cmpl	\$32, %edx
+**	jnb	.L14
+**	cmpl	\$16, %edx
+**	jnb	.L5
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L10
+**	movb	%sil, \(%rdi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	movl	%edx, %edx
+**	vmovdqu	%ymm0, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**.L10:
+**	vzeroupper
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-4c.c b/gcc/testsuite/gcc.target/i386/builtin-memset-4c.c
new file mode 100644
index 00000000000..965a299f0db
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-4c.c
@@ -0,0 +1,81 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$63, %rdx
+**	jbe	.L13
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	vpbroadcastq	%rax, %ymm0
+**	cmpl	\$32, %edx
+**	jnb	.L14
+**	cmpl	\$16, %edx
+**	jnb	.L5
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L10
+**	movb	%sil, \(%rdi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	movl	%edx, %edx
+**	vmovdqu	%ymm0, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**.L10:
+**	vzeroupper
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-4a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-5a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-5a.c
new file mode 100644
index 00000000000..d66f9e96742
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-5a.c
@@ -0,0 +1,93 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$127, %rdx
+**	jbe	.L12
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L12:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	movq	%rax, %xmm0
+**	punpcklqdq	%xmm0, %xmm0
+**	cmpl	\$16, %edx
+**	jnb	.L13
+**	cmpl	\$8, %edx
+**	jnb	.L6
+**	cmpl	\$4, %edx
+**	jnb	.L7
+**	cmpl	\$1, %edx
+**	ja	.L8
+**	jb	.L1
+**	movb	%sil, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L13:
+**	movups	%xmm0, \(%rdi\)
+**	cmpl	\$32, %edx
+**	ja	.L5
+**	movl	%edx, %edx
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movups	%xmm0, 16\(%rdi\)
+**	cmpl	\$64, %edx
+**	ja	.L14
+**	movl	%edx, %edx
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	movups	%xmm0, -32\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	movl	%edx, %edx
+**	movups	%xmm0, 32\(%rdi\)
+**	movups	%xmm0, 48\(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rdx\)
+**	movups	%xmm0, -32\(%rdi,%rdx\)
+**	movups	%xmm0, -48\(%rdi,%rdx\)
+**	movups	%xmm0, -64\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dest, int c, __SIZE_TYPE__ n)
+{
+  if (n < 128)
+    __builtin_memset (dest, c, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-5b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-5b.c
new file mode 100644
index 00000000000..a6a04cd2651
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-5b.c
@@ -0,0 +1,93 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$127, %rdx
+**	jbe	.L15
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L15:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	vmovq	%rax, %xmm1
+**	vpbroadcastq	%xmm1, %ymm0
+**	cmpl	\$32, %edx
+**	jnb	.L16
+**	cmpl	\$16, %edx
+**	jnb	.L6
+**	cmpl	\$8, %edx
+**	jnb	.L7
+**	cmpl	\$4, %edx
+**	jnb	.L8
+**	cmpl	\$1, %edx
+**	ja	.L9
+**	jb	.L12
+**	movb	%sil, \(%rdi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	vmovdqu	%ymm0, \(%rdi\)
+**	cmpl	\$64, %edx
+**	ja	.L5
+**	movl	%edx, %edx
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	%ymm0, 32\(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vmovdqu	%ymm0, -64\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**.L12:
+**	vzeroupper
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-5a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-5c.c b/gcc/testsuite/gcc.target/i386/builtin-memset-5c.c
new file mode 100644
index 00000000000..7e91aa865e6
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-5c.c
@@ -0,0 +1,89 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$127, %rdx
+**	jbe	.L14
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	movabsq	\$72340172838076673, %rcx
+**	movzbl	%sil, %eax
+**	imulq	%rcx, %rax
+**	vpbroadcastq	%rax, %zmm0
+**	cmpl	\$64, %edx
+**	jnb	.L15
+**	cmpl	\$32, %edx
+**	jnb	.L5
+**	cmpl	\$16, %edx
+**	jnb	.L6
+**	cmpl	\$8, %edx
+**	jnb	.L7
+**	cmpl	\$4, %edx
+**	jnb	.L8
+**	cmpl	\$1, %edx
+**	ja	.L9
+**	jb	.L11
+**	movb	%sil, \(%rdi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L15:
+**	movl	%edx, %edx
+**	vmovdqu8	%zmm0, \(%rdi\)
+**	vmovdqu8	%zmm0, -64\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%edx, %edx
+**	vmovdqu	%ymm0, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%edx, %edx
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%edx, %edx
+**	movq	%rax, \(%rdi\)
+**	movq	%rax, -8\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%edx, %edx
+**	movl	%eax, \(%rdi\)
+**	movl	%eax, -4\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**.L9:
+**	movl	%edx, %edx
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rdx\)
+**	vzeroupper
+**	ret
+**.L11:
+**	vzeroupper
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-5a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-6a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-6a.c
new file mode 100644
index 00000000000..f4a76ade2d1
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-6a.c
@@ -0,0 +1,109 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$247, %rsi
+**	jbe	.L16
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	movq	%rdi, %rax
+**	cmpl	\$16, %esi
+**	jnb	.L17
+**	cmpl	\$8, %esi
+**	jnb	.L6
+**	cmpl	\$4, %esi
+**	jnb	.L7
+**	cmpl	\$1, %esi
+**	ja	.L8
+**	jb	.L1
+**	movb	\$0, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L17:
+**	pxor	%xmm0, %xmm0
+**	cmpl	\$32, %esi
+**	ja	.L5
+**	movl	%esi, %esi
+**	movups	%xmm0, \(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	cmpl	\$128, %esi
+**	ja	.L10
+**	movups	%xmm0, \(%rdi\)
+**	movups	%xmm0, 16\(%rdi\)
+**	cmpl	\$64, %esi
+**	jbe	.L11
+**	movups	%xmm0, 32\(%rdi\)
+**	movups	%xmm0, 48\(%rdi\)
+**.L14:
+**	movl	%esi, %esi
+**	movups	%xmm0, -16\(%rdi,%rsi\)
+**	movups	%xmm0, -32\(%rdi,%rsi\)
+**	movups	%xmm0, -48\(%rdi,%rsi\)
+**	movups	%xmm0, -64\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%esi, %esi
+**	movq	\$0, \(%rdi\)
+**	movq	\$0, -8\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movl	%esi, %edx
+**.L12:
+**	subl	\$64, %edx
+**	movups	%xmm0, \(%rax\)
+**	addq	\$64, %rax
+**	movups	%xmm0, -48\(%rax\)
+**	movups	%xmm0, -32\(%rax\)
+**	movups	%xmm0, -16\(%rax\)
+**	cmpl	\$64, %edx
+**	ja	.L12
+**	jmp	.L14
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%esi, %esi
+**	movl	\$0, \(%rdi\)
+**	movl	\$0, -4\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%esi, %esi
+**	movups	%xmm0, -16\(%rdi,%rsi\)
+**	movups	%xmm0, -32\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	xorl	%eax, %eax
+**	movl	%esi, %esi
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rsi\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+void
+foo (char *dst, __SIZE_TYPE__ n)
+{
+  if (n <= 247)
+    __builtin_memset(dst, 0, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-6b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-6b.c
new file mode 100644
index 00000000000..75b8d40a09f
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-6b.c
@@ -0,0 +1,97 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$247, %rsi
+**	jbe	.L14
+**.L12:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L14:
+**	vpxor	%xmm0, %xmm0, %xmm0
+**	cmpl	\$32, %esi
+**	jnb	.L15
+**	cmpl	\$16, %esi
+**	jnb	.L6
+**	cmpl	\$8, %esi
+**	jnb	.L7
+**	cmpl	\$4, %esi
+**	jnb	.L8
+**	cmpl	\$1, %esi
+**	ja	.L9
+**	jb	.L12
+**	movb	\$0, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L15:
+**	vmovdqu	%ymm0, \(%rdi\)
+**	cmpl	\$64, %esi
+**	ja	.L5
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	vmovdqu	%ymm0, 32\(%rdi\)
+**	cmpl	\$128, %esi
+**	ja	.L16
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -64\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%esi, %esi
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, 64\(%rdi\)
+**	vmovdqu	%ymm0, 96\(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -64\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -96\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -128\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%esi, %esi
+**	movq	\$0, \(%rdi\)
+**	movq	\$0, -8\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%esi, %esi
+**	movl	\$0, \(%rdi\)
+**	movl	\$0, -4\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	xorl	%eax, %eax
+**	movl	%esi, %esi
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rsi\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-6a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-6c.c b/gcc/testsuite/gcc.target/i386/builtin-memset-6c.c
new file mode 100644
index 00000000000..d93a518bbe2
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-6c.c
@@ -0,0 +1,91 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$247, %rsi
+**	jbe	.L15
+**.L13:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L15:
+**	vpxor	%xmm0, %xmm0, %xmm0
+**	cmpl	\$64, %esi
+**	jnb	.L16
+**	cmpl	\$32, %esi
+**	jnb	.L6
+**	cmpl	\$16, %esi
+**	jnb	.L7
+**	cmpl	\$8, %esi
+**	jnb	.L8
+**	cmpl	\$4, %esi
+**	jnb	.L9
+**	cmpl	\$1, %esi
+**	ja	.L10
+**	jb	.L13
+**	movb	\$0, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	vmovdqu8	%zmm0, \(%rdi\)
+**	cmpl	\$128, %esi
+**	ja	.L5
+**	movl	%esi, %esi
+**	vmovdqu8	%zmm0, -64\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%esi, %esi
+**	vmovdqu8	%zmm0, 64\(%rdi\)
+**	vmovdqu8	%zmm0, -64\(%rdi,%rsi\)
+**	vmovdqu8	%zmm0, -128\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%esi, %esi
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%esi, %esi
+**	movq	\$0, \(%rdi\)
+**	movq	\$0, -8\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	movl	%esi, %esi
+**	movl	\$0, \(%rdi\)
+**	movl	\$0, -4\(%rdi,%rsi\)
+**	ret
+**.L10:
+**	xorl	%eax, %eax
+**	movl	%esi, %esi
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rsi\)
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-6a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-7a.c b/gcc/testsuite/gcc.target/i386/builtin-memset-7a.c
new file mode 100644
index 00000000000..ca7ffba0b23
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-7a.c
@@ -0,0 +1,108 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$247, %rsi
+**	jbe	.L16
+**.L1:
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	movq	%rdi, %rax
+**	cmpl	\$16, %esi
+**	jnb	.L17
+**	cmpl	\$8, %esi
+**	jnb	.L6
+**	cmpl	\$4, %esi
+**	jnb	.L7
+**	cmpl	\$1, %esi
+**	ja	.L8
+**	jb	.L1
+**	movb	\$-1, \(%rdi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L17:
+**	pcmpeqd	%xmm0, %xmm0
+**	cmpl	\$32, %esi
+**	ja	.L5
+**	movl	%esi, %esi
+**	movups	%xmm0, \(%rdi\)
+**	movups	%xmm0, -16\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	cmpl	\$128, %esi
+**	ja	.L10
+**	movups	%xmm0, \(%rdi\)
+**	movups	%xmm0, 16\(%rdi\)
+**	cmpl	\$64, %esi
+**	jbe	.L11
+**	movups	%xmm0, 32\(%rdi\)
+**	movups	%xmm0, 48\(%rdi\)
+**.L14:
+**	movl	%esi, %esi
+**	movups	%xmm0, -16\(%rdi,%rsi\)
+**	movups	%xmm0, -32\(%rdi,%rsi\)
+**	movups	%xmm0, -48\(%rdi,%rsi\)
+**	movups	%xmm0, -64\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%esi, %esi
+**	movq	\$-1, \(%rdi\)
+**	movq	\$-1, -8\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L10:
+**	movl	%esi, %edx
+**.L12:
+**	subl	\$64, %edx
+**	movups	%xmm0, \(%rax\)
+**	addq	\$64, %rax
+**	movups	%xmm0, -48\(%rax\)
+**	movups	%xmm0, -32\(%rax\)
+**	movups	%xmm0, -16\(%rax\)
+**	cmpl	\$64, %edx
+**	ja	.L12
+**	jmp	.L14
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%esi, %esi
+**	movl	\$-1, \(%rdi\)
+**	movl	\$-1, -4\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L11:
+**	movl	%esi, %esi
+**	movups	%xmm0, -16\(%rdi,%rsi\)
+**	movups	%xmm0, -32\(%rdi,%rsi\)
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	\$-1, %eax
+**	movl	%esi, %esi
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rsi\)
+**	ret
+**...
+*/
+
+void
+foo (char *dst, __SIZE_TYPE__ n)
+{
+  if (n <= 247)
+    __builtin_memset(dst, -1, n);
+}
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-7b.c b/gcc/testsuite/gcc.target/i386/builtin-memset-7b.c
new file mode 100644
index 00000000000..1054e5f0c2e
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-7b.c
@@ -0,0 +1,103 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v3 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$247, %rsi
+**	jbe	.L15
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L15:
+**	vpcmpeqd	%ymm0, %ymm0, %ymm0
+**	cmpl	\$32, %esi
+**	jnb	.L16
+**	cmpl	\$16, %esi
+**	jnb	.L6
+**	cmpl	\$8, %esi
+**	jnb	.L7
+**	cmpl	\$4, %esi
+**	jnb	.L8
+**	cmpl	\$1, %esi
+**	ja	.L9
+**	jb	.L12
+**	movb	\$-1, \(%rdi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	vmovdqu	%ymm0, \(%rdi\)
+**	cmpl	\$64, %esi
+**	ja	.L5
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	vmovdqu	%ymm0, 32\(%rdi\)
+**	cmpl	\$128, %esi
+**	ja	.L17
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -64\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%esi, %esi
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L17:
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, 64\(%rdi\)
+**	vmovdqu	%ymm0, 96\(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -64\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -96\(%rdi,%rsi\)
+**	vmovdqu	%ymm0, -128\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%esi, %esi
+**	movq	\$-1, \(%rdi\)
+**	movq	\$-1, -8\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%esi, %esi
+**	movl	\$-1, \(%rdi\)
+**	movl	\$-1, -4\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	movl	\$-1, %eax
+**	movl	%esi, %esi
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**.L12:
+**	vzeroupper
+**	ret
+**...
+*/
+
+#include "builtin-memset-7a.c"
diff --git a/gcc/testsuite/gcc.target/i386/builtin-memset-7c.c b/gcc/testsuite/gcc.target/i386/builtin-memset-7c.c
new file mode 100644
index 00000000000..6c99eafea20
--- /dev/null
+++ b/gcc/testsuite/gcc.target/i386/builtin-memset-7c.c
@@ -0,0 +1,99 @@
+/* { dg-do compile } */
+/* { dg-options "-O2 -march=x86-64-v4 -minline-all-stringops" } */
+/* Keep labels and directives ('.cfi_startproc', '.cfi_endproc').  */
+/* { dg-final { check-function-bodies "**" "" "" { target lp64 } {^\t?\.} } } */
+
+/*
+**foo:
+**.LFB0:
+**	.cfi_startproc
+**	cmpq	\$247, %rsi
+**	jbe	.L16
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L16:
+**	vpxor	%xmm0, %xmm0, %xmm0
+**	vpternlogd	\$0xFF, %zmm0, %zmm0, %zmm0
+**	cmpl	\$64, %esi
+**	jnb	.L17
+**	cmpl	\$32, %esi
+**	jnb	.L6
+**	cmpl	\$16, %esi
+**	jnb	.L7
+**	cmpl	\$8, %esi
+**	jnb	.L8
+**	cmpl	\$4, %esi
+**	jnb	.L9
+**	cmpl	\$1, %esi
+**	ja	.L10
+**	jb	.L13
+**	movb	\$-1, \(%rdi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L17:
+**	vmovdqu8	%zmm0, \(%rdi\)
+**	cmpl	\$128, %esi
+**	ja	.L5
+**	movl	%esi, %esi
+**	vmovdqu8	%zmm0, -64\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L5:
+**	movl	%esi, %esi
+**	vmovdqu8	%zmm0, 64\(%rdi\)
+**	vmovdqu8	%zmm0, -64\(%rdi,%rsi\)
+**	vmovdqu8	%zmm0, -128\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L6:
+**	movl	%esi, %esi
+**	vmovdqu	%ymm0, \(%rdi\)
+**	vmovdqu	%ymm0, -32\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L7:
+**	movl	%esi, %esi
+**	vmovdqu	%xmm0, \(%rdi\)
+**	vmovdqu	%xmm0, -16\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L8:
+**	movl	%esi, %esi
+**	movq	\$-1, \(%rdi\)
+**	movq	\$-1, -8\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**	.p2align 4,,10
+**	.p2align 3
+**.L9:
+**	movl	%esi, %esi
+**	movl	\$-1, \(%rdi\)
+**	movl	\$-1, -4\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**.L10:
+**	movl	\$-1, %eax
+**	movl	%esi, %esi
+**	movw	%ax, \(%rdi\)
+**	movw	%ax, -2\(%rdi,%rsi\)
+**	vzeroupper
+**	ret
+**.L13:
+**	vzeroupper
+**	ret
+**	.cfi_endproc
+**...
+*/
+
+#include "builtin-memset-7a.c"
-- 
2.54.0