[llvm-branch-commits] [llvm] 9946b16 - [lit] Use os.cpu_count() to cleanup TODO
Author: Julian Lettner
Date: 2021-01-25T11:44:18-08:00
New Revision: 9946b169c379daee603436a4753acfef8be373dd
URL:
https://github.com/llvm/llvm-project/commit/9946b169c379daee603436a4753acfef8be373dd
DIFF:
https://github.com/llvm/llvm-project/commit/9946b169c379daee603436a4753acfef8be373dd.diff
LOG: [lit] Use os.cpu_count() to cleanup TODO
We can now use Python3. Let's use `os.cpu_count()` to cleanup this
helper.
Differential Revision: https://reviews.llvm.org/D94734
Added:
Modified:
llvm/utils/lit/lit/cl_arguments.py
llvm/utils/lit/lit/run.py
llvm/utils/lit/lit/util.py
Removed:
diff --git a/llvm/utils/lit/lit/cl_arguments.py
b/llvm/utils/lit/lit/cl_arguments.py
index ce49c3c48a97..ddb4d25cbba1 100644
--- a/llvm/utils/lit/lit/cl_arguments.py
+++ b/llvm/utils/lit/lit/cl_arguments.py
@@ -23,7 +23,7 @@ def parse_args():
metavar="N",
help="Number of workers used for testing",
type=_positive_int,
-default=lit.util.detectCPUs())
+default=lit.util.usable_core_count())
parser.add_argument("--config-prefix",
dest="configPrefix",
metavar="NAME",
diff --git a/llvm/utils/lit/lit/run.py b/llvm/utils/lit/lit/run.py
index 5aef77e5f605..255e63329432 100644
--- a/llvm/utils/lit/lit/run.py
+++ b/llvm/utils/lit/lit/run.py
@@ -110,7 +110,7 @@ def _update_test(self, local_test, remote_test):
# threads counts toward the current process limit. Try to raise the (soft)
# process limit so that tests don't fail due to resource exhaustion.
def _increase_process_limit(self):
-ncpus = lit.util.detectCPUs()
+ncpus = lit.util.usable_core_count()
desired_limit = self.workers * ncpus * 2 # the 2 is a safety factor
# Importing the resource module will likely fail on Windows.
diff --git a/llvm/utils/lit/lit/util.py b/llvm/utils/lit/lit/util.py
index d7afbdabcff9..0d8f0040f41c 100644
--- a/llvm/utils/lit/lit/util.py
+++ b/llvm/utils/lit/lit/util.py
@@ -109,32 +109,23 @@ def to_unicode(s):
return s
-# TODO(yln): multiprocessing.cpu_count()
-# TODO(python3): len(os.sched_getaffinity(0)) and os.cpu_count()
-def detectCPUs():
-"""Detects the number of CPUs on a system.
-
-Cribbed from pp.
+def usable_core_count():
+"""Return the number of cores the current process can use, if supported.
+Otherwise, return the total number of cores (like `os.cpu_count()`).
+Default to 1 if undetermined.
"""
-# Linux, Unix and MacOS:
-if hasattr(os, 'sysconf'):
-if 'SC_NPROCESSORS_ONLN' in os.sysconf_names:
-# Linux & Unix:
-ncpus = os.sysconf('SC_NPROCESSORS_ONLN')
-if isinstance(ncpus, int) and ncpus > 0:
-return ncpus
-else: # OSX:
-return int(subprocess.check_output(['sysctl', '-n', 'hw.ncpu'],
- stderr=subprocess.STDOUT))
-# Windows:
-if 'NUMBER_OF_PROCESSORS' in os.environ:
-ncpus = int(os.environ['NUMBER_OF_PROCESSORS'])
-if ncpus > 0:
-# With more than 32 processes, process creation often fails with
-# "Too many open files". FIXME: Check if there's a better fix.
-return min(ncpus, 32)
-return 1 # Default
+try:
+n = len(os.sched_getaffinity(0))
+except AttributeError:
+n = os.cpu_count() or 1
+
+# On Windows, with more than 32 processes, process creation often fails
with
+# "Too many open files". FIXME: Check if there's a better fix.
+if platform.system() == 'Windows':
+return min(n, 32)
+
+return n
def mkdir(path):
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [compiler-rt] 8f5ec45 - [Sanitizer][Darwin] Fix test for macOS 11+ point releases
Author: Julian Lettner
Date: 2021-01-12T15:23:43-08:00
New Revision: 8f5ec4593754a58a4feb835a9d44d59c655bd0d1
URL:
https://github.com/llvm/llvm-project/commit/8f5ec4593754a58a4feb835a9d44d59c655bd0d1
DIFF:
https://github.com/llvm/llvm-project/commit/8f5ec4593754a58a4feb835a9d44d59c655bd0d1.diff
LOG: [Sanitizer][Darwin] Fix test for macOS 11+ point releases
This test wrongly asserted that the minor version is always 0 when
running on macOS 11 and above.
Added:
Modified:
compiler-rt/lib/sanitizer_common/tests/sanitizer_mac_test.cpp
Removed:
diff --git a/compiler-rt/lib/sanitizer_common/tests/sanitizer_mac_test.cpp
b/compiler-rt/lib/sanitizer_common/tests/sanitizer_mac_test.cpp
index 090947eceb4a..b149567949b5 100644
--- a/compiler-rt/lib/sanitizer_common/tests/sanitizer_mac_test.cpp
+++ b/compiler-rt/lib/sanitizer_common/tests/sanitizer_mac_test.cpp
@@ -56,12 +56,18 @@ TEST(SanitizerMac, GetMacosAlignedVersion) {
#else
TEST(SanitizerMac, GetMacosAlignedVersion) {
MacosVersion vers = GetMacosAlignedVersion();
- u16 kernel_major = GetDarwinKernelVersion().major;
- bool macos_11 = (kernel_major >= 20);
- u16 expected_major = macos_11 ? (kernel_major - 9) : 10;
- u16 expected_minor = macos_11 ? 0 : (kernel_major - 4);
- EXPECT_EQ(vers.major, expected_major);
- EXPECT_EQ(vers.minor, expected_minor);
+ std::ostringstream oss;
+ oss << vers.major << '.' << vers.minor;
+ std::string actual = oss.str();
+
+ char buf[100];
+ size_t len = sizeof(buf);
+ int res = sysctlbyname("kern.osproductversion", buf, &len, nullptr, 0);
+ ASSERT_EQ(res, KERN_SUCCESS);
+ std::string expected(buf);
+
+ // Prefix match
+ ASSERT_EQ(expected.compare(0, actual.size(), actual), 0);
}
#endif
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [compiler-rt] 84de4fa - GetMacosAlignedVersion() fails if sysctl is not setup
Author: Julian Lettner
Date: 2021-01-15T11:42:25-08:00
New Revision: 84de4faf4cae2885056c608db8256e9f039050b3
URL:
https://github.com/llvm/llvm-project/commit/84de4faf4cae2885056c608db8256e9f039050b3
DIFF:
https://github.com/llvm/llvm-project/commit/84de4faf4cae2885056c608db8256e9f039050b3.diff
LOG: GetMacosAlignedVersion() fails if sysctl is not setup
`GetMacosAlignedVersion()` fails for ASan-ified launchd because the
sanitizer initialization code runs before `sysctl` has been setup by
launchd. In this situation, `sysctl kern.osproductversion` returns a
non-empty string that does not match our expectations of a
well-formatted version string.
Retrieving the kernel version (via `sysctl kern.osrelease`) still works,
so we can use it to add a fallback for this corner case.
Differential Revision: https://reviews.llvm.org/D94190
Added:
Modified:
compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
Removed:
diff --git a/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
b/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
index 6fe6991bc816..2b53d7d730d7 100644
--- a/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
+++ b/compiler-rt/lib/sanitizer_common/sanitizer_mac.cpp
@@ -620,6 +620,23 @@ constexpr u16 GetOSMajorKernelOffset() {
using VersStr = char[64];
+static uptr ApproximateOSVersionViaKernelVersion(VersStr vers) {
+ u16 kernel_major = GetDarwinKernelVersion().major;
+ u16 offset = GetOSMajorKernelOffset();
+ CHECK_GE(kernel_major, offset);
+ u16 os_major = kernel_major - offset;
+
+ const char *format = "%d.0";
+ if (TARGET_OS_OSX) {
+if (os_major >= 16) { // macOS 11+
+ os_major -= 5;
+} else { // macOS 10.15 and below
+ format = "10.%d";
+}
+ }
+ return internal_snprintf(vers, sizeof(VersStr), format, os_major);
+}
+
static void GetOSVersion(VersStr vers) {
uptr len = sizeof(VersStr);
if (SANITIZER_IOSSIM) {
@@ -633,17 +650,19 @@ static void GetOSVersion(VersStr vers) {
} else {
int res =
internal_sysctlbyname("kern.osproductversion", vers, &len, nullptr, 0);
-if (res) {
- // Fallback for XNU 17 (macOS 10.13) and below that do not provide the
- // `kern.osproductversion` property.
- u16 kernel_major = GetDarwinKernelVersion().major;
- u16 offset = GetOSMajorKernelOffset();
- CHECK_LE(kernel_major, 17);
- CHECK_GE(kernel_major, offset);
- u16 os_major = kernel_major - offset;
-
- auto format = TARGET_OS_OSX ? "10.%d" : "%d.0";
- len = internal_snprintf(vers, len, format, os_major);
+
+// XNU 17 (macOS 10.13) and below do not provide the sysctl
+// `kern.osproductversion` entry (res != 0).
+bool no_os_version = res != 0;
+
+// For launchd, sanitizer initialization runs before sysctl is setup
+// (res == 0 && len != strlen(vers), vers is not a valid version).
However,
+// the kernel version `kern.osrelease` is available.
+bool launchd = (res == 0 && internal_strlen(vers) < 3);
+if (launchd) CHECK_EQ(internal_getpid(), 1);
+
+if (no_os_version || launchd) {
+ len = ApproximateOSVersionViaKernelVersion(vers);
}
}
CHECK_LT(len, sizeof(VersStr));
@@ -681,7 +700,7 @@ static void MapToMacos(u16 *major, u16 *minor) {
}
static MacosVersion GetMacosAlignedVersionInternal() {
- VersStr vers;
+ VersStr vers = {};
GetOSVersion(vers);
u16 major, minor;
@@ -707,7 +726,7 @@ MacosVersion GetMacosAlignedVersion() {
}
DarwinKernelVersion GetDarwinKernelVersion() {
- VersStr vers;
+ VersStr vers = {};
uptr len = sizeof(VersStr);
int res = internal_sysctlbyname("kern.osrelease", vers, &len, nullptr, 0);
CHECK_EQ(res, 0);
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [llvm] 975086b - Remove obsolete TODOs
Author: Julian Lettner
Date: 2021-01-22T12:03:03-08:00
New Revision: 975086b10a6f1ce5a9b78783f085c0da454c30bc
URL:
https://github.com/llvm/llvm-project/commit/975086b10a6f1ce5a9b78783f085c0da454c30bc
DIFF:
https://github.com/llvm/llvm-project/commit/975086b10a6f1ce5a9b78783f085c0da454c30bc.diff
LOG: Remove obsolete TODOs
Remove a few of my own TODOs that I will not have time to fix from lit
code.
Added:
Modified:
llvm/utils/lit/lit/cl_arguments.py
Removed:
diff --git a/llvm/utils/lit/lit/cl_arguments.py
b/llvm/utils/lit/lit/cl_arguments.py
index 591d4f9aaafb..ce49c3c48a97 100644
--- a/llvm/utils/lit/lit/cl_arguments.py
+++ b/llvm/utils/lit/lit/cl_arguments.py
@@ -116,7 +116,7 @@ def parse_args():
dest="maxIndividualTestTime",
help="Maximum time to spend running a single test (in seconds). "
"0 means no time limit. [Default: 0]",
-type=_non_negative_int) # TODO(yln): --[no-]test-timeout, instead
of 0 allowed
+type=_non_negative_int)
execution_group.add_argument("--max-failures",
help="Stop execution after the given number of failures.",
type=_positive_int)
@@ -135,15 +135,15 @@ def parse_args():
metavar="N",
help="Maximum number of tests to run",
type=_positive_int)
-selection_group.add_argument("--max-time", #TODO(yln): --timeout
+selection_group.add_argument("--max-time",
dest="timeout",
metavar="N",
help="Maximum time to spend testing (in seconds)",
type=_positive_int)
-selection_group.add_argument("--shuffle", # TODO(yln): --order=random
-help="Run tests in random order", # default or 'by-path' (+
isEarlyTest())
+selection_group.add_argument("--shuffle",
+help="Run tests in random order",
action="store_true")
-selection_group.add_argument("-i", "--incremental", # TODO(yln):
--order=failing-first
+selection_group.add_argument("-i", "--incremental",
help="Run modified and failing tests first (updates mtimes)",
action="store_true")
selection_group.add_argument("--filter",
@@ -151,7 +151,7 @@ def parse_args():
type=_case_insensitive_regex,
help="Only run tests with paths matching the given regular
expression",
default=os.environ.get("LIT_FILTER", ".*"))
-selection_group.add_argument("--num-shards", # TODO(yln): --shards N/M
+selection_group.add_argument("--num-shards",
dest="numShards",
metavar="M",
help="Split testsuite into M pieces and only run one",
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [compiler-rt] 8b0bd54 - [sanitizer][Darwin] Suppress -Wno-non-virtual-dtor warning
Author: Julian Lettner
Date: 2021-01-05T17:09:18-08:00
New Revision: 8b0bd54d0ec968df28ccc58bbb537a7b7c074ef2
URL:
https://github.com/llvm/llvm-project/commit/8b0bd54d0ec968df28ccc58bbb537a7b7c074ef2
DIFF:
https://github.com/llvm/llvm-project/commit/8b0bd54d0ec968df28ccc58bbb537a7b7c074ef2.diff
LOG: [sanitizer][Darwin] Suppress -Wno-non-virtual-dtor warning
Suppress the warning:
```
'fake_shared_weak_count' has virtual functions but non-virtual destructor
[-Wnon-virtual-dtor]
```
The warning has been recently enabled [1], but the associated cleanup
missed this instance in Darwin code [2].
[1] 9c31e12609e1935eb84a2497ac08a49e3139859a
[2] d48f2d7c02743571075bb7812bb4c9e634e51ed1
Differential Revision: https://reviews.llvm.org/D94139
Added:
Modified:
compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
Removed:
diff --git a/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
b/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
index aa29536d8616..ed10fccc980a 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
@@ -438,6 +438,7 @@ struct fake_shared_weak_count {
virtual void on_zero_shared() = 0;
virtual void _unused_0x18() = 0;
virtual void on_zero_shared_weak() = 0;
+ virtual ~fake_shared_weak_count() = 0; // suppress -Wnon-virtual-dtor
};
} // namespace
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [compiler-rt] 10904dd - [TSan] Refactor ExternalAccess() to avoid unnecessary pop/push tag [NFC]
Author: Julian Lettner
Date: 2023-03-22T16:38:11-07:00
New Revision: 10904ddca3ce5471c159fe7e6b7441b0a0443d49
URL:
https://github.com/llvm/llvm-project/commit/10904ddca3ce5471c159fe7e6b7441b0a0443d49
DIFF:
https://github.com/llvm/llvm-project/commit/10904ddca3ce5471c159fe7e6b7441b0a0443d49.diff
LOG: [TSan] Refactor ExternalAccess() to avoid unnecessary pop/push tag [NFC]
* Avoid unnecessary frame & tag push/pops if memory access is ignored
* Rename function and add comment to make it clearer what the code does
* Make helper functions static and move inside `#if !SANITIZER_GO`
Differential Revision: https://reviews.llvm.org/D146670
Added:
Modified:
compiler-rt/lib/tsan/rtl/tsan_external.cpp
Removed:
diff --git a/compiler-rt/lib/tsan/rtl/tsan_external.cpp
b/compiler-rt/lib/tsan/rtl/tsan_external.cpp
index 463f32d7fdc31..98abff54e2b28 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_external.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_external.cpp
@@ -46,10 +46,6 @@ const char *GetReportHeaderFromTag(uptr tag) {
return tag_data ? tag_data->header : nullptr;
}
-void InsertShadowStackFrameForTag(ThreadState *thr, uptr tag) {
- FuncEntry(thr, (uptr)®istered_tags[tag]);
-}
-
uptr TagFromShadowStackFrame(uptr pc) {
uptr tag_count = atomic_load(&used_tags, memory_order_relaxed);
void *pc_ptr = (void *)pc;
@@ -60,16 +56,26 @@ uptr TagFromShadowStackFrame(uptr pc) {
#if !SANITIZER_GO
-void ExternalAccess(void *addr, uptr caller_pc, uptr tsan_caller_pc, void *tag,
-AccessType typ) {
+// We need to track tags for individual memory accesses, but there is no space
+// in the shadow cells for them. Instead we push/pop them onto the thread
+// traces and ignore the extra tag frames when printing reports.
+static void PushTag(ThreadState *thr, uptr tag) {
+ FuncEntry(thr, (uptr)®istered_tags[tag]);
+}
+static void PopTag(ThreadState *thr) { FuncExit(thr); }
+
+static void ExternalAccess(void *addr, uptr caller_pc, uptr tsan_caller_pc,
+ void *tag, AccessType typ) {
CHECK_LT(tag, atomic_load(&used_tags, memory_order_relaxed));
+ bool in_ignored_lib;
+ if (caller_pc && libignore()->IsIgnored(caller_pc, &in_ignored_lib))
+return;
+
ThreadState *thr = cur_thread();
if (caller_pc) FuncEntry(thr, caller_pc);
- InsertShadowStackFrameForTag(thr, (uptr)tag);
- bool in_ignored_lib;
- if (!caller_pc || !libignore()->IsIgnored(caller_pc, &in_ignored_lib))
-MemoryAccess(thr, tsan_caller_pc, (uptr)addr, 1, typ);
- FuncExit(thr);
+ PushTag(thr, (uptr)tag);
+ MemoryAccess(thr, tsan_caller_pc, (uptr)addr, 1, typ);
+ PopTag(thr);
if (caller_pc) FuncExit(thr);
}
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
[llvm-branch-commits] [compiler-rt] c32d809 - [TSan] Ensure we can compile the runtime with older SDKs
Author: Julian Lettner
Date: 2020-02-05T10:57:21-08:00
New Revision: c32d809e9cae8da7d3016b6cb30e2a2a9c9e2762
URL:
https://github.com/llvm/llvm-project/commit/c32d809e9cae8da7d3016b6cb30e2a2a9c9e2762
DIFF:
https://github.com/llvm/llvm-project/commit/c32d809e9cae8da7d3016b6cb30e2a2a9c9e2762.diff
LOG: [TSan] Ensure we can compile the runtime with older SDKs
One of my changes [1] included in this release silently bumped the
minimal macOS SDK required for building the TSan runtime to SDK 10.12.
Let's ensure release 10 does not unexpectedly break builders with old
SDKs and add proper minimal SDK checking in CMake for subsequent
releases.
This fix `#ifdef`s out interceptors for newer APIs. Note that the
resulting TSan runtime is less complete: when these newer APIs are used
TSan will report false positives.
Fixes llvm 10 release blocker: #44682
https://bugs.llvm.org/show_bug.cgi?id=44682
[1] 894abb46f891cba2e0ef581650f27f512a7824b4
Reviewed By: dmajor
Differential Revision: https://reviews.llvm.org/D74059
Added:
Modified:
compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
Removed:
diff --git a/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
b/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
index aa29536d8616..91584914d868 100644
--- a/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
+++ b/compiler-rt/lib/tsan/rtl/tsan_interceptors_mac.cpp
@@ -23,9 +23,12 @@
#include
#include
#include
-#include
#include
+#if defined(__has_include) && __has_include()
+#include
+#endif
+
#if defined(__has_include) && __has_include()
#include
#endif // #if defined(__has_include) && __has_include()
@@ -247,6 +250,8 @@ TSAN_INTERCEPTOR(void, os_lock_unlock, void *lock) {
REAL(os_lock_unlock)(lock);
}
+#if defined(__has_include) && __has_include()
+
TSAN_INTERCEPTOR(void, os_unfair_lock_lock, os_unfair_lock_t lock) {
if (!cur_thread()->is_inited || cur_thread()->is_dead) {
return REAL(os_unfair_lock_lock)(lock);
@@ -286,6 +291,8 @@ TSAN_INTERCEPTOR(void, os_unfair_lock_unlock,
os_unfair_lock_t lock) {
REAL(os_unfair_lock_unlock)(lock);
}
+#endif // #if defined(__has_include) && __has_include()
+
#if defined(__has_include) && __has_include()
TSAN_INTERCEPTOR(void, xpc_connection_set_event_handler,
___
llvm-branch-commits mailing list
llvm-branch-commits@lists.llvm.org
https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-branch-commits
