Dear R Developers, I hope you are doing well.
I re-sent my package and I still see: 1. https://win-builder.r-project.org/incoming_pretest/redatam_2.0.3_20241112_165458/specialChecks/clang-san/outputs.txt 2. https://win-builder.r-project.org/incoming_pretest/redatam_2.0.3_20241112_165458/specialChecks/gcc-san/outputs.txt As it was discussed in: 1. "Possible false negative for compiled C++ code in CRAN checks" (this R developers mailing list) 2. "AddressSanitizer Error (alloc-dealloc-mismatch, operator new vs free) in R Package with C++ Code Using libc++ and CLANG-ASAN/UBSAN" (Stackoverflow, https://stackoverflow.com/questions/79171799/addresssanitizer-error-alloc-dealloc-mismatch-operator-new-vs-free-in-r-packa?noredirect=1#comment139621299_79171799) 3. "clang-asan: use clang 19 as CRAN does" (GitHub, https://github.com/r-hub/containers/pull/79#issuecomment-2470390769) The package was tested with Clang 19 on Fedora 36 and Ubuntu 22, and as S.O. reads "it is likely a false positive in the container used by R-Hub as discussed in this [#598] issue at its repo but not a false positive but an actual error as shown in the setup at CRAN (different container)." The R-Hub container was fixed with a pull request I sent, and now it passes the checks. The CRAN errors were addressed as mentioned in the mailing list. I am also using "-Wall -O0 -pedantic" and "-UDEBUG -g" to check locally, and I am also using all the available R-Hub images (27 different configurations, as in https://github.com/pachadotdev/open-redatam/actions/runs/11801021816) I would really appreciate some guidance here before re-submitting. Best wishes, ���� Mauricio "Pach��" Vargas Sep��lveda PhD Student, Political Science University of Toronto ________________________________ From: Mauricio Vargas Sepulveda <m.sepulv...@mail.utoronto.ca> Sent: November 12, 2024 11:01 AM To: Ivan Krylov <ikry...@disroot.org> Cc: r-package-devel@r-project.org <r-package-devel@r-project.org> Subject: Re: [R-pkg-devel] Possible false negative for compiled C++ code in CRAN checks Dear Prof. Dr. Kryov, Thanks a lot. Apologies if the email has an odd format, I have a modified Outlook UI that makes everything "curious". After taking each of the suggestions, now I get this with Clang-ASAN on R-Hub: > 0 errors | 0 warnings | 2 note > Package was archived on CRAN > CRAN repository db overrides: X-CRAN-Comment: Archived on 2024-11-06 for > repeated policy violation. Repeatedy spamming a team member's personal email > address in HTML. > checking compilation flags used ... NOTE Compilation used the following > non-portable flag(s): ��-Wp,-D_FORTIFY_SOURCE=3�� This is great. Thanks a lot. I would like to add you to AUTHORS.md or as "ctb" to list your contribution. Hopefully it can be re-accepted. The first note was because I was asking about this same issue when CRAN notified me. Now I make sure to use non-HTML email when sending R-related emails. So far, this is the only useful communication I received about this. On Stack Overflow I only got "-1". I added the changes here: https://github.com/pachadotdev/open-redatam/commit/d8955195d4e13fd7794de56d24b4238b7a7ec26f Best wishes, ���� Mauricio "Pach��" Vargas Sep��lveda PhD Student, Political Science University of Toronto ________________________________________ From: Ivan Krylov <ikry...@disroot.org> Sent: November 12, 2024 5:56 AM To: Mauricio Vargas Sepulveda <m.sepulv...@mail.utoronto.ca> Cc: r-package-devel@r-project.org <r-package-devel@r-project.org> Subject: Re: [R-pkg-devel] Possible false negative for compiled C++ code in CRAN checks �� Tue, 12 Nov 2024 04:02:17 +0000 Mauricio Vargas Sepulveda <m.sepulv...@mail.utoronto.ca> ��ڧ�֧�: > For v2.0.3 I implemented a full refactor, where I simplified the code > considerably while trying to solve what looks to be a false positive. > > Now I implemented a "patch" based on the suggestion: I see the change on GitHub [1], made back on October 22nd, but I am not seeing these checks in your latest submission to CRAN [2] from November 7th. Before you have implemented [1], have you been able to reproduce the errors that your package received on CRAN, that is, the container-overflow and the null object pointers? I've just noticed that FuzzyVariableParser::ParseAllVariables is using std::thread::hardware_concurrency(). CRAN uses a shared computer with many cores to test many packages at the same time, so the code most likely needs to limit itself to two threads during tests and examples [3]. > SUMMARY: AddressSanitizer: alloc-dealloc-mismatch > (/opt/R/devel-asan/lib/R/bin/exec/R+0xc6306) (BuildId: > 178e357df79b1589a38c1949da5e5f022d4bb535) in free This is an error you're getting when testing with an R-hub image. R-hub is not CRAN. You've performed good investigative work to reduce the issue you're seeing on R-hub for the StackOverflow question [4], but it looks like this particular problem is already reported to R-hub developers [5] and it's not what got your latest CRAN submission archived. When your latest submission, redatam_2.0.3.tar.gz, was archived, you should have received an e-mail from CRAN. Was it about the package failing automatic checks? Were there any other comments? Did the e-mail highlight any issues in particular? I do see the problem in [6], but it's hard to diagnose by itself. I was able to reproduce it (but only after compiling everything with -shared-san and providing the necessary LD_LIBRARY_PATH, otherwise packages using C++ failed to load; I also had to manually set UBSAN_OPTIONS=print_stacktrace=1 to find out where the overflow originates): > read_redatam(paste(dout, "cg15.dic", sep = "/")) Opening dictionary file... /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/basic_string.h:1272:9: runtime error: addition of unsigned offset to 0x7f38f12bcc30 overflowed to 0x7f38f12bcc2f #0 0x7f38e859a15f in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::operator[](unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/basic_string.h:1272:9 #1 0x7f38e859a15f in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::back() /usr/bin/../lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/basic_string.h:1353:9 #2 0x7f38e859a15f in RedatamLib::ListExporter::ListExporter(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&) /root/redatam.Rcheck/00_pkg_src/redatam/src/redatamlib/exporters/RListExporter.cpp:16:21 #3 0x7f38e85a7d60 in RedatamLib::RedatamDatabase::ExportRLists() const /root/redatam.Rcheck/00_pkg_src/redatam/src/redatamlib/entities/RedatamDatabase.cpp:23:16 #4 0x7f38e85b2224 in export_redatam_to_list_(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>) /root/redatam.Rcheck/00_pkg_src/redatam/src/main.cpp:13:15 #5 0x7f38e85b3068 in _redatam_export_redatam_to_list_ /root/redatam.Rcheck/00_pkg_src/redatam/src/cpp11.cpp:12:27 #6 0x55b5da67bf6a in R_doDotCall /root/R/src/main/../../../R-svn/src/main/dotcode.c:754:11 The corresponding lines of the source code are: ListExporter::ListExporter(const std::string &outputDirectory) : m_path(outputDirectory) { if ('/' != m_path.back()) { // <-- here m_path.append("/"); } } Looks like the code is calling .back() on an empty string. This also needs to be prevented somehow. To summarise, I recommend to do the following for the next submission: - make sure the fix for the UBSan issues from [7] is included and note it in the submission comments - avoid using more than two threads during R CMD check - fix the empty string issue from [6] and note it in the submission comments - fix any other issues noted in the CRAN correspondence and note it in the submission comments - do not mention the alloc-free-mismatch problem in the submission comments Good luck! I hope this helps and there won't be any more issues hiding in the code. -- Best regards, Ivan [1] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fpachadotdev%2Fopen-redatam%2Fcommit%2F5399332f26fde8029cc7f78e0f889dfd983504c2&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049707683%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=Y9GdK8O0MMGX%2FBjLqpU7da7VMGJXfeYv3C7v4PHvN%2FI%3D&reserved=0<https://github.com/pachadotdev/open-redatam/commit/5399332f26fde8029cc7f78e0f889dfd983504c2> [2] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcran.r-project.org%2Fincoming%2Farchive%2Fredatam_2.0.3.tar.gz&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049727294%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kBArgyzK12m3U%2F3f2U%2BIDweyYyznZQEFhjXZ9by4WRA%3D&reserved=0<https://cran.r-project.org/incoming/archive/redatam_2.0.3.tar.gz> [3] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcontributor.r-project.org%2Fcran-cookbook%2Fcode_issues.html%23using-more-than-2-cores&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049737222%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=grg1UU%2FKj6BhySwAV9aLn%2BqzN78l8BuQTDy71%2BOCGmY%3D&reserved=0<https://contributor.r-project.org/cran-cookbook/code_issues.html#using-more-than-2-cores> [4] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fstackoverflow.com%2Fq%2F79171799&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049747065%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=wC3%2BnFVKhJuppgiJlX6OMZyKRcZnCRIz%2BftqrRPBztc%3D&reserved=0<https://stackoverflow.com/q/79171799> [5] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fr-hub%2Frhub%2Fissues%2F598&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049756791%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=%2FVDNBLbGM9%2ByHGlpgXRBuWWhviPq%2FHdGxCYyXCs6ys0%3D&reserved=0<https://github.com/r-hub/rhub/issues/598> There's no actual alloc-free-mismatch, it's a false positive due to the way libc++ (correctly) uses the delete operator: https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fllvm%2Fllvm-project%2Fissues%2F59432&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049766535%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=8RXeYSqX09BjLczvg5qO%2FP7joiYeDahKMCB83e4x8YQ%3D&reserved=0<https://github.com/llvm/llvm-project/issues/59432> [6] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Fredatam_2.0.3_20241107_154750%2FspecialChecks%2Fclang-san%2Fsummary.txt&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049776187%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=U%2FPeu7qjoLu%2BGjN3sx%2FZ43%2BS5waMgMfV26FI9R40%2BvM%3D&reserved=0<https://win-builder.r-project.org/incoming_pretest/redatam_2.0.3_20241107_154750/specialChecks/clang-san/summary.txt> [7] https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fclang-ASAN%2Fredatam%2F00check.log&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049785895%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=HGE%2BuM83P0TBqCHrBEiX2twRbdKe3vHPbez8%2B5tI9CY%3D&reserved=0<https://www.stats.ox.ac.uk/pub/bdr/memtests/clang-ASAN/redatam/00check.log> https://can01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.stats.ox.ac.uk%2Fpub%2Fbdr%2Fmemtests%2Fgcc-UBSAN%2Fredatam%2F00check.log&data=05%7C02%7Cm.sepulveda%40mail.utoronto.ca%7C1e1a8fc469b049d82c1508dd0308b0d1%7C78aac2262f034b4d9037b46d56c55210%7C0%7C0%7C638670058049795583%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OaIvDVSUXmxIHEwMBpLOlgNkpgsNjXlBwJsYWXcBaho%3D&reserved=0<https://www.stats.ox.ac.uk/pub/bdr/memtests/gcc-UBSAN/redatam/00check.log> [[alternative HTML version deleted]]
______________________________________________ R-package-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-package-devel
