[Python-Dev] Debugging of native extensions on windows
Hello! I am dropping this mail to bring up an issue which cost me three good evenings of time. Now that i figured it out i believe it is quite a serious usability problem. Gist of the problem: i have some C++ code wrapped with SWIG, so a native extension. As with all software - there was a bug. However, no matter what i did - i could not debug it in a native debugger. I ran ".venv/Scripts/python.exe script.py" in a native debugger and breakpoints would not be hit, application would crash eventually. This was especially confusing, because exact same setup worked just fine on linux. I eventually stumbled on to process list showing ".venv/Scripts/python.exe" having spawned a subprocess... Which led me to "PC/launcher.c" which is what ".venv/Scripts/python.exe" really is. I cant find much information about this behavior in documentation even after the fact. All in all, this was quite confusing. So now every time i want to debug a native extension i have to start a program and then attach a debugger to it, instead of just hitting "Debug" button in IDE. It gets worse if crash happens immediately, which means i have to resort to things like adding a message box somewhere to block execution and give me enough time to attach the debugger. It works in the end, but user experience is really not great. But whats worse - this is such a non-obvious behavior that many more people may trip on it and waste their time. Documenting this behavior would be of little help too, as there is no clear path from the issue to the documentation on the matter... So there it is. I am sure it is the way it is for a good reason. However, this is a very error-prone process which is likely to waste people's time. So maybe this behavior could be reconsidered? Or maybe there is a solution already, which escaped me? -- -- Rokas Kupstys ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/2BHNPL7DHLPRQYMN7FQBJQRZ6H6SDIVE/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Debugging of native extensions on windows
Hi Rokas The typical solution (which I myself use frequently) is to enable your debugger to attach to child processes automatically. It can make things a bit noisier, but it's generally manageable, especially if you've got breakpoints set in your own code. Another option is to not use the virtual environment, but set PYTHONPATH to your environment's Lib\site-packages directory and then run the base interpreter directly. Most of the time, this will be identical, but it avoids the extra process. Unfortunately, for a variety of reasons, we can't get away from the redirector process as long as virtual environments keep their current design. As changing the design would be just as disruptive, we've not done anything yet, nor do we have any plans to change anything. Finally, most discussion about Python occurs at https://discuss.python.org/ these days. You'll likely find more help and discussion over there. Cheers, Steve On 3/13/2023 3:25 PM, Rokas Kupstys wrote: Hello! I am dropping this mail to bring up an issue which cost me three good evenings of time. Now that i figured it out i believe it is quite a serious usability problem. Gist of the problem: i have some C++ code wrapped with SWIG, so a native extension. As with all software - there was a bug. However, no matter what i did - i could not debug it in a native debugger. I ran ".venv/Scripts/python.exe script.py" in a native debugger and breakpoints would not be hit, application would crash eventually. This was especially confusing, because exact same setup worked just fine on linux. I eventually stumbled on to process list showing ".venv/Scripts/python.exe" having spawned a subprocess... Which led me to "PC/launcher.c" which is what ".venv/Scripts/python.exe" really is. I cant find much information about this behavior in documentation even after the fact. All in all, this was quite confusing. So now every time i want to debug a native extension i have to start a program and then attach a debugger to it, instead of just hitting "Debug" button in IDE. It gets worse if crash happens immediately, which means i have to resort to things like adding a message box somewhere to block execution and give me enough time to attach the debugger. It works in the end, but user experience is really not great. But whats worse - this is such a non-obvious behavior that many more people may trip on it and waste their time. Documenting this behavior would be of little help too, as there is no clear path from the issue to the documentation on the matter... So there it is. I am sure it is the way it is for a good reason. However, this is a very error-prone process which is likely to waste people's time. So maybe this behavior could be reconsidered? Or maybe there is a solution already, which escaped me? ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/424LIHYHRQWM5VLQF2PAMLKGCNCKSJDF/ Code of Conduct: http://python.org/psf/codeofconduct/
[Python-Dev] Re: Debugging of native extensions on windows
Is it easier to simply run python outside a virtualenv? They are great, but maybe when debugging an extension module, it's not so hard to just not use it :-) You also might want to give conda environments a try -- they include Python, so probably won't have the same issue. -CHB On Mon, Mar 13, 2023 at 4:58 PM Steve Dower wrote: > Hi Rokas > > The typical solution (which I myself use frequently) is to enable your > debugger to attach to child processes automatically. It can make things > a bit noisier, but it's generally manageable, especially if you've got > breakpoints set in your own code. > > Another option is to not use the virtual environment, but set PYTHONPATH > to your environment's Lib\site-packages directory and then run the base > interpreter directly. Most of the time, this will be identical, but it > avoids the extra process. > > Unfortunately, for a variety of reasons, we can't get away from the > redirector process as long as virtual environments keep their current > design. As changing the design would be just as disruptive, we've not > done anything yet, nor do we have any plans to change anything. > > Finally, most discussion about Python occurs at > https://discuss.python.org/ these days. You'll likely find more help and > discussion over there. > > Cheers, > Steve > > On 3/13/2023 3:25 PM, Rokas Kupstys wrote: > > Hello! > > > > I am dropping this mail to bring up an issue which cost me three good > > evenings of time. Now that i figured it out i believe it is quite a > > serious usability problem. > > > > Gist of the problem: i have some C++ code wrapped with SWIG, so a native > > extension. As with all software - there was a bug. However, no matter > > what i did - i could not debug it in a native debugger. I ran > > ".venv/Scripts/python.exe script.py" in a native debugger and > > breakpoints would not be hit, application would crash eventually. This > > was especially confusing, because exact same setup worked just fine on > > linux. I eventually stumbled on to process list showing > > ".venv/Scripts/python.exe" having spawned a subprocess... Which led me > > to "PC/launcher.c" which is what ".venv/Scripts/python.exe" really is. I > > cant find much information about this behavior in documentation even > > after the fact. All in all, this was quite confusing. So now every time > > i want to debug a native extension i have to start a program and then > > attach a debugger to it, instead of just hitting "Debug" button in IDE. > > It gets worse if crash happens immediately, which means i have to resort > > to things like adding a message box somewhere to block execution and > > give me enough time to attach the debugger. It works in the end, but > > user experience is really not great. But whats worse - this is such a > > non-obvious behavior that many more people may trip on it and waste > > their time. Documenting this behavior would be of little help too, as > > there is no clear path from the issue to the documentation on the > matter... > > > > So there it is. I am sure it is the way it is for a good reason. > > However, this is a very error-prone process which is likely to waste > > people's time. So maybe this behavior could be reconsidered? Or maybe > > there is a solution already, which escaped me? > > > > ___ > Python-Dev mailing list -- [email protected] > To unsubscribe send an email to [email protected] > https://mail.python.org/mailman3/lists/python-dev.python.org/ > Message archived at > https://mail.python.org/archives/list/[email protected]/message/424LIHYHRQWM5VLQF2PAMLKGCNCKSJDF/ > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, PhD (Chris) Python Language Consulting - Teaching - Scientific Software Development - Desktop GUI and Web Development - wxPython, numpy, scipy, Cython ___ Python-Dev mailing list -- [email protected] To unsubscribe send an email to [email protected] https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/[email protected]/message/R3FSGSRCYIRYQL43TVJOKSSKA6YWEMZ3/ Code of Conduct: http://python.org/psf/codeofconduct/
