OK, time to call it for the night (or morning). Can't believe it's 6:30 already.
I've advised Iris to flick over to pogstest later on, if she remembers, as I've released a version which ignores cpu_time checking - at least I believe I have. See how far it gets. Hopefully the other device on pogstest picks up one of those tasks too. Cheers Daniel On Fri, Aug 9, 2013 at 6:05 AM, Daniel Carrion <[email protected]> wrote: > DEBUG: START > DEBUG: - process_tree_cpu_time(pid) > DEBUG: -- procinfo_setup(pm) > DEBUG: --- dir = opendir(/proc) > DEBUG: --- find_children(pm) > DEBUG: -- procinfo_app(procinfo, NULL, pm, NULL) > DEBUG: END > 15:56:00 (13190): wrapper: running worker (2.fit) > DEBUG: START > DEBUG: - process_tree_cpu_time(pid) > DEBUG: -- procinfo_setup(pm) > DEBUG: --- dir = opendir(/proc) > SIGSEGV: segmentation violation > > More verbose output - when I was distributing version with log output in loop: > > http://akira.onburde.net/burdetest/result.php?resultid=8371 > > > > On Fri, Aug 9, 2013 at 5:20 AM, David Anderson <[email protected]>wrote: > >> Good! Next question: is it crashing in find_children()? >> >> >> On 08-Aug-2013 12:08 PM, Daniel Carrion wrote: >> >>> Hi David >>> >>> Two of these reported back on my test server: >>> >>> DEBUG: START >>> DEBUG: - process_tree_cpu_time(pid) >>> DEBUG: -- procinfo_setup(pm) >>> DEBUG: -- procinfo_app(procinfo, NULL, pm, NULL) >>> DEBUG: END >>> DEBUG: START >>> DEBUG: - process_tree_cpu_time(pid) >>> DEBUG: -- procinfo_setup(pm) >>> SIGSEGV: segmentation violation >>> >>> Adding more logging in procinfo_setup in procinfo_unix.cpp and >>> generating more work. >>> >>> Cheers >>> >>> Daniel >>> >>> >>> >>> On Thu, Aug 8, 2013 at 5:32 PM, David Anderson <[email protected] >>> <mailto:[email protected]**>> wrote: >>> >>> If it's reproducible and happens quickly, >>> one can find the location of the crash by putting in lots of >>> printf()s, >>> although of course it's tedious. >>> I can help with this if needed. >>> -- David >>> >>> >>> On 07-Aug-2013 11:20 PM, Daniel Carrion wrote: >>> >>> Hi David >>> >>> Thanks for that explanation. >>> >>> It would definitely be useful to get a stack trace. >>> Unfortunately, there >>> is only >>> one person with this problem volunteering their phone time and I >>> cannot >>> get an >>> Android stack trace from their phone. Nothing when they run >>> logcat or >>> look in >>> /data/tombstones. This seems to be a known problem with new >>> Android >>> releases. I >>> actually think they pulled it out. The NativeBOINC stack trace >>> (which uses >>> ptrace I think) doesn't give us anything useful. It would be >>> handy to attach >>> gdbserver to the task but probably not feasible. I'll continue >>> to think >>> of ways >>> to get a useful stack trace. >>> >>> I released a version on pogstest that omits checking cpu_time >>> just to >>> see if it >>> runs through on the problematic device. I will put more >>> printf()s in the >>> next >>> release. I'm guessing this time in functions called in >>> lib/procinfo.cpp? >>> I'll >>> probably release this version to the test project site I setup >>> as I can more >>> easily control the worker task (shorter). >>> >>> Regards >>> >>> Daniel >>> >>> >>> On Thu, Aug 8, 2013 at 3:19 PM, David Anderson < >>> [email protected] >>> <mailto:[email protected]**> >>> <mailto:[email protected] <mailto:[email protected]**>__>> >>> wrote: >>> >>> I couldn't immediately see the problem. >>> A stack trace would help; >>> all we know is it's crashing somewhere in >>> process_tree_cpu_time(). >>> >>> Let me explain how this code works, in case anyone else >>> wants to >>> review it. >>> The purpose of process_tree_cpu_time(pid) is to get the >>> current CPU >>> time >>> of the given process and all its descendants. >>> Unix doesn't provide an easy way to do this; >>> getrusage() only reports on children that have exited. >>> So instead we enumerate the set of all processes (using >>> /proc), >>> find the ones that are descendants of pid, >>> and add up their CPU time. >>> >>> The functions involved are: >>> procinfo_setup() >>> scans /proc, and build a data structure, >>> namely a std::map that maps pid to PROCINFO >>> (a structure that describes a process, and which >>> contains >>> a std::vector of the PIDs of its children) >>> procinfo_app() >>> given a PID, sum the CPU time of its descendants. >>> The summing is done by a recursive function >>> add_child_totals(). >>> add_child_totals(), BTW, has a safeguard that prevents >>> infinite recursion >>> if the process tree has a cycle (i.e. is not actually >>> a tree); >>> this happens sometimes in Windows. >>> >>> Daniel, if you want to put in more printf()s would could >>> try to narrow >>> the crash down more among these functions. >>> >>> -- David >>> >>> >>> On 07-Aug-2013 8:46 PM, Daniel Carrion wrote: >>> >>> Hello >>> >>> Iris has attached to my test project site. Looks like >>> all tasks >>> exited >>> out at >>> task.cpu_time as per before. >>> >>> See task stderr: >>> http://akira.onburde.net/____**burdetest/results.php?hostid=_** >>> ___2&offset=0&show_names=0&__**state=__6&appid=<http://akira.onburde.net/____burdetest/results.php?hostid=____2&offset=0&show_names=0&__state=__6&appid=> >>> <http://akira.onburde.net/__**burdetest/results.php?hostid=_** >>> _2&offset=0&show_names=0&**state=__6&appid=<http://akira.onburde.net/__burdetest/results.php?hostid=__2&offset=0&show_names=0&state=__6&appid=> >>> > >>> >>> >>> >>> <http://akira.onburde.net/__**burdetest/results.php?hostid=_** >>> _2&offset=0&show_names=0&**state=__6&appid=<http://akira.onburde.net/__burdetest/results.php?hostid=__2&offset=0&show_names=0&state=__6&appid=> >>> <http://akira.onburde.net/**burdetest/results.php?hostid=** >>> 2&offset=0&show_names=0&state=**6&appid=<http://akira.onburde.net/burdetest/results.php?hostid=2&offset=0&show_names=0&state=6&appid=> >>> >> >>> >>> I'm going to pull this section out to confirm that it >>> is this >>> function >>> call >>> causing problems and release new version. >>> >>> Cheers >>> >>> Daniel >>> >>> On Thu, Aug 8, 2013 at 7:43 AM, Daniel Carrion >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto: >>> [email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**>__> >>> wrote: >>> >>> Oh and Thanks again David for helping out. >>> >>> >>> On Thu, Aug 8, 2013 at 7:42 AM, Daniel Carrion >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto: >>> [email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**>__> >>> wrote: >>> >>> Hopefully we can get a few more occurrences in case >>> that was a >>> one off. >>> If it >>> occurs again I might just have it spit out before and >>> after >>> that point and >>> remove the "first 180 second" threshold. Every 10 >>> seconds won't >>> make the >>> stderr.txt file too full. >>> >>> I've got a test project with POGS like tasks set up >>> here: >>> >>> http://akira.onburde.net/____**burdetest/<http://akira.onburde.net/____burdetest/> >>> >>> <http://akira.onburde.net/__**burdetest/<http://akira.onburde.net/__burdetest/> >>> > >>> >>> >>> <http://akira.onburde.net/__**burdetest/<http://akira.onburde.net/__burdetest/> >>> >>> <http://akira.onburde.net/**burdetest/<http://akira.onburde.net/burdetest/>>>. >>> At the moment the worker >>> >>> (fit_sed) is a >>> dummy app that outputs like fit_sed but runs shorter. >>> I'll see >>> if Iris wants >>> to jump on there for a few runs to see if it faults >>> with short >>> tasks as >>> well. >>> If pogstest has to be brought down next week I can >>> continue >>> troubleshooting >>> on this site. >>> >>> Cheers >>> >>> Daniel >>> >>> >>> On Thu, Aug 8, 2013 at 7:12 AM, David Anderson >>> <[email protected] <mailto:[email protected]**> >>> <mailto:[email protected] <mailto: >>> [email protected]**>__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] >>> <mailto:[email protected]**>__>__>> >>> wrote: >>> >>> That's a help. I'll take a close look at that code. -- >>> David >>> >>> >>> On 07-Aug-2013 12:39 PM, Daniel Carrion wrote: >>> >>> Hey David/Kevin >>> >>> Please see attached. Managed to catch one. Looks like >>> it was >>> calling >>> task.cpu_time(). >>> >>> Watching out for more to see if it consistently happens >>> at this >>> point. My >>> test user base is minimal for this so have to wait a >>> bit. I'm >>> pretty much >>> relying on Iris' device and one other. >>> >>> Regards >>> >>> Daniel >>> >>> On Wed, Aug 7, 2013 at 2:35 PM, Daniel Carrion >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto: >>> [email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**>__>__> >>> wrote: >>> >>> Actually, I'll have the output cycle as the stuff we're >>> interested in is >>> during the polling loop. >>> >>> >>> On Wed, Aug 7, 2013 at 2:13 PM, Daniel Carrion >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto: >>> [email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**>__>__> >>> wrote: >>> >>> Hi David >>> >>> I'm just testing this out now. A bit worried about the >>> size of this >>> stderr.txt file on users devices. Could end up over >>> 50MB. I'll >>> give people a >>> heads up before releasing. >>> >>> Regards >>> >>> Daniel >>> >>> >>> On Wed, Aug 7, 2013 at 4:31 AM, David Anderson >>> <[email protected] <mailto:[email protected]**> >>> <mailto:[email protected] <mailto: >>> [email protected]**>__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__>__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__>__>__>> >>> wrote: >>> >>> The code below (execv()) is executed in the child >>> process. It >>> looks like >>> what's getting the SIGSEGV is the parent process. I'd >>> try putting a >>> fprintf(stderr, ...) at the start and end of >>> TASK::poll(), and >>> before the >>> call to task.cpu_time(), and before the call to >>> boinc_report_app_start(). The >>> problem is likely in one of those. -- David >>> >>> >>> On 06-Aug-2013 7:47 AM, Daniel Carrion wrote: >>> >>> Hey David/Kevin >>> >>> Iris has run a couple jobs on the test instances along >>> with a >>> couple of >>> others. Here's the output of one of his tasks: >>> >>> >>> http://54.208.29.24/pogstest/_**_______result.php?resultid=294<http://54.208.29.24/pogstest/________result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**______result.php?resultid=294<http://54.208.29.24/pogstest/______result.php?resultid=294> >>> > >>> >>> <http://54.208.29.24/pogstest/** >>> ______result.php?resultid=294<http://54.208.29.24/pogstest/______result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**____result.php?resultid=294<http://54.208.29.24/pogstest/____result.php?resultid=294> >>> >> >>> >>> <http://54.208.29.24/pogstest/** >>> ______result.php?resultid=294<http://54.208.29.24/pogstest/______result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**____result.php?resultid=294<http://54.208.29.24/pogstest/____result.php?resultid=294> >>> > >>> <http://54.208.29.24/pogstest/** >>> ____result.php?resultid=294<http://54.208.29.24/pogstest/____result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**__result.php?resultid=294<http://54.208.29.24/pogstest/__result.php?resultid=294> >>> >>> >>> >>> >>> >>> <http://54.208.29.24/pogstest/** >>> ______result.php?resultid=294<http://54.208.29.24/pogstest/______result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**____result.php?resultid=294<http://54.208.29.24/pogstest/____result.php?resultid=294> >>> > >>> <http://54.208.29.24/pogstest/** >>> ____result.php?resultid=294<http://54.208.29.24/pogstest/____result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**__result.php?resultid=294<http://54.208.29.24/pogstest/__result.php?resultid=294> >>> >> >>> <http://54.208.29.24/pogstest/** >>> ____result.php?resultid=294<http://54.208.29.24/pogstest/____result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**__result.php?resultid=294<http://54.208.29.24/pogstest/__result.php?resultid=294> >>> > >>> <http://54.208.29.24/pogstest/** >>> __result.php?resultid=294<http://54.208.29.24/pogstest/__result.php?resultid=294> >>> >>> <http://54.208.29.24/pogstest/**result.php?resultid=294<http://54.208.29.24/pogstest/result.php?resultid=294> >>> >>>> >>> >>> Seems as though it's crashing as it's going to execute >>> task? >>> >>> 08:05:25 (5741): wrapper (main): task poll begin >>> 08:05:25 >>> (6818): wrapper >>> (TASK::run): in child proc of the fork 08:05:25 (6818): >>> wrapper >>> (TASK::run): >>> construct argv 08:05:25 (6818): wrapper (TASK::run): >>> set up env >>> variables >>> 08:05:25 (6818): wrapper (TASK::run): executing app >>> SIGSEGV: >>> segmentation >>> violation >>> >>> From modified code (TASK::run): >>> >>> 808 809 if (nvars > 0) { 810 >>> set_up_env_vars(&env_vars, >>> nvars); 811 >>> fprintf(stderr, "%s wrapper (TASK::run): executing app >>> with >>> vars\n", 812 >>> boinc_msg_prefix(buf, sizeof(buf)) 813 ); >>> 814 >>> retval >>> = execve(app_path, argv, env_vars); 815 } else >>> { 816 >>> fprintf(stderr, "%s wrapper (TASK::run): executing >>> app\n", 817 >>> boinc_msg_prefix(buf, sizeof(buf)) 818 ); >>> 819 >>> retval >>> = execv(app_path, argv); 820 } >>> >>> >>> I guess it could be coming from wrapper during poll but >>> it >>> seems like it has >>> something to do with fit_sed starting. Possibly some >>> memory >>> allocation >>> problems as the app is starting? It's proving quite >>> difficult >>> to get dumps >>> out of people's phones. >>> >>> I'm going to continue prodding and poking and try and >>> get a >>> stack-trace into >>> stderr from the wrapper itself. I'll also try different >>> fit_sed >>> compilation >>> as well, including a C port. >>> >>> I just wish I could fire up gdb server on their phones >>> and >>> attach over the >>> internet =D. >>> >>> Regards >>> >>> Daniel >>> >>> On Sat, Aug 3, 2013 at 12:23 AM, Daniel Carrion >>> <[email protected] <mailto:[email protected]> >>> <mailto:[email protected] <mailto: >>> [email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>** >>> >__> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>>**> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] <mailto:[email protected]>> >>> >>> <mailto:[email protected] <mailto: >>> [email protected]> >>> <mailto:[email protected] >>> <mailto:[email protected]>>**>__>__>__> >>> wrote: >>> >>> Thanks David, I'll have a look into that... >>> >>> I'll have the "debug ready" wrapper to push onto the >>> test >>> instance on >>> Monday. >>> Hopefully we can then grab a few users having this >>> problem to >>> jump on >>> and try >>> it. >>> >>> I'll have to think of a way of getting those crash >>> dumps out >>> without having >>> to use NativeBOINC . Some Android devices seem to drop >>> in >>> /data/tombstones >>> and most just log crash dump data directly to system >>> log for >>> viewing via >>> logcat. Could probably get the rooted users to open a >>> terminal >>> and leave >>> "logcat -s DEBUG > /sdcard/Download/logcat.txt" running >>> before >>> jobs run >>> so we >>> get dumps in the attached format (this is just me >>> testing/causing segfault). >>> Non rooted can either call this via adb or try an app >>> that >>> essentially does >>> the same thing. >>> >>> Sorry, just me thinking out loud. >>> >>> Regards >>> >>> Daniel >>> >>> On Fri, Aug 2, 2013 at 4:28 PM, David Anderson >>> <[email protected] <mailto:[email protected]**> >>> <mailto:[email protected] <mailto: >>> [email protected]**>__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__>__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__>__>__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__>__> >>> >>> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__> >>> <mailto:[email protected] <mailto: >>> [email protected]**> >>> <mailto:[email protected] <mailto:[email protected]** >>> >__>__>__>__>> >>> >>> >>> wrote: >>> >>> >>> >>> On 01-Aug-2013 11:15 PM, Daniel Carrion wrote: >>> >>> >>> David, is there anything else you suggest for debugging >>> purpose? E.g. >>> Catching SIGILL and SIGSEGV somehow? >>> >>> >>> That should do it; catching the signals would help only >>> if we >>> can then >>> generate a stack trace. In principle backtrace(3) can >>> do this, >>> but it may >>> not work on Anroid, see >>> http://stackoverflow.com/_____**_____questions/10864882/____** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/__________questions/10864882/____stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**____questions/10864882/__** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/________questions/10864882/__stacktrace-______arm-linux-gcc> >>> **> >>> >>> <http://stackoverflow.com/____**____questions/10864882/__** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/________questions/10864882/__stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> **>__> >>> >>> >>> >>> >>> >>> <http://stackoverflow.com/____**____questions/10864882/__** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/________questions/10864882/__stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> **> >>> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> >>> >>> >>> >>> >>> <http://stackoverflow.com/____**____questions/10864882/__** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/________questions/10864882/__stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> **> >>> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> >> >>> >>> >>> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> > >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> <http://stackoverflow.com/__**questions/10864882/stacktrace-** >>> __arm-linux-gcc<http://stackoverflow.com/__questions/10864882/stacktrace-__arm-linux-gcc> >>> >>>> >>> >>> >>> >>> >>> >>> <http://stackoverflow.com/____**____questions/10864882/__** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/________questions/10864882/__stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> **> >>> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> >> >>> >>> >>> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> > >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> <http://stackoverflow.com/__**questions/10864882/stacktrace-** >>> __arm-linux-gcc<http://stackoverflow.com/__questions/10864882/stacktrace-__arm-linux-gcc> >>> >>> >>> >>> >>> >>> <http://stackoverflow.com/____**__questions/10864882/** >>> stacktrace-______arm-linux-gcc<http://stackoverflow.com/______questions/10864882/stacktrace-______arm-linux-gcc> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> > >>> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> <http://stackoverflow.com/__**questions/10864882/stacktrace-** >>> __arm-linux-gcc<http://stackoverflow.com/__questions/10864882/stacktrace-__arm-linux-gcc> >>> >> >>> >>> <http://stackoverflow.com/____**questions/10864882/stacktrace-** >>> ____arm-linux-gcc<http://stackoverflow.com/____questions/10864882/stacktrace-____arm-linux-gcc> >>> <http://stackoverflow.com/__**questions/10864882/stacktrace-** >>> __arm-linux-gcc<http://stackoverflow.com/__questions/10864882/stacktrace-__arm-linux-gcc> >>> > >>> <http://stackoverflow.com/__**questions/10864882/stacktrace-** >>> __arm-linux-gcc<http://stackoverflow.com/__questions/10864882/stacktrace-__arm-linux-gcc>< >>> http://stackoverflow.com/**questions/10864882/stacktrace-**arm-linux-gcc<http://stackoverflow.com/questions/10864882/stacktrace-arm-linux-gcc> >>> >>>>> >>> >>> -- David >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> > _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
