Why is no one asking the question, In all the YEARS we have been using BOINC/SETI@Home, why has NOT ONE of the other MILLIONS of users seen this problem? Isn't it possible that is an owner-specific problem? I realize that the process is tedious and time-consuming, but what would happen if the user reinstalled Windows, on a clean new or reformatted hard disk or SSD, then installed BOINC and its applications, then installed the all the user's other applications one-by-one until the problem re-appeared, if ever. What was the last thing that changed before this problem appeared?
Charles Elliott -----Original Message----- From: boinc_dev [mailto:[email protected]] On Behalf Of Robert Miles Sent: Monday, October 26, 2015 9:45 PM To: [email protected] Subject: [boinc_dev] Problems with cached memory under 64-bit Windows What I'm seeing indicates that is what's supposed to happen, but not quite what is actually happening. It seems to preload memory just as you expect, but NOT free much of the preloaded memory of BOINC-related files when it is under memory pressure because a 32-bit program or an attempt to start a 32-bit program needs it. It does seem to free preloaded memory as needed when a 64-bit workunit needs it, and starts up 64-bit programs fast enough that I'd also expect it to free preloaded memory as needed for 64-bit programs started from the console. If BOINC has no control over this, then the most that BOINC developers can do is to allow reducing the total size of the project directories, as I previously requested, and try to help persuade Microsoft to allow more control over SuperFetch and provide a way to see just what SuperFetch has stored in the preloaded memory. I don't especially want to start removing the more memory-hungry BOINC projects from my computers, except for the few projects that allow downloading only 64-bit workunits, which seems to be the only option I currently have. Many of the programs I want to run from the console simply aren't available in 64-bit forms. -----Original Message Date: Tue, 27 Oct 2015 00:10:46 +0000 From: Rom Walton <[email protected]> Subject: Re: [boinc_dev] Problems with cached memory under 64-bit Windows From my understanding Superfetch tries to use all the memory on a machine. It preloads various programs and files into memory just in case you want you use them. If Windows finds itself under memory pressure, it just forgets about the stuff in the cache to make room for the stuff you want at that moment. As far as I know, there isn't anything a program can do to control that behavior in Windows. ----- Rom -----Original Message----- From: boinc_dev [mailto:[email protected]] On Behalf Of Robert Miles Sent: Monday, October 26, 2015 5:15 PM To: [email protected] Subject: [boinc_dev] Problems with cached memory under 64-bit Windows Probably, but finding a name for the cause of the problem offers little help in fixing it. ---- Robert -----Original Message-----Date: Mon, 26 Oct 2015 14:56:54 +0000 From: Rom Walton <[email protected]> To: Robert Miles <[email protected]>, Subject: Re: [boinc_dev] Problems with cached memory under 64-bit Windows I believe you are running into SuperFetch. See: http://www.osnews.com/story/21471/SuperFetch_How_it_Works_Myths ----- Rom -----Original Message----- From: boinc_dev [mailto:[email protected]] On Behalf Of Robert Miles Sent: Monday, October 26, 2015 10:51 AM To: [email protected] Subject: [boinc_dev] Problems with cached memory under 64-bit Windows BOINC appears to have a problem with filling the cached memory in a way that prevents Windows from using that cached memory for any 32-bit programs until the next restart or reboot. Problem seen under 64-bit Windows Vista and 64-bit Windows 7; 64-bit Windows 10 appears to have a similar problem even if it does not call anything cached memory; I do not have any running other versions of Windows to check. 64-bit workunits appear to be able to get cached memory freed if they need it. I do not run 64-bit programs from the console often enough to tell if such 64-bit console programs can also free cached memory or not. One of my computers with 8 GB of memory and 64-bit Windows Vista is now often unable to start its email program unless I have shut down BOINC. Another computer, with 16 GB memory and Windows 10, does not have as obvious a problem yet, but the email program is often slow to start or run there. Suspending BOINC to free CPU use does not help. My Windows Vista computer currently has 5319 of its 8190 blocks of physical memory labelled as cached, and is slow to respond when I start a 32-bit program with significant use of memory. Still, the help files do not appear to explain what cached memory is, whether it is useful to reduce the amount of cached memory, and if so, how to do this. So many files on the hard drives that defragmentation, full virus scans, and other operations on all the files take 3 to 5 days, even with nothing else running. These are mostly message files from using Windows Mail and Windows Live Mail to read newsgroups, and save most of the messages read or even downloaded but not yet read. This is so long that I seldom run any operations that require reading all the files. Quick virus scans find nothing of interest, though. The motherboard appears to be at the maximum amount of physical memory it can handle. Could several of you join me in asking Microsoft to add an explanation of whether it is useful to reduce the amount of memory cached, and if so, how to reduce the cached memory? I'd like for them to provide a program to show more about what is being cached, including any file names, so that users can look harder at the files cached most often. For example, I'd like to check if files that a program used months ago but but not more recently are getting stored in the cache every time that program runs. I've found information on what memory caches are supposed to be used for (faster access to items likely to be used again), but it looks like BOINC does not allow enough of the cached memory to be freed when 32-bit programs need more memory. One idea of how to change BOINC to reduce the number of BOINC-related files in cached memory: BOINC already has a feature for allowing frequently used files to be stored in project directories so they can be saved for any future workunits that need them. A new feature could be added to allow BOINC servers to tell BOINC clients that some of the files in the project directories are not expected to be used again, and therefore should be deleted from the project directories so that they cannot be loaded into cached memory in the future. This may need some thought about what to do if a project mistakenly lists a file as no longer needed, but some later workunit tries to use it anyway. Note that both the server and the client will need to have sufficiently recent software for this new feature will work, so it should be designed in a way that will allow older BOINC software to ignore it if only one end of the connection supports the feature. I have not found a definite way to duplicate this problem, but the following method seems likely: Start with a computer with 8 GB of memory or less, and running 64-bit Windows. Install a recent version of BOINC, if it isn't already installed. Connect it to each of the following BOINC projects, and run each of their applications at least once: World Community Grid rosetta@home ralph@home PrimeGrid Milkyway@Home GPUGRID FiND@Home Einstein@Home Albert@Home These projects were the ones that use the most disk space on my computers, and therefore probably have the most space used in their project directories. My computers are connected to enough other projects to have about 20 projects in all connected. Now, with all but one of the CPU cores running 32-bit CPU workunits, and a GPU workunit also running, start trying to start some 32-bit program that uses a lot of memory. _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. ------------------------------ Message: 4 Date: Mon, 26 Oct 2015 17:30:46 -0700 From: Eric J Korpela <[email protected]> To: Rom Walton <[email protected]> Cc: "[email protected]" <[email protected]> Subject: Re: [boinc_dev] Problems with cached memory under 64-bit Windows Message-ID: <cakfuvzz+sqs+jxmrmtg9wd6gaarfga+rse+a0hfup_xw6_g...@mail.gmail.com> Content-Type: text/plain; charset=UTF-8 Cached pages are at minimum priority with the exception of a small number of critical pages. If there's a significant amount of cached pages, it probably means that nothing else has asked for the memory. I don't know if dirty pages not currently mapped count in the cached category under windows, but if they do that would explain a delay in new allocations. (ie. something did a lot of disk writes without a Flush) I've always assumed that "Available" is "Cached minus Dirty", but a lot of that part of Windows memory management is not well documented. But in general "Cached" should "Available" because it's discardable without I/O and shouldn't cause any performance penalty. My guess is that the slow start of 32 bit apps is that there's something slowing up Wow64. Perhaps it's not being cached by SuperFetch because it's used rarely. On Mon, Oct 26, 2015 at 5:10 PM, Rom Walton <[email protected]> wrote: > From my understanding Superfetch tries to use all the memory on a > machine. It preloads various programs and files into memory just in > case you want you use them. If Windows finds itself under memory > pressure, it just forgets about the stuff in the cache to make room > for the stuff you want at that moment. > > As far as I know, there isn't anything a program can do to control > that behavior in Windows. > > ----- Rom > > -----Original Message----- > From: boinc_dev [mailto:[email protected]] On Behalf > Of Robert Miles > Sent: Monday, October 26, 2015 5:15 PM > To: [email protected] > Subject: [boinc_dev] Problems with cached memory under 64-bit Windows > > Probably, but finding a name for the cause of the problem offers > little help in fixing it. > > ---- Robert > > -----Original Message-----Date: Mon, 26 Oct 2015 14:56:54 +0000 > From: Rom Walton <[email protected]> > To: Robert Miles <[email protected]>, > Subject: Re: [boinc_dev] Problems with cached memory under 64-bit > Windows > > I believe you are running into SuperFetch. > > See: > http://www.osnews.com/story/21471/SuperFetch_How_it_Works_Myths > > ----- Rom > > -----Original Message----- > From: boinc_dev [mailto:[email protected]] On Behalf > Of Robert Miles > Sent: Monday, October 26, 2015 10:51 AM > To: [email protected] > Subject: [boinc_dev] Problems with cached memory under 64-bit Windows > > BOINC appears to have a problem with filling the cached memory in a > way that prevents Windows from using that cached memory for any 32-bit > programs until the next restart or reboot. Problem seen under 64-bit > Windows Vista and 64-bit Windows 7; 64-bit Windows 10 appears to have > a similar problem even if it does not call anything cached memory; I > do not have any running other versions of Windows to check. > > 64-bit workunits appear to be able to get cached memory freed if they > need it. I do not run 64-bit programs from the console often enough > to tell if such 64-bit console programs can also free cached memory or not. > > One of my computers with 8 GB of memory and 64-bit Windows Vista is > now often unable to start its email program unless I have shut down BOINC. > Another computer, with 16 GB memory and Windows 10, does not have as > obvious a problem yet, but the email program is often slow to start or > run there. > Suspending BOINC to free CPU use does not help. > > My Windows Vista computer currently has 5319 of its 8190 blocks of > physical memory labelled as cached, and is slow to respond when I > start a 32-bit program with significant use of memory. > > Still, the help files do not appear to explain what cached memory is, > whether it is useful to reduce the amount of cached memory, and if so, > how to do this. > > So many files on the hard drives that defragmentation, full virus > scans, and other operations on all the files take 3 to 5 days, even > with nothing else running. These are mostly message files from using > Windows Mail and Windows Live Mail to read newsgroups, and save most > of the messages read or even downloaded but not yet read. This is so > long that I seldom run any operations that require reading all the > files. Quick virus scans find nothing of interest, though. > > The motherboard appears to be at the maximum amount of physical memory > it can handle. > > Could several of you join me in asking Microsoft to add an explanation > of whether it is useful to reduce the amount of memory cached, and if > so, how to reduce the cached memory? I'd like for them to provide a > program to show more about what is being cached, including any file > names, so that users can look harder at the files cached most often. > For example, I'd like to check if files that a program used months ago > but but not more recently are getting stored in the cache every time > that program runs. > > I've found information on what memory caches are supposed to be used > for (faster access to items likely to be used again), but it looks > like BOINC does not allow enough of the cached memory to be freed when > 32-bit programs need more memory. > > One idea of how to change BOINC to reduce the number of BOINC-related > files in cached memory: BOINC already has a feature for allowing > frequently used files to be stored in project directories so they can > be saved for any future workunits that need them. A new feature could > be added to allow BOINC servers to tell BOINC clients that some of the > files in the project directories are not expected to be used again, > and therefore should be deleted from the project directories so that > they cannot be loaded into cached memory in the future. > This may need some thought about what to do if a project mistakenly > lists a file as no longer needed, but some later workunit tries to use it anyway. > Note that both the server and the client will need to have > sufficiently recent software for this new feature will work, so it > should be designed in a way that will allow older BOINC software to > ignore it if only one end of the connection supports the feature. > > I have not found a definite way to duplicate this problem, but the > following method seems likely: Start with a computer with 8 GB of > memory or less, and running 64-bit Windows. Install a recent version > of BOINC, if it isn't already installed. Connect it to each of the > following BOINC projects, and run each of their applications at least once: > > World Community Grid > rosetta@home > ralph@home > PrimeGrid > Milkyway@Home > GPUGRID > FiND@Home > Einstein@Home > Albert@Home > > These projects were the ones that use the most disk space on my > computers, and therefore probably have the most space used in their > project directories. > My computers are connected to enough other projects to have about 20 > projects in all connected. > > Now, with all but one of the CPU cores running 32-bit CPU workunits, > and a GPU workunit also running, start trying to start some 32-bit > program that uses a lot of memory. > _______________________________________________ > boinc_dev mailing list > [email protected] > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev > To unsubscribe, visit the above URL and (near bottom of page) enter > your email address. > _______________________________________________ > boinc_dev mailing list > [email protected] > http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev > To unsubscribe, visit the above URL and (near bottom of page) enter > your email address. > ------------------------------ Subject: Digest Footer _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev ------------------------------ End of boinc_dev Digest, Vol 137, Issue 14 ****************************************** _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address. _______________________________________________ boinc_dev mailing list [email protected] http://lists.ssl.berkeley.edu/mailman/listinfo/boinc_dev To unsubscribe, visit the above URL and (near bottom of page) enter your email address.
