The count of handles seems not to be the problem!
But, there is a limit for available filedescrptors per process on windows
(msvcrt.dll) - 2048 !
I made a test with the following code
#####
use Tie::File;
our %k;
for (0...10000) {
$k{$_} = [];
}
for (sort {$a <=> $b} keys %k) {
tie @{$k{$_}}, 'Tie::File', "c:/tmp/test/$_";
push @{$k{$_}}, "$_";
print "$_\n";
}
####
this creates 2044 files (not the expected 10001). 2044 + STDOUT + STDIN +
STDERR + script = 2048
So, if assp tries (and retries for ever) to use a BerkeleyDB - but the
folder not extsts - and BerkeleyDB.pm does for what ever reason not close
files/filehandles/... - this will lead in to a 'too many opened files'
error.
Thomas
Von: "Thomas Eckardt" <[email protected]>
An: "ASSP development mailing list" <[email protected]>
Datum: 11.10.2021 09:58
Betreff: Re: [Assp-test] Multiple topics: Griplist vs GripList,
stuck open handles to dbd-error on windows, more.
1) - this will be fixed
2) - not default settings needs to be checked by the admin
3) - BDB-error.txt files are used to record BerkeleyDB errors - a handle
is keeped opened by each thread for each BDB-environment (db) as long as
the database-env is opened by the thread
most times such handles are opened until the thread is stopped (assp
ends)
4)
a) yes
b) yes
c)
Scoring is done using gripValencePB and the resulting message/ip - score
is calculated as follows:
if the grip value is < 0.3 : -int(((0.3 - grip value) / 0.3) *
gripValencePB)
if the grip value is > 0.7 : int(((grip value - 0.7) / 0.3) *
gripValencePB)
grip values between 0.3 and 0.7 are ignored.
windows handles (IMHO):
system-max handles : ~ 2**24 (>16.000.000)
process/thread max handles : ~ 10.000 (configurable in the registry -
hex(2710))
use Testlimit.exe / Testlimit64.exe -h
How ever - the c-library used by the process sets the handle (and other)
limits! If a valid but not-default-system c-lib is found in the PATH, it
will be used (with there internal limit settings).
assp never uses more than 2000 handles (typical less than 1000) on any
system watched by me
most handles (~ 1.000.000) are used permanently by mysqld on windows
Thomas
Von: "K Post" <[email protected]>
An: "ASSP development mailing list"
<[email protected]>
Datum: 09.10.2021 16:21
Betreff: [Assp-test] Multiple topics: Griplist vs GripList, stuck
open handles to dbd-error on windows, more.
Several related items here:
Bug? Rebuild process still uploading griplist, even if disabled, due
possibly to case error in code.
ASSP not checking for valid griplist, if an invalid folder name is entered
On windows, Rebuild process leaving \tmpDB\rebuildDB\BDB-error.txt handle
open (wider issue with DBD-error.txt files getting stuck open on
Windows?)
Griplist clarification request
1) I've got noGriplistUpload and noGriplistDownload both checked in the
GUI. But I noticed that ,at the end of my rebuild log, it's still doing
the upload
Uploading Griplist via Direct Connection
rebuildspamdb.pm has
return if $main::noGripListUpload;
before the upload happens, but I think there's a case mistake there.
I believe it should be Griplist, with a lower case L.
2) In trying to figure out my ASSP on Windows crashes due to too many open
files, I've started using Sysinternals Handle utility to look at open
handles. Yesterday I saw 1300+ handles open to
d:\assp\tmpDB\Griplist\BDB-error.txt
I believe that had something to do with me having an invalid griplist
database name, "d/griplist" but the d folder didn't exist on my new server
config!! My fault there for sure, not sure how d/ prefixed it, but it
appears ASSP didn't check to see that the folder existed. Shame on me for
missing my config file, but it might be good for ASSP to warn or create
the folder if it doesn't exist.
I cleared out the griplist database file to test and restarted. I don't
see any more open griplist\dbd-error.txt handles, even though I'm getting
many many bad SMTP servers connecting with early TLS. Good. I'll
probably put the griplist database back (obviously with a valid
filename!!) See #3....
3) Windows (both 2012 and 2019) might not be closing Berkely error files
correctly in general.
When the new version started logging the new SSL errors, I >believe<
that's when it started trying to access the non-existent "d/" folder
config error in m assp.cfg. Every time an early TLS line was caught,
BerkeleyDB would keep the error file open. Strangely, the error file was
0kb.
Now that I've cleared out the griplist entry, I don't get those
/tmpdb/griplist/dbd-error.txt open handles. However, after my rebuild
over night, I see and open handle to:
tmpDB\rebuildDB\BDB-error.txt
about 9+ hours after rebuild.
Is that normal? Might Windows not be closing handles correctly in
general?
4) While I'm at it, could some clarification be provided as to the
function of the griplist? (and please correct me if anything I say here
is incorrect!!) I've searched like crazy over the last couple of days,
but can't find the answers I'm looking for.
The griplist is an ip scoring database correct? I know it was originally
called the grey-ip-list, 15+ years ago, but then greylisting became common
language for delaying, so the original grayiplist started being called the
"griplist" to avoid confusion.
Note: the gui still says:
GreyIPlist Database (griplist)
The file with the current Grey-IP-List database -- make this blank if you
don't use it.
If I'm understanding griplist correctly, I think the gui should be
reworded. Maybe some explanation added too??
There's also the optional upload and download concept of the griplist.
This appears to send the local griplist to sourceforge, it gets processed
by whatever you've got running on the backend, and then I can (also
optionally) download another griplist which is based on all ASSP user
data. If you don't upload, you can't download, and that's fair - you need
to contribute to benefit from the group.
The charity that I work for has a pretty poorly thought out privacy policy
that requires me to jump through all kinds of hoops when sharing >any<
information. It's frustrating for sure. That means I cannot upload our
griplist without petitioning an internal committee. I'm thinking I want
to do that, but need to fully understand the griplist first.
If we don't upload/download, the griplist is stlil maintained locally,
just only with my data right?
If I get approval to share the ip data and download the griplist, that
downloaded griplist is merged with my local griplist?
And in the end, how is a match on the griplist scored? Let's say an IP is
on the griplist as being a really bad IP. What score does a message get?
Is that configurable?
As always, thank you
Ken
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test
DISCLAIMER:
*******************************************************
This email and any files transmitted with it may be confidential, legally
privileged and protected in law and are intended solely for the use of the
individual to whom it is addressed.
This email was multiple times scanned for viruses. There should be no
known virus in this email!
*******************************************************
_______________________________________________
Assp-test mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/assp-test
