On 1/15/21 11:53 AM, Pierre Rogier wrote:
Hi Thierry,
I was rather thinking about the key and value duplication when
querying the DB:
When using bdb functions that is done implicitly.
bdb either copies the values in the DBT buffer or it alloc/realloc it
When mimicking bdb behavior with LDBM we will have to do that
explicitly in the LDBM plugin:
LDMB returns a memory mapped address that may be ummapped
once the txn is ended. So we must copy the result before
closing the txn.
If we have a read txn that protects the full operation lifespan, then
we could directly use the mapped address without needing to duplicate
them.
ah okay ! nice.
Just a question, if we have a txn covering a search with large candidate
list (unindexed), does that mean that by default each db key/value will
remain mapped until txn commit ?
thanks
thierry
Pierre
On Fri, Jan 15, 2021 at 10:53 AM thierry bordaz <[email protected]
<mailto:[email protected]>> wrote:
On 1/14/21 12:32 PM, Pierre Rogier wrote:
Hi William,
> It's a scenario we will need to fix via your BE work because of
the MVCC transaction model that
> LMDB will force us to adopt :)
As I see things in the early phases the lmdb read txn will
probably only be managed at the db plugin level rather than at
backend level. That means that we will have the same
inconsistency risk than today (i.e as if using bdb and the
implicit txn).
The txn model redesign you are speaking about should only occur
in one of the last phases (once bdb does no more coexists with lmdb).
It must be done because it could provide a serious performance
boost for read operations (IMHO, In most cases we could avoid to
duplicate the db data)
Pierre, what duplicate are you thinking of ? str2entry ?
But we should not do it while bdb is still around because of the
risk of lock issue and excessive retries.
Note I put a phasing section in
https://directory.fedoraproject.org/docs/389ds/design/backend-redesign-phase3.html#phasing
explaining that. But I guess I should move it within Ludwig's
document that englobs it.
Pierre
On Thu, Jan 14, 2021 at 12:01 AM William Brown <[email protected]
<mailto:[email protected]>> wrote:
> On 13 Jan 2021, at 21:24, Pierre Rogier <[email protected]
<mailto:[email protected]>> wrote:
>
> Thank you Willian,
> So far your scenario (entry found when reading base entry
but no more existing when computing the candidates) is the
only one that matches the symptoms.
It's a scenario we will need to fix via your BE work because
of the MVCC transaction model that LMDB will force us to
adopt :)
> And that triggered a thought:
> We cannot do anything for SUBTREE and ONE_LEVEL searches
> because the fact that the base entry id is not in the
candidate may be normal
> but IMHO we should improve the BASE search case.
> In this case the candidate list is directly set to the base
entry id
> ==> if the candidate entry (in
ldbm_back_next_search_entry) is not found and the scope is
BASE then we should return a LDAP_NO_SUCH_ENTRY error ..
I suspect that Mark has seen this email and submitted a PR to
resolve this exact case :)
>
> Pierre
>
>
> On Wed, Jan 13, 2021 at 1:45 AM William Brown
<[email protected] <mailto:[email protected]>> wrote:
> Hey there,
>
> https://github.com/389ds/389-ds-base/pull/4525/files
>
> I had a look and I can see a few possible contributing
factors, but without a core and the exact state I can't be
sure if this is correct. It's all just hypothetical from
reading the code.
>
>
> The crash is in deref_do_deref_attr() which is called as
part of deref_pre_entry(). This is the
SLAPI_PLUGIN_PRE_ENTRY_FN which is called by
"./ldap/servers/slapd/result.c:1488: rc =
plugin_call_plugins(pb, SLAPI_PLUGIN_PRE_ENTRY_FN);"
>
>
> I think what's important here is that the search is
conducted in ./ldap/servers/slapd/opshared.c:818 rc =
(*be->be_search)(pb); Is *not* in a transaction. That means
that while the single search in be_search() is consistent due
to an implied transaction, the subsequent search in
deref_pre_entry() is likely conducted in a seperate
transaction. This allows for other operations to potentially
interleave and cause changes - modrdn or delete would
certainly be candidates to cause a DN to be remove between
these two points. It would be extremely hard to reproduce as
a race condition of course.
>
>
> A question you asked is why don't we get a "no such entry"
error or similar? I think that this is because
build_candidate_list in ldbm_search.c doesn't actually create
an error if the base_candidates list is empty, because an IDL
is allocated with a value of 0 (no matching entries). this
allows the search to proceed, and there are no errors, and
the result set is set to NULL with size 0. I can't see where
LDAP_NO_SUCH_OBJECT is set in this process, but without
looking further into it, my suspicion is that entries of size
0 WONT return an error condition to internal_search_pb, so
it's valid for this to be empty.
>
> Anyway, again, this is just reading the code for 20
minutes, and is not a complete in depth investigation, but
maybe it's some ideas about what happened?
>
> Hope it helps :)
>
>
>
> —
> Sincerely,
>
> William Brown
>
> Senior Software Engineer, 389 Directory Server
> SUSE Labs, Australia
> _______________________________________________
> 389-devel mailing list -- [email protected]
<mailto:[email protected]>
> To unsubscribe send an email to
[email protected]
<mailto:[email protected]>
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
>
>
> --
> --
>
> 389 Directory Server Development Team
> _______________________________________________
> 389-devel mailing list -- [email protected]
<mailto:[email protected]>
> To unsubscribe send an email to
[email protected]
<mailto:[email protected]>
> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs, Australia
_______________________________________________
389-devel mailing list -- [email protected]
<mailto:[email protected]>
To unsubscribe send an email to
[email protected]
<mailto:[email protected]>
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
--
--
389 Directory Server Development Team
_______________________________________________
389-devel mailing list [email protected]
<mailto:[email protected]>
To unsubscribe send an email [email protected]
<mailto:[email protected]>
Fedora Code of
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/[email protected]
--
--
389 Directory Server Development Team
_______________________________________________
389-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
