Re: [DISCUSS] CommitLog default disk access mode
I think what we should do is provide options, and let users make their own decisions. The default behavior should not be modified until some future released version. Pawar, Amit 于2023年10月16日周一 15:51写道： > [Public] > > Hi, > > > > CommitLog uses mmap (memory mapped ) segments by default. Direct-IO > feature is proposed through new PR[1] to improve the CommitLog IO speed. > Enabling this by default could be useful feature to address IO bottleneck > seen during peak load. > > > > Need your input regarding changing this default. Please suggest. > > > > https://issues.apache.org/jira/browse/CASSANDRA-18464 > > > > thanks, > > Amit Pawar > > > > [1] - https://github.com/apache/cassandra/pull/2777 > -- you are the apple of my eye !
New episode of the Apache Cassandra(R) Corner podcast
s2e11 - Sarma Pydipally (You may have to download it to play) https://drive.google.com/file/d/1X98HjR0G3wnRaUUuUUvsDyvx05QZYXXN/view?usp=drive_link It will remain in staging for 72 hours, going live (assuming no objections) by Thursday, October 19th. If anyone should have any questions or comments, or if you want to be a guest, please reach out to me. I am looking for additional guests. So if you know someone who would be a great guest, please let me know! Thanks, everyone! Aaron
ci-cassandra 5.0 and trunk now uses in-tree scripts
FYI CASSANDRA-18665 has been committed. This makes our ci-cassandra.a.o stage jobs on 5.0 and trunk branches build and test with the in-tree script instead of cassandra-builds. This is a significant change, lots of testing happened, but I'm still expecting some trouble (like everytime we touch ci-cassandra.a.o). Reach out if there's anything. Next step is the standalone Jenkinsfile (which will also fix devbranch on ci-cassandra.a.o on 5.0/trunk) and Jenkinsfile on k8s (repeatable CI, so anyone can repeat ci-cassandra on their own k8s cluster).
Re: [DISCUSS] CommitLog default disk access mode
I haven't looked at the patch, but at a high level, defaulting to direct I/O for commit logs makes a lot of sense to me. On 2023/10/16 06:34:05 "Pawar, Amit" wrote: > [Public] > > Hi, > > CommitLog uses mmap (memory mapped ) segments by default. Direct-IO feature > is proposed through new PR[1] to improve the CommitLog IO speed. Enabling > this by default could be useful feature to address IO bottleneck seen during > peak load. > > Need your input regarding changing this default. Please suggest. > > https://issues.apache.org/jira/browse/CASSANDRA-18464 > > thanks, > Amit Pawar > > [1] - https://github.com/apache/cassandra/pull/2777 >
Re: [DISCUSS] CommitLog default disk access mode
I have some plans to (eventually) use the commit log as memtable payload storage (ie memtables would reference the commit log entries directly, storing only indexing info), and to back first level of sstables by reference to commit log entries. This will permit us to deliver not only much bigger memtables (cutting compaction throughput requirements by the ratio of size increase - so pretty dramatically), and faster flushing (so better behaviour ling write bursts), but also a fairly cheap and simple way to support MVCC - which will be helpful for transaction throughput. There is also a new commit log (“journal”) coming with Accord, that the rest of C* may or may not transition to. I only say this because this makes the utility of direct IO for commit log suspect, as we will be reading from the files as a matter of course should this go ahead; and we may end up relying on a different commit log implementation before long anyway. This is obviously a big suggestion and is not guaranteed to transpire, and probably won’t within the next year or so, but it should perhaps form some minimal part of any calculus. If the patch is otherwise simple and beneficial I don’t have anything against it, and the use of direct IO could well be of benefit eg in compaction - and also in future if we manage to bring a page management in process. So laying foundations there could be of benefit, even if the commit log eventually does not use it. > On 16 Oct 2023, at 17:00, Jon Haddad wrote: > > I haven't looked at the patch, but at a high level, defaulting to direct I/O > for commit logs makes a lot of sense to me. > >> On 2023/10/16 06:34:05 "Pawar, Amit" wrote: >> [Public] >> >> Hi, >> >> CommitLog uses mmap (memory mapped ) segments by default. Direct-IO feature >> is proposed through new PR[1] to improve the CommitLog IO speed. Enabling >> this by default could be useful feature to address IO bottleneck seen during >> peak load. >> >> Need your input regarding changing this default. Please suggest. >> >> https://issues.apache.org/jira/browse/CASSANDRA-18464 >> >> thanks, >> Amit Pawar >> >> [1] - https://github.com/apache/cassandra/pull/2777 >>
Re: [DISCUSS] CommitLog default disk access mode
Glad you brought up compaction here - I think there would be a significant benefit to moving compaction to direct i/o. On 2023/10/16 16:14:28 Benedict wrote: > I have some plans to (eventually) use the commit log as memtable payload > storage (ie memtables would reference the commit log entries directly, > storing only indexing info), and to back first level of sstables by reference > to commit log entries. This will permit us to deliver not only much bigger > memtables (cutting compaction throughput requirements by the ratio of size > increase - so pretty dramatically), and faster flushing (so better behaviour > ling write bursts), but also a fairly cheap and simple way to support MVCC - > which will be helpful for transaction throughput. > > There is also a new commit log (“journal”) coming with Accord, that the rest > of C* may or may not transition to. > > I only say this because this makes the utility of direct IO for commit log > suspect, as we will be reading from the files as a matter of course should > this go ahead; and we may end up relying on a different commit log > implementation before long anyway. > > This is obviously a big suggestion and is not guaranteed to transpire, and > probably won’t within the next year or so, but it should perhaps form some > minimal part of any calculus. If the patch is otherwise simple and beneficial > I don’t have anything against it, and the use of direct IO could well be of > benefit eg in compaction - and also in future if we manage to bring a page > management in process. So laying foundations there could be of benefit, even > if the commit log eventually does not use it. > > > On 16 Oct 2023, at 17:00, Jon Haddad wrote: > > > > I haven't looked at the patch, but at a high level, defaulting to direct > > I/O for commit logs makes a lot of sense to me. > > > >> On 2023/10/16 06:34:05 "Pawar, Amit" wrote: > >> [Public] > >> > >> Hi, > >> > >> CommitLog uses mmap (memory mapped ) segments by default. Direct-IO > >> feature is proposed through new PR[1] to improve the CommitLog IO speed. > >> Enabling this by default could be useful feature to address IO bottleneck > >> seen during peak load. > >> > >> Need your input regarding changing this default. Please suggest. > >> > >> https://issues.apache.org/jira/browse/CASSANDRA-18464 > >> > >> thanks, > >> Amit Pawar > >> > >> [1] - https://github.com/apache/cassandra/pull/2777 > >> >
Re: [DISCUSS] CommitLog default disk access mode
I haven't looked at the patch yet so take whatever I say here with a pinch of salt. Philosophically, defaults should not change unless there is a clear demonstrable benefit in majority cases for our users. In this case DirectIO should have clear benefits. That said, this is a new feature and I would personally default it to off. We should document it and allow for our users to enable it. This derisks the project in case there is an inadvertent change in behavior. Dinesh > On Oct 15, 2023, at 11:34 PM, Pawar, Amit wrote: > > [Public] > > > Hi, > > CommitLog uses mmap (memory mapped ) segments by default. Direct-IO feature > is proposed through new PR[1] to improve the CommitLog IO speed. Enabling > this by default could be useful feature to address IO bottleneck seen during > peak load. > > Need your input regarding changing this default. Please suggest. > > https://issues.apache.org/jira/browse/CASSANDRA-18464 > > thanks, > Amit Pawar > > [1] - https://github.com/apache/cassandra/pull/2777
Re: [DISCUSS] CommitLog default disk access mode
*Glad you brought up compaction here - I think there would be a significant benefit to moving compaction to direct i/o.* +1. Would love to see this get traction. On Mon, 16 Oct 2023 at 19:38, Jon Haddad wrote: > Glad you brought up compaction here - I think there would be a significant > benefit to moving compaction to direct i/o. > > > On 2023/10/16 16:14:28 Benedict wrote: > > I have some plans to (eventually) use the commit log as memtable payload > storage (ie memtables would reference the commit log entries directly, > storing only indexing info), and to back first level of sstables by > reference to commit log entries. This will permit us to deliver not only > much bigger memtables (cutting compaction throughput requirements by the > ratio of size increase - so pretty dramatically), and faster flushing (so > better behaviour ling write bursts), but also a fairly cheap and simple way > to support MVCC - which will be helpful for transaction throughput. > > > > There is also a new commit log (“journal”) coming with Accord, that the > rest of C* may or may not transition to. > > > > I only say this because this makes the utility of direct IO for commit > log suspect, as we will be reading from the files as a matter of course > should this go ahead; and we may end up relying on a different commit log > implementation before long anyway. > > > > This is obviously a big suggestion and is not guaranteed to transpire, > and probably won’t within the next year or so, but it should perhaps form > some minimal part of any calculus. If the patch is otherwise simple and > beneficial I don’t have anything against it, and the use of direct IO could > well be of benefit eg in compaction - and also in future if we manage to > bring a page management in process. So laying foundations there could be of > benefit, even if the commit log eventually does not use it. > > > > > On 16 Oct 2023, at 17:00, Jon Haddad > wrote: > > > > > > I haven't looked at the patch, but at a high level, defaulting to > direct I/O for commit logs makes a lot of sense to me. > > > > > >> On 2023/10/16 06:34:05 "Pawar, Amit" wrote: > > >> [Public] > > >> > > >> Hi, > > >> > > >> CommitLog uses mmap (memory mapped ) segments by default. Direct-IO > feature is proposed through new PR[1] to improve the CommitLog IO speed. > Enabling this by default could be useful feature to address IO bottleneck > seen during peak load. > > >> > > >> Need your input regarding changing this default. Please suggest. > > >> > > >> https://issues.apache.org/jira/browse/CASSANDRA-18464 > > >> > > >> thanks, > > >> Amit Pawar > > >> > > >> [1] - https://github.com/apache/cassandra/pull/2777 > > >> > > >
Re: [DISCUSS] CommitLog default disk access mode
Strongly agree with this point of view that direct IO can bring great benefits. I have reviewed part of the code, and my preliminary judgment is that it is not very common and limited in some situations, for example, it works for commitlog's write path only for this patch.So I suggest that the default value should not be modified until the entire function is comprehensive and stable, and then modified in a future version. Sam 于2023年10月17日周二 05:39写道： > > *Glad you brought up compaction here - I think there would be a > significant benefit to moving compaction to direct i/o.* > > +1. Would love to see this get traction. > > On Mon, 16 Oct 2023 at 19:38, Jon Haddad > wrote: > >> Glad you brought up compaction here - I think there would be a >> significant benefit to moving compaction to direct i/o. >> >> >> On 2023/10/16 16:14:28 Benedict wrote: >> > I have some plans to (eventually) use the commit log as memtable >> payload storage (ie memtables would reference the commit log entries >> directly, storing only indexing info), and to back first level of sstables >> by reference to commit log entries. This will permit us to deliver not only >> much bigger memtables (cutting compaction throughput requirements by the >> ratio of size increase - so pretty dramatically), and faster flushing (so >> better behaviour ling write bursts), but also a fairly cheap and simple way >> to support MVCC - which will be helpful for transaction throughput. >> > >> > There is also a new commit log (“journal”) coming with Accord, that the >> rest of C* may or may not transition to. >> > >> > I only say this because this makes the utility of direct IO for commit >> log suspect, as we will be reading from the files as a matter of course >> should this go ahead; and we may end up relying on a different commit log >> implementation before long anyway. >> > >> > This is obviously a big suggestion and is not guaranteed to transpire, >> and probably won’t within the next year or so, but it should perhaps form >> some minimal part of any calculus. If the patch is otherwise simple and >> beneficial I don’t have anything against it, and the use of direct IO could >> well be of benefit eg in compaction - and also in future if we manage to >> bring a page management in process. So laying foundations there could be of >> benefit, even if the commit log eventually does not use it. >> > >> > > On 16 Oct 2023, at 17:00, Jon Haddad >> wrote: >> > > >> > > I haven't looked at the patch, but at a high level, defaulting to >> direct I/O for commit logs makes a lot of sense to me. >> > > >> > >> On 2023/10/16 06:34:05 "Pawar, Amit" wrote: >> > >> [Public] >> > >> >> > >> Hi, >> > >> >> > >> CommitLog uses mmap (memory mapped ) segments by default. Direct-IO >> feature is proposed through new PR[1] to improve the CommitLog IO speed. >> Enabling this by default could be useful feature to address IO bottleneck >> seen during peak load. >> > >> >> > >> Need your input regarding changing this default. Please suggest. >> > >> >> > >> https://issues.apache.org/jira/browse/CASSANDRA-18464 >> > >> >> > >> thanks, >> > >> Amit Pawar >> > >> >> > >> [1] - https://github.com/apache/cassandra/pull/2777 >> > >> >> > >> > -- you are the apple of my eye !
