Re: [DISCUSS] CASSANDRA-18940 SAI post-filtering reads don't update local table latency metrics
Thanks for the feedback. To wrap this up we will introduce a new SAI local read metric (name to be decided later) to record read count and latency and these reads to be kept separate from existing local range read metrics. On Fri, 1 Dec 2023 at 19:10, Jeremiah Jordan wrote: > Again I am coming at this from the operator/end user perspective. > Creating a metrics dashboard, and then I am looking at those metrics to > understand what my queries are doing. We have coordinator query level > metrics, and then we have lower level table metrics on the replicas. I > want to be able to draw a line from this set of coordinator query metrics, > to that set of table metrics, and be able to understand how they are > affecting each other for a given query. > > The best would be for SAI / Indexes to have their very own sets of all the > metrics to understand how many rows are read by a given SAI query, and how > that turns into the over all time for the query, and how long those > individual reads were taking, etc. > > But at the very least I want all of that separate from the metrics for my > regular point reads. > > And yes putting the individual point read metrics into the range metrics > would be strange. But rolling up the time to get all the rows and rolling > that into the Range metrics could possibly make sense. Still strange. So > again SAI specific metrics seem the best to me, rather than shoe horning > them into the existing metrics. > > -Jeremiah > > On Dec 1, 2023 at 1:04:47 PM, Caleb Rackliffe > wrote: > >> Right. SAI queries are distributed range queries that produce local >> single-partition reads. They should absolutely not be recorded in the local >> range read latency metric. I'm fine ultimately with a new metric or the >> existing local single-partition read metric. >> >> On Fri, Dec 1, 2023 at 1:02 PM J. D. Jordan >> wrote: >> >>> At the coordinator level SAI queries fall under Range metrics. I would >>> either put them under the same at the lower level or in a new SAI metric. >>> >>> It would be confusing to have the top level coordinator query metrics in >>> Range and the lower level in Read. >>> >>> On Dec 1, 2023, at 12:50 PM, Caleb Rackliffe >>> wrote: >>> >>> >>> So the plan would be to have local "Read" and "Range" remain unchanged >>> in TableMetrics, but have a third "SAIRead" (?) just for SAI post-filtering >>> read SinglePartitionReadCommands? I won't complain too much if that's what >>> we settle on, but it just depends on how much this is a metric for >>> ReadCommand subclasses operating at the node-local level versus something >>> we think we should link conceptually to a user query. SAI queries will >>> produce a SinglePartitionReadCommand per matching primary key, so that >>> definitely won't work for the latter. >>> >>> @Mike On a related note, we now have "PartitionReads" and "RowsFiltered" >>> in TableQueryMetrics. Should the former just be removed, given a.) it >>> actually is rows now not partitions and b.) "RowsFiltered" seems like it'll >>> be almost the same thing now? (I guess if we ever try batching rows reads >>> per partition, it would come in handy again...) >>> >>> On Fri, Dec 1, 2023 at 12:30 PM J. D. Jordan >>> wrote: >>> I prefer option 2. It is much easier to understand and roll up two metrics than to do subtractive dashboards. SAI reads are already “range reads” for the client level metrics, not regular reads. So grouping them into the regular read metrics at the lower level seems confusing to me in that sense as well. As an operator I want to know how my SAI reads and normal reads are performing latency wise separately. -Jeremiah On Dec 1, 2023, at 11:15 AM, Caleb Rackliffe wrote: Option 1 would be my preference. Seems both useful to have a single metric for read load against the table and a way to break out SAI reads specifically. On Fri, Dec 1, 2023 at 11:00 AM Mike Adamson wrote: > Hi, > > We are looking at adding SAI post-filtering reads to the local table > metrics and would like some feedback on the best approach. > > We don't think that SAI reads are that special so they can be included > in the table latencies, but how do we handle the global counts and the SAI > counts? Do we need to maintain a separate count of SAI reads? We feel the > answer to this is yes so how do we do the counting? There are two options > (others welcome): > > 1. All reads go into the current global count and we have a separate > count for SAI specific reads. So non-SAI reads = global count - SAI count > 2. We leave the exclude the SAI reads from the current global count so > total reads = global count + SAI count > > Our preference is for option 1 above. Does anyone have any strong > views / opinions on this? > > > > -- > [image: DataStax Logo Square]
Re: [VOTE] Release Apache Cassandra 5.0-beta1 (take2)
+1 Kind Regards, Brandon On Fri, Dec 1, 2023 at 7:32 AM Mick Semb Wever wrote: > > > Proposing the test build of Cassandra 5.0-beta1 for release. > > sha1: 87fd1fa88a0c859cc32d9f569ad09ad0b345e465 > Git: https://github.com/apache/cassandra/tree/5.0-beta1-tentative > Maven Artifacts: > https://repository.apache.org/content/repositories/orgapachecassandra-1321/org/apache/cassandra/cassandra-all/5.0-beta1/ > > The Source and Build Artifacts, and the Debian and RPM packages and > repositories, are available here: > https://dist.apache.org/repos/dist/dev/cassandra/5.0-beta1/ > > The vote will be open for 72 hours (longer if needed). Everyone who has > tested the build is invited to vote. Votes by PMC members are considered > binding. A vote passes if there are at least three binding +1s and no -1's. > > [1]: CHANGES.txt: > https://github.com/apache/cassandra/blob/5.0-beta1-tentative/CHANGES.txt > [2]: NEWS.txt: > https://github.com/apache/cassandra/blob/5.0-beta1-tentative/NEWS.txt
[DISCUSS] CASSANDRA-19104: Standardize tablestats formatting and data units
Tablestats currently reports output in a mixed format which is neither ideal for human readability nor machine readability. Spaces are also inconsistently used and 13 digit numbers w/out commas or larger units are complicated to read. For example, 'Bytes repaired / un-repaired / pending' uses KiB, MiB units, but 'Space used live / total' uses bytes. Space used (live): 1463210998523 Space used (total): 1463210998523 Bytes repaired: 0.000KiB Bytes unrepaired: 4315.386GiB Bytes pending repair: 0.000KiB Given tablestats supports a machine readable formatting with the -f format option for json or yaml output, this Jira proposes: - standardizing the output to be human readable (-H) as default and - eliminating the current mixed mode of formatting. The above example would become: Space used (live): 1463.210 GiB Space used (total): 1463.210 GiB Bytes repaired: 0.000 KiB Bytes unrepaired: 4315.386 GiB Bytes pending repair: 0.000 KiB Existing machine readable formatting (with -f) will be unchanged. More detailed examples can be found in the Jira CASSANDRA-19104 and associated google spreadsheet detailing the existing and proposed output: https://tinyurl.com/38edebjd We welcome feedback and thoughts on this.
Re: [DISCUSS] CASSANDRA-19104: Standardize tablestats formatting and data units
Given how fundamental disk space is to operational competence, it's hard to imagine anyone actually parsing this output instead of using at least something like the machine readable option. The switch is rather simple if they need to make it, and they will be better for it, so I support tidying up this output so new users have a better experience. Kind Regards, Brandon On Mon, Dec 4, 2023 at 8:44 AM Brad wrote: > > Tablestats currently reports output in a mixed format which is neither ideal > for human readability nor machine readability. Spaces are also > inconsistently used and 13 digit numbers w/out commas or larger units are > complicated to read. > > For example, 'Bytes repaired / un-repaired / pending' uses KiB, MiB units, > but 'Space used live / total' uses bytes. > > Space used (live): 1463210998523 > Space used (total): 1463210998523 > > Bytes repaired: 0.000KiB > Bytes unrepaired: 4315.386GiB > Bytes pending repair: 0.000KiB > > Given tablestats supports a machine readable formatting with the -f format > option for json or yaml output, this Jira proposes: > > standardizing the output to be human readable (-H) as default and > eliminating the current mixed mode of formatting. > > The above example would become: > > Space used (live): 1463.210 GiB > Space used (total): 1463.210 GiB > > Bytes repaired: 0.000 KiB > Bytes unrepaired: 4315.386 GiB > Bytes pending repair: 0.000 KiB > > Existing machine readable formatting (with -f) will be unchanged. More > detailed examples can be found in the Jira CASSANDRA-19104 and associated > google spreadsheet detailing the existing and proposed output: > https://tinyurl.com/38edebjd > > We welcome feedback and thoughts on this. >
Re: [DISCUSS] CASSANDRA-19104: Standardize tablestats formatting and data units
This looks great, I'd consider limiting the number of significant digits to 3 in the human readable format. In the above example it would translate to: Space used (live): 1.46 TiB Space used (total): 1.46 TiB Bytes repaired: 0.00 KiB Bytes unrepaired: 4.31 TiB Bytes pending repair: 0.000 KiB I just think with human readable format we just expect to have a grasp view of the stats and 4th significant digit has very little meaning in that case. thanks, Jacek
Re: [DISCUSS] CASSANDRA-19104: Standardize tablestats formatting and data units
Thanks, Jacek. Using three significant digits for disk space is a good suggestion. On Mon, Dec 4, 2023 at 9:58 AM Jacek Lewandowski < lewandowski.ja...@gmail.com> wrote: > This looks great, > > I'd consider limiting the number of significant digits to 3 in the human > readable format. In the above example it would translate to: > > Space used (live): 1.46 TiB > Space used (total): 1.46 TiB > > Bytes repaired: 0.00 KiB > Bytes unrepaired: 4.31 TiB > Bytes pending repair: 0.000 KiB > > I just think with human readable format we just expect to have a grasp > view of the stats and 4th significant digit has very little meaning in that > case. > > > thanks, > Jacek > >
Re: [DISCUSS] CASSANDRA-19104: Standardize tablestats formatting and data units
Why not change the option so that -H will operate as it does now while -Hn (where n is a digit) will limit the number of decimal places to n. On Mon, Dec 4, 2023 at 5:11 PM Brad wrote: > Thanks, Jacek. Using three significant digits for disk space is a good > suggestion. > > On Mon, Dec 4, 2023 at 9:58 AM Jacek Lewandowski < > lewandowski.ja...@gmail.com> wrote: > >> This looks great, >> >> I'd consider limiting the number of significant digits to 3 in the human >> readable format. In the above example it would translate to: >> >> Space used (live): 1.46 TiB >> Space used (total): 1.46 TiB >> >> Bytes repaired: 0.00 KiB >> Bytes unrepaired: 4.31 TiB >> Bytes pending repair: 0.000 KiB >> >> I just think with human readable format we just expect to have a grasp >> view of the stats and 4th significant digit has very little meaning in that >> case. >> >> >> thanks, >> Jacek >> >>
