Community Track CFP Open for Community Over Code (Formerly ApacheCon)

[sharanf]

Hi All

This is a note to let you know that we are running a Community Tracks 
for Community Over Code (Formerly ApacheCon 
). If you haven’t seen the announcement 
then please take a look. The event will take place in Halifax, Nova 
Scotia, Canada from 7^th - 10^th October 2023. (And yes the dates are 
correct – it starts on a Saturday!)


Having a track means that our community has the opportunity to participate.
*
*So why should you submit something for the Community Track CFP?

The ASF and our communities are always evolving and growing. We are 
still attracting new projects and new contributors. It’s this diversity 
that makes us sustainable so it becomes important that we hear from the 
people within all our communities. What topics could you talk about and 
submit? Your experience of being part of the ASF is important.


How and why did you become involved with the ASF? What does it mean to 
you to be a Contributor, Committer, Member or Officer in the Foundation? 
How do you understand and interpret our collaborative culture? What are 
your real life stories adopting it? Did you or your project have any 
challenges to overcome?


All those with knowledge of branding, licencing, governance - please 
feel free to submit something that will share your knowledge with others.


How diverse and inclusive are our communities? Do you have any data or 
opinions about where we are now and what our future communities could 
evolve into?


The Community Track is all about our communities sharing their thoughts 
and experiences with others, so that they can learn from it - so please 
submit if you have something you want share. And if it’s your first time 
submitting – then no problem, we welcome all submissions. (And if you'd 
like to submit something to any other track-- then that's fine too!. 
There is no reason you can't submit muitiple talks to multiple tracks)


So get thinking and when you are ready please use the CFP link below to 
make your submission.


Submit to the CFP 

You can find more details on the website. https://communityovercode.org/

Looking forward to your submissions :-)

Thanks
Sharan and Swapnil

[jira] [Created] (COMDEV-522) RocketMQ DLedger Controller Performance Optimization

[Rongtong Jin (Jira)]

Rongtong Jin created COMDEV-522:
---

 Summary: RocketMQ DLedger Controller Performance Optimization
 Key: COMDEV-522
 URL: https://issues.apache.org/jira/browse/COMDEV-522
 Project: Community Development
  Issue Type: Task
  Components: Comdev, GSoC/Mentoring ideas
Reporter: Rongtong Jin


*Apache RocketMQ*

Apache RocketMQ is a distributed messaging and streaming platform with low 
latency, high performance and reliability, trillion-level capacity, and 
flexible scalability.
Page: 
[https://rocketmq.apache.org|https://rocketmq.apache.org/]([https://rocketmq.apache.org/])
Repo: [https://github.com/apache/rocketmq]([https://github.com/apache/rocketmq])

*Background*

RocketMQ 5.0 introduced a new component, the controller, which controls the 
high availability master-slave switch in multi-replica scenarios. It uses the 
DLedger Raft library as a consensus replication state machine for metadata. As 
a completely independent component, it can run normally in some scenarios, but 
in large-scale clusters, it is necessary to maintain a large number of broker 
groups, which is a great challenge for operational capabilities and resource 
waste. When dealing with a large number of Broker groups, we need to optimize 
performance in large-scale scenarios, leveraging the high-performance writing 
of DLedger itself and performing some optimization for the current Controller 
architecture.

*Task*

1. Polish the usage of DLedger

Currently, on the Controller side, a task queue single thread is used for 
requesting reads and writes to DLedger, that is, only one read/write request 
can be processed at a time. However, DLedger itself implements many 
optimizations for multi-client reads and writes and can ensure linear 
consistency reading. However, now all read and write processing is performed 
using a single logical DLedger client, which will become a serious performance 
bottleneck in large-scale scenarios.

2. Optimization of DLedger features usage

DLedger itself can implement many optimizations, such as ReadIndex read and 
FollowerRead read. After implementation, we can fully leverage the performance 
of reads. Currently, all Broker nodes communicate with the Leader node of the 
Controller. In large-scale scenarios, this will cause the requests of each 
Controller group to be concentrated on the Leader node, and the other Follower 
nodes will not share the request processing of the Leader, which will cause 
single-point performance bottlenecks for the Leader.

3. Full asynchronous + parallel processing

Currently, DLedger itself is fully asynchronous, but on the Controller side, 
all requests for the DLedger side are synchronized, and many Controller-side 
operations are performed synchronously in a single thread, such as heartbeat 
checks and other timed tasks. In large-scale scenarios, the logic of these 
single-threaded synchronous operations will block a large number of requests 
from Broker-side, so asynchronous + parallel processing can be used for 
optimization.

4. Correctness testing and performance testing

After completing the above optimizations, it is necessary to conduct 
correctness testing on the new version and use distributed chaos testing 
frameworks such as OpenChaos to verify correct operation under fault scenarios 
such as network partition and random crashes.
After completing the correctness testing, a detailed performance testing report 
can be produced by comparing the new and old versions.

*Skills Required*
 - Strong interest in message middleware and distributed storage systems
 - Proficient in Java development
 - In-depth understanding of distributed consensus algorithms
 - In-depth understanding of the high-availability module of RockeetMQ and the 
DLedger library
 - Understanding of distributed chaos testing and performance testing.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org

[jira] [Updated] (COMDEV-522) RocketMQ DLedger Controller Performance Optimization

[Rongtong Jin (Jira)]

 [ 
https://issues.apache.org/jira/browse/COMDEV-522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rongtong Jin updated COMDEV-522:

Description: 
*Apache RocketMQ*

Apache RocketMQ is a distributed messaging and streaming platform with low 
latency, high performance and reliability, trillion-level capacity, and 
flexible scalability.
Page: [https://rocketmq.apache.org|https://rocketmq.apache.org/]
Repo: [https://github.com/apache/rocketmq]

*Background*

RocketMQ 5.0 introduced a new component, the controller, which controls the 
high availability master-slave switch in multi-replica scenarios. It uses the 
DLedger Raft library as a consensus replication state machine for metadata. As 
a completely independent component, it can run normally in some scenarios, but 
in large-scale clusters, it is necessary to maintain a large number of broker 
groups, which is a great challenge for operational capabilities and resource 
waste. When dealing with a large number of Broker groups, we need to optimize 
performance in large-scale scenarios, leveraging the high-performance writing 
of DLedger itself and performing some optimization for the current Controller 
architecture.

*Task*

1. Polish the usage of DLedger

Currently, on the Controller side, a task queue single thread is used for 
requesting reads and writes to DLedger, that is, only one read/write request 
can be processed at a time. However, DLedger itself implements many 
optimizations for multi-client reads and writes and can ensure linear 
consistency reading. However, now all read and write processing is performed 
using a single logical DLedger client, which will become a serious performance 
bottleneck in large-scale scenarios.

2. Optimization of DLedger features usage

DLedger itself can implement many optimizations, such as ReadIndex read and 
FollowerRead read. After implementation, we can fully leverage the performance 
of reads. Currently, all Broker nodes communicate with the Leader node of the 
Controller. In large-scale scenarios, this will cause the requests of each 
Controller group to be concentrated on the Leader node, and the other Follower 
nodes will not share the request processing of the Leader, which will cause 
single-point performance bottlenecks for the Leader.

3. Full asynchronous + parallel processing

Currently, DLedger itself is fully asynchronous, but on the Controller side, 
all requests for the DLedger side are synchronized, and many Controller-side 
operations are performed synchronously in a single thread, such as heartbeat 
checks and other timed tasks. In large-scale scenarios, the logic of these 
single-threaded synchronous operations will block a large number of requests 
from Broker-side, so asynchronous + parallel processing can be used for 
optimization.

4. Correctness testing and performance testing

After completing the above optimizations, it is necessary to conduct 
correctness testing on the new version and use distributed chaos testing 
frameworks such as OpenChaos to verify correct operation under fault scenarios 
such as network partition and random crashes.
After completing the correctness testing, a detailed performance testing report 
can be produced by comparing the new and old versions.

*Skills Required*
 - Strong interest in message middleware and distributed storage systems
 - Proficient in Java development
 - In-depth understanding of distributed consensus algorithms
 - In-depth understanding of the high-availability module of RockeetMQ and the 
DLedger library
 - Understanding of distributed chaos testing and performance testing.

  was:
*Apache RocketMQ*

Apache RocketMQ is a distributed messaging and streaming platform with low 
latency, high performance and reliability, trillion-level capacity, and 
flexible scalability.
Page: 
[https://rocketmq.apache.org|https://rocketmq.apache.org/]([https://rocketmq.apache.org/])
Repo: [https://github.com/apache/rocketmq]([https://github.com/apache/rocketmq])

*Background*

RocketMQ 5.0 introduced a new component, the controller, which controls the 
high availability master-slave switch in multi-replica scenarios. It uses the 
DLedger Raft library as a consensus replication state machine for metadata. As 
a completely independent component, it can run normally in some scenarios, but 
in large-scale clusters, it is necessary to maintain a large number of broker 
groups, which is a great challenge for operational capabilities and resource 
waste. When dealing with a large number of Broker groups, we need to optimize 
performance in large-scale scenarios, leveraging the high-performance writing 
of DLedger itself and performing some optimization for the current Controller 
architecture.

*Task*

1. Polish the usage of DLedger

Currently, on the Controller side, a task queue single thread is used for 
requesting reads and writes to DLedger, that is, only one read/write request 
can be process

[jira] [Commented] (COMDEV-384) Tasks in the help page seems to have not been updated for a long time

[Sebb (Jira)]

[ 
https://issues.apache.org/jira/browse/COMDEV-384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702189#comment-17702189
 ] 

Sebb commented on COMDEV-384:
-

There was an issue with searching the database, now resolved.

> Tasks in the help page seems to have not been updated for a long time
> -
>
> Key: COMDEV-384
> URL: https://issues.apache.org/jira/browse/COMDEV-384
> Project: Community Development
>  Issue Type: Improvement
>  Components: Help Wanted
>Reporter: ningtao
>Priority: Major
>
> https://helpwanted.apache.org/
> First, chose “Programming and Development”,  pick  “Java language”, then 
> “Found 26 items you might be interested in”,  but these tasks looks too old,  
> 25 task is created at 2017, one task created at 2019. And some of these tasks 
> are already in “closed” status. 
> I guess there are a lots of Java project need help in Apache, just not 
> include by Help Wanted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
For additional commands, e-mail: dev-h...@community.apache.org