This is an automated email from the ASF dual-hosted git repository. lhotari pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/pulsar-site.git
commit a11327ae06a37324a6c7d25da8eb1483583bd23e Author: Lari Hotari <[email protected]> AuthorDate: Thu Jun 11 23:52:27 2026 +0300 Update contribute docs for the Gradle build on apache/pulsar master Pulsar master has migrated from Maven to Gradle (PIP-463). Describe the development environment setup for the master branch only and align the development docs with the in-repo guides (CONTRIBUTING.md, ARCHITECTURE.md, CODING.md, AGENTS.md): - setup-buildtools: JDK 21/25 via SDKMAN (21.0.11-amzn / 25.0.3-amzn); the Gradle wrapper is bundled, so there is no separate build tool to install. Keep a note on Maven for maintenance branches and the release process. - setup-building: Gradle build/lint/test commands, the module-name-vs-directory gotcha, docker image build; maintenance branches (branch-4.2 and earlier) keep using Maven per their README - setup-ide: Gradle-based IntelliJ and VS Code setup; with build/run delegated to Gradle, no manual annotation processing or pre-build workarounds are needed - setup-debugging: build from source with ./gradlew assemble - testing-licenses: spotlessApply/rat for license headers and checkBinaryLicense for bundled-dependency LICENSE/NOTICE coverage - develop-coding-conventions: align with CODING.md on master (slog logging, async, testing conventions, backward compatibility) and link to it as the canonical reference - about: add a table of the in-repo developer guides, including AGENTS.md and the ASF generative tooling guidance - personal-ci: document the fork CI loop and /pulsarbot rerun - setup-git: refresh the maintenance branch examples Pulsar does not officially support Windows: run Pulsar in Docker, and for development use the most recent WSL2 version (legacy WSL is not supported), where the Linux instructions apply as-is. IntelliJ and VS Code (via the WSL extension) support developing in a WSL2 environment. --- contribute/about.md | 12 ++++ contribute/develop-coding-conventions.md | 95 +++++++++++++++++++------------- contribute/personal-ci.md | 12 ++-- contribute/setup-building.md | 70 +++++++++++++++++++---- contribute/setup-buildtools.md | 52 ++++++++++------- contribute/setup-debugging.md | 2 +- contribute/setup-git.md | 7 +-- contribute/setup-ide.md | 87 ++++++----------------------- contribute/testing-licenses.md | 16 +++++- 9 files changed, 202 insertions(+), 151 deletions(-) diff --git a/contribute/about.md b/contribute/about.md index 56fbde38b9f..4a2347a8866 100644 --- a/contribute/about.md +++ b/contribute/about.md @@ -14,6 +14,18 @@ The content of the contribution guide can be found in the left sidebar menu. ::: +## Developer guides in the apache/pulsar repository + +In addition to this contribution guide, the [apache/pulsar](https://github.com/apache/pulsar) repository contains in-repo developer guides for the `master` branch: + +| Doc | Use for | +|-----|---------| +| [`CONTRIBUTING.md`](https://github.com/apache/pulsar/blob/master/CONTRIBUTING.md) | Local dev workflow: building, running tests and test groups, integration tests, Personal CI, PR conventions, security reporting. | +| [`ARCHITECTURE.md`](https://github.com/apache/pulsar/blob/master/ARCHITECTURE.md) | Big-picture module map, the Gradle build infrastructure and how to change build files, the concurrency model, and the `pip/` proposals. | +| [`CODING.md`](https://github.com/apache/pulsar/blob/master/CODING.md) | Coding conventions: style, async/`CompletableFuture`, concurrency, logging, dependencies, backward compatibility, testing, and the review checklist. | +| [`SECURITY.md`](https://github.com/apache/pulsar/blob/master/SECURITY.md) | Reporting a vulnerability and disclosure hygiene. | +| [`AGENTS.md`](https://github.com/apache/pulsar/blob/master/AGENTS.md) | Guidance for AI coding assistants (Claude Code, Copilot, Cursor, Gemini, Codex, …), including the ASF [Generative Tooling guidance](https://www.apache.org/legal/generative-tooling.html): a human contributor must always review, verify, and take responsibility for every contribution. | + ## Channels To engage the Pulsar developers community, you can join the following channels. diff --git a/contribute/develop-coding-conventions.md b/contribute/develop-coding-conventions.md index 01a326b4815..437b6b9096c 100644 --- a/contribute/develop-coding-conventions.md +++ b/contribute/develop-coding-conventions.md @@ -3,7 +3,15 @@ id: develop-coding-conventions title: Coding conventions --- -The guidelines help to encourage consistency and best practices among people working on Apache Pulsar code base. You should observe the guidelines unless there is compelling reason to ignore them. Pulsar uses checkstyle to enforce coding style, refer to our [checkstyle rules](https://github.com/apache/pulsar/blob/master/buildtools/src/main/resources/pulsar/checkstyle.xml) for all enforced checkstyle rules. +The guidelines help to encourage consistency and best practices among people working on Apache Pulsar code base. You should observe the guidelines unless there is compelling reason to ignore them. + +The **canonical coding reference** for the Pulsar `master` branch is [`CODING.md`](https://github.com/apache/pulsar/blob/master/CODING.md) in the apache/pulsar repository. It is the always-up-to-date version of these conventions and covers each topic in more depth, with code examples. This page summarizes the key points. + +:::note + +Some conventions below — most notably [slog](https://github.com/merlimat/slog) logging — apply to the `master` branch only. When contributing to a maintenance branch, follow the conventions of the surrounding code in that branch. + +::: ## Java code style @@ -12,73 +20,84 @@ Apache Pulsar code follows the [Sun Java Coding Convention](http://www.oracle.co * Indentation should be **4 spaces**. Tabs should never be used. * Use curly braces even for single-line ifs and elses. * No @author tags in any javadoc. -* Use try-with-resources blocks whenever is possible. -* **TODO**s should be associated to at least one issue. - -## Dependencies - -Apache Pulsar uses the following libraries a lot: +* Prefer imports over fully qualified class names in code. Use a fully qualified class name only when needed to disambiguate a name collision that imports cannot resolve. +* Every **TODO** must reference a GitHub issue, e.g. `// TODO: https://github.com/apache/pulsar/issues/XXXX`. -* [Guava](https://github.com/google/guava) as a fundamental core library -* [Netty](http://netty.io/) for network communications and memory buffer management. +Pulsar uses checkstyle to enforce coding style, refer to the [checkstyle rules](https://github.com/apache/pulsar/blob/master/buildtools/src/main/resources/pulsar/checkstyle.xml) for all enforced checkstyle rules. Lombok is enabled in the codebase. -Use these libraries whenever possible rather than introducing more dependencies. +## Logging -Dependencies are bundled with our binary distributions, so you need to attach the relevant licenses when adding new dependencies. +See [`CODING.md` → Logging](https://github.com/apache/pulsar/blob/master/CODING.md#logging) for the details. -## Future +* On `master`, prefer **[slog](https://github.com/merlimat/slog)** (`io.github.merlimat.slog`) via Lombok's **`@CustomLog`**. **SLF4J is deprecated** for new code; never use `System.out` or `System.err`. +* **Default new logs to `TRACE`/`DEBUG`, not `INFO`** — Pulsar overuses `INFO` and floods production logs. Reserve `INFO` for low-frequency lifecycle and state-change events. Use `DEBUG` in a way where it could be enabled in production without causing too many log entries; use `TRACE` for more detailed information. +* Attach data as **structured attributes** — `log.info().attr("topic", topic).log("Published")` — not interpolated into the message string. +* For expensive `DEBUG`/`TRACE` values, don't guard with `isDebugEnabled()`/`isTraceEnabled()`; use slog's lazy form — `log.debug().attr("dump", () -> expensiveDump()).log("...")`. +* Avoid logging on hot paths, and stack traces at `INFO` or lower. -`CompletableFuture` introduce in Java 8 is preferred over Guava's `ListenableFuture`. Use `CompletableFuture` whenever possible. +## Asynchronous programming -## Memory +See [`CODING.md` → Asynchronous programming](https://github.com/apache/pulsar/blob/master/CODING.md#asynchronous-programming) for the details and code examples. -Use netty `ByteBuf` over `java.nio.ByteBuffer` for internal usage. As Pulsar uses Netty Buffer for memory management. +* `CompletableFuture` is preferred over Guava's `ListenableFuture` for new code. +* **A method returning `CompletableFuture` must not throw synchronously.** Propagate failures — including argument-validation failures — through the returned future with `CompletableFuture.failedFuture(e)`. +* **Never block on event-loop / async-execution threads** — no `Thread.sleep`, `Future.get()`, `CompletableFuture.join()`, or blocking IO. +* Avoid nested futures (`CompletableFuture<CompletableFuture<T>>`); flatten with `thenCompose`. Prefer `OrderedExecutor` for ordered asynchronous work. +* Limit concurrency and handle backpressure when firing many async operations. -## Logging +## Dependencies -* Logging should be taken seriously. Please take the time to access the logs when making a change to ensure that the important things are getting logged and there is no junk there. -* Logging statements should be complete sentences with proper capitalization that are written to be read by a person not necessarily familiar with the source code. -* All logs should be done with **SLF4j**, never `System.out` or `System.err`. +Prefer existing dependencies over new libraries. Pulsar commonly uses Apache Commons / [Guava](https://github.com/google/guava) (utilities), FastUtil (type-specific collections), JCTools (concurrent structures), RoaringBitmap (compressed bitsets), Caffeine (caching), Jackson (JSON), Prometheus / OpenTelemetry (metrics), and [Netty](http://netty.io/) (networking and buffers). -### Logging levels +A new dependency must be justified (why existing ones are insufficient), and dependencies are bundled with the binary distributions, so the bundled-dependency `LICENSE`/`NOTICE` files must be updated — verify with `./gradlew checkBinaryLicense`. See [License header](testing-licenses.md). -* **INFO** is the level you should assume the software will be run in. INFO messages are things which are not bad but which the user will definitely want to know about every time they occur. -* **TRACE** and **DEBUG** are both things you turn on when something is wrong, and you want to figure out what is going on. **DEBUG** should not be so fine-grained that it will seriously affect performance of the program. **TRACE** can be anything. You should wrap DEBUG and TRACE statements in the `if (logger.isDebugEnabled())` or `if (logger.isTraceEnabled())` check to avoid performance degradation. -* **WARN** and **ERROR** indicate something that is *BAD*. Use WARN if you aren't totally sure it is bad, and ERROR if you are. +## Resource and memory management -Log the stack traces at **ERROR** level, but never at **INFO** level or below. You can log at **WARN** level if you are interested in debugging. +* Always close resources (streams, connections, executors, buffers) — prefer try-with-resources. +* On internal networking/messaging paths, prefer Netty `ByteBuf` over `java.nio.ByteBuffer` unless an external API requires it; release ref-counted buffers you allocate. +* Don't hand-optimize allocation away — Pulsar runs on ZGC, where short-lived allocations are cheap. Don't add new Netty `Recycler` usage; see [`CODING.md` → Resource and memory management](https://github.com/apache/pulsar/blob/master/CODING.md#resource-and-memory-management). ## Monitoring * Any new features should come with appropriate metrics, so monitoring the feature is working correctly. * Those metrics should be taken seriously and only export useful metrics that would be used on production on monitoring/alerting healthy of the system, or troubleshooting problems. -## Unit tests +## Testing conventions + +See [`CODING.md` → Testing conventions](https://github.com/apache/pulsar/blob/master/CODING.md#testing-conventions) for the details. -* New changes should come with unit tests that verify the functionality being added. -* Unit tests should test the least amount of code possible. Don't start the whole server unless there is no other way to test a single class or small group of classes in isolation. -* Tests should not depend on any external resources. They need to set up and teardown their own stuff. -* It is okay to use the filesystem and network in tests since that's our business, but you need to clean up them after test. -* DO NOT use sleep or other timing assumptions in tests. It is wrong and will fail intermittently on any test server with other things going on that causes delays. -* You'd better add a timeout value to all test cases, to prevent a build from completing indefinitely. For example, `@Test(timeout=60000)`. +* Tests use **TestNG + Mockito**. Prefer **AssertJ** assertions (with descriptions) over TestNG asserts; use **Awaitility** for async conditions instead of `sleep` timing, with timeouts to prevent hangs. +* Every feature or bug fix needs **deterministic** tests for edge and failure cases. A bug-fix test must fail on the unpatched code for the real reason. +* **No reflection into private state** (`WhiteboxImpl`, `setAccessible(true)`). Expose a package-private `@VisibleForTesting` accessor and put the test in the same package. +* New integration-style tests should extend `SharedPulsarBaseTest`, use `getNamespace()` and `newTopicName()`, and never hardcode namespace/topic names. +* Close and release what the test allocates; a `ByteBuf` leak detected by the pooled allocator is a real bug. +* Validate performance optimizations with a JMH benchmark under `microbench/`. + +For how to *run* tests (scoping with `--tests`, test groups, integration tests), see [Setup and building](setup-building.md) and [`CONTRIBUTING.md` → Running tests](https://github.com/apache/pulsar/blob/master/CONTRIBUTING.md#running-tests). ## Configuration -* When you use the config files, think of the names from the very beginning. +* When adding configuration options, use clear, descriptive names and provide sensible defaults. * If you run the program without tuning parameters, use the default values. * All configuration settings should be added accordingly in the [default configuration file](https://github.com/apache/pulsar/tree/master/conf) directory and documented accordingly. ## Concurrency +See [`CODING.md` → Concurrency](https://github.com/apache/pulsar/blob/master/CODING.md#concurrency) for the details, including the Java-Memory-Model guidance. + Apache Pulsar is a low latency system, it is implemented as a purely asynchronous service, which is accomplished as follows: -* All public classes should be **thread-safe**. +* All public classes should be **thread-safe**. If a class is not thread-safe, it should be annotated [@NotThreadSafe](https://github.com/misberner/jsr-305/blob/master/ri/src/main/java/javax/annotation/concurrent/NotThreadSafe.java). +* Protect shared mutable state; prefer fine-grained synchronization. Prefer the **single-writer principle** (a given piece of state mutated by only one thread) to avoid concurrent mutation entirely. +* **Minimize work while holding a lock** — never call out to listener/callback code while holding a lock. * Prefer using [OrderedExecutor](https://github.com/apache/bookkeeper/blob/master/bookkeeper-common/src/main/java/org/apache/bookkeeper/common/util/OrderedExecutor.java) for executing any asynchronous actions. The mutations to the same instance should be submitted to the same thread to execute. -* If synchronization and locking are required, they should be in a fine granularity way. -* All threads should have proper meaningful name. -* If a class is not thread-safe, it should be annotated [@NotThreadSafe](https://github.com/misberner/jsr-305/blob/master/ri/src/main/java/javax/annotation/concurrent/NotThreadSafe.java). The instances that use this class is responsible for its synchronization. +* All threads should have proper meaningful name. When creating thread pools, prefer Netty's `io.netty.util.concurrent.DefaultThreadFactory`. +* **Prefer immutable objects**, and conform to the Java Memory Model — a field accessed by more than one thread needs explicit visibility (`volatile`, or the same lock guarding every read *and* write). ## Backwards compatibility -* Wire protocol should support backwards compatibility to enable no-downtime upgrades. This means the servers **MUST** be able to support requests from both old and new clients simultaneously. -* Metadata formats and data formats should support backwards compatibility. +See [`CODING.md` → Backward compatibility](https://github.com/apache/pulsar/blob/master/CODING.md#backward-compatibility) for the details. + +* Changes must not break public APIs, client compatibility, wire-protocol compatibility, or serialized/metadata formats. The servers **MUST** be able to support requests from both old and new clients simultaneously to enable no-downtime upgrades. +* **Plugin / SPI extension points are public API** — interfaces selected by a `*ClassName` configuration setting (e.g. `AuthorizationProvider`, `EntryFilter`, `BrokerInterceptor`) have third-party implementations. Changing such an interface generally needs a PIP and must not land in maintenance-branch backports. +* Introduce new or changed behaviour behind a backward-compatible default — opt-in via configuration rather than silently changing existing deployments. diff --git a/contribute/personal-ci.md b/contribute/personal-ci.md index c91b827b4a4..8b7baf1f2f0 100644 --- a/contribute/personal-ci.md +++ b/contribute/personal-ci.md @@ -12,13 +12,17 @@ When you create a pull request from your fork, GitHub Actions provides a separat 3. You can iterate and fix issues faster in your own environment The workflow is simple: -1. Test your changes thoroughly in your fork using Personal CI -2. Once tests pass consistently, notify maintainers for final PR review +1. Keep your local `master` up-to-date with `apache/pulsar` and rebase your feature branch on it. +2. Push the feature branch to your fork to trigger CI runs there. CI runs against the PR opened in your own fork (it is normal to have a PR open in the fork *and* a PR for the same branch open in `apache/pulsar` at the same time). +3. Monitor CI status on the fork and fix failures. +4. Open the PR to `apache/pulsar` only after the fork's CI is green. + +Once the PR to `apache/pulsar` has been opened, stop rebasing as part of this loop: bring in upstream changes by merging `apache/pulsar` `master` into the PR branch instead. Rebasing a PR branch once the PR is open rewrites history and disrupts reviewers. See [`CONTRIBUTING.md` → Pull requests](https://github.com/apache/pulsar/blob/master/CONTRIBUTING.md#pull-requests). Some important notes about testing: - Pulsar has known [flaky tests](https://github.com/apache/pulsar/issues?q=is%3Aissue%20state%3Aopen%20flaky-test) that may require multiple CI runs -- Use the "Rerun failed jobs" button in GitHub Actions to retry failed workflows -- For test failures related to your changes, debug locally by running specific tests in your IDE +- In your own fork, use the "Rerun failed jobs" button in GitHub Actions to retry failed workflows. On a PR in `apache/pulsar`, comment `/pulsarbot rerun` to re-run the failed jobs once the workflow run has completed — don't push empty "trigger CI" commits. +- For test failures related to your changes, debug locally by running specific tests in your IDE or with a scoped `./gradlew :<module>:test --tests "..."` run (see [Setup and building](setup-building.md)) Critical requirement: Always create pull requests from a unique feature branch, not from your fork's `master` branch. The Personal CI process only works with feature branches. For example: diff --git a/contribute/setup-building.md b/contribute/setup-building.md index ab6c6de1bc3..749bb026eec 100644 --- a/contribute/setup-building.md +++ b/contribute/setup-building.md @@ -3,25 +3,30 @@ id: setup-building title: Setup and building --- +This page describes building the Pulsar `master` branch, which uses a **Gradle** build (migrated from Maven via [PIP-463](https://github.com/apache/pulsar/blob/master/pip/pip-463.md)). + +:::note + +Maintenance branches (`branch-4.2` and earlier) continue to use the Maven build (`./mvnw`). When working on a maintenance branch, follow the build instructions in that branch's `README.md`. + +::: + ## Prerequisites | Dependency | Description | |------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | Git | The source code of Pulsar is hosted on GitHub as a git repository. To work with the git repository, please [install git](https://git-scm.com/downloads). We highly recommend that you also [set up a Git mergetool](setup-git.md#mergetool) for resolving merge conflicts. | -| JDK | The source code of Pulsar is primarily written in Java. Therefore, you need a working Java Development Kit (JDK) to build it. It recommended to use SDKMAN to install Corretto OpenJDK 21 and 17, see ["Setting up JDKs and Maven using SDKMAN"](setup-buildtools.md) for details. | -| Maven | The source code of Pulsar is managed by [Apache Maven](https://maven.apache.org/) The recommended Maven version is 3.9.9. It is recommended to use SDKMAN to install Maven, see ["Setting up JDKs and Maven using SDKMAN"](setup-buildtools.md) for details. | +| JDK | The source code of Pulsar is primarily written in Java. Building the `master` branch requires **JDK 21 or 25** (the bytecode targets Java 17). It is recommended to use SDKMAN to install Corretto OpenJDK, see ["Setting up JDKs using SDKMAN"](setup-buildtools.md) for details. | | Zip | The build process requires Zip as a utility tool. | +| Docker | Required only for building docker images and running the container-based integration tests. | -:::note +There is no separate build tool to install: the repository includes the [Gradle Wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), so a Gradle installation is not needed. -On Windows, replace `./mvnw` with `mvnw.cmd` in the commands below. +:::note Windows -::: +Pulsar does not officially support Windows. To run Pulsar on Windows, run it in Docker — see [Run Pulsar in Docker](https://pulsar.apache.org/docs/4.0.x/getting-started-docker/). -:::note - -Pulsar does not support running server on Windows yet, you have to use Docker to run Pulsar. -Please consider checking [Run Pulsar In Docker](https://pulsar.apache.org/docs/3.1.x/getting-started-docker/) +For developing Pulsar on Windows, using [WSL2 (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install) is strongly recommended. Use the most recent WSL2 version; legacy WSL (WSL 1) is not supported. Inside WSL2, follow the Linux instructions on this page as-is (`./gradlew`). IntelliJ IDEA supports [developing in a WSL2 environment](https://www.jetbrains.com/help/idea/how-to-use-wsl-development-environment-in-product.html). ::: @@ -41,14 +46,47 @@ cd pulsar ## Build -Compile and install to local Maven repository: +Compile and assemble everything, or a single module: + +```bash +./gradlew assemble +./gradlew :pulsar-broker:assemble +``` + +Lint / verify (license headers, formatting, checkstyle) — run before pushing: ```bash -./mvnw clean install -DskipTests +./gradlew rat spotlessCheck checkstyleMain checkstyleTest +./gradlew spotlessApply # auto-fix license headers/formatting ``` +For the Gradle build infrastructure and how to change build files (convention plugins, version catalog, configuration-cache rules), see [`ARCHITECTURE.md` → Build infrastructure](https://github.com/apache/pulsar/blob/master/ARCHITECTURE.md#build-infrastructure) in the apache/pulsar repository. + +## Run tests + +Always scope test runs with `--tests` — running a whole module's test task is slow: + +```bash +# Run a single test class +./gradlew :pulsar-client-original:test --tests "ConsumerBuilderImplTest" +# Run a single test method +./gradlew :pulsar-client-original:test --tests "ConsumerBuilderImplTest.<methodName>" +# Run all tests in a specific package +./gradlew :pulsar-broker:test --tests "org.apache.pulsar.broker.admin.*" +``` + +:::note + +Several Gradle project paths do not match their directory name because the Maven artifactId is preserved — for example, directory `pulsar-client/` is the Gradle project `:pulsar-client-original`. Check `settings.gradle.kts` when a path is ambiguous. See [`ARCHITECTURE.md` → Module name vs. directory name gotcha](https://github.com/apache/pulsar/blob/master/ARCHITECTURE.md#module-name-vs-directory-name-gotcha). + +::: + +For test groups, test-related build properties, container-based integration tests, and running the full CI pipeline, see [`CONTRIBUTING.md` → Running tests](https://github.com/apache/pulsar/blob/master/CONTRIBUTING.md#running-tests) in the apache/pulsar repository and the [Personal CI guide](personal-ci.md). + ## Run +Start a standalone Pulsar service (broker + bookie + metadata in one JVM): + ```bash bin/pulsar standalone ``` @@ -57,4 +95,12 @@ bin/pulsar standalone ```bash bin/pulsar-shell -``` \ No newline at end of file +``` + +## Build docker images + +Build the `apachepulsar/pulsar:latest` and `apachepulsar/pulsar-all:latest` docker images: + +```bash +./gradlew docker # or docker-all +``` diff --git a/contribute/setup-buildtools.md b/contribute/setup-buildtools.md index 6075cceb536..ae503871b9b 100644 --- a/contribute/setup-buildtools.md +++ b/contribute/setup-buildtools.md @@ -1,42 +1,54 @@ --- id: setup-buildtools -title: Setting up JDKs and Maven using SDKMAN +title: Setting up JDKs using SDKMAN --- -## Setting up JDKs and Maven using SDKMAN +## Build tool requirements for Pulsar master branch + +Building the Pulsar `master` branch requires **JDK 21 or JDK 25** (the bytecode targets Java 17). There is no separate build tool to install: Pulsar uses a [Gradle](https://gradle.org/) build (migrated from Maven via [PIP-463](https://github.com/apache/pulsar/blob/master/pip/pip-463.md)), and the repository includes the [Gradle Wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html) — use `./gradlew`. On Windows, developing inside WSL2 is strongly recommended (see [Setup [...] + +The instructions below cover installing and managing JDK versions with [SDKMAN](https://sdkman.io/). + +:::note + +Maintenance branches (`branch-4.2` and earlier) and the [release process](release-process.md) continue to use the Maven build. When needed for those, install Maven with SDKMAN (`sdk i maven 3.9.9`) and additional JDK versions the same way as below (e.g. `sdk i java 17.0.19-amzn` for branches that build with JDK 17). + +::: + +## Setting up JDKs using SDKMAN ### Install SDKMAN See https://sdkman.io/install for detailed instructions. -### Install JDK versions 21 and 17 +### Install a JDK In Pulsar development, we use [Amazon Corretto OpenJDK](https://docs.aws.amazon.com/corretto/) to build Pulsar. -- JDK 21 for Pulsar version >= 3.3 - - code will be compiled for Java 17 with Java 21 - - Pulsar docker images are running Java 21 since 3.3.0 -- JDK 17 for Pulsar version >= 2.11 -- JDK 8 or 11 for Pulsar version < 2.11 +- JDK 21 or 25 is required for building the Pulsar `master` branch + - code is compiled for Java 17 bytecode + - Pulsar docker images run Java 21 -#### Installing Amazon Corretto OpenJDK versions 21 and 17 using SDKMAN. +#### Installing Amazon Corretto OpenJDK using SDKMAN ```shell # find out most recent Amazon Corretto release sdk l java |grep amzn -# install -sdk i java 21.0.9-amzn -sdk i java 17.0.17-amzn -# switching between versions -sdk u java 17.0.17-amzn -sdk u java 21.0.9-amzn +# install Java 21 +sdk i java 21.0.11-amzn +# install Java 25 +sdk i java 25.0.3-amzn +# install Java 17, optional step, not needed for master branch development +sdk i java 17.0.19-amzn # adding / updating aliases cd ~/.sdkman/candidates/java -rm -f 17 && ln -s 17.0.17-amzn 17 -rm -f 21 && ln -s 21.0.9-amzn 21 +rm -f 17 && ln -s 17.0.19-amzn 17 # optional step, not needed for master branch development +rm -f 21 && ln -s 21.0.11-amzn 21 +rm -f 25 && ln -s 25.0.3-amzn 25 # switching between versions using aliases sdk u java 17 sdk u java 21 +sdk u java 25 ``` #### Setting up Java version auto-switching with SDKMAN (optional) @@ -56,8 +68,6 @@ git config --global core.excludesfile $HOME/.gitignore_global echo java=21 > .sdkmanrc && cd $PWD ``` -### Install Maven +## Gradle command-line completion (optional) -```shell -sdk i maven 3.9.9 -``` \ No newline at end of file +For a better developer experience, install [Gradle command-line completion](https://docs.gradle.org/current/userguide/command_line_interface.html#sec:command_line_completion) ([gradle-completion installation instructions](https://github.com/gradle/gradle-completion?tab=readme-ov-file#gradle-completion)) for bash and zsh shells. diff --git a/contribute/setup-debugging.md b/contribute/setup-debugging.md index a8392b05fcf..5410ad7ec65 100644 --- a/contribute/setup-debugging.md +++ b/contribute/setup-debugging.md @@ -65,7 +65,7 @@ Clone and compile Pulsar from source code and run Pulsar in standalone mode with ```bash git clone https://github.com/apache/pulsar cd pulsar -mvn -Pcore-modules,-main -T 1C install -DskipTests -Dspotbugs.skip=true +./gradlew assemble OPTS="-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005" ./bin/pulsar standalone -nss -nfw ``` diff --git a/contribute/setup-git.md b/contribute/setup-git.md index dd577fc2012..37881f33ac8 100644 --- a/contribute/setup-git.md +++ b/contribute/setup-git.md @@ -73,7 +73,7 @@ git remote add forked https://github.com/your_github_id/pulsar Add separate working directories that share the local git repository ```shell -for branch in branch-3.3 branch-3.2 branch-3.1 branch-3.0; do +for branch in branch-4.2 branch-4.0 branch-3.0; do git worktree add ../pulsar-$branch $branch done ``` @@ -81,9 +81,8 @@ done After this you would have these directories in the same level as the original checked out `pulsar` directory: ``` -pulsar-branch-3.3 -pulsar-branch-3.2 -pulsar-branch-3.1 +pulsar-branch-4.2 +pulsar-branch-4.0 pulsar-branch-3.0 ``` diff --git a/contribute/setup-ide.md b/contribute/setup-ide.md index 6c697712d76..3894bf072c8 100644 --- a/contribute/setup-ide.md +++ b/contribute/setup-ide.md @@ -3,46 +3,32 @@ id: setup-ide title: Setting up an IDE --- -Apache Pulsar is using [lombok](https://projectlombok.org/), so you have to ensure your IDE setup with required plugins. +These instructions apply to the Pulsar `master` branch, which uses a Gradle build. Apache Pulsar is using [lombok](https://projectlombok.org/), so you have to ensure your IDE setup with required plugins. -## IntelliJ IDEA +:::note Windows -### Configure Project JDK to JDK 21 +Pulsar does not officially support Windows. For developing Pulsar on Windows, using [WSL2 (Windows Subsystem for Linux)](https://learn.microsoft.com/en-us/windows/wsl/install) is strongly recommended — use the most recent WSL2 version; legacy WSL (WSL 1) is not supported. IntelliJ IDEA supports [developing in a WSL2 environment](https://www.jetbrains.com/help/idea/how-to-use-wsl-development-environment-in-product.html), and VS Code supports WSL2 via the [WSL extension](https://code.visua [...] -1. Open **Project Settings**. Click **File** → **Project Structure** → **Project Settings** → **Project**. -2. Select the JDK version. From the JDK version drop-down list, select **Download JDK...** or choose an existing recent Java 21 JDK version [installed by SDKMAN](setup-buildtools.md). -3. In the download dialog, select version **17** and vendor **Amazon Corretto**. +::: -### Configure Java version for Maven +## IntelliJ IDEA -1. Open Maven Importing Settings. Click **Settings** → **Build, Execution, Deployment** → **Build Tools** → **Maven** → **Importing**. -2. For **JDK for Importer** setting, select **Use Project JDK**. This uses the Java 21 JDK for running Maven when importing the project. -3. Ensure that the JRE setting in **Maven** → **Runner** dialog is set to **Use Project JDK**. +### Open the project -:::caution +Open the Pulsar source directory in IntelliJ IDEA (**File** → **Open** and select the cloned `pulsar` directory). IntelliJ detects the Gradle build from `settings.gradle.kts` and imports the project automatically. -Some configuration in the Maven build is conditional based on the JDK version. Incorrect configuration gets chosen when the "JDK for Importer" isn't the same as the "Project JDK". +### Configure the JDK -::: +Building the `master` branch requires JDK 21 or 25 (see ["Setting up JDKs using SDKMAN"](setup-buildtools.md)). -### Configure annotation processing +1. Set the project JDK: Click **File** → **Project Structure** → **Project Settings** → **Project** and select a Java 21 JDK. From the JDK version drop-down list you can also choose **Download JDK...** and select version **21** and vendor **Amazon Corretto**. +2. Set the Gradle JVM: Click **Settings** → **Build, Execution, Deployment** → **Build Tools** → **Gradle** and set **Gradle JVM** to **Project SDK**. -1. Open Annotation Processors Settings. Click **Settings** → **Build, Execution, Deployment** → **Compiler** → **Annotation Processors**. -2. Select the following buttons: - 1. **Enable annotation processing** - 2. **Obtain processors from project classpath** - 3. Store generated sources relative to: **Module output directory** -3. Set the generated source directories to be equal to the Maven directories: - 1. Set "Production sources directory:" to "generated-sources". - 2. Set "Test sources directory:" to "generated-test-sources". -4. Click **OK**. -5. Enable the lombok plugin in intelliJ. +### Build and run delegation to Gradle -### Configure VM options for compiler process to avoid StackOverflowError with Lombok +Keep the default setting **Build and run using: Gradle** and **Run tests using: Gradle** (**Settings** → **Build, Execution, Deployment** → **Build Tools** → **Gradle**). With Gradle delegation, annotation processing (Lombok) and code generation (protobuf / lightproto) are handled by the Gradle build, so no separate annotation-processing configuration is needed. -1. Open Compiler Settings. Click **Settings** → **Build, Execution, Deployment** → **Compiler** -2. Under **Build Process**, set **Shared VM options** to `-Xss1500k` -3. Click **OK**. +Ensure the Lombok plugin is enabled in IntelliJ (it is bundled with recent IntelliJ IDEA versions). ### Configure code style @@ -56,7 +42,7 @@ Some configuration in the Maven build is conditional based on the JDK version. I 1. Install the Checkstyle-IDEA plugin. 2. Open Checkstyle Settings. Click **Settings** → **Tools** → **Checkstyle**. -3. Set **Checkstyle version** to **10.14.2**. +3. Set **Checkstyle version** to the version used by the build (defined as `checkstyle` in [`gradle/libs.versions.toml`](https://github.com/apache/pulsar/blob/master/gradle/libs.versions.toml)), or the nearest version available in the plugin. 4. Set **Scan scope** to **Only Java sources (including tests)**. 5. Click **+** button in the **Configuration** section to open a dialog to choose the checkstyle config file. 1. Enter a **Description**. For example, Pulsar. @@ -73,36 +59,13 @@ You can scan individual files by activating CheckStyle UI in the left side bar.  -### Further configuration - -* When working on the Pulsar core modules in IntelliJ, reduce the number of active projects in IntelliJ to speed up IDE actions and reduce unrelated IDE warnings. - * In IntelliJ's Maven UI's tree view under "Profiles" - * Activate "core-modules" Maven profile - * De-activate "main" Maven profile - * Run the "Reload All Maven Projects" action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ "Search Everywhere" window that gets activated by pressing the **Shift** key twice. -* Run the "Generate Sources and Update Folders For All Projects" action from the Maven UI toolbar. You can also find the action by the name in the IntelliJ "Search Everywhere" window that gets activated by pressing the **Shift** key twice. Running the action takes about 10 minutes for all projects. This is faster when the "core-modules" profile is the only active profile. - -### Troubleshooting - -* In the case of compilation errors with missing Protobuf classes, ensure to run the "Generate Sources and Update Folders For All Projects" action. -* When all the Pulsar source code doesn't compile properly in IntelliJ and there are compilation errors: - * Use the "core-modules" profile if working on the Pulsar core modules since the source code for those modules can be compiled in IntelliJ. - * Sometimes it might help to mark a specific project ignored in IntelliJ Maven UI by right-clicking the project name and select **Ignore Projects** from the menu. - * Currently, it is not always possible to run unit tests directly from the IDE because of the compilation issues. As a workaround, individual test classes can be run by using the `mvn test -Dtest=TestClassName` command. -* The above steps have all been performed, but a test still won't run. - * In this case, try the following steps: - 1. Close IntelliJ. - 2. Run `mvn clean install -DskipTests` on the command line. - 3. Reopen IntelliJ. - * If that still doesn't work: - 1. Verify Maven is using a supported version. Currently, the supported version of Maven is specified in the `<requireMavenVersion>` section of the root `pom.xml` file. - 2. Try "restart and clear caches" in IntelliJ and repeat the above steps to reload projects and generate sources. - ## Visual Studio Code (VS Code) Before starting, make sure you have installed the [Java Extension Pack](https://marketplace.visualstudio.com/items?itemName=vscjava.vscode-java-pack) in VS Code. -Since multiple versions of Java are used for Pulsar development, it is recommended to use [SDKMAN](https://sdkman.io/installation) to manage different versions of Java. The separate guide [how to setup build tools](setup-buildtools.md) explains how to install Java 17 and 21 using SDKMAN. +On Windows, use WSL2 with VS Code's [WSL extension](https://code.visualstudio.com/docs/remote/wsl), which runs the development environment inside WSL2 while the VS Code UI runs on Windows. + +It is recommended to use [SDKMAN](https://sdkman.io/installation) to manage Java versions. The separate guide [how to setup build tools](setup-buildtools.md) explains how to install Java 21 using SDKMAN. Once you have installed the Java versions using SDKMAN, you can add (or modify) the following settings to your VS Code User level `settings.json` file. Please check [VS Code documentation](https://code.visualstudio.com/docs/getstarted/settings) for more details. The simplest way to open the settings file is to run the `Preferences: Open Settings (JSON)` command from the Command Palette (`Ctrl+Shift+P` or `Cmd+Shift+P` on Mac). @@ -115,10 +78,6 @@ Once you have installed the Java versions using SDKMAN, you can add (or modify) "name": "JavaSE-21", "path": "~/.sdkman/candidates/java/21", "default": true - }, - { - "name": "JavaSE-17", - "path": "~/.sdkman/candidates/java/17" } ], "java.autobuild.enabled": false, @@ -128,20 +87,10 @@ Once you have installed the Java versions using SDKMAN, you can add (or modify) } ``` -If the `java.jdt.ls.java.home` setting doesn't point to a Java 21 JDK, you must remove `-XX:+ZGenerational` from `java.jdt.ls.vmargs` setting since Java 21 is the first version that supports generational ZGC. - The `java.autobuild.enabled` setting is set to `false` since building the Pulsar project in VS Code takes very long time. The `java.debug.settings.onBuildFailureProceed` is set to `true` so that tests can be run even when there are individual build failures. -Before running tests, you need to build the project manually at least once using the following command: - -```shell -mvn -Pcore-modules,-main -T 1C clean install -DskipTests -Dspotbugs.skip=true -Dcheckstyle.skip=true -Dlicense.skip=true -DnarPluginPhase=none -``` - -This will make the protobuf / lightproto generated classes available to the tests run in the IDE. Without this there will be errors at runtime about missing classes or Mockito related errors. - For troubleshooting, please check [Language support for Java extension documentation](https://github.com/redhat-developer/vscode-java/wiki/Troubleshooting). Adding `"java.transport": "stdio"` to the settings can help display errors in the error log if the problem is related to the language server. ## Eclipse diff --git a/contribute/testing-licenses.md b/contribute/testing-licenses.md index bd869411863..a7a4c507148 100644 --- a/contribute/testing-licenses.md +++ b/contribute/testing-licenses.md @@ -3,8 +3,20 @@ id: testing-licenses title: License header --- -All code contributed to Pulsar will be licensed under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). You need to ensure every new files you are adding have the right license header. You can add license header to your files by running following command: +All code contributed to Pulsar will be licensed under [Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0). You need to ensure every new files you are adding have the right license header. On the `master` branch, license headers are enforced by Spotless and the Apache Rat check. You can add license headers to your files by running the following command: ```bash -mvn initialize license:format +./gradlew spotlessApply +``` + +Verify license headers (and formatting / checkstyle) before pushing: + +```bash +./gradlew rat spotlessCheck checkstyleMain checkstyleTest +``` + +When you add or change a runtime dependency, the bundled-dependency `LICENSE`/`NOTICE` files of the binary distributions must be updated as well. Verify the coverage with: + +```bash +./gradlew checkBinaryLicense ```
