This is an automated email from the ASF dual-hosted git repository.
weichiu pushed a commit to branch HDDS-9225-website-v2
in repository https://gitbox.apache.org/repos/asf/ozone-site.git
The following commit(s) were added to refs/heads/HDDS-9225-website-v2 by this
push:
new 7e856d712 HDDS-14286. [Website v2] [Docs] [Core Concepts] Write
pipelines (#224)
7e856d712 is described below
commit 7e856d71210ed00c5ce3e0eaf7d2a618912f172f
Author: Bolin Lin <[email protected]>
AuthorDate: Tue Jan 13 01:55:19 2026 -0500
HDDS-14286. [Website v2] [Docs] [Core Concepts] Write pipelines (#224)
---
cspell.yaml | 2 +
.../02-replication/02-write-pipelines.md | 297 ++++++++++++++++++++-
static/img/replication/containers.svg | 5 +
static/img/replication/erasure-coding.svg | 26 ++
static/img/replication/pipelines.svg | 16 ++
static/img/replication/ratis.svg | 125 +++++++++
6 files changed, 469 insertions(+), 2 deletions(-)
diff --git a/cspell.yaml b/cspell.yaml
index 783480f38..d7f878876 100644
--- a/cspell.yaml
+++ b/cspell.yaml
@@ -172,6 +172,8 @@ words:
- snapshottable
- reencryption
- JNI
+- checksummed
+- ISA
- doublebuffer
# Company names for "Who Uses Ozone" page
- Shopee
diff --git a/docs/03-core-concepts/02-replication/02-write-pipelines.md
b/docs/03-core-concepts/02-replication/02-write-pipelines.md
index 607313472..7fd4984fa 100644
--- a/docs/03-core-concepts/02-replication/02-write-pipelines.md
+++ b/docs/03-core-concepts/02-replication/02-write-pipelines.md
@@ -1,5 +1,298 @@
+---
+sidebar_label: Write Pipelines
+---
+
# Write Pipelines
-**TODO:** File a subtask under
[HDDS-9857](https://issues.apache.org/jira/browse/HDDS-9857) and complete this
page or section.
+Write pipelines are a fundamental component of Apache Ozone's storage
architecture, enabling reliable data storage across distributed nodes. This
document provides a comprehensive overview of write pipelines, covering both
replication and erasure coding approaches, their architecture, implementation
details, and usage patterns.
+
+## What are Write Pipelines?
+
+Write pipelines are groups of Datanodes that work together as a unit to store
and replicate data in Ozone. They serve as the foundation for Ozone's data
redundancy strategy, providing:
+
+- A coordinated path for write operations across multiple nodes
+- Consistency guarantees for data replication
+- Efficient management of data distribution and storage
+
+The Storage Container Manager (SCM) is responsible for creating and managing
write pipelines, selecting appropriate Datanodes based on factors like
availability, capacity, and network topology.
+
+## Pipeline Types
+
+Ozone supports different types of write pipelines to accommodate various
durability and storage efficiency requirements:
+
+### 1. Ratis Pipelines (Replicated)
+
+Ratis pipelines use the [Apache Ratis](https://ratis.apache.org/)
implementation of the Raft consensus protocol for strongly consistent
replication.
+
+- **Structure**: Typically consists of three Datanodes (one leader, multiple
followers)
+- **Consistency**: Provides strong consistency through synchronous replication
+- **Durability**: Data is fully replicated on all nodes in the pipeline
+- **Use Case**: Default replication strategy for most Ozone deployments
+
+
+
+#### Ratis Pipeline V1: Async API
+
+The original Ozone replication pipeline (V1) uses the Ratis Async API for data
replication across multiple Datanodes:
+
+1. Client buffers data locally until a certain threshold is reached
+2. Data is sent to the leader Datanode in the pipeline
+3. Leader replicates data to follower Datanodes
+4. Once a quorum of Datanodes acknowledge the write, the operation is
considered successful
+
+This approach ensures data consistency but has some limitations in terms of
network topology awareness and buffer handling efficiency.
+
+#### Ratis Pipeline V2: Streaming API
+
+The newer Streaming Write Pipeline (V2) in Ozone leverages the Ratis Streaming
API to provide significant performance improvements:
+
+**Key Enhancements:**
+
+- Better network topology awareness
+- Elimination of unnecessary buffer copying (Netty zero copy)
+- Improved CPU and disk utilization on Datanodes
+- More efficient parallelism in data processing
+
+The Streaming Write Pipeline introduces a new network protocol that enables
direct streaming of data from client to Datanodes, reducing overhead and
improving throughput.
+
+**Configuration:**
+
+To enable the Streaming Write Pipeline (V2), configure these properties in
`ozone-site.xml`:
+
+```xml
+<property>
+ <name>hdds.container.ratis.datastream.enabled</name>
+ <value>true</value>
+ <description>Enable data stream of container</description>
+</property>
+
+<property>
+ <name>hdds.container.ratis.datastream.port</name>
+ <value>9855</value>
+ <description>The datastream port number of container</description>
+</property>
+
+<property>
+ <name>ozone.fs.datastream.enabled</name>
+ <value>true</value>
+ <description>Enable filesystem write via ratis streaming</description>
+</property>
+```
+
+### 2. Erasure Coded Pipelines
+
+Erasure coded (EC) pipelines use mathematical techniques to achieve data
durability with lower storage overhead than full replication.
+
+- **Structure**: Distributes data chunks and parity chunks across multiple
Datanodes
+- **Efficiency**: Requires less storage space than full replication (e.g., 50%
overhead instead of 200%)
+- **Recovery**: Can reconstruct data from a subset of available chunks
+- **Use Case**: Optimized for large data sets where storage efficiency is
critical
+
+
+
+#### Striping and Data Layout
+
+EC in Ozone uses a striping data model where:
+
+1. Data is divided into fixed-size chunks (typically 1MB)
+2. The chunks are organized into stripes
+3. For each stripe, parity chunks are computed
+4. The chunks are distributed across Datanodes
+
+For example, with an RS (Reed-Solomon) 6-3 configuration:
+
+- Data is split into 6 data chunks
+- 3 parity chunks are computed
+- These 9 chunks together form a "Stripe"
+- Multiple stripes using the same set of Datanodes form a "BlockGroup"
+
+This approach provides 50% storage overhead compared to 200% with 3x
replication, while maintaining similar durability guarantees.
+
+#### EC Configuration
+
+To use Erasure Coding in Ozone, you can configure EC at the bucket level or
per key:
+
+**Setting EC at bucket creation:**
+
+```bash
+ozone sh bucket create <bucket path> --type EC --replication rs-6-3-1024k
+```
+
+**Changing EC configuration for an existing bucket:**
+
+```bash
+ozone sh bucket set-replication-config <bucket path> --type EC --replication
rs-3-2-1024k
+```
+
+**Setting EC when creating a key:**
+
+```bash
+ozone sh key put <Ozone Key Object Path> <Local File> --type EC --replication
rs-6-3-1024k
+```
+
+Supported EC configurations include:
+
+- `RS-3-2-1024k`: Reed-Solomon with 3 data chunks, 2 parity chunks, 1MB chunk
size
+- `RS-6-3-1024k`: Reed-Solomon with 6 data chunks, 3 parity chunks, 1MB chunk
size (recommended)
+- `XOR-2-1-1024k`: XOR coding with 2 data chunks, 1 parity chunk, 1MB chunk
size
+
+## Pipeline Lifecycle
+
+Write pipelines follow a well-defined lifecycle, managed by the Storage
Container Manager:
+
+1. **Creation**: SCM selects appropriate Datanodes and creates a pipeline
+2. **Active**: Pipeline accepts write operations and manages replication
+3. **Closing**: Pipeline stops accepting new writes when it reaches capacity
limits
+4. **Closed**: Pipeline becomes read-only after all write operations complete
+5. **Recovery/Reconstruction**: If a node fails, SCM may initiate recovery
procedures
+
+## How Write Pipelines Work
+
+The write operation in Ozone follows these steps through the pipeline:
+
+1. **Client Request**: Client contacts the Ozone Manager (OM) to create or
write to a key
+2. **Block Allocation**: OM requests block allocation from SCM
+3. **Pipeline Selection**: SCM selects an appropriate pipeline for the write
operation
+4. **Data Transfer**: Client streams data directly to the leader Datanode in
the pipeline
+5. **Replication**: For Ratis pipelines, the leader replicates data to
followers using the Raft protocol; for EC pipelines, the client distributes
different chunks to different Datanodes
+6. **Acknowledgment**: Once all replicas are written, the client receives
confirmation
+
+
+
+### Implementation Details
+
+#### Replication Pipeline Implementation
+
+The replication write pipeline is implemented through several key classes:
+
+- **BlockOutputStream**: Base class that manages the overall write process
+- **RatisBlockOutputStream**: Implements the Ratis-specific functionality for
replication
+- **CommitWatcher**: Tracks commit status across all Datanodes in the pipeline
+- **XceiverClient**: Handles communication with Datanodes
+
+The write process follows these steps:
+
+1. Client creates a `BlockOutputStream` for the allocated block
+2. Data is written in chunks, which are buffered locally
+3. When buffer fills or flush is triggered, data is written to Datanodes
+4. Each chunk is assigned a sequential index and checksummed
+5. After all data is written, a putBlock operation finalizes the block
+
+#### EC Pipeline Implementation
+
+EC write pipeline implementation involves several key components:
+
+- **ECKeyOutputStream**: Main client-side class that manages EC writes
+- **ECChunkBuffers**: Maintains buffers for both data and parity chunks
+- **ECBlockOutputStreamEntry**: Manages Datanode connections and write
operations
+- **RawErasureEncoder**: Performs the mathematical encoding to generate parity
chunks
+
+The EC write process follows these steps:
+
+1. **Data Buffering**: Client buffers incoming data into chunks
+2. **Stripe Formation**: When all data chunks for a stripe are filled, parity
is generated
+3. **Parallel Write**: Data and parity chunks are written to different
Datanodes
+4. **Commit**: After all chunks are written, the stripe is committed
+
+## Containers and Pipelines
+
+Containers are the fundamental storage units in Ozone, and their relationship
with pipelines is essential to understand:
+
+- Each container (typically 5GB) is associated with a specific pipeline
+- Multiple containers can exist within a single pipeline
+- When a container is in the OPEN state, it actively receives data via its
pipeline
+- Once a container is CLOSED, its data can be accessed via any replica node
+
+## Comparing Replication and Erasure Coding
+
+| Feature | Replication (RATIS/THREE) | Erasure Coding (RS-6-3) |
+| ------- | ------------------------- | ----------------------- |
+| Storage Overhead | 200% (3x copies) | 50% (9 chunks for 6 data chunks) |
+| Write Performance | Higher throughput for small writes | Better for large
sequential writes |
+| Read Performance | Consistent performance, any replica can serve | Slightly
lower for intact data, reconstruction penalty for lost chunks |
+| CPU Usage | Lower | Higher (encoding/decoding overhead) |
+| Network Bandwidth | Higher during writes | Lower during writes |
+| Minimum Nodes | 3 | Depends on config (9 for RS-6-3) |
+| Use Cases | Hot data, random access, small files | Warm/cold data, large
files, archival |
+
+**When to use Replication:**
+
+- For frequently accessed "hot" data
+- For workloads with small random writes
+- When raw write performance is critical
+- When CPU resources are limited
+
+**When to use Erasure Coding:**
+
+- For "warm" or "cold" data with lower access frequency
+- For large files with sequential access patterns
+- To optimize storage costs while maintaining durability
+- For archival or backup storage
+
+## Advanced Topics
+
+### Error Handling and Recovery
+
+Both write pipelines implement sophisticated error handling:
+
+**Replication Pipeline:**
+
+- Uses Ratis consensus protocol to handle failures
+- Automatically recovers from minority Datanode failures
+- Supports pipeline reconstruction if leader fails
+- Implements idempotent operations for retries
+
+**EC Pipeline:**
+
+- Tracks failures at the individual chunk level
+- Can retry specific chunk writes to failed Datanodes
+- Maintains stripe status to ensure consistency
+- Implements checksum validation for data integrity
+
+### Performance Considerations
+
+**Optimizing Replication Write Pipeline:**
+
+- Adjust buffer sizes based on workload (`hdds.client.buffer.size.max`)
+- Configure flush periods for write-heavy workloads
+- Use Streaming Pipeline (V2) for high-throughput scenarios
+- Consider network topology when placing Datanodes
+
+**Optimizing EC Write Pipeline:**
+
+- Choose EC configuration based on workload characteristics
+- Enable ISA-L hardware acceleration for better performance
+- Adjust chunk size for optimal performance
+- Balance between parallelism and overhead
+
+### Monitoring and Metrics
+
+Both write pipelines expose metrics that can be monitored through Ozone's
Prometheus endpoint or JMX interface:
+
+**Key Metrics for Replication Pipeline:**
+
+- Write throughput
+- Average chunk write latency
+- Pipeline creation time
+- Ratis consensus latency
+
+**Key Metrics for EC Pipeline:**
+
+- Encode time per stripe
+- Chunk distribution latency
+- Success rate of first-time stripe writes
+- Parity calculation overhead
+
+## Benefits of Write Pipelines
+
+The pipeline architecture in Ozone provides several key benefits:
+
+1. **Reliability**: Automatic failure detection and recovery mechanisms
+2. **Consistency**: Strong consistency guarantees for data replication
+3. **Scalability**: Efficient management of write operations across thousands
of nodes
+4. **Flexibility**: Support for different replication strategies depending on
needs
+5. **Performance**: Optimized data flow paths that minimize network overhead
+
+## Conclusion
-Provide a high level summary of write pipelines and their purpose. Technically
this is just for context and deep understanding is not required by most users
of the system. The System Internals section of the docs will have much more
details on write pipelines.
+Write pipelines form the backbone of Apache Ozone's data redundancy
architecture, ensuring data is reliably stored and replicated across the
cluster. By understanding how write pipelines work, administrators and users
can make informed decisions about their Ozone deployment and effectively tune
it for specific use cases. Whether you need the raw performance of replication
or the storage efficiency of erasure coding, Ozone's write pipelines provide
the foundation for durable and consisten [...]
diff --git a/static/img/replication/containers.svg
b/static/img/replication/containers.svg
new file mode 100644
index 000000000..ae7bba836
--- /dev/null
+++ b/static/img/replication/containers.svg
@@ -0,0 +1,5 @@
+<svg xmlns="http://www.w3.org/2000/svg"; width="65" height="65" viewBox="0 0 16
16" fill="#357500">
+ <g transform="scale(0.8)">
+ <path d="M2.95.4a1 1 0 0 1 .8-.4h8.5a1 1 0 0 1 .8.4l2.85 3.8a.5.5 0 0 1
.1.3V15a1 1 0 0 1-1 1H1a1 1 0 0 1-1-1V4.5a.5.5 0 0 1 .1-.3zM7.5 1H3.75L1.5
4h6zm1 0v3h6l-2.25-3zM15 5H1v10h14z"/>
+ </g>
+</svg>
\ No newline at end of file
diff --git a/static/img/replication/erasure-coding.svg
b/static/img/replication/erasure-coding.svg
new file mode 100644
index 000000000..393db51de
--- /dev/null
+++ b/static/img/replication/erasure-coding.svg
@@ -0,0 +1,26 @@
+<svg width="150" height="70" viewBox="0 0 100 70"
xmlns="http://www.w3.org/2000/svg";>
+ <title>Erasure Coding Icon - Object Split</title>
+ <desc>Abstract representation of an object being split into data and parity
blocks.</desc>
+ <g transform="translate(5, -1)">
+
+ <!-- Original Object -->
+ <rect x="35" y="5" width="30" height="15" fill="#306900" rx="2"/>
+
+ <!-- Lines showing splitting from the object -->
+ <line x1="40" y1="20" x2="15" y2="40" stroke="#357500" stroke-width="1.5"/>
<!-- Data line -->
+ <line x1="45" y1="20" x2="35" y2="40" stroke="#357500" stroke-width="1.5"/>
<!-- Data line -->
+ <line x1="50" y1="20" x2="55" y2="40" stroke="#357500" stroke-width="1.5"/>
<!-- Data line -->
+ <line x1="55" y1="20" x2="75" y2="40" stroke="#96d26e" stroke-width="1.5"/>
<!-- Parity line -->
+ <line x1="60" y1="20" x2="95" y2="40" stroke="#96d26e" stroke-width="1.5"/>
<!-- Parity line -->
+
+ <!-- Data Fragments -->
+ <rect x="10" y="40" width="10" height="12" fill="#357500" rx="1"/>
+ <rect x="30" y="40" width="10" height="12" fill="#357500" rx="1"/>
+ <rect x="50" y="40" width="10" height="12" fill="#357500" rx="1"/>
+
+ <!-- Parity Fragments -->
+ <rect x="70" y="40" width="10" height="12" fill="none" stroke="#96d26e"
stroke-width="1.5" rx="1"/>
+ <rect x="90" y="40" width="10" height="12" fill="none" stroke="#96d26e"
stroke-width="1.5" rx="1"/>
+ </g>
+
+</svg>
\ No newline at end of file
diff --git a/static/img/replication/pipelines.svg
b/static/img/replication/pipelines.svg
new file mode 100644
index 000000000..ce1a20367
--- /dev/null
+++ b/static/img/replication/pipelines.svg
@@ -0,0 +1,16 @@
+<?xml version="1.0" encoding="utf-8"?>
+<svg version="1.1" id="Icons" xmlns="http://www.w3.org/2000/svg";
xmlns:xlink="http://www.w3.org/1999/xlink";
+ viewBox="0 0 28.8 28.8" width="90" height="90" xml:space="preserve">
+<style type="text/css">
+
.st0{fill:none;stroke:#357500;stroke-width:1.6;stroke-linecap:round;stroke-linejoin:round;stroke-miterlimit:10;}
+
.st1{fill:none;stroke:#357500;stroke-width:1.6;stroke-linejoin:round;stroke-miterlimit:10;}
+
.st2{fill:none;stroke:#357500;stroke-width:1.6;stroke-linecap:round;stroke-miterlimit:10;}
+</style>
+ <g transform="scale(0.9)">
+ <path class="st0" d="M11,7H5C3.9,7,3,6.1,3,5V3h10v2C13,6.1,12.1,7,11,7z"/>
+ <path class="st0"
d="M18,13v6c0,0.6-0.4,1-1,1h-2c-0.6,0-1-0.4-1-1v-6c0-0.6,0.4-1,1-1h2C17.6,12,18,12.4,18,13z"/>
+ <path class="st0"
d="M29,29H19v-2c0-1.1,0.9-2,2-2h6c1.1,0,2,0.9,2,2V29z"/>
+ <path class="st0"
d="M14,13h-1.8c-0.6,0-1.2-0.5-1.2-1.2V7H5v6v0c0,3.3,2.7,6,6,6h0h3"/>
+ <path class="st0"
d="M18,19h1.8c0.6,0,1.2,0.5,1.2,1.2V25h6v-6v0c0-3.3-2.7-6-6-6h-3"/>
+</g>
+</svg>
\ No newline at end of file
diff --git a/static/img/replication/ratis.svg b/static/img/replication/ratis.svg
new file mode 100644
index 000000000..e7187a76d
--- /dev/null
+++ b/static/img/replication/ratis.svg
@@ -0,0 +1,125 @@
+<?xml version="1.0" encoding="UTF-8" standalone="no"?>
+<svg
+ xmlns:dc="http://purl.org/dc/elements/1.1/";
+ xmlns:cc="http://creativecommons.org/ns#";
+ xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#";
+ xmlns:svg="http://www.w3.org/2000/svg";
+ xmlns="http://www.w3.org/2000/svg";
+ xmlns:sodipodi="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd";
+ xmlns:inkscape="http://www.inkscape.org/namespaces/inkscape";
+ id="svg4899"
+ version="1.1"
+ viewBox="0 0 34 36"
+ height="36mm"
+ width="34mm"
+ sodipodi:docname="ratis.svg"
+ inkscape:version="0.92.4 5da689c313, 2019-01-14">
+ <sodipodi:namedview
+ pagecolor="#ffffff"
+ bordercolor="#666666"
+ borderopacity="1"
+ objecttolerance="10"
+ gridtolerance="10"
+ guidetolerance="10"
+ inkscape:pageopacity="0"
+ inkscape:pageshadow="2"
+ inkscape:window-width="636"
+ inkscape:window-height="506"
+ id="namedview21"
+ showgrid="false"
+ inkscape:zoom="0.90509668"
+ inkscape:cx="81.367613"
+ inkscape:cy="119.96512"
+ inkscape:window-x="640"
+ inkscape:window-y="543"
+ inkscape:window-maximized="0"
+ inkscape:current-layer="svg4899" />
+ <defs
+ id="defs4893" />
+ <metadata
+ id="metadata4896">
+ <rdf:RDF>
+ <cc:Work
+ rdf:about="">
+ <dc:format>image/svg+xml</dc:format>
+ <dc:type
+ rdf:resource="http://purl.org/dc/dcmitype/StillImage"; />
+ <dc:title></dc:title>
+ </cc:Work>
+ </rdf:RDF>
+ </metadata>
+ <g
+ id="layer1"
+ transform="translate(-1.971788,-1.2114122)">
+ <g
+ transform="matrix(0.35277777,0,0,-0.35277777,7.9375016,11.237401)"
+ id="g64">
+ <path
+ id="path66"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="M 0,0 34,17"
+ inkscape:connector-curvature="0" />
+ </g>
+ <g
+ transform="matrix(0.35277777,0,0,-0.35277777,31.044446,24.113791)"
+ id="g68">
+ <path
+ id="path70"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="M 0,0 -66,37"
+ inkscape:connector-curvature="0" />
+ </g>
+ <g
+ transform="matrix(0.35277777,0,0,-0.35277777,16.580556,22.702678)"
+ id="g72">
+ <path
+ id="path74"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="M 0,0 -24,32"
+ inkscape:connector-curvature="0" />
+ </g>
+ <path
+ id="path76"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:0.7055555;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="M 12.170836,15.470735 H 4.0569456 V 7.3568457 h 8.1138904 z"
+ inkscape:connector-curvature="0" />
+ <path
+ id="path78"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:0.7055555;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="M 10.406946,34.873512 H 6.1736116 v -4.233333 h 4.2333344 z"
+ inkscape:connector-curvature="0" />
+ <g
+ transform="matrix(0.35277777,0,0,-0.35277777,8.1139596,32.404031)"
+ id="g80">
+ <path
+ id="path82"
+ style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:none"
+ d="M 0,0 V 56 Z"
+ inkscape:connector-curvature="0" />
+ </g>
+ <g
+ transform="matrix(0.35277777,0,0,-0.35277777,8.1138886,32.404068)"
+ id="g84">
+ <path
+ id="path86"
+
style="fill:none;stroke:#62cbd5;stroke-width:2;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="M 0,0 V 56"
+ inkscape:connector-curvature="0" />
+ </g>
+ <path
+ id="path88"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:0.7055555;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="m 18.520836,24.642957 h -4.23334 v -4.233334 h 4.23334 z"
+ inkscape:connector-curvature="0" />
+ <path
+ id="path90"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:0.7055555;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="m 32.984726,26.054068 h -4.23334 v -4.233334 h 4.23334 z"
+ inkscape:connector-curvature="0" />
+ <path
+ id="path92"
+
style="fill:#62cbd5;fill-opacity:1;fill-rule:nonzero;stroke:#62cbd5;stroke-width:0.7055555;stroke-linecap:butt;stroke-linejoin:miter;stroke-miterlimit:10;stroke-dasharray:none;stroke-opacity:1"
+ d="m 22.048616,7.3568457 h -4.23334 v -4.233333 h 4.23334 z"
+ inkscape:connector-curvature="0" />
+ </g>
+</svg>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
