GitHub user ZTE-EBASE edited a discussion: Add LZO compression support to gpfdist
### Description This PR adds **LZO compressed file format read/write support** to the `gpfdist` utility in Apache Cloudberry. **Key Insight**: LZO is specifically designed for speed over compression ratio. It's ideal for scenarios where decompression throughput matters more than storage savings - such as large-scale data loading into MPP databases like Cloudberry. #### Core Features **Read Support** (.lzo → Cloudberry): - ✅ **Block-level decompression**: Processes LZO data block by block, extracting uncompressed length and compressed length from each block header - ✅ **Checksum verification**: Validates data integrity using Adler32 and CRC32 checksums embedded in LZO blocks - ✅ **Enhanced error reporting**: Provides detailed, actionable error messages for various failure scenarios (truncated files, corrupt data, invalid format, checksum mismatches) **Write Support** (Cloudberry → .lzo): - ✅ **Table data unloading**: Exports table data to LZO-compressed external files - ✅ **Standard LZOP format output**: Generates files compatible with standard lzop tool and Hadoop LZO readers - ✅ **Configurable compression level**: Supports LZO compression levels (default: level 1, fastest; configurable up to level 9) **Future Potential** - Hadoop Ecosystem Integration: - 🔮 Direct query of Hive tables stored in LZO-compressed format on HDFS - 🔮 Seamless integration with Spark LZO Parquet output - 🔮 Accelerated data loading from Sqoop LZO imports ### Use case/motivation Scenario 1: High-Volume Data Loading LZO is a lightweight compression algorithm optimized for decompression speed (~500MB/s), making it ideal for scenarios requiring rapid data ingestion. Compared to BZip2 (~10MB/s) and GZIP (~100MB/s), LZO provides the fastest decompression while maintaining a reasonable compression ratio. Scenario 2: Hadoop Ecosystem Integration Many Hadoop ecosystem tools natively support LZO compression: 1. **Hive**: Supports LZO-compressed text and sequence files 2. **Spark**: Can write LZO-compressed output (via hadoop-lzo library) 3. **Sqoop**: Commonly uses LZO for bulk imports from RDBMS to HDFS ## References ### Technical Documentation - [LZO Official Website](https://www.oberhumer.com/opensource/lzo/) - [LZOP File Format Specification](http://www.lzop.org/) - [Hadoop LZO Compression Guide](https://github.com/twitter/lzo-guide) ### Cloudberry Documentation - [GPFDIST Official Docs](https://cloudberry.apache.org/docs/admin_guide/gpfdist.html) - [External Tables Guide](https://cloudberry.apache.org/docs/admin_guide/external-tables.html) ### Related Projects - [Apache Cloudberry GitHub](https://github.com/apache/cloudberry) - [Greenplum Database (upstream)](https://github.com/greenplum-db/gpdb) ### Related issues _No response_ ### Are you willing to submit a PR? - [X] Yes I am willing to submit a PR! GitHub link: https://github.com/apache/cloudberry/discussions/1886 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
