laskoviymishka commented on code in PR #25: URL: https://github.com/apache/iceberg-verification/pull/25#discussion_r4596783632
########## table-spec/partition-transforms/README.md: ########## @@ -50,7 +50,9 @@ Consumers subscribe by subdirectory: taking modulo `N`. - `truncate/` - integer, long, decimal, string, and binary truncation. - `year/`, `month/`, `day/`, and `hour/` - temporal ordinals from the Unix - epoch. `day` expected values are logical dates. + epoch. `day` expected values are logical dates. A pre-epoch value maps to the + period that contains it, so `1969-12-31T23:59:59.999999` is hour `-1` and + `1968-12-31` is year `-2`; rounding toward zero fails these cases. Review Comment: Could this say that the spec leaves the direction open? The transform rows only say "as hours from 1970-01-01 00:00:00" ([spec](https://github.com/apache/iceberg/blob/0cb5041b9c7afc046e84406d2785ed79cd54fa46/format/spec.md?plain=1#L652-L655)), and CONTRIBUTING asks a surface README to name the questions its inputs depend on. As I read Java's `DateTimeUtil` ([L215-L255](https://github.com/apache/iceberg/blob/9a11874cd731bfdf97a00fda2486e4c652d10c5e/api/src/main/java/org/apache/iceberg/util/DateTimeUtil.java#L215-L255)), it also lands on floor, via a one-unit shift and a `- 1` rather than `floorDiv`. ```suggestion `1968-12-31` is year `-2`; rounding toward zero fails these cases. The specification does not state a direction for negative values; these cases agree with Java's `DateTimeUtil`, which also rounds down. ``` ########## dev/validate-fixtures.py: ########## @@ -370,28 +389,115 @@ def _bucket_errors(path, doc): if murmur3_x86_32(data) != expected_hash: raise AssertionError("Murmur3 does not reproduce an Appendix B vector") + +def bucket_value(count, source_type, value): + return (murmur3_x86_32(encode_for_hash(source_type, value)) & 0x7FFFFFFF) % count + + +def truncate_value(width, source_type, value): + """Truncate Transform Details; remainders are positive, as Python's % is for W > 0.""" + if source_type in ("int", "long"): + if source_type == "int": + if type(value) is not int or not -(1 << 31) <= value < (1 << 31): + raise ValueError("int must be a 32-bit JSON integer") + number, bits = value, 32 + else: + if not isinstance(value, str) or not re.fullmatch(r"-?(0|[1-9][0-9]*)", value): + raise ValueError("long must be a decimal string") + number, bits = int(value), 64 + if not -(1 << 63) <= number < (1 << 63): + raise ValueError("long must fit 64 bits") + truncated = number - number % width + if truncated < -(1 << (bits - 1)): + raise ValueError("the specification does not settle integer overflow") + return truncated if source_type == "int" else str(truncated) + if source_type.startswith("decimal("): + precision, scale, unscaled = parse_decimal(source_type, value) + truncated = unscaled - unscaled % width + if len(str(abs(truncated))) > precision: + raise ValueError("the specification does not settle decimal overflow") + return format_decimal(truncated, scale) + if source_type == "string": + if not isinstance(value, str): + raise ValueError("string must be a JSON string") + if any(0xD800 <= ord(char) <= 0xDFFF for char in value): + raise ValueError("string must hold Unicode scalar values") + return value[:width] + if source_type == "binary": + if not isinstance(value, str) or not re.fullmatch(r"(?:[0-9A-F]{2})*", value): + raise ValueError("binary must be uppercase hexadecimal bytes") + return value[: 2 * width] + raise ValueError(f"unsupported truncate source type: {source_type}") + + +def temporal_value(transform, source_type, value): + """year, month, day, and hour as ordinals from the Unix epoch, rounded down.""" + if not isinstance(value, str): + raise ValueError(f"{source_type} must be an ISO string") + if source_type == "date": + if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value): + raise ValueError("date must use YYYY-MM-DD") + day = date.fromisoformat(value) + hours = None + else: + match = TIMESTAMP.fullmatch(value) + if not match or bool(match[3]) != source_type.startswith("timestamptz"): + raise ValueError("timestamp zone must match source type") + if match[2] and len(match[2]) > (9 if source_type.endswith("_ns") else 6): + raise ValueError("timestamp exceeds source type precision") + nanoseconds = epoch_nanos(value) + hours = nanoseconds // 3_600_000_000_000 + day = date(1970, 1, 1) + timedelta(days=nanoseconds // 86_400_000_000_000) + if transform == "year": + return day.year - 1970 + if transform == "month": + return (day.year - 1970) * 12 + day.month - 1 + if transform == "day": + return day.isoformat() + if transform == "hour" and hours is not None: + return hours + raise ValueError(f"unsupported {transform} source type: {source_type}") + + +def expected_value(transform, source_type, value): + """Recompute a transform result from the spec, or None for an unchecked transform.""" + if value is None: + return None + bucket = re.fullmatch(r"bucket\[([1-9][0-9]*)\]", transform) + if bucket: + return bucket_value(int(bucket[1]), source_type, value) + truncate = re.fullmatch(r"truncate\[([1-9][0-9]*)\]", transform) + if truncate: + return truncate_value(int(truncate[1]), source_type, value) + if transform in ("year", "month", "day", "hour"): + return temporal_value(transform, source_type, value) + return None + + +RECOMPUTED = re.compile(r"^(bucket\[|truncate\[|year$|month$|day$|hour$)") Review Comment: Could `RECOMPUTED` and `expected_value` share one list? A transform added only to `expected_value` is skipped here with no output. The reverse mismatch fails misleadingly: with the regex extended to `identity`, the check reports `expected null, found 5`. A sentinel from `expected_value` for unchecked transforms, in place of the trailing `return None`, would leave one place to edit. ########## dev/validate-fixtures.py: ########## @@ -370,28 +389,115 @@ def _bucket_errors(path, doc): if murmur3_x86_32(data) != expected_hash: raise AssertionError("Murmur3 does not reproduce an Appendix B vector") + +def bucket_value(count, source_type, value): + return (murmur3_x86_32(encode_for_hash(source_type, value)) & 0x7FFFFFFF) % count + + +def truncate_value(width, source_type, value): + """Truncate Transform Details; remainders are positive, as Python's % is for W > 0.""" + if source_type in ("int", "long"): + if source_type == "int": + if type(value) is not int or not -(1 << 31) <= value < (1 << 31): + raise ValueError("int must be a 32-bit JSON integer") + number, bits = value, 32 + else: + if not isinstance(value, str) or not re.fullmatch(r"-?(0|[1-9][0-9]*)", value): + raise ValueError("long must be a decimal string") + number, bits = int(value), 64 + if not -(1 << 63) <= number < (1 << 63): + raise ValueError("long must fit 64 bits") + truncated = number - number % width + if truncated < -(1 << (bits - 1)): + raise ValueError("the specification does not settle integer overflow") + return truncated if source_type == "int" else str(truncated) + if source_type.startswith("decimal("): + precision, scale, unscaled = parse_decimal(source_type, value) + truncated = unscaled - unscaled % width + if len(str(abs(truncated))) > precision: + raise ValueError("the specification does not settle decimal overflow") + return format_decimal(truncated, scale) + if source_type == "string": + if not isinstance(value, str): + raise ValueError("string must be a JSON string") + if any(0xD800 <= ord(char) <= 0xDFFF for char in value): + raise ValueError("string must hold Unicode scalar values") + return value[:width] + if source_type == "binary": + if not isinstance(value, str) or not re.fullmatch(r"(?:[0-9A-F]{2})*", value): + raise ValueError("binary must be uppercase hexadecimal bytes") + return value[: 2 * width] + raise ValueError(f"unsupported truncate source type: {source_type}") + + +def temporal_value(transform, source_type, value): + """year, month, day, and hour as ordinals from the Unix epoch, rounded down.""" + if not isinstance(value, str): + raise ValueError(f"{source_type} must be an ISO string") + if source_type == "date": + if not re.fullmatch(r"\d{4}-\d{2}-\d{2}", value): + raise ValueError("date must use YYYY-MM-DD") + day = date.fromisoformat(value) + hours = None + else: + match = TIMESTAMP.fullmatch(value) + if not match or bool(match[3]) != source_type.startswith("timestamptz"): + raise ValueError("timestamp zone must match source type") + if match[2] and len(match[2]) > (9 if source_type.endswith("_ns") else 6): + raise ValueError("timestamp exceeds source type precision") + nanoseconds = epoch_nanos(value) + hours = nanoseconds // 3_600_000_000_000 Review Comment: Should this say why it accepts negative sub-microsecond `timestamp_ns` values that `epoch_units` refuses? `epoch_units("1969-12-31T23:59:59.999999999", True)` raises "need a spec rounding rule" ([L282-L286](https://github.com/apache/iceberg-verification/blob/d12a6369d6ac8b40bfb74170e21c6487adfc5520/dev/validate-fixtures.py#L282-L286)), while `temporal_value("hour", "timestamp_ns", …)` returns `-1`. As I read it, the bucket guard is about the nanos-to-micros hash input, so a one-line comment would keep someone from aligning the two later. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
