amogh-jahagirdar commented on code in PR #10784: URL: https://github.com/apache/iceberg/pull/10784#discussion_r1702188447
########## api/src/main/java/org/apache/iceberg/actions/RepairManifests.java: ########## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iceberg.actions; + +import org.apache.iceberg.ManifestFile; + +/** An action that will repair manifests. Implementations should produce a new set of manifests. */ +public interface RepairManifests extends SnapshotUpdate<RepairManifests, RepairManifests.Result> { + + /** + * Configuration method for removing duplicate file entries and removing files which no longer + * exist in storage + */ + RepairManifests repairEntryStats(); + + /** + * Configuration method for removing duplicate file entries and removing files which no longer + * exist in storage + */ + RepairManifests repairFileEntries(); + + /** + * Configuration option for determining the rewritten and added manifests without actually + * committing the operation to the table + * + * @return this for method chaining + */ + RepairManifests dryRun(); + + interface Result { + /** Returns rewritten manifests. */ + Iterable<ManifestFile> rewrittenManifests(); + + /** Returns added manifests. */ + Iterable<ManifestFile> addedManifests(); Review Comment: I was getting more into the implementation and I'm skeptical having `addedManifests` is really required in the result for repair. I think in the original PR it was just being based off rewrite manifests. But in the end for a manifest that's being repaired for a duplicate entry or missing file or incorret stats, these are existing manifests which are being rewritten. I'm hard pressed to believe there's a case where we will be adding new manifests as part of the repair? I'm thinking about removing this. If it ends up being required we can add a default implementation later? cc @danielcweeks @szehon-ho @RussellSpitzer @nastra ########## api/src/main/java/org/apache/iceberg/actions/RepairManifests.java: ########## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iceberg.actions; + +import org.apache.iceberg.ManifestFile; + +/** An action that will repair manifests. Implementations should produce a new set of manifests. */ +public interface RepairManifests extends SnapshotUpdate<RepairManifests, RepairManifests.Result> { + + /** + * Configuration method for removing duplicate file entries and removing files which no longer + * exist in storage + */ + RepairManifests repairEntryStats(); + + /** + * Configuration method for removing duplicate file entries and removing files which no longer + * exist in storage + */ + RepairManifests repairFileEntries(); + + /** + * Configuration option for determining the rewritten and added manifests without actually + * committing the operation to the table + * + * @return this for method chaining + */ + RepairManifests dryRun(); + + interface Result { Review Comment: I've also removed "added manifests", because as I was working through the implementation a bit more, I came to the conclusion that it's not needed. I think it was just a copy from the rewrite manifests results but my thinking is that for repairing we're always working with some existing manifests and producing new manifests with rewritten contents but the "source" is always some existing manifest. For repair there isn't really an "added" manifest in the sense of how the rewrite procedure defines it. cc @danielcweeks @szehon-ho @RussellSpitzer . It's also something we could add later on with a default implementation if we realize we need it. ########## api/src/main/java/org/apache/iceberg/actions/RepairManifests.java: ########## @@ -0,0 +1,62 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iceberg.actions; + +import org.apache.iceberg.ManifestFile; + +/** An action that will repair manifests. Implementations should produce a new set of manifests. */ +public interface RepairManifests extends SnapshotUpdate<RepairManifests, RepairManifests.Result> { + + /** + * Configuration method for removing duplicate file entries and removing files which no longer + * exist in storage + */ + RepairManifests repairEntryStats(); + + /** + * Configuration method for removing duplicate file entries and removing files which no longer + * exist in storage + */ + RepairManifests repairFileEntries(); + + /** + * Configuration option for determining the rewritten and added manifests without actually + * committing the operation to the table + * + * @return this for method chaining + */ + RepairManifests dryRun(); + + interface Result { Review Comment: Good point, I don't think we should assume everything was corrected since it involves a footer read, some computation, and in all that flow things can go wrong in that for an individual file but not for another. I've added two result methods for this: ``` /** Returns the number of manifest entries for which stats were incorrect */ long entryStatsIncorrect(); /** Returns the number of manifest entries for which stats were corrected */ long entryStatsRepairedCount(); ``` I *think* it's important to distinguish between how many were repaired and how many were incorrect to begin with. WIth only the repair count, if it's 0 a user wouldn't be able to distinguish "was everything actually OK with the stats" vs "Hey a bunch of entry stats are incorrect but we couldn't actually do the repair". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@iceberg.apache.org For additional commands, e-mail: issues-h...@iceberg.apache.org
