kkrugler opened a new issue #7082:
URL: https://github.com/apache/incubator-pinot/issues/7082
In `SegmentDeletionManager.removeAgedDeletedSegments()`, there’s this bit of
code:
``` java
try {
// Check that the directory for deleted segments exists.
if (!pinotFS.isDirectory(deletedDirURI)) {
LOGGER.warn("Deleted segment directory {} does not exist or it is
not directory.", deletedDirURI.toString());
return;
}
```
But if we’ve never deleted any segments, then this "Deleted_Segments"
directory won’t exist yet. So shouldn’t this check be “if it exists && it’s not
a directory”? Otherwise we get regular warnings in our log files, which trigger
an alert, etc, etc.
Also the `HadoopPinotFS.isDirectory()` call throws a `FileNotFoundException`
if called on a non-existent file/directory, which isn’t caught in this method,
bubbles, up, and generates a `Could not get file status for
hdfs:///path/to/Deleted_Segments` error.
@mayankshriv said:
> 1. What is the expected behavior wrt Deleted Segments directory? Is it
that if directory does not exist then we don't save, or is it that we always
save, in which case we should create this directory in ControllerStarter.
> 2. Depending on what's the consensus on 1, we can either assert or make
this a debug message after adding Ken's suggested check, instead of warn.
> IMHO, we should not force DeletedSegments directory to be created (unless
user specifies in a conf), and make this a debug message, along with exists()
check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org
For additional commands, e-mail: commits-h...@pinot.apache.org
