Hi David! A FileSystem class is an abstraction for the file system. It doesn't make sense to do an hsync on a file system (should the file system sync all files currently open / just the user's etc.) . With appropriate flags maybe you can make it make sense, but we don't have that functionality.
When you create() a file in the file system, you get back a FSDataOutputStream on which you can call the hsync() method . Doesn't that make more sense? On that method call, the buffer from the client is flushed to the pipeline, and an RPC goes to the NameNode to update the length <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java#L3146> (which is a fair amount of work). This is obviously not a scalable solution. Perhaps you might want to look at Kafka? If you don't need it to scale, and are fine with hammering the NameNode (really you shouldn't be), then maybe HDFS inotify <https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInotifyEventInputStream.java> can help? HTH, Ravi On Wed, Aug 9, 2017 at 5:37 AM, David Robison <[email protected]> wrote: > I understand that, when writing to a file, I can force it to update its > length on the namenode by using the following command: > > > > ((DFSOutputStream) imageWriter.getWrappedStream() > ).hsync(EnumSet.of(SyncFlag.UPDATE_LENGTH)); > > > > Is there a way to force the update without having to open a > DFSOutputStream? Can I do this from the FileSystem class or some other Java > class? The reason for this is that I am mostly writing to HDFS and only > occasionally reading. However, when I go to read, I am most often reading > the most recent data written (reading the end of the file not the > beginning). If I could force the length update at the time of reading that > would save time by not having to make sure I update the length every time I > write to the file (which is about once per second). > > > > Thanks, David > > > > *David R Robison* > > *Senior Systems Engineer* > > O. +1 512 247 3700 <(512)%20247-3700> > > M. +1 757 286 0022 <(757)%20286-0022> > > [email protected] > > *www.psgglobal.net <http://www.psgglobal.net/>* > > [image: cid:[email protected]] > > *Prometheus Security Group Global, Inc.* > > 3019 Alvin Devane Boulevard > > Building 4, Suite 450 > > Austin, TX 78741 > > [image: cid:[email protected]] > > >
