Max file size is not configurable directly but other settings could affect max
file size, such as maximum number of blocks per file setting
dfs.namenode.fs-limits.max-blocks-per-file. This prevents the creation of
extremely large files which can degrade performance.
<property> <name>dfs.namenode.fs-limits.max-blocks-per-file</name>
<value>1048576</value> <description>Maximum number of blocks per file,
enforced by the Namenode on write. This prevents the creation of
extremely large files which can degrade
performance.</description></property>
Space Quotas, Storage Type Quotas may also be set.
https://hadoop.apache.org/docs/r3.0.3/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html
https://www.informit.com/articles/article.aspx?p=2755708&seqNum=4
On Thursday, May 14, 2020, 09:19:44 a.m. UTC, J M
<[email protected]> wrote:
Hi,
I don't have much knowledge about Hadoop/HDFS, my question can be simple, or
not...
Then, I have a Hadoop/HDFS environment, but my disks are not very big.
One applicacion is writing in files. But, sometimes the disk is filled with
large file sizes.
Then, my question is:
Exist any form to limitating the maximum file sizes written in HDFS?
I was thinking of something like:
When a file have a size of >= 1Gb, then new data written to this file, cause
that the first data written to this file deleted. In this way the file size
would always be limited, as a rolled file.
Howto do this task?
Regards,Cesar Jorge
