Author: buildbot
Date: Wed Dec 17 07:19:49 2014
New Revision: 933087
Log:
Production update by buildbot for camel
Modified:
websites/production/camel/content/book-component-appendix.html
websites/production/camel/content/book-in-one-page.html
websites/production/camel/content/cache/main.pageCache
websites/production/camel/content/download.html
websites/production/camel/content/hdfs.html
websites/production/camel/content/hdfs2.html
Modified: websites/production/camel/content/book-component-appendix.html
==============================================================================
--- websites/production/camel/content/book-component-appendix.html (original)
+++ websites/production/camel/content/book-component-appendix.html Wed Dec 17
07:19:49 2014
@@ -1415,11 +1415,11 @@ template.send("direct:alias-verify&
</div>
</div>
<p>The <strong>cxf:</strong> component provides integration with <a
shape="rect" href="http://cxf.apache.org";>Apache CXF</a> for connecting to
JAX-WS services hosted in CXF.</p><p><style type="text/css">/*<![CDATA[*/
-div.rbtoc1418714315607 {padding: 0px;}
-div.rbtoc1418714315607 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418714315607 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800713901 {padding: 0px;}
+div.rbtoc1418800713901 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800713901 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style></p><div class="toc-macro rbtoc1418714315607">
+/*]]>*/</style></p><div class="toc-macro rbtoc1418800713901">
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-CXFComponent">CXF
Component</a>
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-URIformat">URI
format</a></li><li><a shape="rect" href="#CXF-Options">Options</a>
<ul class="toc-indentation"><li><a shape="rect"
href="#CXF-Thedescriptionsofthedataformats">The descriptions of the
dataformats</a>
@@ -4827,9 +4827,13 @@ template.sendBodyAndHeader("direct:
</div></div><h3 id="BookComponentAppendix-URIformat.24">URI format</h3><div
class="code panel pdl" style="border-width: 1px;"><div class="codeContent
panelContent pdl">
<script class="theme: Default; brush: java; gutter: false"
type="syntaxhighlighter"><![CDATA[hdfs://hostname[:port][/path][?options]
]]></script>
-</div></div><p>You can append query options to the URI in the following
format, <code>?option=value&option=value&...</code><br clear="none">
The path is treated in the following way:</p><ol><li>as a consumer, if it's a
file, it just reads the file, otherwise if it represents a directory it scans
all the file under the path satisfying the configured pattern. All the files
under that directory must be of the same type.</li><li>as a producer, if at
least one split strategy is defined, the path is considered a directory and
under that directory the producer creates a different file per split named
using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a>.</li></ol><h3
id="BookComponentAppendix-Options.19">Options</h3><div
class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th
colspan="1" rowspan="1" class="confluenceTh"><p> Name </p></th><th colspan="1"
rowspan="1" class="confluenceTh"><p> Default Value </p></th><th colspan="1"
rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>overwrite</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The file can be
overwritten </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>append</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>false</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> Append to existing file.
Notice that not all HDFS file systems support the append option.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>bufferSize</code> </p></td><td colspan="1" rowspan="1" cla
ss="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The buffer size used by HDFS </p></td></tr><tr><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>replication</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>3</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The HDFS
replication factor </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>blockSize</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>67108864</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> The size of the HDFS blocks
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>fileType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>NORMAL_FILE</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> It can be SEQUENCE_FILE, MAP_FILE,
ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop </p></td></tr>
<tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>fileSystemType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>HDFS</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> It can be LOCAL for local filesystem
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>keyType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>NULL</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The type for the key in case of sequence or map files.
See below. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>valueType</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>TEXT</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence
or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>splitStrategy</code> </p></td><td colspan="1"
rowsp
an="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> A string describing the strategy on how to split the
file based on different criteria. See below. </p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>openedSuffix</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>opened</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> When a file is
opened for reading/writing the file is renamed with this suffix to avoid to
read it during the writing phase. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>readSuffix</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>read</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> Once the file has been read is renamed
with this suffix to avoid to read it again. </p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>initialDelay</code> </p></td><td co
lspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> For the consumer, how much to
wait (milliseconds) before to start scanning the directory.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>delay</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The interval (milliseconds) between the directory
scans. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>pattern</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>*</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The pattern used for scanning the directory
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>chunkSize</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1"
class="co
nfluenceTd"><p> When reading a normal file, this is split into chunks
producing a message per chunk. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>connectOnStartup</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <strong>Camel 2.9.3/2.10.1:</strong>
Whether to connect to the HDFS file system on starting the producer/consumer.
If <code>false</code> then the connection is created on-demand. Notice that
HDFS may take up till 15 minutes to establish a connection, as it has hardcoded
45 x 20 sec redelivery. By setting this option to <code>false</code> allows
your application to startup, and not block for up till 15 minutes.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>owner</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <strong>Camel 2.13/2
.12.4:</strong> The file owner must match this owner for the consumer to
pickup the file. Otherwise the file is skipped.
</p></td></tr></tbody></table></div>
-</div><h4 id="BookComponentAppendix-KeyTypeandValueType">KeyType and
ValueType</h4><ul><li>NULL it means that the key or the value is
absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a
BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer
class</li><li>INT for writing java integer</li><li>FLOAT for writing java
float</li><li>LONG for writing java long</li><li>DOUBLE for writing java
double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used
with everything else, for example, in Camel a file is sent around as an
InputStream, int this case is written in a sequence file or a map file as a
sequence of bytes.</p><h3
id="BookComponentAppendix-SplittingStrategy">Splitting Strategy</h3><p>In the
current version of Hadoop opening a file in append mode is disabled since it's
not very reliable. So, for the moment, it's only possible to create new files.
The Camel HDFS endpoint tries to solve this problem in this way:</p><ul><
li>If the split strategy option has been defined, the hdfs path will be used
as a directory and files will be created using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting
condition is met, a new file is created.<br clear="none"> The splitStrategy
option is defined as a string with the following syntax:<br clear="none">
splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where
<ST> can be:</p><ul><li>BYTES a new file is created, and the old is
closed when the number of written bytes is more than
<value></li><li>MESSAGES a new file is created, and the old is closed
when the number of written messages is more than <value></li><li>IDLE a
new file is created, and the old is closed when no writing happened in the last
<value> milliseconds</li></ul> <div class="aui-message warning
shadowed information-macro">
+</div></div><p>You can append query options to the URI in the following
format, <code>?option=value&option=value&...</code><br clear="none">
The path is treated in the following way:</p><ol><li>as a consumer, if it's a
file, it just reads the file, otherwise if it represents a directory it scans
all the file under the path satisfying the configured pattern. All the files
under that directory must be of the same type.</li><li>as a producer, if at
least one split strategy is defined, the path is considered a directory and
under that directory the producer creates a different file per split named
using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a>.</li></ol> <div
class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning">Icon</span>
+ <div class="message-content">
+ <p><span>When consuming from hdfs then in normal
mode, a file is split into chunks, producing a message per chunk. You can
configure the size of the chunk using the chunkSize option. If you want to read
from hdfs and write to a regular file using the file component, then you can
use the fileMode=Append to append each of the chunks together.</span></p>
+ </div>
+ </div>
+<p> </p><h3 id="BookComponentAppendix-Options.19">Options</h3><div
class="confluenceTableSmall"><div class="table-wrap"><table
class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1"
class="confluenceTh"><p>Name</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Default Value</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>overwrite</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>The file can be
overwritten</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>append</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>false</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>Append to existing file. Notice that not all HDFS file
systems support the append option.</p></td></tr><tr><td colspan="1" rowspan="1"
class=
"confluenceTd"><p><code>bufferSize</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>The buffer size used by HDFS</p></td></tr><tr><td
colspan="1" rowspan="1"
class="confluenceTd"><p><code>replication</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>3</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The HDFS replication
factor</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>blockSize</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>67108864</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>The size of the HDFS
blocks</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>fileType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>NORMAL_FILE</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>It can be SEQUENCE_FILE,
MAP_FILE
, ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>fileSystemType</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><code>HDFS</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>It can be LOCAL for local
filesystem</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>keyType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>NULL</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or
map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>valueType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>TEXT</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or
map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>splitStrategy</co
de></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>A string describing the strategy on how to split the
file based on different criteria. See below.</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>openedSuffix</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><code>opened</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>When a file is opened for
reading/writing the file is renamed with this suffix to avoid to read it during
the writing phase.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>readSuffix</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>read</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>Once the file has been read is renamed with
this suffix to avoid to read it again.</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>initialDelay</code>
</p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>For the consumer, how much to wait (milliseconds)
before to start scanning the directory.</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>delay</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The interval (milliseconds) between the
directory scans.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>pattern</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>*</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The pattern used for scanning the
directory</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>chunkSize</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1"
rowspan="1" class="confluenc
eTd"><p>When reading a normal file, this is split into chunks producing a
message per chunk.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>connectOnStartup</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><strong>Camel 2.9.3/2.10.1:</strong>
Whether to connect to the HDFS file system on starting the producer/consumer.
If <code>false</code> then the connection is created on-demand. Notice that
HDFS may take up till 15 minutes to establish a connection, as it has hardcoded
45 x 20 sec redelivery. By setting this option to <code>false</code> allows
your application to startup, and not block for up till 15
minutes.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>owner</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><strong>Camel 2.13/2.12.4:</strong> Th
e file owner must match this owner for the consumer to pickup the file.
Otherwise the file is skipped.</p></td></tr></tbody></table></div></div><h4
id="BookComponentAppendix-KeyTypeandValueType">KeyType and
ValueType</h4><ul><li>NULL it means that the key or the value is
absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a
BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer
class</li><li>INT for writing java integer</li><li>FLOAT for writing java
float</li><li>LONG for writing java long</li><li>DOUBLE for writing java
double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used
with everything else, for example, in Camel a file is sent around as an
InputStream, int this case is written in a sequence file or a map file as a
sequence of bytes.</p><h3
id="BookComponentAppendix-SplittingStrategy">Splitting Strategy</h3><p>In the
current version of Hadoop opening a file in append mode is disabled since it's
not very relia
ble. So, for the moment, it's only possible to create new files. The Camel
HDFS endpoint tries to solve this problem in this way:</p><ul><li>If the split
strategy option has been defined, the hdfs path will be used as a directory and
files will be created using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting
condition is met, a new file is created.<br clear="none"> The splitStrategy
option is defined as a string with the following syntax:<br clear="none">
splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where
<ST> can be:</p><ul><li>BYTES a new file is created, and the old is
closed when the number of written bytes is more than
<value></li><li>MESSAGES a new file is created, and the old is closed
when the number of written messages is more than <value></li><li>IDLE a
new file is created, and the old is closed when no writing happened in the last
<value> milliseconds</li></u
l> <div class="aui-message warning shadowed information-macro">
<span class="aui-icon icon-warning">Icon</span>
<div class="message-content">
<p>note that this strategy currently requires
either setting an IDLE value or setting the HdfsConstants.HDFS_CLOSE header to
false to use the BYTES/MESSAGES configuration...otherwise, the file will be
closed with each message</p>
@@ -4838,9 +4842,7 @@ template.sendBodyAndHeader("direct:
<p>for example:</p><div class="code panel pdl" style="border-width: 1px;"><div
class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false"
type="syntaxhighlighter"><![CDATA[hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
]]></script>
-</div></div><p>it means: a new file is created either when it has been idle
for more than 1 second or if more than 5 bytes have been written. So, running
<code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have
been created.</p><h3 id="BookComponentAppendix-MessageHeaders.4">Message
Headers</h3><p>The following headers are supported by this component:</p><h4
id="BookComponentAppendix-Produceronly">Producer only</h4><div
class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th
colspan="1" rowspan="1" class="confluenceTh"><p> Header </p></th><th
colspan="1" rowspan="1" class="confluenceTh"><p> Description
</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>CamelFileName</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <strong>Camel 2.13:</strong> Specifies the name of the
file to write (relative to the endpoint path). The name can be a
<code>String</code> or an <a shape="rect" href="expression.html">Expression</a>
object. Only relevant when not using a split strategy.
</p></td></tr></tbody></table></div>
-</div><h3 id="BookComponentAppendix-Controllingtoclosefilestream">Controlling
to close file stream</h3><p><strong>Available as of Camel
2.10.4</strong></p><p>When using the <a shape="rect" href="hdfs.html">HDFS</a>
producer <strong>without</strong> a split strategy, then the file output stream
is by default closed after the write. However you may want to keep the stream
open, and only explicitly close the stream later. For that you can use the
header <code>HdfsConstants.HDFS_CLOSE</code> (value =
<code>"CamelHdfsClose"</code>) to control this. Setting this value to a boolean
allows you to explicit control whether the stream should be closed or
not.</p><p>Notice this does not apply if you use a split strategy, as there are
various strategies that can control when the stream is closed.</p><h3
id="BookComponentAppendix-UsingthiscomponentinOSGi">Using this component in
OSGi</h3><p>This component is fully functional in an OSGi environment, however,
it requires some actions from the user.
Hadoop uses the thread context class loader in order to load resources.
Usually, the thread context classloader will be the bundle class loader of the
bundle that contains the routes. So, the default configuration files need to be
visible from the bundle class loader. A typical way to deal with it is to keep
a copy of core-default.xml in your bundle root. That file can be found in the
hadoop-common.jar.</p>
+</div></div><p>it means: a new file is created either when it has been idle
for more than 1 second or if more than 5 bytes have been written. So, running
<code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have
been created.</p><h3 id="BookComponentAppendix-MessageHeaders.4">Message
Headers</h3><p>The following headers are supported by this component:</p><h4
id="BookComponentAppendix-Produceronly">Producer only</h4><div
class="confluenceTableSmall"><div class="table-wrap"><table
class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1"
class="confluenceTh"><p>Header</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>CamelFileName</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13:</strong>
Specifies the name of the file to write (relative to the endpoint path). The
name can be a <code>String</code> or an <a shape="rect" href="
expression.html">Expression</a> object. Only relevant when not using a split
strategy.</p></td></tr></tbody></table></div></div><h3
id="BookComponentAppendix-Controllingtoclosefilestream">Controlling to close
file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When
using the <a shape="rect" href="hdfs.html">HDFS</a> producer
<strong>without</strong> a split strategy, then the file output stream is by
default closed after the write. However you may want to keep the stream open,
and only explicitly close the stream later. For that you can use the header
<code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>)
to control this. Setting this value to a boolean allows you to explicit control
whether the stream should be closed or not.</p><p>Notice this does not apply if
you use a split strategy, as there are various strategies that can control when
the stream is closed.</p><h3
id="BookComponentAppendix-UsingthiscomponentinOSGi">Using this component
in OSGi</h3><p>This component is fully functional in an OSGi environment,
however, it requires some actions from the user. Hadoop uses the thread context
class loader in order to load resources. Usually, the thread context
classloader will be the bundle class loader of the bundle that contains the
routes. So, the default configuration files need to be visible from the bundle
class loader. A typical way to deal with it is to keep a copy of
core-default.xml in your bundle root. That file can be found in the
hadoop-common.jar.</p>
<h2 id="BookComponentAppendix-HibernateComponent">Hibernate Component</h2>
<p>The <strong>hibernate:</strong> component allows you to work with databases
using Hibernate as the object relational mapping technology to map POJOs to
database tables. The <strong>camel-hibernate</strong> library is provided by
the <a shape="rect" class="external-link"
href="http://code.google.com/p/camel-extra/"; rel="nofollow">Camel Extra</a>
project which hosts all *GPL related components for Camel.</p>
Modified: websites/production/camel/content/book-in-one-page.html
==============================================================================
--- websites/production/camel/content/book-in-one-page.html (original)
+++ websites/production/camel/content/book-in-one-page.html Wed Dec 17 07:19:49
2014
@@ -4126,11 +4126,11 @@ While not actual tutorials you might fin
</div>
</div>
<h2 id="BookInOnePage-Preface">Preface</h2><p>This tutorial aims to guide the
reader through the stages of creating a project which uses Camel to facilitate
the routing of messages from a JMS queue to a <a shape="rect"
class="external-link" href="http://www.springramework.org";
rel="nofollow">Spring</a> service. The route works in a synchronous fashion
returning a response to the client.</p><p><style type="text/css">/*<![CDATA[*/
-div.rbtoc1418732329449 {padding: 0px;}
-div.rbtoc1418732329449 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418732329449 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800724259 {padding: 0px;}
+div.rbtoc1418800724259 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800724259 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style></p><div class="toc-macro rbtoc1418732329449">
+/*]]>*/</style></p><div class="toc-macro rbtoc1418800724259">
<ul class="toc-indentation"><li><a shape="rect"
href="#Tutorial-JmsRemoting-TutorialonSpringRemotingwithJMS">Tutorial on Spring
Remoting with JMS</a></li><li><a shape="rect"
href="#Tutorial-JmsRemoting-Preface">Preface</a></li><li><a shape="rect"
href="#Tutorial-JmsRemoting-Prerequisites">Prerequisites</a></li><li><a
shape="rect"
href="#Tutorial-JmsRemoting-Distribution">Distribution</a></li><li><a
shape="rect" href="#Tutorial-JmsRemoting-About">About</a></li><li><a
shape="rect" href="#Tutorial-JmsRemoting-CreatetheCamelProject">Create the
Camel Project</a>
<ul class="toc-indentation"><li><a shape="rect"
href="#Tutorial-JmsRemoting-UpdatethePOMwithDependencies">Update the POM with
Dependencies</a></li></ul>
</li><li><a shape="rect" href="#Tutorial-JmsRemoting-WritingtheServer">Writing
the Server</a>
@@ -6316,11 +6316,11 @@ So we completed the last piece in the pi
<style type="text/css">/*<![CDATA[*/
-div.rbtoc1418732330791 {padding: 0px;}
-div.rbtoc1418732330791 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418732330791 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800724844 {padding: 0px;}
+div.rbtoc1418800724844 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800724844 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style><div class="toc-macro rbtoc1418732330791">
+/*]]>*/</style><div class="toc-macro rbtoc1418800724844">
<ul class="toc-indentation"><li><a shape="rect"
href="#Tutorial-AXIS-Camel-TutorialusingAxis1.4withApacheCamel">Tutorial using
Axis 1.4 with Apache Camel</a>
<ul class="toc-indentation"><li><a shape="rect"
href="#Tutorial-AXIS-Camel-Prerequisites">Prerequisites</a></li><li><a
shape="rect"
href="#Tutorial-AXIS-Camel-Distribution">Distribution</a></li><li><a
shape="rect"
href="#Tutorial-AXIS-Camel-Introduction">Introduction</a></li><li><a
shape="rect" href="#Tutorial-AXIS-Camel-SettinguptheprojecttorunAxis">Setting
up the project to run Axis</a>
<ul class="toc-indentation"><li><a shape="rect"
href="#Tutorial-AXIS-Camel-Maven2">Maven 2</a></li><li><a shape="rect"
href="#Tutorial-AXIS-Camel-wsdl">wsdl</a></li><li><a shape="rect"
href="#Tutorial-AXIS-Camel-ConfiguringAxis">Configuring Axis</a></li><li><a
shape="rect" href="#Tutorial-AXIS-Camel-RunningtheExample">Running the
Example</a></li></ul>
@@ -19229,11 +19229,11 @@ template.send("direct:alias-verify&
</div>
</div>
<p>The <strong>cxf:</strong> component provides integration with <a
shape="rect" href="http://cxf.apache.org";>Apache CXF</a> for connecting to
JAX-WS services hosted in CXF.</p><p><style type="text/css">/*<![CDATA[*/
-div.rbtoc1418732349169 {padding: 0px;}
-div.rbtoc1418732349169 ul {list-style: disc;margin-left: 0px;}
-div.rbtoc1418732349169 li {margin-left: 0px;padding-left: 0px;}
+div.rbtoc1418800730865 {padding: 0px;}
+div.rbtoc1418800730865 ul {list-style: disc;margin-left: 0px;}
+div.rbtoc1418800730865 li {margin-left: 0px;padding-left: 0px;}
-/*]]>*/</style></p><div class="toc-macro rbtoc1418732349169">
+/*]]>*/</style></p><div class="toc-macro rbtoc1418800730865">
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-CXFComponent">CXF
Component</a>
<ul class="toc-indentation"><li><a shape="rect" href="#CXF-URIformat">URI
format</a></li><li><a shape="rect" href="#CXF-Options">Options</a>
<ul class="toc-indentation"><li><a shape="rect"
href="#CXF-Thedescriptionsofthedataformats">The descriptions of the
dataformats</a>
@@ -22641,9 +22641,13 @@ template.sendBodyAndHeader("direct:
</div></div><h3 id="BookInOnePage-URIformat.25">URI format</h3><div
class="code panel pdl" style="border-width: 1px;"><div class="codeContent
panelContent pdl">
<script class="theme: Default; brush: java; gutter: false"
type="syntaxhighlighter"><![CDATA[hdfs://hostname[:port][/path][?options]
]]></script>
-</div></div><p>You can append query options to the URI in the following
format, <code>?option=value&option=value&...</code><br clear="none">
The path is treated in the following way:</p><ol><li>as a consumer, if it's a
file, it just reads the file, otherwise if it represents a directory it scans
all the file under the path satisfying the configured pattern. All the files
under that directory must be of the same type.</li><li>as a producer, if at
least one split strategy is defined, the path is considered a directory and
under that directory the producer creates a different file per split named
using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a>.</li></ol><h3
id="BookInOnePage-Options.40">Options</h3><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th
colspan="1" rowspan="1" class="confluenceTh"><p> Name </p></th><th colspan="1"
rowspan="1" class="confluenceTh"><p> Default Value </p></th><th colspan="1"
rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>overwrite</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The file can be
overwritten </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>append</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>false</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> Append to existing file.
Notice that not all HDFS file systems support the append option.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>bufferSize</code> </p></td><td colspan="1" rowspan="1" cla
ss="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The buffer size used by HDFS </p></td></tr><tr><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>replication</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>3</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The HDFS
replication factor </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>blockSize</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>67108864</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> The size of the HDFS blocks
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>fileType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>NORMAL_FILE</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> It can be SEQUENCE_FILE, MAP_FILE,
ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop </p></td></tr>
<tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>fileSystemType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>HDFS</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> It can be LOCAL for local filesystem
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>keyType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>NULL</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The type for the key in case of sequence or map files.
See below. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>valueType</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>TEXT</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence
or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>splitStrategy</code> </p></td><td colspan="1"
rowsp
an="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> A string describing the strategy on how to split the
file based on different criteria. See below. </p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>openedSuffix</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>opened</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> When a file is
opened for reading/writing the file is renamed with this suffix to avoid to
read it during the writing phase. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>readSuffix</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>read</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> Once the file has been read is renamed
with this suffix to avoid to read it again. </p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>initialDelay</code> </p></td><td co
lspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> For the consumer, how much to
wait (milliseconds) before to start scanning the directory.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>delay</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The interval (milliseconds) between the directory
scans. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>pattern</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>*</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The pattern used for scanning the directory
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>chunkSize</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1"
class="co
nfluenceTd"><p> When reading a normal file, this is split into chunks
producing a message per chunk. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>connectOnStartup</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <strong>Camel 2.9.3/2.10.1:</strong>
Whether to connect to the HDFS file system on starting the producer/consumer.
If <code>false</code> then the connection is created on-demand. Notice that
HDFS may take up till 15 minutes to establish a connection, as it has hardcoded
45 x 20 sec redelivery. By setting this option to <code>false</code> allows
your application to startup, and not block for up till 15 minutes.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>owner</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <strong>Camel 2.13/2
.12.4:</strong> The file owner must match this owner for the consumer to
pickup the file. Otherwise the file is skipped.
</p></td></tr></tbody></table></div>
-</div><h4 id="BookInOnePage-KeyTypeandValueType">KeyType and
ValueType</h4><ul><li>NULL it means that the key or the value is
absent</li><li>BYTE for writing a byte, the java Byte class is mapped into a
BYTE</li><li>BYTES for writing a sequence of bytes. It maps the java ByteBuffer
class</li><li>INT for writing java integer</li><li>FLOAT for writing java
float</li><li>LONG for writing java long</li><li>DOUBLE for writing java
double</li><li>TEXT for writing java strings</li></ul><p>BYTES is also used
with everything else, for example, in Camel a file is sent around as an
InputStream, int this case is written in a sequence file or a map file as a
sequence of bytes.</p><h3 id="BookInOnePage-SplittingStrategy">Splitting
Strategy</h3><p>In the current version of Hadoop opening a file in append mode
is disabled since it's not very reliable. So, for the moment, it's only
possible to create new files. The Camel HDFS endpoint tries to solve this
problem in this way:</p><ul><li>If the split
strategy option has been defined, the hdfs path will be used as a directory
and files will be created using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting
condition is met, a new file is created.<br clear="none"> The splitStrategy
option is defined as a string with the following syntax:<br clear="none">
splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where
<ST> can be:</p><ul><li>BYTES a new file is created, and the old is
closed when the number of written bytes is more than
<value></li><li>MESSAGES a new file is created, and the old is closed
when the number of written messages is more than <value></li><li>IDLE a
new file is created, and the old is closed when no writing happened in the last
<value> milliseconds</li></ul> <div class="aui-message warning
shadowed information-macro">
+</div></div><p>You can append query options to the URI in the following
format, <code>?option=value&option=value&...</code><br clear="none">
The path is treated in the following way:</p><ol><li>as a consumer, if it's a
file, it just reads the file, otherwise if it represents a directory it scans
all the file under the path satisfying the configured pattern. All the files
under that directory must be of the same type.</li><li>as a producer, if at
least one split strategy is defined, the path is considered a directory and
under that directory the producer creates a different file per split named
using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a>.</li></ol> <div
class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning">Icon</span>
+ <div class="message-content">
+ <p><span>When consuming from hdfs then in normal
mode, a file is split into chunks, producing a message per chunk. You can
configure the size of the chunk using the chunkSize option. If you want to read
from hdfs and write to a regular file using the file component, then you can
use the fileMode=Append to append each of the chunks together.</span></p>
+ </div>
+ </div>
+<p> </p><h3 id="BookInOnePage-Options.40">Options</h3><div
class="confluenceTableSmall"><div class="table-wrap"><table
class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1"
class="confluenceTh"><p>Name</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Default Value</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>overwrite</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>The file can be
overwritten</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>append</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>false</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>Append to existing file. Notice that not all HDFS file
systems support the append option.</p></td></tr><tr><td colspan="1" rowspan="1"
class="conflue
nceTd"><p><code>bufferSize</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>The buffer size used by HDFS</p></td></tr><tr><td
colspan="1" rowspan="1"
class="confluenceTd"><p><code>replication</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>3</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The HDFS replication
factor</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>blockSize</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>67108864</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>The size of the HDFS
blocks</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>fileType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>NORMAL_FILE</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>It can be SEQUENCE_FILE,
MAP_FILE, ARRAY_
FILE, or BLOOMMAP_FILE, see Hadoop</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>fileSystemType</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><code>HDFS</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>It can be LOCAL for local
filesystem</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>keyType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>NULL</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or
map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>valueType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>TEXT</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or
map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>splitStrategy</code></p><
/td><td colspan="1" rowspan="1" class="confluenceTd"><p> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>A string describing the
strategy on how to split the file based on different criteria. See
below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>openedSuffix</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>opened</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>When a file is opened for reading/writing
the file is renamed with this suffix to avoid to read it during the writing
phase.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>readSuffix</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>read</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>Once the file has been read is renamed with
this suffix to avoid to read it again.</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>initialDelay</code></p></td
><td colspan="1" rowspan="1"
>class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1"
>class="confluenceTd"><p>For the consumer, how much to wait (milliseconds)
>before to start scanning the directory.</p></td></tr><tr><td colspan="1"
>rowspan="1" class="confluenceTd"><p><code>delay</code></p></td><td
>colspan="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td
>colspan="1" rowspan="1" class="confluenceTd"><p>The interval (milliseconds)
>between the directory scans.</p></td></tr><tr><td colspan="1" rowspan="1"
>class="confluenceTd"><p><code>pattern</code></p></td><td colspan="1"
>rowspan="1" class="confluenceTd"><p><code>*</code></p></td><td colspan="1"
>rowspan="1" class="confluenceTd"><p>The pattern used for scanning the
>directory</p></td></tr><tr><td colspan="1" rowspan="1"
>class="confluenceTd"><p><code>chunkSize</code></p></td><td colspan="1"
>rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1"
>rowspan="1" class="confluenceTd"><p>
When reading a normal file, this is split into chunks producing a message per
chunk.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>connectOnStartup</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><strong>Camel 2.9.3/2.10.1:</strong>
Whether to connect to the HDFS file system on starting the producer/consumer.
If <code>false</code> then the connection is created on-demand. Notice that
HDFS may take up till 15 minutes to establish a connection, as it has hardcoded
45 x 20 sec redelivery. By setting this option to <code>false</code> allows
your application to startup, and not block for up till 15
minutes.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>owner</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><strong>Camel 2.13/2.12.4:</strong> The file o
wner must match this owner for the consumer to pickup the file. Otherwise the
file is skipped.</p></td></tr></tbody></table></div></div><h4
id="BookInOnePage-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL
it means that the key or the value is absent</li><li>BYTE for writing a byte,
the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence
of bytes. It maps the java ByteBuffer class</li><li>INT for writing java
integer</li><li>FLOAT for writing java float</li><li>LONG for writing java
long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java
strings</li></ul><p>BYTES is also used with everything else, for example, in
Camel a file is sent around as an InputStream, int this case is written in a
sequence file or a map file as a sequence of bytes.</p><h3
id="BookInOnePage-SplittingStrategy">Splitting Strategy</h3><p>In the current
version of Hadoop opening a file in append mode is disabled since it's not very
reliable. So, for the moment,
it's only possible to create new files. The Camel HDFS endpoint tries to
solve this problem in this way:</p><ul><li>If the split strategy option has
been defined, the hdfs path will be used as a directory and files will be
created using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting
condition is met, a new file is created.<br clear="none"> The splitStrategy
option is defined as a string with the following syntax:<br clear="none">
splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where
<ST> can be:</p><ul><li>BYTES a new file is created, and the old is
closed when the number of written bytes is more than
<value></li><li>MESSAGES a new file is created, and the old is closed
when the number of written messages is more than <value></li><li>IDLE a
new file is created, and the old is closed when no writing happened in the last
<value> milliseconds</li></ul> <div class="aui-me
ssage warning shadowed information-macro">
<span class="aui-icon icon-warning">Icon</span>
<div class="message-content">
<p>note that this strategy currently requires
either setting an IDLE value or setting the HdfsConstants.HDFS_CLOSE header to
false to use the BYTES/MESSAGES configuration...otherwise, the file will be
closed with each message</p>
@@ -22652,9 +22656,7 @@ template.sendBodyAndHeader("direct:
<p>for example:</p><div class="code panel pdl" style="border-width: 1px;"><div
class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false"
type="syntaxhighlighter"><![CDATA[hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
]]></script>
-</div></div><p>it means: a new file is created either when it has been idle
for more than 1 second or if more than 5 bytes have been written. So, running
<code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have
been created.</p><h3 id="BookInOnePage-MessageHeaders.4">Message
Headers</h3><p>The following headers are supported by this component:</p><h4
id="BookInOnePage-Produceronly">Producer only</h4><div
class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th
colspan="1" rowspan="1" class="confluenceTh"><p> Header </p></th><th
colspan="1" rowspan="1" class="confluenceTh"><p> Description
</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>CamelFileName</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <strong>Camel 2.13:</strong> Specifies the name of the
file to write (relative to the endpoint path). The name can be a
<code>String</code> or an <a shape="rect" href="expression.html">Expression</a>
object. Only relevant when not using a split strategy.
</p></td></tr></tbody></table></div>
-</div><h3 id="BookInOnePage-Controllingtoclosefilestream">Controlling to close
file stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When
using the <a shape="rect" href="hdfs.html">HDFS</a> producer
<strong>without</strong> a split strategy, then the file output stream is by
default closed after the write. However you may want to keep the stream open,
and only explicitly close the stream later. For that you can use the header
<code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>)
to control this. Setting this value to a boolean allows you to explicit control
whether the stream should be closed or not.</p><p>Notice this does not apply if
you use a split strategy, as there are various strategies that can control when
the stream is closed.</p><h3 id="BookInOnePage-UsingthiscomponentinOSGi">Using
this component in OSGi</h3><p>This component is fully functional in an OSGi
environment, however, it requires some actions from the user. Hadoop uses the
thread context class loader in order to load resources. Usually, the thread
context classloader will be the bundle class loader of the bundle that contains
the routes. So, the default configuration files need to be visible from the
bundle class loader. A typical way to deal with it is to keep a copy of
core-default.xml in your bundle root. That file can be found in the
hadoop-common.jar.</p>
+</div></div><p>it means: a new file is created either when it has been idle
for more than 1 second or if more than 5 bytes have been written. So, running
<code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have
been created.</p><h3 id="BookInOnePage-MessageHeaders.4">Message
Headers</h3><p>The following headers are supported by this component:</p><h4
id="BookInOnePage-Produceronly">Producer only</h4><div
class="confluenceTableSmall"><div class="table-wrap"><table
class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1"
class="confluenceTh"><p>Header</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>CamelFileName</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13:</strong>
Specifies the name of the file to write (relative to the endpoint path). The
name can be a <code>String</code> or an <a shape="rect" href="expression.html"
>Expression</a> object. Only relevant when not using a split
>strategy.</p></td></tr></tbody></table></div></div><h3
>id="BookInOnePage-Controllingtoclosefilestream">Controlling to close file
>stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using
>the <a shape="rect" href="hdfs.html">HDFS</a> producer
><strong>without</strong> a split strategy, then the file output stream is by
>default closed after the write. However you may want to keep the stream open,
>and only explicitly close the stream later. For that you can use the header
><code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>)
>to control this. Setting this value to a boolean allows you to explicit
>control whether the stream should be closed or not.</p><p>Notice this does
>not apply if you use a split strategy, as there are various strategies that
>can control when the stream is closed.</p><h3
>id="BookInOnePage-UsingthiscomponentinOSGi">Using this component in
>OSGi</h3><p>This component is
fully functional in an OSGi environment, however, it requires some actions
from the user. Hadoop uses the thread context class loader in order to load
resources. Usually, the thread context classloader will be the bundle class
loader of the bundle that contains the routes. So, the default configuration
files need to be visible from the bundle class loader. A typical way to deal
with it is to keep a copy of core-default.xml in your bundle root. That file
can be found in the hadoop-common.jar.</p>
<h2 id="BookInOnePage-HibernateComponent">Hibernate Component</h2>
<p>The <strong>hibernate:</strong> component allows you to work with databases
using Hibernate as the object relational mapping technology to map POJOs to
database tables. The <strong>camel-hibernate</strong> library is provided by
the <a shape="rect" class="external-link"
href="http://code.google.com/p/camel-extra/"; rel="nofollow">Camel Extra</a>
project which hosts all *GPL related components for Camel.</p>
Modified: websites/production/camel/content/cache/main.pageCache
==============================================================================
Binary files - no diff available.
Modified: websites/production/camel/content/download.html
==============================================================================
--- websites/production/camel/content/download.html (original)
+++ websites/production/camel/content/download.html Wed Dec 17 07:19:49 2014
@@ -84,7 +84,7 @@
<tbody>
<tr>
<td valign="top" width="100%">
-<div class="wiki-content maincontent"><h2 id="Download-LatestReleases">Latest
Releases</h2><div class="sectionColumnWrapper"><div class="sectionMacro"><div
class="sectionMacroRow"><div class="columnMacro"
style="width:20%;min-width:20%;max-width:20%;"><div
style="padding-right:20px;float:left;"><p><img class="confluence-embedded-image
confluence-external-resource"
src="http://camel.apache.org/images/camel-box-small.png";
data-image-src="http://camel.apache.org/images/camel-box-small.png";></p></div></div><div
class="columnMacro" style="width:80%;min-width:80%;max-width:80%;"><div
style="min-height:200px"><p>Grab these releases while they are
hot!</p><ul><li><span>The latest release for Camel 2.14.x is <a shape="rect"
href="camel-2140-release.html">Camel 2.14.1 Release</a>.</span></li><li>The
latest release for Camel 2.13.x is <a shape="rect"
href="camel-2133-release.html">Camel 2.13.3 Release</a>.</li><li>The latest
release for Camel 2.12.x is <a shape="rect" href="camel-2125-release.
html">Camel 2.12.5 Release</a>. (This is last planned release for 2.12.x which
now is no longer maintained)</li><li>Camel versions 2.11.x and older are no
longer actively developed.</li></ul></div></div></div></div></div><h2
id="Download-OlderReleases">Older Releases</h2><p>See <a shape="rect"
href="download-archives.html">Download Archives</a></p><h2
id="Download-Gettingthelatestdistributions">Getting the latest
distributions</h2> <div class="aui-message hint shadowed information-macro">
+<div class="wiki-content maincontent"><h2 id="Download-LatestReleases">Latest
Releases</h2><div class="sectionColumnWrapper"><div class="sectionMacro"><div
class="sectionMacroRow"><div class="columnMacro"
style="width:20%;min-width:20%;max-width:20%;"><div
style="padding-right:20px;float:left;"><p><img class="confluence-embedded-image
confluence-external-resource"
src="http://camel.apache.org/images/camel-box-small.png";
data-image-src="http://camel.apache.org/images/camel-box-small.png";></p></div></div><div
class="columnMacro" style="width:80%;min-width:80%;max-width:80%;"><div
style="min-height:200px"><p>Grab these releases while they are
hot!</p><ul><li><span>The latest release for Camel 2.14.x is <a shape="rect"
href="camel-2141-release.html">Camel 2.14.1 Release</a>.</span></li><li>The
latest release for Camel 2.13.x is <a shape="rect"
href="camel-2133-release.html">Camel 2.13.3 Release</a>.</li><li>The latest
release for Camel 2.12.x is <a shape="rect" href="camel-2125-release.
html">Camel 2.12.5 Release</a>. (This is last planned release for 2.12.x which
now is no longer maintained)</li><li>Camel versions 2.11.x and older are no
longer actively developed.</li></ul></div></div></div></div></div><h2
id="Download-OlderReleases">Older Releases</h2><p>See <a shape="rect"
href="download-archives.html">Download Archives</a></p><h2
id="Download-Gettingthelatestdistributions">Getting the latest
distributions</h2> <div class="aui-message hint shadowed information-macro">
<p class="title">The URLs below use redirection</p>
<span class="aui-icon icon-hint">Icon</span>
<div class="message-content">
Modified: websites/production/camel/content/hdfs.html
==============================================================================
--- websites/production/camel/content/hdfs.html (original)
+++ websites/production/camel/content/hdfs.html Wed Dec 17 07:19:49 2014
@@ -96,9 +96,13 @@
</div></div><h3 id="HDFS-URIformat">URI format</h3><div class="code panel pdl"
style="border-width: 1px;"><div class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false"
type="syntaxhighlighter"><![CDATA[hdfs://hostname[:port][/path][?options]
]]></script>
-</div></div><p>You can append query options to the URI in the following
format, <code>?option=value&option=value&...</code><br clear="none">
The path is treated in the following way:</p><ol><li>as a consumer, if it's a
file, it just reads the file, otherwise if it represents a directory it scans
all the file under the path satisfying the configured pattern. All the files
under that directory must be of the same type.</li><li>as a producer, if at
least one split strategy is defined, the path is considered a directory and
under that directory the producer creates a different file per split named
using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a>.</li></ol><h3
id="HDFS-Options">Options</h3><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th
colspan="1" rowspan="1" class="confluenceTh"><p> Name </p></th><th colspan="1"
rowspan="1" class="confluenceTh"><p> Default Value </p></th><th colspan="1"
rowspan="1" class="confluenceTh"><p> Description </p></th></tr><tr><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>overwrite</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>true</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The file can be
overwritten </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>append</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>false</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> Append to existing file.
Notice that not all HDFS file systems support the append option.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>bufferSize</code> </p></td><td colspan="1" rowspan="1" cla
ss="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The buffer size used by HDFS </p></td></tr><tr><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>replication</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> <code>3</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> The HDFS
replication factor </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>blockSize</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>67108864</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> The size of the HDFS blocks
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>fileType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>NORMAL_FILE</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> It can be SEQUENCE_FILE, MAP_FILE,
ARRAY_FILE, or BLOOMMAP_FILE, see Hadoop </p></td></tr>
<tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>fileSystemType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>HDFS</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> It can be LOCAL for local filesystem
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>keyType</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>NULL</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The type for the key in case of sequence or map files.
See below. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>valueType</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>TEXT</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> The type for the key in case of sequence
or map files. See below. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>splitStrategy</code> </p></td><td colspan="1"
rowsp
an="1" class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> A string describing the strategy on how to split the
file based on different criteria. See below. </p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>openedSuffix</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> <code>opened</code>
</p></td><td colspan="1" rowspan="1" class="confluenceTd"><p> When a file is
opened for reading/writing the file is renamed with this suffix to avoid to
read it during the writing phase. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>readSuffix</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>read</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> Once the file has been read is renamed
with this suffix to avoid to read it again. </p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>initialDelay</code> </p></td><td co
lspan="1" rowspan="1" class="confluenceTd"><p> <code>0</code> </p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p> For the consumer, how much to
wait (milliseconds) before to start scanning the directory.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>delay</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>0</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The interval (milliseconds) between the directory
scans. </p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>pattern</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>*</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> The pattern used for scanning the directory
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>chunkSize</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>4096</code> </p></td><td colspan="1" rowspan="1"
class="co
nfluenceTd"><p> When reading a normal file, this is split into chunks
producing a message per chunk. </p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p> <code>connectOnStartup</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <code>true</code> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p> <strong>Camel 2.9.3/2.10.1:</strong>
Whether to connect to the HDFS file system on starting the producer/consumer.
If <code>false</code> then the connection is created on-demand. Notice that
HDFS may take up till 15 minutes to establish a connection, as it has hardcoded
45 x 20 sec redelivery. By setting this option to <code>false</code> allows
your application to startup, and not block for up till 15 minutes.
</p></td></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>owner</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <strong>Camel 2.13/2
.12.4:</strong> The file owner must match this owner for the consumer to
pickup the file. Otherwise the file is skipped.
</p></td></tr></tbody></table></div>
-</div><h4 id="HDFS-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL
it means that the key or the value is absent</li><li>BYTE for writing a byte,
the java Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence
of bytes. It maps the java ByteBuffer class</li><li>INT for writing java
integer</li><li>FLOAT for writing java float</li><li>LONG for writing java
long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java
strings</li></ul><p>BYTES is also used with everything else, for example, in
Camel a file is sent around as an InputStream, int this case is written in a
sequence file or a map file as a sequence of bytes.</p><h3
id="HDFS-SplittingStrategy">Splitting Strategy</h3><p>In the current version of
Hadoop opening a file in append mode is disabled since it's not very reliable.
So, for the moment, it's only possible to create new files. The Camel HDFS
endpoint tries to solve this problem in this way:</p><ul><li>If the split
strategy option ha
s been defined, the hdfs path will be used as a directory and files will be
created using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a splitting
condition is met, a new file is created.<br clear="none"> The splitStrategy
option is defined as a string with the following syntax:<br clear="none">
splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where
<ST> can be:</p><ul><li>BYTES a new file is created, and the old is
closed when the number of written bytes is more than
<value></li><li>MESSAGES a new file is created, and the old is closed
when the number of written messages is more than <value></li><li>IDLE a
new file is created, and the old is closed when no writing happened in the last
<value> milliseconds</li></ul> <div class="aui-message warning
shadowed information-macro">
+</div></div><p>You can append query options to the URI in the following
format, <code>?option=value&option=value&...</code><br clear="none">
The path is treated in the following way:</p><ol><li>as a consumer, if it's a
file, it just reads the file, otherwise if it represents a directory it scans
all the file under the path satisfying the configured pattern. All the files
under that directory must be of the same type.</li><li>as a producer, if at
least one split strategy is defined, the path is considered a directory and
under that directory the producer creates a different file per split named
using the configured <a shape="rect"
href="uuidgenerator.html">UuidGenerator</a>.</li></ol> <div
class="aui-message warning shadowed information-macro">
+ <span class="aui-icon icon-warning">Icon</span>
+ <div class="message-content">
+ <p><span>When consuming from hdfs then in normal
mode, a file is split into chunks, producing a message per chunk. You can
configure the size of the chunk using the chunkSize option. If you want to read
from hdfs and write to a regular file using the file component, then you can
use the fileMode=Append to append each of the chunks together.</span></p>
+ </div>
+ </div>
+<p> </p><h3 id="HDFS-Options">Options</h3><div
class="confluenceTableSmall"><div class="table-wrap"><table
class="confluenceTable"><tbody><tr><th colspan="1" rowspan="1"
class="confluenceTh"><p>Name</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Default Value</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>overwrite</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>The file can be
overwritten</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>append</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>false</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>Append to existing file. Notice that not all HDFS file
systems support the append option.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><c
ode>bufferSize</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>4096</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>The buffer size used by HDFS</p></td></tr><tr><td
colspan="1" rowspan="1"
class="confluenceTd"><p><code>replication</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>3</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The HDFS replication
factor</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>blockSize</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>67108864</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>The size of the HDFS
blocks</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>fileType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>NORMAL_FILE</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>It can be SEQUENCE_FILE,
MAP_FILE, ARRAY_FILE, or BLO
OMMAP_FILE, see Hadoop</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>fileSystemType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>HDFS</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>It can be LOCAL for local
filesystem</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>keyType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>NULL</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or
map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>valueType</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>TEXT</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The type for the key in case of sequence or
map files. See below.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>splitStrategy</code></p></td><td cols
pan="1" rowspan="1" class="confluenceTd"><p> </p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>A string describing the strategy on how to
split the file based on different criteria. See below.</p></td></tr><tr><td
colspan="1" rowspan="1"
class="confluenceTd"><p><code>openedSuffix</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>opened</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>When a file is opened for reading/writing
the file is renamed with this suffix to avoid to read it during the writing
phase.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>readSuffix</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>read</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>Once the file has been read is renamed with
this suffix to avoid to read it again.</p></td></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>initialDelay</code></p></td><td
colspan
="1" rowspan="1" class="confluenceTd"><p><code>0</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p>For the consumer, how much to
wait (milliseconds) before to start scanning the
directory.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>delay</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>0</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p>The interval (milliseconds) between the directory
scans.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>pattern</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>*</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>The pattern used for scanning the
directory</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>chunkSize</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>4096</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p>When reading
a normal file, this is split into chunks producing a message per
chunk.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>connectOnStartup</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>true</code></p></td><td colspan="1"
rowspan="1" class="confluenceTd"><p><strong>Camel 2.9.3/2.10.1:</strong>
Whether to connect to the HDFS file system on starting the producer/consumer.
If <code>false</code> then the connection is created on-demand. Notice that
HDFS may take up till 15 minutes to establish a connection, as it has hardcoded
45 x 20 sec redelivery. By setting this option to <code>false</code> allows
your application to startup, and not block for up till 15
minutes.</p></td></tr><tr><td colspan="1" rowspan="1"
class="confluenceTd"><p><code>owner</code></p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p><strong>Camel 2.13/2.12.4:</strong> The file owner must
ma
tch this owner for the consumer to pickup the file. Otherwise the file is
skipped.</p></td></tr></tbody></table></div></div><h4
id="HDFS-KeyTypeandValueType">KeyType and ValueType</h4><ul><li>NULL it means
that the key or the value is absent</li><li>BYTE for writing a byte, the java
Byte class is mapped into a BYTE</li><li>BYTES for writing a sequence of bytes.
It maps the java ByteBuffer class</li><li>INT for writing java
integer</li><li>FLOAT for writing java float</li><li>LONG for writing java
long</li><li>DOUBLE for writing java double</li><li>TEXT for writing java
strings</li></ul><p>BYTES is also used with everything else, for example, in
Camel a file is sent around as an InputStream, int this case is written in a
sequence file or a map file as a sequence of bytes.</p><h3
id="HDFS-SplittingStrategy">Splitting Strategy</h3><p>In the current version of
Hadoop opening a file in append mode is disabled since it's not very reliable.
So, for the moment, it's only possible to create
new files. The Camel HDFS endpoint tries to solve this problem in this
way:</p><ul><li>If the split strategy option has been defined, the hdfs path
will be used as a directory and files will be created using the configured <a
shape="rect" href="uuidgenerator.html">UuidGenerator</a></li><li>Every time a
splitting condition is met, a new file is created.<br clear="none"> The
splitStrategy option is defined as a string with the following syntax:<br
clear="none">
splitStrategy=<ST>:<value>,<ST>:<value>,*</li></ul><p>where
<ST> can be:</p><ul><li>BYTES a new file is created, and the old is
closed when the number of written bytes is more than
<value></li><li>MESSAGES a new file is created, and the old is closed
when the number of written messages is more than <value></li><li>IDLE a
new file is created, and the old is closed when no writing happened in the last
<value> milliseconds</li></ul> <div class="aui-message warning
shadowed informa
tion-macro">
<span class="aui-icon icon-warning">Icon</span>
<div class="message-content">
<p>note that this strategy currently requires
either setting an IDLE value or setting the HdfsConstants.HDFS_CLOSE header to
false to use the BYTES/MESSAGES configuration...otherwise, the file will be
closed with each message</p>
@@ -107,9 +111,7 @@
<p>for example:</p><div class="code panel pdl" style="border-width: 1px;"><div
class="codeContent panelContent pdl">
<script class="theme: Default; brush: java; gutter: false"
type="syntaxhighlighter"><![CDATA[hdfs://localhost/tmp/simple-file?splitStrategy=IDLE:1000,BYTES:5
]]></script>
-</div></div><p>it means: a new file is created either when it has been idle
for more than 1 second or if more than 5 bytes have been written. So, running
<code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have
been created.</p><h3 id="HDFS-MessageHeaders">Message Headers</h3><p>The
following headers are supported by this component:</p><h4
id="HDFS-Produceronly">Producer only</h4><div class="confluenceTableSmall">
-<div class="table-wrap"><table class="confluenceTable"><tbody><tr><th
colspan="1" rowspan="1" class="confluenceTh"><p> Header </p></th><th
colspan="1" rowspan="1" class="confluenceTh"><p> Description
</p></th></tr><tr><td colspan="1" rowspan="1" class="confluenceTd"><p>
<code>CamelFileName</code> </p></td><td colspan="1" rowspan="1"
class="confluenceTd"><p> <strong>Camel 2.13:</strong> Specifies the name of the
file to write (relative to the endpoint path). The name can be a
<code>String</code> or an <a shape="rect" href="expression.html">Expression</a>
object. Only relevant when not using a split strategy.
</p></td></tr></tbody></table></div>
-</div><h3 id="HDFS-Controllingtoclosefilestream">Controlling to close file
stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using
the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong>
a split strategy, then the file output stream is by default closed after the
write. However you may want to keep the stream open, and only explicitly close
the stream later. For that you can use the header
<code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>)
to control this. Setting this value to a boolean allows you to explicit control
whether the stream should be closed or not.</p><p>Notice this does not apply if
you use a split strategy, as there are various strategies that can control when
the stream is closed.</p><h3 id="HDFS-UsingthiscomponentinOSGi">Using this
component in OSGi</h3><p>This component is fully functional in an OSGi
environment, however, it requires some actions from the user. Hadoop uses the
thread context cl
ass loader in order to load resources. Usually, the thread context classloader
will be the bundle class loader of the bundle that contains the routes. So, the
default configuration files need to be visible from the bundle class loader. A
typical way to deal with it is to keep a copy of core-default.xml in your
bundle root. That file can be found in the hadoop-common.jar.</p></div>
+</div></div><p>it means: a new file is created either when it has been idle
for more than 1 second or if more than 5 bytes have been written. So, running
<code>hadoop fs -ls /tmp/simple-file</code> you'll see that multiple files have
been created.</p><h3 id="HDFS-MessageHeaders">Message Headers</h3><p>The
following headers are supported by this component:</p><h4
id="HDFS-Produceronly">Producer only</h4><div class="confluenceTableSmall"><div
class="table-wrap"><table class="confluenceTable"><tbody><tr><th colspan="1"
rowspan="1" class="confluenceTh"><p>Header</p></th><th colspan="1" rowspan="1"
class="confluenceTh"><p>Description</p></th></tr><tr><td colspan="1"
rowspan="1" class="confluenceTd"><p><code>CamelFileName</code></p></td><td
colspan="1" rowspan="1" class="confluenceTd"><p><strong>Camel 2.13:</strong>
Specifies the name of the file to write (relative to the endpoint path). The
name can be a <code>String</code> or an <a shape="rect"
href="expression.html">Expression</a> obje
ct. Only relevant when not using a split
strategy.</p></td></tr></tbody></table></div></div><h3
id="HDFS-Controllingtoclosefilestream">Controlling to close file
stream</h3><p><strong>Available as of Camel 2.10.4</strong></p><p>When using
the <a shape="rect" href="hdfs.html">HDFS</a> producer <strong>without</strong>
a split strategy, then the file output stream is by default closed after the
write. However you may want to keep the stream open, and only explicitly close
the stream later. For that you can use the header
<code>HdfsConstants.HDFS_CLOSE</code> (value = <code>"CamelHdfsClose"</code>)
to control this. Setting this value to a boolean allows you to explicit control
whether the stream should be closed or not.</p><p>Notice this does not apply if
you use a split strategy, as there are various strategies that can control when
the stream is closed.</p><h3 id="HDFS-UsingthiscomponentinOSGi">Using this
component in OSGi</h3><p>This component is fully functional in an OSGi environme
nt, however, it requires some actions from the user. Hadoop uses the thread
context class loader in order to load resources. Usually, the thread context
classloader will be the bundle class loader of the bundle that contains the
routes. So, the default configuration files need to be visible from the bundle
class loader. A typical way to deal with it is to keep a copy of
core-default.xml in your bundle root. That file can be found in the
hadoop-common.jar.</p></div>
</td>
<td valign="top">
<div class="navigation">
![http://camel.apache.org/images/camel-box-small.png"](http://camel.apache.org/images/camel-box-small.png"){kind=link}