Stream caching

While stream types (like StreamSource, InputStream and Reader) are commonly used in messaging for performance reasons, they also have an important drawback: they can only be read once. In order to be able to work with message content multiple times, the stream needs to be cached.

Streams are caching in memory. In Camel 2.0, large stream messages (over 64 Kb) will be cached in a temporary file instead – Camel itself will handle deleting the temporary file once the cached stream is no longer necessary.

In Camel 2.0 stream cache is default disabled out of the box.
In Camel 1.x stream cache is default enabled out of the box.

StreamCache Affects your payload object
The StreamCache will affect your payload object as it will replace the Stream payload with a org.apache.camel.StreamCache object.
This StreamCache is capable of being re-readable and thus possible to better be routed within Camel using redelivery or Content Based Router or the likes.

However to not change the payload under the covers without the end user really knowing we changed the default in Camel 2.0 to disabled. So in Camel 2.0 you have to explicit enable it if you want to use it.

Enabling stream caching

In Apache Camel, you can explicitly enable stream caching for a single route with the streamCaching DSL method:

from("jbi:service:http://foo.bar.org/MyService")
    .streamCaching()
    .to("jbi:service:http://foo.bar.org/MyOtherService");


In Spring XML you enable it by setting the streamCache="true" attribute on the route tag.

<route streamCache="true">
   <from uri="jbi:service:http://foo.bar.org/MyService"/>
   <to uri="jbi:service:http://foo.bar.org/MyOtherService"/>
</route>



Scopes
StreamCache supports the global and per route scope. So by setting the streamCache attribute on camelContext you can enable/disable it globally.

<camelContext streamCache="true">
   ...
</camelContext>



The route scope is configured by the streamCache attribute on the <route> tag such as:

<route streamCache="true">
   <from uri="jbi:service:http://foo.bar.org/MyService"/>
   <to uri="jbi:service:http://foo.bar.org/MyOtherService"/>
</route>



You can mix and match for instance you can enable it globally and disable it on a particular route such as:

<camelContext streamCache="true">
  <route>
    <from uri="jbi:service:http://foo.bar.org/MyService"/>
    <to uri="jbi:service:http://foo.bar.org/MyOtherService"/>
  </route>
  
  <route streamCache="false">
    <from uri="jms:queue:foo"/>
    <to uri="jms:queue:bar"/>
  </route>

</camelContext>



Enabling from Java DSL
In Camel 2.0 you can enable stream cache by setting the property on CamelContext, for instance in a RouteBuilder class:

  context.setStreamCache(true);

  from("jbi:service:http://foo.bar.org/MyService")
     .to("jbi:service:http://foo.bar.org/MyOtherService");



Disable stream caching explicitly
If you have enabled stream caching globally on the CamelContext and you want to disable it for certain routes in a selective manner, you can use the following syntax.

Spring DSL:

<camelContext streamCache="true">
  <route streamCache="false">
    <from uri="jetty:http://0.0.0.0:9090"/>
    <to uri="file:target/incoming"/>
  </route>
</camelContext>



Java DSL:

  context.setStreamCache(true);

  from("jetty:http://0.0.0.0:9090").noStreamCaching()
     .to("file:target/incoming");



Implicitly enabled for multicast and dead letter channel (Camel 1.x)
Some EIPs require that the message content can be read multiple times. Stream caching will be automatically enabled when using these EIPs in your routes:

	Multicast  will implicitly cache streams to ensure that all the endpoints can access the message content
	Dead Letter Channel  uses stream caching to ensure that the message content can actually be read again when redelivering a message




Streaming cache to files
When stream cache is enabled it will by default spool big streams to files instead of keeping them in memory. The default threshold is 64kb but you can configure it with the following properties:




 Property 
 Default 
 Description 


 CamelCachedOutputStreamBufferSize 
 2kb 
 Camel 2.9.4, 2.10.2, 2.11.0: Size in bytes of the buffer used in the stream. 


 CamelCachedOutputStreamThreshold 
 64kb 
 Size in bytes when the stream should be spooled to disk instead of keeping in memory. Use a value of 0 or negative to disable it all together so streams is always kept in memory regardless of their size. 


 CamelCachedOutputStreamOutputDirectory 
 java.io.tmpdir 
 Base directory where temporary files for spooled streams should be stored. 


 CamelCachedOutputStreamCipherTransformation 
 null 
 Camel 2.11.0: If set, the temporary files are encrypted using the specified cipher transformation. 





You set these properties on the CamelContext as shown below:


context.getProperties().put(CachedOutputStream.TEMP_DIR, "/tmp/cachedir");
context.getProperties().put(CachedOutputStream.THRESHOLD, "1024");
context.getProperties().put(CachedOutputStream.BUFFER_SIZE, "4096");
// to enable encryption using RC4
// context.getProperties().put(CachedOutputStream.CIPHER_TRANSFORMATION, "RC4");



Disabling spooling to disk
Available as of *Camel 1.6.2/2.0

You can disable spooling to disk by setting a threshold of 0 or a negative value.

// disable spooling to disk
context.getProperties().put(CachedOutputStream.THRESHOLD, "0");



How it works?
In order to determine if a type requires caching, we leverage the type converter  feature.  Any type that requires stream caching can be converted into an org.apache.camel.StreamCache instance.

Change Notification Preferences

View Online | View Changes | Add Comment

[CONF] Apache Camel > Stream caching

Stream caching

Page edited by Aki Yoshida

Changes (2)

Full Content

Stream caching

Enabling stream caching

Scopes

Enabling from Java DSL

Disable stream caching explicitly

Implicitly enabled for multicast and dead letter channel (Camel 1.x)

Streaming cache to files

Disabling spooling to disk

How it works?

Reply via email to