Correct optional fields can be omitted from the Heka message. The uuid and timestamp should remain required and set by the message creator, otherwise: - If the message creator was delayed in sending the messages to Heka when the backlog was processed all messages would basically be given the same timestamp. - If the creator sent duplicates of the same message it would be very difficult to correctly de-dupe without a uuid and timestamp.
If you have your own protobuf format nothing prevents you from writing your own decoder and transforming the result into a Heka message. Trink ----- Original Message ----- > From: "bruno binet" <[email protected]> > To: "heka" <[email protected]> > Sent: Tuesday, June 24, 2014 5:14:58 AM > Subject: [heka] using the protobuf decoder > Hi, > I'd like to use heka to forward and process the data generated by an > application. This is a new application so I'm free to choose any > serialization format for data sent to Heka. Since Heka uses protobuf as the > data format to communicate between multiple instances, I think this could be > a good thing to go with protocol buffers. > My question is: given the current message.proto file ( > https://github.com/mozilla-services/heka/blob/0ce27cb548df1fb31f3def56e3a9b6c1e28dcb77/message/message.proto > ), many fields seem to be optional, so it means I don't have to set them > (heka won't fail if these fields are not set)? > And two fields are required in the "Message" message: > required bytes uuid = 1 ; > required int64 timestamp = 2 ; // nanoseconds since UNIX epoch > I would like those fields to be optional so that heka could set a value for > them if they are missing (it would set the timestamp to the current time and > generate a new uuid). > Woult it be possible? > May be this does not make sense if the protocol buffer decoder is only meant > to be used internally, but in this case, having a way to specify a custom > protocol buffer decoder may be a good thing to have? (we could specify a > custom proto file to load) > Thanks, > Bruno > _______________________________________________ > Heka mailing list > [email protected] > https://mail.mozilla.org/listinfo/heka
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
