How to convert datetime format into different format in Fluentd (td-agent)
If we need to change datetime format in a record in Fluentd (td-agent), we need to use "record_transformer" plugin which is a built-in plugin.In this article, we will convert "2017-12-19 16:09:31" into "2017-12-19T16:09:31+09:00" (ISO 8601 format). As far as I know, some NoSQL databases take only ISO 8601 format. If we have datetime value in a log file or something else, it's better to convert it into ISO 8601 format.
Convert datetime format into ISO 8601 format in Fluentd
Since Fluentd (td-agent) supports Ruby language feature, we can convert datetime format into ISO 8601 format.In the following configuration, we convert "datetime_received" field value into ISO 8601 format. Ruby language has "iso8601" method, we just need to call the method, so that we can convert our datetime format into ISO 8601 format.
<filter tail_ex.linelog>
@type record_transformer
enable_ruby true
auto_typecast true
<record>
datetime_received ${require 'time'; Time.parse(record["datetime_received"].to_s).iso8601.to_s}
</record>
</filter>
When the following datetime format is given,
[Before convert] "2017-12-19 16:09:31"
we get the following datetime format (ISO 8601 format).
[After converted] "2017-12-19T16:09:31+09:00"
