How to convert datetime format into different format in Fluentd (td-agent)

How to convert datetime format into different format in Fluentd (td-agent)

If we need to change datetime format in a record in Fluentd (td-agent), we need to use "record_transformer" plugin which is a built-in plugin.

In this article, we will convert "2017-12-19 16:09:31" into "2017-12-19T16:09:31+09:00" (ISO 8601 format). As far as I know, some NoSQL databases take only ISO 8601 format. If we have datetime value in a log file or something else, it's better to convert it into ISO 8601 format.


Convert datetime format into ISO 8601 format in Fluentd

Since Fluentd (td-agent) supports Ruby language feature, we can convert datetime format into ISO 8601 format.

In the following configuration, we convert "datetime_received" field value into ISO 8601 format. Ruby language has "iso8601" method, we just need to call the method, so that we can convert our datetime format into ISO 8601 format.

<filter tail_ex.linelog>
  @type record_transformer
  enable_ruby true
  auto_typecast true
  <record>
    datetime_received ${require 'time'; Time.parse(record["datetime_received"].to_s).iso8601.to_s}
  </record>
</filter>

When the following datetime format is given,
[Before convert]
"2017-12-19 16:09:31"

we get the following datetime format (ISO 8601 format).
[After converted]
"2017-12-19T16:09:31+09:00"