Logstash filters

It is not always a good thing to have options, many of them. I tend to start thinking about all the combinations, the plus and minus for all aspects. And how about the future, what possible negative consequences could it have if I choose a instead b now, how hard would it be to change back?

Diving into logstash, and not to mention logstash-forwarder (lumberjack) is a daunting task. Its not difficult or hard to understand, by dealing with all the choises are.

Recently I had a dilemma, no big thing, but anyway. When to set "type" ?

But the real question is why setting type? Well, the typical use it to tell logstash how to deal and process the data. Do we really need the the type setting? Not really, but it simplifies the configuration and makes it more readable too.

I hastly setup logstash-forwarder on a webserver with large amount traffic and by not really thinking about any technical/architectual decisions the type on the client. When working through the pipline and finally configuring on logstash I noticed that type was allready set, but not to exactly what fitted my need.

Type was set on client to apache-access, the access log needs their own type declaration since the log format is different from eg. error log. But on the logstash I had set this to the more general type 'apache'. I could not just change this since logstash was allready receiving data from other servers in production.

So back to options. A neat thing with logstash-forward is the annotation of the object sent, if data comes from a log file, the object is annotated whence it came from. Then with some grok'ing it's easy to filter objects based on not only the set type, but also source file name.

Eg.

filter {
  if [type] == "apache" {
    grok { 
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      match => { "file" => "%{GREEDYDATA}.access.log" }
    }
  }
}