Sending log files to Logstash with Elastic's Filebeat

When initially running an application on only one server, keeping log files on the filesystem is usually sufficient. You can easily parse and read through old log files directly from the server itself.

However once your logging requirements increase, it is very useful to centralize your logging into Logstash - this usually lets you store log data for longer, and to easily filter log data by different parameters (eg application version, user groups, etc). 

You may find your logging requirements increase in one of the following scenarios:

1. Your application grows to multiple servers - you now need a way to parse logging statements without switching between server instances.
2. Your logging output grows - once your log files are too large to handle with grep etc, Logstash lets you filter larger amounts of logging data.

Setting up a Logstash instance is beyond the scope of this article. However, let's discuss setting up Filebeat - this is a useful utility which periodically sweeps through your log files and sends them all to Logstash. It lets you specify which log files to watch, and also optionally append custom variables to your log statements.

First, get started using apt or yum to install Filebeat.

Next, replace the default filebeat config at /etc/filebeat/filebeat.yml with your own configuration. Here is what I use:

############################# Filebeat ######################################

filebeat:
  prospectors:
    -
      paths:
        - /path/to/my/application/log/*.log

      # Possible options are:
      # * log: Reads every line of the log file (default)
      # * stdin: Reads the standard in
      input_type: log

      # Optional additional fields. These field can be freely picked
      # to add additional information to the crawled log files for filtering
      fields:
        app_version: 3.2

      # Set to true to store the additional fields as top level fields instead
      # of under the "fields" sub-dictionary. In case of name conflicts with the
      # fields added by Filebeat itself, the custom fields overwrite the default
      # fields.
      fields_under_root: true

output:
  logstash:
    hosts: ["127.0.0.1:5044"]

    # Number of workers per Logstash host.
    worker: 1

    # Set gzip compression level.
    compression_level: 3


############################# Logging #########################################

logging:
  to_files: true

  files:
    path: /path/to/my/logs
    name: filebeat.log

    # Configure log file size limit. If limit is reached, log file will be
    # automatically rotated
    rotateeverybytes: 10485760 # = 10MB

    # Number of rotated log files to keep. Oldest files will be deleted first.
    #keepfiles: 7

  # Available log levels are: critical, error, warning, info, debug
  level: info

Notice I am sending a custom field called `app_version` with my log messages.

And finally I add an upstart script to /etc/init/filebeat.conf like this:

description "Filebeat"
    start on runlevel [2345]
    stop on runlevel [!2345]
    respawn

pre-start exec /etc/init.d/filebeat start
post-stop exec /etc/init.d/filebeat stop

Now when we run `service filebeat start`, Filebeat monitors the log files and sends them to logstash periodically. It also appends a custom field which I specified. And finally, it also outputs its own log locally to filebeat.log for further inspection.