how to reload the skipped old files in ELK

Great man, this is working flawlessly! I’m loading a whole year from apache logs like this:

cd /path/to/stored/logs/
for log in $(ls -1);do 
      zcat $log | /opt/logstash-forwarder/bin/logstash-forwarder -config backfill_apache.conf -spool-size 100;
done

I am using zcat because they are gziped.

My conf: backfill_apache.conf

{
  "network": {
    "servers": [ "logstash:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "apache" }
    }
   ]
}

Checkout the blog post that I linked. By default, logstash-forwarder doesn’t ingest new logs, but we can backfill the old logs.

cat /home/lab/data/samplexml.log | /opt/logstash-forwarder/bin/logstash-forwarder -config temp.conf -spool-size 100 -log-to-syslog

where temp.conf is:

{
  "network": {
    "servers": [ "localhost:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "logfile" }
    }
   ]
}

This is configuration allows logstash-forwarder to read logs piped in through stdin.