how to reload the skipped old files in ELK

December 17, 2014 6.6k views
thiru
By:
thiru

how to reload the skipped old files in ELK and need to know where the file information
will be saved in Elasticsearch

2 comments
  • asb MOD December 17, 2014

    Hi! Could you be a little more specific? Any additional information that you could provide would help us figure out your problem. How are you inputting logs into logstash? Are you using logstash-forwarder or the file input? This blog post show how you can load old logs using logstash-forwarder.

  • thiru December 18, 2014

    hi ,

    thanks for the reply @asb ,

    logstash - forwarder is using as input and i have deleted the all old data file in
    elastic search inside /var/lib/elasticsearch/nodes/0/indices,while trying to reload few file using logstash - forwarder, not able load its was skipping as old file .
    the configuration : Ubuntu 14.04
    elasticsearch=1.1.1
    kibana-3.0.1
    logstash=1.4.2
    logstash-forwarder_0.3.1

    Logstash Forwarder configuration file

    {
  "network": {
    "servers": [ "localhost:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [
       "/home/lab/logs/mp.log","/home/lab/data/samplexml.log"      
       ],
      "fields": { "type": "logfile" }
    }
   ]
}

    below lines in the syslog

    Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889356 Loading registrar data
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889520 Skipping old file: /home/lab/logs/mp.log
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889626 Loading registrar data
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889723 Skipping old file: /home/lab/data/samplexml.log
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889802 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt

    input & filter configuration

    input {
  lumberjack {
    port => 5000
    type =>"logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "logfile" {
    grok {
      match => { "message" => "%{POSINT:ident} \[%{TIMESTAMP_ISO8601:time}]\ %{GREEDYDATA:message}" }
      add_field => [ "received_at", "%{time}" ]
      add_field => [ "received_from", "%{host}" ]
    }

  }

}
Log In to Comment
2 Answers
asb MOD December 18, 2014

Checkout the blog post that I linked. By default, logstash-forwarder doesn't ingest new logs, but we can backfill the old logs.

cat /home/lab/data/samplexml.log | /opt/logstash-forwarder/bin/logstash-forwarder -config temp.conf -spool-size 100 -log-to-syslog

where temp.conf is:

{
  "network": {
    "servers": [ "localhost:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "logfile" }
    }
   ]
}

This is configuration allows logstash-forwarder to read logs piped in through stdin.

Reply
  • thiru December 22, 2014

    great job,thanks for the solution its working fine

    file name displaying as '-' we can't keep the same file name(loading file name) ?

Mansor September 4, 2015

Great man, this is working flawlessly!
I'm loading a whole year from apache logs like this:

cd /path/to/stored/logs/
for log in $(ls -1);do 
      zcat $log | /opt/logstash-forwarder/bin/logstash-forwarder -config backfill_apache.conf -spool-size 100;
done

I am using zcat because they are gziped.

My conf:
backfill_apache.conf

{
  "network": {
    "servers": [ "logstash:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "apache" }
    }
   ]
}
Reply
Have another answer? Share your knowledge.