Question

how to reload the skipped old files in ELK

Posted December 17, 2014 8.4k views

how to reload the skipped old files in ELK and need to know where the file information
will be saved in Elasticsearch

Add a comment
thiru
By:
thiru
Add a comment
2 comments
  • asb December 17, 2014
    Reply Report

    Hi! Could you be a little more specific? Any additional information that you could provide would help us figure out your problem. How are you inputting logs into logstash? Are you using logstash-forwarder or the file input? This blog post show how you can load old logs using logstash-forwarder.

  • thiru December 18, 2014
    Reply Report

    hi ,

    thanks for the reply @asb ,

    logstash - forwarder is using as input and i have deleted the all old data file in
    elastic search inside /var/lib/elasticsearch/nodes/0/indices,while trying to reload few file using logstash - forwarder, not able load its was skipping as old file .
    the configuration : Ubuntu 14.04
    elasticsearch=1.1.1
    kibana-3.0.1
    logstash=1.4.2
    logstash-forwarder_0.3.1

    Logstash Forwarder configuration file

    {
  "network": {
    "servers": [ "localhost:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [
       "/home/lab/logs/mp.log","/home/lab/data/samplexml.log"      
       ],
      "fields": { "type": "logfile" }
    }
   ]
}

    below lines in the syslog

    Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889356 Loading registrar data
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889520 Skipping old file: /home/lab/logs/mp.log
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889626 Loading registrar data
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889723 Skipping old file: /home/lab/data/samplexml.log
Dec 16 07:18:10  :10-05:00 ogstash-forwarder[15910]: 2014/12/15 07:18:10.889802 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt

    input & filter configuration

    input {
  lumberjack {
    port => 5000
    type =>"logs"
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
  if [type] == "logfile" {
    grok {
      match => { "message" => "%{POSINT:ident} \[%{TIMESTAMP_ISO8601:time}]\ %{GREEDYDATA:message}" }
      add_field => [ "received_at", "%{time}" ]
      add_field => [ "received_from", "%{host}" ]
    }

  }

}

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

×
2 answers
asb December 18, 2014

Checkout the blog post that I linked. By default, logstash-forwarder doesn’t ingest new logs, but we can backfill the old logs.

cat /home/lab/data/samplexml.log | /opt/logstash-forwarder/bin/logstash-forwarder -config temp.conf -spool-size 100 -log-to-syslog

where temp.conf is:

{
  "network": {
    "servers": [ "localhost:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "logfile" }
    }
   ]
}

This is configuration allows logstash-forwarder to read logs piped in through stdin.

Reply Report
  • thiru December 22, 2014

    great job,thanks for the solution its working fine

    file name displaying as ’-’ we can’t keep the same file name(loading file name) ?

    Reply Report
Mansor September 4, 2015

Great man, this is working flawlessly!
I’m loading a whole year from apache logs like this:

cd /path/to/stored/logs/
for log in $(ls -1);do 
      zcat $log | /opt/logstash-forwarder/bin/logstash-forwarder -config backfill_apache.conf -spool-size 100;
done

I am using zcat because they are gziped.

My conf:
backfill_apache.conf

{
  "network": {
    "servers": [ "logstash:5000" ],
    "timeout": 15,
    "ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"
  },
  "files": [
    {
      "paths": [ "-" ],
      "fields": { "type": "apache" }
    }
   ]
}
Reply Report
Submit an Answer

Looking for something else?

Ask a new question Search for more help