how to write a grok pattern for nginx's log

Posted August 21, 2015 13.6k views
NginxSystem Tools

Hello everyone !
I have a problem in my elk environment.

  1. nginx log: - - [18/Aug/2015:03:30:01 +0800] "POST /?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9 HTTP/1.0"200 23 lotteryid=14&issue=20150818-043&flag=gettime """Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" 0.140 0.027 -
  1. log format:
log_format access '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent $request_body "$http_referer"' '
"$http_user_agent"  $http_x_forwarded_for $host $request_time $upstream_response_time' " $upstream_cache_status";

now , I should how to write this grok pattern for logstash ?

Please help me …

edited by asb

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

Submit an Answer
1 answer

The following grok pattern would match the log message as you resented it:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\"%{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NOTSPACE:querystring} (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})%{QS:agent}  %{IPORHOST:forwardedfor} %{IPORHOST:host} %{NUMBER:upstreamresponse} (?:-|%{NUMBER:cache})

I was able to put that together using the great Grok Constructor tool. Using its Matcher, you can test and tweak your patterns.

For a deeper dive into writing filters for Logstash, check out:

by Mitchell Anicas
One way to increase the effectiveness of your Logstash setup is to collect important application logs and structure the log data by employing filters. In this guide, we will focus primarily on how to add filters for various common application logs.