how to write a grok pattern for nginx's log

August 21, 2015 8.7k views
Nginx System Tools

Hello everyone !
I have a problem in my elk environment.

  1. nginx log: - - [18/Aug/2015:03:30:01 +0800] "POST /?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9 HTTP/1.0"200 23 lotteryid=14&issue=20150818-043&flag=gettime """Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" 0.140 0.027 -
  1. log format:
log_format access '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent $request_body "$http_referer"' '
"$http_user_agent"  $http_x_forwarded_for $host $request_time $upstream_response_time' " $upstream_cache_status";

now , I should how to write this grok pattern for logstash ?

Please help me …

edited by asb
1 Answer

The following grok pattern would match the log message as you resented it:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\"%{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NOTSPACE:querystring} (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})%{QS:agent}  %{IPORHOST:forwardedfor} %{IPORHOST:host} %{NUMBER:upstreamresponse} (?:-|%{NUMBER:cache})

I was able to put that together using the great Grok Constructor tool. Using its Matcher, you can test and tweak your patterns.

For a deeper dive into writing filters for Logstash, check out:

by Mitchell Anicas
One way to increase the effectiveness of your Logstash setup is to collect important application logs and structure the log data by employing filters. In this guide, we will focus primarily on how to add filters for various common application logs.
Have another answer? Share your knowledge.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!