Question

how to write a grok pattern for nginx's log

Hello everyone ! I have a problem in my elk environment. Follows:

  1. nginx log:
124.134.229.30 - - [18/Aug/2015:03:30:01 +0800] "POST /?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9 HTTP/1.0"200 23 lotteryid=14&issue=20150818-043&flag=gettime "http://x8shi.com/?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9""Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"  124.134.229.30 x8shi.com 0.140 0.027 -
  1. log format:
log_format access '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent $request_body "$http_referer"' '
"$http_user_agent"  $http_x_forwarded_for $host $request_time $upstream_response_time' " $upstream_cache_status";

now , I should how to write this grok pattern for logstash ?

Please help me …

Subscribe
Share

Submit an answer
You can type!ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

These answers are provided by our Community. If you find them useful, show some love by clicking the heart. If you run into issues leave a comment, or add your own answer to help others.

The following grok pattern would match the log message as you resented it:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\"%{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NOTSPACE:querystring} (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})%{QS:agent}  %{IPORHOST:forwardedfor} %{IPORHOST:host} %{NUMBER:upstreamresponse} (?:-|%{NUMBER:cache})

I was able to put that together using the great Grok Constructor tool. Using its Matcher, you can test and tweak your patterns.

For a deeper dive into writing filters for Logstash, check out: