how to write a grok pattern for nginx's log

Question

Hello everyone !I have a problem in my elk environment.Follows: nginx log: 124.134.229.30 - - [18/Aug/2015:03:30:01 +0800] "POST /?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9 HTTP/1.0"200 23 lotteryid=14&issue=20150818-043&flag=gettime "http://x8shi.com/?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9""Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36" 124.134.229.30 x8shi.com 0.140 0.027 - log format: log_format access '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent $request_body "$http_referer"' ' "$http_user_agent" $http_x_forwarded_for $host $request_time $upstream_response_time' " $upstream_cache_status"; now , I should how to write this grok pattern for logstash ? Please help me …

Andrew SB · Answer

The following grok pattern would match the log message as you resented it:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\"%{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NOTSPACE:querystring} (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})%{QS:agent}  %{IPORHOST:forwardedfor} %{IPORHOST:host} %{NUMBER:upstreamresponse} (?:-|%{NUMBER:cache})

I was able to put that together using the great Grok Constructor tool. Using its Matcher, you can test and tweak your patterns.

For a deeper dive into writing filters for Logstash, check out:

Adding Logstash Filters To Improve Centralized Logging

Related

Question

how to write a grok pattern for nginx's log

Related