how to write a grok pattern for nginx's log

August 21, 2015 8.7k views
Nginx System Tools
tom1990
By:
tom1990

Hello everyone !
I have a problem in my elk environment.
Follows:

  1. nginx log:
124.134.229.30 - - [18/Aug/2015:03:30:01 +0800] "POST /?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9 HTTP/1.0"200 23 lotteryid=14&issue=20150818-043&flag=gettime "http://x8shi.com/?nav=24xsc&flaglot=hnquick5&tit=%E6%B2%B3%E5%86%855%E5%88%86%E5%BD%A9""Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/31.0.1650.63 Safari/537.36"  124.134.229.30 x8shi.com 0.140 0.027 -
  1. log format:
log_format access '$remote_addr - $remote_user [$time_local] "$request"' '$status $body_bytes_sent $request_body "$http_referer"' '
"$http_user_agent"  $http_x_forwarded_for $host $request_time $upstream_response_time' " $upstream_cache_status";

now , I should how to write this grok pattern for logstash ?

Please help me …

edited by asb
Sign In to Comment
1 Answer
asb MOD June 23, 2016

The following grok pattern would match the log message as you resented it:

NGUSERNAME [a-zA-Z\.\@\-\+_%]+
NGUSER %{NGUSERNAME}
%{IPORHOST:clientip} %{NGUSER:ident} %{NGUSER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|-)\"%{NUMBER:response} (?:%{NUMBER:bytes}|-) %{NOTSPACE:querystring} (?:"(?:%{URI:referrer}|-)"|%{QS:referrer})%{QS:agent}  %{IPORHOST:forwardedfor} %{IPORHOST:host} %{NUMBER:upstreamresponse} (?:-|%{NUMBER:cache})

I was able to put that together using the great Grok Constructor tool. Using its Matcher, you can test and tweak your patterns.

For a deeper dive into writing filters for Logstash, check out:

Adding Logstash Filters To Improve Centralized Logging
by Mitchell Anicas
One way to increase the effectiveness of your Logstash setup is to collect important application logs and structure the log data by employing filters. In this guide, we will focus primarily on how to add filters for various common application logs.
Report
Have another answer? Share your knowledge.

You can type !ref in this text area to quickly search our full set of tutorials, documentation & marketplace offerings and insert the link!

Related