Posted by: ibmadmin | February 8, 2017

IBM Websphere Datapower – Latency Log parser for Splunk

The following Regular expression Splunk query will help to extract the Datapower Latency log tokens. This query will be helpful to generate the Splunk reports based on the different domains and datapower components like web service proxy, xml filewall and multi protocol gateway etc…

Explanation of Arguments found in the Latency log message
Position Argument
1 request header read
2 request header sent
3 front side transform begun
4 front side transform complete
5 entire request transmitted
6 front side style-sheet ready
7 front side parsing complete
8 response header received
9 response headers sent
10 back side transform begun
11 back side transform complete
12 response transmitted
13 back side style-sheet read
14 back side parsing complete
15 back side connection attempted
16 back side connection completed

Splunk Query with Regular expression field extraction:

index=datapower_index latency <your datapower domain name> <datapower component name> NOT <Exclution String> earliest=”2/7/2017:07:00:00″ latest=”2/8/2017:7:00:00″| rex field=_raw “^(?P<DP_Date_Time>\w+\s+\d+\s+\d+:\d+:\d+) (?P<DP_DeviceName>[^ ]+) (?P<DP_Domain>[^ ]+) \[(?P<DP_UID>.*)\] (?P<DP_ServiceName>[^:]+).*: trans\((?P<DP_TRANSID>.*)\).*Latency:[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_ENTIRE_REQ_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_SYTLE_READY>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_PARSING_COM>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_RECVD>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BS_STYLE_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSPC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCA>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCC>[0-9]*) \[(?P<DP_Backside_URL>.*)\]”|stats count(DP_LATENCY_TIME_RES_TRS) as TotalTrans perc75(DP_LATENCY_TIME_RES_TRS) as 75P, perc80(DP_LATENCY_TIME_RES_TRS) as 80P,perc85(DP_LATENCY_TIME_RES_TRS) as 85P, perc90(DP_LATENCY_TIME_RES_TRS) as 90P, perc95(DP_LATENCY_TIME_RES_TRS) as 95P, perc98(DP_LATENCY_TIME_RES_TRS) as 98P,perc99(DP_LATENCY_TIME_RES_TRS) as 99P,avg(DP_LATENCY_TIME_RES_TRS) as AvgResTime, min(DP_LATENCY_TIME_RES_TRS) as MinResTime,max(DP_LATENCY_TIME_RES_TRS) as maxResTime by DP_ServiceName,DP_Backside_URL|sort DP_ServiceName AvgResTime DESC

index=datapower_index latency <your datapower domain name> <datapower component name> NOT <Exclution String> earliest="2/7/2017:07:00:00" latest="2/8/2017:7:00:00"| rex field=_raw "^(?P<DP_Date_Time>\w+\s+\d+\s+\d+:\d+:\d+) (?P<DP_DeviceName>[^ ]+) (?P<DP_Domain>[^ ]+) \[(?P<DP_UID>.*)\] (?P<DP_ServiceName>[^:]+).*: trans\((?P<DP_TRANSID>.*)\).*Latency:[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_ENTIRE_REQ_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_SYTLE_READY>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_PARSING_COM>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_RECVD>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BS_STYLE_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSPC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCA>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCC>[0-9]*) \[(?P<DP_Backside_URL>.*)\]"|stats count(DP_LATENCY_TIME_RES_TRS) as TotalTrans perc75(DP_LATENCY_TIME_RES_TRS) as 75P, perc80(DP_LATENCY_TIME_RES_TRS) as 80P,perc85(DP_LATENCY_TIME_RES_TRS) as 85P, perc90(DP_LATENCY_TIME_RES_TRS) as 90P, perc95(DP_LATENCY_TIME_RES_TRS) as 95P, perc98(DP_LATENCY_TIME_RES_TRS) as 98P,perc99(DP_LATENCY_TIME_RES_TRS) as 99P,avg(DP_LATENCY_TIME_RES_TRS) as AvgResTime, min(DP_LATENCY_TIME_RES_TRS) as MinResTime,max(DP_LATENCY_TIME_RES_TRS) as maxResTime by DP_ServiceName,DP_Backside_URL|sort DP_ServiceName AvgResTime DESC


Regular Expression alone to check in https://regex101.com website:

^(?P<DP_Date_Time>\w+\s+\d+\s+\d+:\d+:\d+) (?P<DP_DeviceName>[^ ]+) (?P<DP_Domain>[^ ]+) \[(?P<DP_UID>.*)\] (?P<DP_ServiceName>[^:]+).*: trans\((?P<DP_TRANSID>.*)\).*Latency:[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_ENTIRE_REQ_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_SYTLE_READY>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_PARSING_COM>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_RECVD>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BS_STYLE_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSPC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCA>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCC>[0-9]*) \[(?P<DP_Backside_URL>.*)\]


Sample Latency Log:

Feb 07 16:26:15 ibmadmin_dp_device_host_name ibmadmin_Domain1 [0x80e00073][latency][info] mpgw(DataPower_ibmadmin_MPGW): trans(113306135) gtid(219725225): Latency:   0  37   0  36  37  31   6 303 329 303 329 329 322 303  36  37 [https://ibmadmin.wordpress.com/2017/02/08/ibm-websphere-datapower-latency-log-parser-for-splunk/]

The following website will help us to build and test the regular expression (regex):

https://regex101.com/

Ref:

https://www.ibm.com/developerworks/community/blogs/aimsupport/entry/analyzing_datapower_latency_log?lang=en

http://www-01.ibm.com/support/docview.wss?uid=swg21239328

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Categories

%d bloggers like this: