The following Regular expression Splunk query will help to extract the Datapower Latency log tokens. This query will be helpful to generate the Splunk reports based on the different domains and datapower components like web service proxy, xml filewall and multi protocol gateway etc…
Explanation of Arguments found in the Latency log message | |
Position | Argument |
1 | request header read |
2 | request header sent |
3 | front side transform begun |
4 | front side transform complete |
5 | entire request transmitted |
6 | front side style-sheet ready |
7 | front side parsing complete |
8 | response header received |
9 | response headers sent |
10 | back side transform begun |
11 | back side transform complete |
12 | response transmitted |
13 | back side style-sheet read |
14 | back side parsing complete |
15 | back side connection attempted |
16 | back side connection completed |
Splunk Query with Regular expression field extraction:
index=datapower_index latency <your datapower domain name> <datapower component name> NOT <Exclution String> earliest=”2/7/2017:07:00:00″ latest=”2/8/2017:7:00:00″| rex field=_raw “^(?P<DP_Date_Time>\w+\s+\d+\s+\d+:\d+:\d+) (?P<DP_DeviceName>[^ ]+) (?P<DP_Domain>[^ ]+) \[(?P<DP_UID>.*)\] (?P<DP_ServiceName>[^:]+).*: trans\((?P<DP_TRANSID>.*)\).*Latency:[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_ENTIRE_REQ_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_SYTLE_READY>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_PARSING_COM>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_RECVD>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BS_STYLE_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSPC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCA>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCC>[0-9]*) \[(?P<DP_Backside_URL>.*)\]”|stats count(DP_LATENCY_TIME_RES_TRS) as TotalTrans perc75(DP_LATENCY_TIME_RES_TRS) as 75P, perc80(DP_LATENCY_TIME_RES_TRS) as 80P,perc85(DP_LATENCY_TIME_RES_TRS) as 85P, perc90(DP_LATENCY_TIME_RES_TRS) as 90P, perc95(DP_LATENCY_TIME_RES_TRS) as 95P, perc98(DP_LATENCY_TIME_RES_TRS) as 98P,perc99(DP_LATENCY_TIME_RES_TRS) as 99P,avg(DP_LATENCY_TIME_RES_TRS) as AvgResTime, min(DP_LATENCY_TIME_RES_TRS) as MinResTime,max(DP_LATENCY_TIME_RES_TRS) as maxResTime by DP_ServiceName,DP_Backside_URL|sort DP_ServiceName AvgResTime DESC
index=datapower_index latency <your datapower domain name> <datapower component name> NOT <Exclution String> earliest="2/7/2017:07:00:00" latest="2/8/2017:7:00:00"| rex field=_raw "^(?P<DP_Date_Time>\w+\s+\d+\s+\d+:\d+:\d+) (?P<DP_DeviceName>[^ ]+) (?P<DP_Domain>[^ ]+) \[(?P<DP_UID>.*)\] (?P<DP_ServiceName>[^:]+).*: trans\((?P<DP_TRANSID>.*)\).*Latency:[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_ENTIRE_REQ_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_SYTLE_READY>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_PARSING_COM>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_RECVD>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BS_STYLE_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSPC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCA>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCC>[0-9]*) \[(?P<DP_Backside_URL>.*)\]"|stats count(DP_LATENCY_TIME_RES_TRS) as TotalTrans perc75(DP_LATENCY_TIME_RES_TRS) as 75P, perc80(DP_LATENCY_TIME_RES_TRS) as 80P,perc85(DP_LATENCY_TIME_RES_TRS) as 85P, perc90(DP_LATENCY_TIME_RES_TRS) as 90P, perc95(DP_LATENCY_TIME_RES_TRS) as 95P, perc98(DP_LATENCY_TIME_RES_TRS) as 98P,perc99(DP_LATENCY_TIME_RES_TRS) as 99P,avg(DP_LATENCY_TIME_RES_TRS) as AvgResTime, min(DP_LATENCY_TIME_RES_TRS) as MinResTime,max(DP_LATENCY_TIME_RES_TRS) as maxResTime by DP_ServiceName,DP_Backside_URL|sort DP_ServiceName AvgResTime DESC Regular Expression alone to check in https://regex101.com website: ^(?P<DP_Date_Time>\w+\s+\d+\s+\d+:\d+:\d+) (?P<DP_DeviceName>[^ ]+) (?P<DP_Domain>[^ ]+) \[(?P<DP_UID>.*)\] (?P<DP_ServiceName>[^:]+).*: trans\((?P<DP_TRANSID>.*)\).*Latency:[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_REQ_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_ENTIRE_REQ_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_SYTLE_READY>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_FS_PARSING_COM>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_RECVD>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_HDR_SENT>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTB>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSTC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_RES_TRS>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BS_STYLE_READ>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSPC>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCA>[0-9]*)[ ]*(?P<DP_LATENCY_TIME_BSCC>[0-9]*) \[(?P<DP_Backside_URL>.*)\]
Sample Latency Log:
Feb 07 16:26:15 ibmadmin_dp_device_host_name ibmadmin_Domain1 [0x80e00073][latency][info] mpgw(DataPower_ibmadmin_MPGW): trans(113306135) gtid(219725225): Latency: 0 37 0 36 37 31 6 303 329 303 329 329 322 303 36 37 [https://ibmadmin.wordpress.com/2017/02/08/ibm-websphere-datapower-latency-log-parser-for-splunk/]
The following website will help us to build and test the regular expression (regex):
Ref:
http://www-01.ibm.com/support/docview.wss?uid=swg21239328