Splunk (http://www.splunk.com) is a popular tool for monitoring and analysing data. This article covers how to read your ZTM's SNMP and Historical Activity data with Splunk.
Requirements
- Knowledge of Splunk, including a basic understanding of its configuration files (in particular inputs.conf, transforms.conf and props.conf)
- Access to Splunk's host machine
- Knowledge of SNMP, including basic understanding of OIDs
- Access to your traffic manager's admin server
The following has been tested on Splunk 4.2.3 and Zeus Traffic Manager 7.4 using a Linux operating system. Unless otherwise stated, all steps are to be completed on Splunk's host machine.
Step 1: Installing net-snmp
The net-snmp project (http://www.net-snmp.org) provides a set of applications that can retrieve SNMP data and notifications. You will need a number of these applications for this guide. If you are unsure as to whether or not net-snmp is installed, open a terminal and type:
- snmpget -V
If you see text similar to "NET-SNMP version: 5.4.3" then it is installed and you are ready to proceed. If not, you can download it from http://www.net-snmp.org/download.html.
Step 2: Configuring net-snmp
We need to provide a Management information base (MIB) so that net-snmp can parse data from the traffic manager. The traffic manager provides a MIB file which you can download from the admin server at System > SNMP > View SNMP MIB. The easiest way to install this file is to put it in /usr/share/snmp/mibs. You must also add the following line to /etc/snmp/snmp.conf:
- mibs +ZXTM-MIB
To test if this has worked, run the following command:
- snmptranslate -On ZXTM-MIB::totalRequests
If it outputs ".1.3.6.1.4.1.7146.1.2.1.127" then everything is working and you can proceed to the next step. If you get any error messages then you might be missing other MIB files. The following table lists possible error messages and their corresponding missing MIB. Put the file in /usr/share/snmp/mibs to fix the problem.
| Error message | Missing file |
| Cannot adopt OID in ZXTM-MIB: zxtmtraps# ::= { zxtmtraps 0 } |
SNMPv2-SMI |
| Did not find 'DisplayString' in module #-1 (/usr/share/snmp/mibs/ZXTM-MIB) |
RFC1213-MIB |
| Cannot find module (INET-ADDRESS-MIB): At line 16 in /usr/share/snmp/mibs/ZXTM-MIB |
INET-ADDRESS-MIB |
Step 3: Reading SNMP Traps
There are two types of SNMP data that you are going to send to Splunk. The first is an SNMP Trap. Splunk provides documentation for fetching this data, however the following amendments must be made for it to work:
1. Add the following option when calling snmptrapd. This will format snmptrapd's output into something that is easier to interpret.
- -F "%#t %q%#v\n"
NOTE: This also removes some information that snmptrapd logs by default. For a full list of possible format strings run man snmptrapd. Changes to this option might require reworking some of the regular expressions below.
2. In Splunk, set the sourcetype of the snmptrapd input to 'snmp-traps'. This will allow us to create extraction rules based on sourcetype.
3. In Splunk, add a field transformation:
- Regular expression: ZXTM\-MIB::([^ ]+) = STRING: "(.+?[^\\])"
- Format: $1::$2
4. In Splunk, add two field extractions. Both will apply to a sourcetype named 'snmp-traps'. The first uses the transform defined above, the second is inline (the trailing comma is important):
- ^[0-9]+ ZXTM\-MIB::(?[^ ]+),
The above instructions will result in your Splunk configuration files looking similar to the following, depending on how you named your transformation and extractions:
| File | Contents |
| props.conf |
1.[snmp-traps]
2.REPORT-snmp-trapValues = snmp-trapValues
3.EXTRACT-snmp-trapType = ^[0-9]+ ZXTM\-MIB::(?<trapType>[^ ]+),
|
| inputs.conf |
1.[monitor://<snmptrapd log file>]
2.sourcetype = snmp-traps
|
| transforms.conf |
1.[snmp-trapValues]
2.REGEX = ZXTM\-MIB::([^ ]+) = STRING: "(.+?[^\\])"
3.FORMAT = $1::$2
|
Finally you need to tell the traffic manager that you want to send SNMP Traps to Splunk. To do this, navigate to System > Alerting on the admin server. Click on 'Manage Actions' and add a new action of type 'SNMP Trap'. Set the traphost to the Splunk server.
To test if everything is working up to this point, click 'Update and test' on the action's page. You should see events in Splunk with a key/value trapType=testaction.
In the main System > Alerting section you can now set which type of event should trigger an SNMP Trap. If you want all events to be reported, select the newly created action from the dropdown box next to All Events.
Step 4: Polling SNMP values
The traffic manager provides a wealth of information that you might want Splunk to use. This step covers setting up a script to poll your traffic manager at regular intervals and parse the information in a way that Splunk can then interpret.
First, you must enable SNMP on your traffic manager. To do this go to the System > SNMP section on the admin server and ensure that snmp!enabled is set to Yes. You must also set a community string, which will be used shortly.
Next, to create scripted data inputs in Splunk, download the script snmppoll.py (NOTE: for security reasons the server renames the script, remove the _.txt suffix when saving) to $SPLUNK_HOME/bin/scripts and make it executable. The script takes the following arguments:
- snmppoll.py <server> <communitystring> <subidentifier>
You must create a separate script input for each subidentifier that you want to poll. A list of subidentifiers can be found in the ZXTM-MIB file. As an example, below is a scripted input that will poll your traffic manager for data about its nodes every three minutes:
- Command: $SPLUNK_HOME/bin/scripts/snmppoll.py <server> <communitystring> nodes
- Interval: 180
- Source name override: snmppoll.py
- Sourcetype: snmp-nodes
NOTE: Substitute $SPLUNK_HOME with the directory in which splunk is installed (normally /opt/splunk)
By default Splunk assumes that any output from a script is a multi-line event. To stop this, add the following source entry to your props.conf:
- [source::snmppoll.py]
- SHOULD_LINEMERGE = false
Step 5: Parsing Historical Activity data
The last type of data you can make available to Splunk is Historical Activity data, which can be downloaded from the Activity > Historical Activity page of the admin server. This data is largely superflous if you are collecting SNMP data as well, but it might still be useful for generating reports or if SNMP is disabled.
Splunk can read many data formats, however it requires some assistance to read the Historical Activity data stored by the traffic manager. Download the script historyparse.py (NOTE: for security reasons the server renames the script, remove the _.txt suffix when saving) - Splunk will not use this directly so you can put it anywhere. The script will convert the data you download from the admin server into a format Splunk can understand. The script should be invoked as follows:
- ./historyparse.py < ztm_data_in > splunk_data_out
You can then upload and index the splunk_data_out file in Splunk. Depending on what group you chose when downloading the data from the Historical Activity tab, the following keys will be made available:
| Group by | Splunk keys |
| Virtual server |
virtualserverName, value |
| Node |
nodeName, value |
| Pool |
poolName, value |
The value key will always correspond to the data you chose to plot (data read from client, data to sent to client or hits)
As an example, if you upload a file to Splunk (with sourcetype=historical-data) containing the Hits per Virtual server, you might want to generate a timechart for a specific virtual server using the following search command:
1. sourcetype="historical-data" virtualserverName="name" | timechart avg(value)
Step 6: Using the data in Splunk
Now all the information is in Splunk you can use its powerful analytical tools to interpret the data in different ways. As an example of what can be done, here are a couple of example reports with search strings visible:
| Number of connection errors |
|
|
| Bytes received since 9:30am |
 |
Appendix A: Example Splunk configuration files
The following configuration files include script inputs for all ZXTM-MIB subidentifiers and a monitor for snmptrapd. The files are provided as a reference only.
