SNMP Trap Source Connector for Confluent Platform
The Kafka Connect SNMP Trap Source Connector is used to receive data (SNMP traps) from devices through SNMP and convert the trap messages into Kafka records.
Simple Network Management Protocol (SNMP) is an application-layer protocol used to manage and monitor network devices and their functions.
SNMP provides a common language for network devices to relay management information within single and multi-vendor environments in a local
area network (LAN) or wide area network (WAN). The most recent iteration of SNMP, version 3, includes security enhancements that authenticate
and encrypt SNMP messages as well as protect packets during transit.
The connector should be configured to listen to PDUs (a SNMP protocol data unit) from an SNMP device.
The connector interprets only those PDUs that are of type trap.
This connector supports both SNMP V2 and V3 protocols and it can be configured using snmp.v3.enabled
parameter.
If the connector is configured to listen an SNMP V3, then the following SNMPv3 options need to be provided.
- Authentication Protocol: Specifies the SNMPv3 authentication protocol or type and its value
can be any of [md5, sha, hmac128sha224, hmac192sha256, hmac256sha384, hmac384sha512].
- Authentication Password: Specifies the SNMPv3 authentication passphrase or password.
- Privacy/Encryption Protocol: Specifies the SNMPv3 privacy/encryption protocol and its
value can be any of [des, 3des, aes, aes128, aes192, aes256].
- Privacy Password: Specifies the SNMPv3 encryption password.
- Security Name: Specifies the SNMPv3 security name or user name.
Prerequisites
The following are required to run the Kafka Connect SNMP Trap Source Connector:
- Kafka Broker: Confluent Platform 3.3.0 or above, or Kafka 0.11.0 or above
- Connect: Confluent Platform 4.0.0 or above, or Kafka 1.0.0 or above
- Java: 1.8
Limitations
- The connector supports only one task because the connector will open a listener port based on
the configurations of
snmp.listen.protocol
, snmp.listen.address
, and snmp.listen.port
. For multiple tasks,
multiple ports need to be opened. Currently this is not supported.
- The authentication protocol with
sha2
is not currently supported.
Install the SNMP Trap Source Connector
You can install this connector by using the instructions or you can
manually download the ZIP file.
Install the connector using Confluent Hub
- Prerequisite
- Confluent Hub Client must be installed. This is installed by default with Confluent Enterprise.
Navigate to your Confluent Platform installation directory and run the following command to install the latest (latest
) connector version. The connector must be installed on every machine where Connect will run.
confluent-hub install confluentinc/kafka-connect-snmp-trap:latest
You can install a specific version by replacing latest
with a version number. For example:
confluent-hub install confluentinc/kafka-connect-snmp-trap:1.0.0-preview
Quick Start
The following uses the SnmpTrapSourceConnector
to receive data (SNMP traps) from devices through SNMP and send them to the Kafka topic.
Install the connector through the Confluent Hub Client.
# run from your Confluent Platform installation directory
confluent-hub install confluentinc/kafka-connect-snmp-trap:latest
Tip
By default, the plugin is installed into share/confluent-hub-components
and the directory is added to the plugin path. If this is the first connector you have installed, you may need to restart the Connect server for the plugin path change to take effect.
Property-based example
Create a snmp-trap-source-quickstart.properties
file with the following contents or use the snmp-trap-source-quickstart.properties
. This configuration is used typically along with standalone workers.:
name=SnmpTrapSourceConnector
tasks.max=1
connector.class=io.confluent.connect.snmp.SnmpTrapSourceConnector
snmp.v3.enabled=true
kafka.topic=snmp-kafka-topic
snmp.batch.size=50
snmp.listen.address=<ip-address to listen trap from>
snmp.listen.port=<port to listen trap from>
auth.password=<Auth-Password>
privacy.password=<privacy-password>
security.name=<security-name>
confluent.topic.bootstrap.servers=localhost:9092
confluent.topic.replication.factor=1
confluent.license=
Tip
The following defines the Confluent license stored in Kafka, so we need the Kafka bootstrap addresses. The replication.factor
may not be larger than the number of Kafka brokers in the destination cluster, so here we set this to ‘1’ for demonstration purposes. Always use at least ‘3’ in production configurations.
Load the SNMP Trap Source Connector.
Caution
You must include a double dash (--
) between the connector name and your flag. For more information,
see this post.
Tip
The command syntax for the Confluent CLI development commands changed in 5.3.0.
These commands have been moved to confluent local
. For example, the syntax for confluent start
is now
confluent local services start
. For more information, see confluent local.
confluent local services connect connector load snmp-trap-source --config snmp-trap-source-quickstart.properties
Important
Don’t use the Confluent CLI in production environments.
Confirm that the connector is in a RUNNING
state.
confluent local services connect connector status snmp-trap-source
The SNMP device should be running and generating PDUs. The connector will listen and push PDUs of type trap to a Kafka topic.
Confirm that the messages are being sent to Kafka.
kafka-avro-console-consumer --bootstrap-server localhost:9092 --property schema.registry.url=http://localhost:8081 --topic snmp-kafka-topic --from-beginning
A sample SNMP PDU of type trap might look like this for sysDescr
Oid. Refer - https://www.alvestrand.no/objectid/1.3.6.1.2.1.1.1.html:
TRAP[
{
contextEngineID=80:00:00:59:03:78:d2:94:b8:9f:95,
contextName=
},
requestID=2058388122,
errorStatus=0,
errorIndex=0,
VBS[
1.3.6.1.2.1.1.1.0 = 24-Port Gigabit Smart Switch with PoE and 4 SFP uplinks
]
]
Data in Kafka topic:
{
"peerAddress":"127.0.0.1/55159",
"securityName":"admin",
"variables":[
{
"oid":"1.3.6.1.2.1.1.1.0",
"type":"octetString",
"counter32":null,
"counter64":null,
"gauge32":null,
"integer":null,
"ipaddress":null,
"null":null,
"objectIdentifier":null,
"octetString":null,
"opaque":null,
"timeticks":null,
"metadata":{
"string":"24-Port Gigabit Smart Switch with PoE and 4 SFP uplinks"
}
}]
}
REST-based example
Use this setting with distributed workers. Write the following JSON to snmp-trap-source-config.json
, configure all of the required values, and use the following command to
post the configuration to one of the distributed connect workers. For more information, see the
Kafka Connect Kafka Connect REST Interface.
{
"name": "SnmpTrapSourceConnector",
"config": {
"name":"SnmpTrapSourceConnector",
"connector.class":"io.confluent.connect.snmp.SnmpTrapSourceConnector",
"tasks.max":"1",
"kafka.topic":"snmp-kafka-topic",
"snmp.v3.enabled":"true",
"snmp.batch.size":"50",
"snmp.listen.address":"<ip-address to listen trap from>",
"snmp.listen.port":"<port to listen trap from>",
"auth.password":"<Auth-Password>",
"privacy.password":"<privacy-password>",
"security.name":"<security-name>",
"confluent.topic.bootstrap.servers":"localhost:9092",
"confluent.topic.replication.factor":"1"
}
}
Use curl
to post the configuration to one of the Kafka Connect Workers. Change http://localhost:8083/
the endpoint of
one of your Kafka Connect worker(s).
curl -sS -X POST -H 'Content-Type: application/json' --data @snmp-trap-source-config.json http://localhost:8083/connectors
Use the following command to update the configuration of existing connector.
curl -s -X PUT -H 'Content-Type: application/json' --data @snmp-trap-source-config.json http://localhost:8083/connectors/snmpTrapSourceConnector/config
Check that the connector started successfully. Review the Connect worker’s log by entering the following:
confluent local services connect log
The SNMP device should be running and generating PDUs. The connector will listen and push PDUs of type trap to Kafka topic.
Record Schema
The source connector creates records in the following format:
Key Schema
The Key is a struct
with the following fields:
Field Name |
Schema Type |
Optional? |
Description |
peerAddress |
string |
mandatory |
Remote address of the host sending the trap. |
Value Schema
The Value is a struct
with the following fields:
Field Name |
Schema Type |
Optional? |
Description |
peerAddress |
string |
mandatory |
Remote address of the host sending the trap. |
securityName |
string |
mandatory |
Community name the event was sent to. |
variables |
array of struct |
mandatory |
Variables for this trap. |
Variable Schema
The Variable is a struct
with the following fields:
Field Name |
Schema Type |
Optional? |
Description |
oid |
string |
mandatory |
OID |
type |
string |
mandatory |
Syntax type for variable binding. |
counter32 |
int32 |
Optional |
Counter32 value. Ranges from 0 to 4294967295. |
counter64 |
int64 |
Optional |
Counter64 value. Ranges from 0 to 18446744073709551615. |
gauge32 |
int32 |
Optional |
Gauge32 value. Ranges from 0 to 4294967295. |
integer |
int32 |
Optional |
Integer value. |
ipaddress |
string |
Optional |
IpAddress value. |
null |
string |
Optional |
null value. |
objectIdentifier |
string |
Optional |
OID value. |
octetString |
string |
Optional |
Octet string value. |
opaque |
string |
Optional |
opaque value. |
timeticks |
int32 |
Optional |
timeticks value. Ranges from 0 to 4294967295. |
metadata |
string |
Optional |
metadata field that contains value object of variables. |
Additional Documentation
SNMP TRAP SOURCE CONNECTOR