SPLUNK SINK CONNECTOR
The Kafka Connect Splunk Sink connector is used to move messages from Apache Kafka® to Splunk.
The connector has the following features:
The Splunk HTTP Event Collector (HEC) receives data from Kafka topics via HTTP or HTTPS connection using an Event Collector token configured in Splunk.
This feature is used to enrich raw data with extra metadata fields. The configured enrichment metadata is indexed along with raw event data by the Splunk software. See Indexed Field Extractions for more information.
Note
Data enrichment for /event HEC endpoint is only available in Splunk Enterprise 6.5 and above.
/event
This feature implements guaranteed delivery by polling Splunk for acknowledgement before committing the Kafka offset.
The following are required to run the Splunk Sink Connector:
HEC Acknowledgement prevents potential data loss but may slow down event ingestion.
You can install this connector by using the instructions or you can manually download the ZIP file.
Navigate to your Confluent Platform installation directory and run the following command to install the latest (latest) connector version. The connector must be installed on every machine where Connect will run.
latest
confluent-hub install splunk/kafka-connect-splunk:latest
You can install a specific version by replacing latest with a version number. For example:
confluent-hub install splunk/kafka-connect-splunk:1.1.1
Download and extract the ZIP file for your connector and then follow the manual connector installation instructions.
The Splunk Sink connector is an open source connector and does not require a Confluent Enterprise License.
For a complete list of configuration properties for this connector, see Splunk Sink Connector Configuration Properties.
For an example of how to get Kafka Connect connected to Confluent Cloud, see Distributed Cluster.
Important
The default port used by a Splunk HEC is 8088. However, the ksqlDB component of Confluent Platform also uses that port. For this quick start, since both Splunk and Confluent Platform will be running, we configure the HEC to use port 8889. If that port is in use by another process, change 8889 to a different, open port.
8088
8889
Start a Splunk Enterprise instance by running the Splunk Docker container.
docker run -d -p 8000:8000 -p 8889:8889 -e "SPLUNK_START_ARGS=--accept-license" -e "SPLUNK_PASSWORD=password" --name splunk splunk/splunk:7.3.0
Open http://localhost:8000 to access Splunk Web. Login with username admin and password password.
admin
password
Configure a Splunk HEC using Splunk Web.
kafka
Note the token value on the “Token has been created successfully” page. This token value is needed for the connector configuration later.
Install the connector through the Confluent Hub Client.
# run from your Confluent Platform installation directory confluent-hub install splunk/kafka-connect-splunk:latest
Start Confluent Platform.
Tip
The command syntax for the Confluent CLI development commands changed in 5.3.0. These commands have been moved to confluent local. For example, the syntax for confluent start is now confluent local services start. For more information, see confluent local.
confluent local
confluent start
confluent local services start
Produce test data to the splunk-qs topic in Kafka.
splunk-qs
echo event 1 | confluent local services kafka produce splunk-qs echo event 2 | confluent local services kafka produce splunk-qs
Create a splunk-sink.properties file with the properties below. Substitute <HEC_TOKEN> with the Splunk HEC token created earlier.
splunk-sink.properties
<HEC_TOKEN>
name=SplunkSink topics=splunk-qs tasks.max=1 connector.class=com.splunk.kafka.connect.SplunkSinkConnector splunk.indexes=main splunk.hec.uri=http://localhost:8889 splunk.hec.token=<HEC_TOKEN> splunk.sourcetypes=my_sourcetype confluent.topic.bootstrap.servers=localhost:9092 confluent.topic.replication.factor=1 value.converter=org.apache.kafka.connect.storage.StringConverter
Start the connector.
Caution
You must include a double dash (--) between the topic name and your flag. For more information, see this post.
--
confluent local services connect connector load splunk --config splunk-sink.properties
In the Splunk UI, verify that data is flowing into your Splunk platform instance by searching using the search parameter source="http:kafka".
source="http:kafka"
Shut down Confluent Platform.
confluent local destroy
Shut down the Docker container.
docker stop splunk docker rm splunk