Preliminary Setup
To add a new connector plugin you must restart Connect. Use the
Confluent CLI command to restart Connect.
Tip
The command syntax for the Confluent CLI development commands changed in 5.3.0.
These commands have been moved to confluent local
. For example, the syntax for confluent start
is now
confluent local services start
. For more information, see confluent local.
confluent local services connect stop && confluent local services connect start
Your output should resemble:
Using CONFLUENT_CURRENT: /Users/username/Sandbox/confluent-snapshots/var/confluent.NuZHxXfq
Starting Zookeeper
Zookeeper is [UP]
Starting Kafka
Kafka is [UP]
Starting Schema Registry
Schema Registry is [UP]
Starting Kafka REST
Kafka REST is [UP]
Starting Connect
Connect is [UP]
Check if the AWS CloudWatch Logs plugin has been installed correctly and picked up
by the plugin loader:
curl -sS localhost:8083/connector-plugins | jq '.[].class' | grep "cloudwatch logs"
Your output should resemble:
"io.confluent.connect.aws.cloudwatch.logs.AwsCloudWatchSourceConnector"
Source Connector Configuration
Start the services using the Confluent CLI:
confluent local services start
Create a configuration file named aws-cloudwatch-logs-source-config.json with the following
contents.
{
"name": "aws-cloudwatch-logs-source",
"config": {
"connector.class": "io.confluent.connect.aws.cloudwatch.logs.AwsCloudWatchSourceConnector",
"tasks.max": "1",
"aws.cloudwatch.logs.url": "https://logs.us-east-2.amazonaws.com",
"aws.cloudwatch.log.group": "my-log-group",
"aws.cloudwatch.log.streams": "my-log-stream",
"name": "aws-cloudwatch-logs-source",
"confluent.topic.bootstrap.servers": "localhost:9092",
"confluent.topic.replication.factor": "1"
}
}
The important configuration parameters used here are:
aws.cloudwatch.logs.url: The endpoint URL that the source connector connects to to pull
the specified logs.
aws.cloudwatch.log.group: The AWS CloudWatch log group under which the log streams are
contained.
aws.cloudwatch.log.streams: A list of AWS CloudWatch log streams from which the logs are
pulled from. The default value is to use all log streams from the configured log group.
tasks.max: The maximum number of tasks that should be created for
this connector.
You may pass your :AWS Credentials to the AWS CloudWatch Logs connector through
your source connector configuration. To pass AWS credentials in the
source configuration set the aws.access.key.id and the aws.secret.key.id: parameters.
"aws.access.key.id":<your-access-key-id>
"aws.secret.access.key":<your-secret-access-key>
Run this command to start the AWS CloudWatch Logs source connector.
Caution
You must include a double dash (--
) between the topic name and your flag. For more information,
see this post.
confluent local services connect connector load aws-cloudwatch-logs-source --config aws-cloudwatch-logs-source-config.json
To check that the connector started successfully view the Connect
worker’s log by running:
confluent local services connect log
Start a Kafka Consumer in a separate terminal session to view the data exported by
the connector into the kafka topic
path/to/confluent/bin/kafka-console-consumer --bootstrap-server localhost:9092 --topic my-log-group.my-log-stream --from-beginning
Finally, stop the Confluent services using the command:
AWS Credentials
By default, the AWS CloudWatch Logs connector looks for AWS credentials in the following
locations and in the following order:
The AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
environment variables accessible to the
Connect worker processes where the connector will be deployed. These variables are
recognized by the AWS CLI and all AWS SDKs (except for the AWS SDK for .NET). You use
export to set these variables.
export AWS_ACCESS_KEY_ID=<your_access_key_id>
export AWS_SECRET_ACCESS_KEY=<your_secret_access_key>
The AWS_ACCESS_KEY
and AWS_SECRET_KEY
can be used instead, but are not recognized by
the AWS CLI.
The aws.accessKeyId
and aws.secretKey
Java system properties on the Connect worker
processes where the connector will be deployed. However, these variables are only recognized by
the AWS SDK for Java and are not recommended.
The ~/.aws/credentials
file located in the home directory of the operating system user
that runs the Connect worker processes. These credentials are recognized by most AWS SDKs
and the AWS CLI. Use the following AWS CLI command to create the credentials file:
You can also manually create the credentials file using a text editor. The file should contain
lines in the following format:
[default]
aws_access_key_id = <your_access_key_id>
aws_secret_access_key = <your_secret_access_key>
Note
When creating the credentials file, make sure that the user creating the credentials file
is the same user that runs the Connect worker processes and that the credentials file is in
this user’s home directory. Otherwise, the AWS CloudWatch Logs connector will not be able
to find the credentials.
See AWS Credentials File Format for additional details.
Choose one of the above to define the AWS credentials that the AWS CloudWatch Logs
connectors use, verify the credentials implementation is set correctly, and then restart all of
the Connect worker processes.
Note
Confluent recommends using either Environment variables or a
Credentials file because these are the most straightforward, and they can
be checked using the AWS CLI tool before running the connector.
Caution
If you configure one of the AWS key and AWS secret key implementations (as
detailed above), credentials can not also be supplied through the following
Credentials Providers or by using the Trusted
Account Credentials
implementation. Attempting to provide credentials using multiple
implementations will cause authentication failure.
Credentials Providers
A credentials provider is a Java class that implements the com.amazon.auth
.AWSCredentialsProvider interface in the AWS Java library and returns AWS credentials
from the environment. By default the AWS CloudWatch Logs connector configuration property
aws.credentials.provider.class
uses the com.amazon.auth.DefaultAWSCredentialsProviderChain class. This class and interface implementation chains together five other
credential provider classes.
The com.amazonaws.auth.DefaultAWSCredentialsProviderChain implementation looks
for credentials in the following order:
Environment variables using the com.amazonaws.auth.EnvironmentVariableCredentialsProvider class implementation. This implementation uses environment
variables AWS_ACCESS_KEY_ID
and AWS_SECRET_ACCESS_KEY
. Environment variables
AWS_ACCESS_KEY
and AWS_SECRET_KEY
are also supported by this implementation; however,
these two variables are only recognized by the AWS SDK for Java and are not recommended.
Java system properties using the com.amazonaws.auth.SystemPropertiesCredentialsProvider class implementation. This implementation uses Java system
properties aws.accessKeyId
and aws.secretKey
.
Credentials file using the com.amazonaws.auth.profile.ProfileCredentialsProvider class implementation. This implementation uses a credentials
file located in the path ~/.aws/credentials
. This credentials provider can be used by most
AWS SDKs and the AWS CLI. Use the following AWS CLI command to create the credentials file:
You can also manually create the credentials file using a text editor. The file should contain lines in the following format:
[default]
aws_access_key_id = <your_access_key_id>
aws_secret_access_key = <your_secret_access_key>
Note
When creating the credentials file, make sure that the user creating the credentials file
is the same user that runs the Connect worker processes and that the credentials file is in
this user’s home directory. Otherwise, the AWS CloudWatch Logs connector will not be able
to find the credentials.
See AWS Credentials File Format for additional details.
Using Trusted Account Credentials
This connector can assume a role and use credentials from a separate trusted
account. This is a default feature provided with recent versions of this
connector that include an updated version of the AWS SDK.
After you create the trust relationship, an IAM user or an application from the trusted account can
use the AWS Security Token Service (AWS STS)
AssumeRole
API operation. This operation provides temporary security credentials that enable
access to AWS resources for the connector. For details, see
Creating a Role to Delegate Permissions to an IAM User.
- Example:
Profile in ~/.aws/credentials:
[default]
role_arn=arn:aws:iam::037803949979:role/kinesis_cross_account_role
source_profile=staging
role_session_name = OPTIONAL_SESSION_NAME
[staging]
aws_access_key_id = <STAGING KEY>
aws_secret_access_key = <STAGING SECRET>
To allow the connector to assume a role with the right permissions, set the
Amazon Resource Name (ARN)
for this role. Additionally, you must choose between source_profile
or credential_source
as the way to get credentials that have permission to assume the role, in the environment where the
connector is running.
Note
When setting up trusted account credentials, be aware that the approach of loading profiles from
both ~/.aws/credentials
and ~/.aws/config
does not work when configuring this connector.
Assumed role settings and credentials must be placed in the ~/.aws/credentials
file.
Using Other Implementations
You can use a different credentials provider. To do this, set the
aws.credentials.provider.class
property to the name of any class that implements the
com.amazon.auth.AWSCredentialsProvider interface.
Important
If you are using a different credentials provider, do not include the aws.access.key.id
and aws.secret.key.id
in the connector configuration file. If these parameters are included,
they will override the custom credentials provider class.
Complete the following steps to use a different credentials provider:
Find or create a Java credentials provider class that implements the com.amazon.auth.
AWSCredentialsProvider interface.
Put the class file in a JAR file.
Place the JAR file in the share/java/kafka-connect-aws-cloudwatch-logs
directory on all
|kconnect| workers.
Restart the Connect workers.
Change the AWS CloudWatch Logs connector property file to use your custom credentials. Add
the provider class entry aws.credentials.provider.class=<className>
in the AWS CloudWatch
Logs connector properties file.
Important
You must use the fully qualified class name in the <className>
entry.