Get Confluent | Sign up for
Confluent Cloud
or download
Confluent Platform
Home
Platform
Cloud
Connectors
Tools
Clients
Download Confluent
Back to home
AMAZON S3 SINK CONNECTOR
Amazon S3 Sink Connector Configuration Properties
Changelog
Changelog
¶
Version 5.4.0
¶
PR-289
- MINOR: Upgrade bugfix versions of dependencies to avoid CVEs
PR-286
- MINOR: Pin hadoop-common version to 2.7.7 and exclude htrace-core dependency
PR-282
- CC-6565: Use parquet.codec and add compression options for Parquet format
PR-281
- CC-6801: Retry effectively on retriable failures while closing files
PR-280
- CC-6766 Handle null-valued records in all three output types
PR-241
- Add Parquet Format
PR-264
- MINOR: New log for SSE
PR-263
- Remove jackson databind version override
PR-250
- CC-4969 MINOR: Remove Unused S3Storage object from TopicPartitionWriter
PR-114
- CC-147: logical types support for HiveSchemaConverter
PR-116
- CC-6565: Add parquet.codec property to enable compression in storage sink connectors
PR-110
- CC-5963: Improved ability to handle schema evolution with different names
Version 5.3.0
¶
PR-244
- CC-4433c: Remove unused dependency on Hive.
PR-243
- CC-4228: Add configuration option to override expect-continue in upload protocol.
Version 5.2.0
¶
PR-220
- CC-3035: Upgrade the S3Mock library so the module builds on Java 8 and Java 11
Version 5.1.0
¶
No changes
Version 5.0.0
¶
PR-181
- MINOR: Update config options to include SSE-C
PR-173
- Extend the existing SSE support to enable Customer (provided) Keys SSE-C
PR-171
- MINOR: Remove MaxPermSize since we dropped support for Java 7
PR-160
- CC-1793. Used full jitter backoff strategy from AWS SDK.
PR-156
- CC-1497. Added logic for passing a subset of configs down to configurable credentials provider
PR-63
- DataUtils::getField returns field
Version 4.1.2
¶
S3 Connector
¶
No changes
Version 4.1.1
¶
S3 Connector
¶
PR-155
- CC-918: Update build to generate plugin package
PR-159
- CC-1791: Added compression options to raw value example
Version 4.1.0
¶
S3 Connector
¶
PR-150
- Make confluent CLI command output shared
PR-148
- Rely on jackson.version from common pom instead of defining it ourselves
PR-149
- Bump Jackson version
PR-140
- Fix for BufferOverflowException in S3OutputStream
PR-106
- Add s3.sse.kms.key.id to specify a particular key in AWS KMS
PR-98
- Add gzip compression support for json and byte array formats
PR-143
- Support for multiple fields in field partitioner
PR-138
- CC-1490: NPE when rotate.interval.ms is set but partitioner is not instance of TimeBasedPartitioner
PR-132
- CC-1214: Import all the AvroData configs to storage sink connectors
PR-123
- CC-1299: Refactor this code to use the AVRO_CODEC config key from storage-common
PR-114
- Update to PowerMock 2.0.x and Mockito 2
PR-48
- RecordField nested timestamp extraction
PR-57
- Issue-53. Multiple field support for partitioning.
PR-55
- CC-1489: NPE on records with null timestamp.
PR-45
- HOTFIX: Consider all the properties in AvroDataConfig.
PR-52
- CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
PR-51
- CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
PR-50
- CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
PR-46
- CC-1299: Create AVRO_CODEC ConfigKey
Version 4.0.1
¶
S3 Connector
¶
PR-148
- Rely on jackson.version from common pom instead of defining it ourselves
PR-149
- Bump Jackson version
PR-140
- Fix for BufferOverflowException in S3OutputStream
PR-138
- CC-1490: NPE when rotate.interval.ms is set but partitioner is not instance of TimeBasedPartitioner
PR-132
- CC-1214: Import all the AvroData configs to storage sink connectors
PR-55
- CC-1489: NPE on records with null timestamp.
PR-45
- HOTFIX: Consider all the properties in AvroDataConfig.
Version 4.0.0
¶
S3 Connector
¶
PR-102
- Allow to write records at the root of the bucket
PR-105
- Add instructions about upstream builds
PR-97
- MINOR: S3 Proxy Settings revisited
PR-103
- Remove unused imports
PR-101
- Remove unused imports
PR-85
- Add ByteArrayFormat
PR-87
- Add s3.canned.acl config
PR-84
- Create CONTRIBUTING.md
PR-92
- CC-1114: Switch to common pom and fix checkstyle issues.
PR-78
- Retry s3 part upload
PR-80
- HOTFIX: Remove refs to schema-generator config.
PR-60
- Add logging when creating a new RecordWriter
PR-44
- HOTFIX: Remove unused parameter in newConfigDef for PartitionerConfig.
PR-41
- HOTFIX: Update dependencies
PR-40
- HOTFIX: Make specific dependencies explicit.
PR-37
- Remove unused imports
PR-35
- Add missing modules to the dependencyManagement pom section so downstream projects will inherit the right version automatically.
PR-31
- Remove schema.generator.class config and have Formats specify their own SchemaGenerator internally
Version 3.3.1
¶
S3 Connector
¶
PR-116
- HOTFIX: Add recommender for schema generator classes only on 3.3.x.
PR-112
- CC-1244: 3.3.1 release notes
PR-108
- HOTFIX: Handle primitive types in AvroFormat
PR-104
- Fix new client creation to cater for URLs and region together
PR-100
- CC-1167: Add recommenders for class type properties
PR-90
- Add upstream project so build are triggered automatically
PR-82
- CC-872: Update default part.size number in S3 docs and more.
PR-77
- Update quickstart to use Confluent CLI.
PR-43
- HOTFIX: Add recommender for schema generator classes only on 3.3.x.
PR-39
- HOTFIX: Check simple class names in recommender.
PR-38
- CC-1153: Isolate config instances and recommenders.
PR-36
- CC-1153: Storage connectors can add recommendations for Class types
PR-33
- Add upstream project so build are triggered automatically
Version 3.3.0
¶
S3 Connector
¶
PR-28
- HOTFIX: Explicitly deactivate licensing profiles by default.
PR-34
- CC-524: Reduce the default s3.part.size to 25MB to avoid OOM with current default jvm heap size.
PR-33
- CC-513: Newline is not appended between records when using JsonFormat.
PR-37
- Enable compression of output Avro files.
PR-38
- CC-530: Exclude storage-common jars when packaging S3 connector.
PR-19
- Move to using io.confluent:common for deps.
PR-22
- HOTFIX: Import recent changes from the hdfs connector.
PR-24
- Convert - to _ in Hive table names.
PR-25
- Include causes with ConfigExceptions caused by catching other exceptions.
PR-26
- Add constructors to SchemaGenerators to support no parameters and a Map config.
PR-29
- Timestamp information should be copied to projected SinkRecord.
Version 3.2.2
¶
S3 Connector
¶
PR-19
- CC-500: Provide exactly-once time-based partitioning in S3
PR-45
- HOTFIX: S3SinkConnector should extend SinkConnector
PR-51
- Allow custom partitioners to have their own configs
PR-27
- HOTFIX: Include a trailing delimiter when verifying data format for Hive
PR-23
- HOTFIX: Add test schema and record builder for records with timestamp field
PR-18
- CC-497: Add timestamp based partitioners.
Version 3.2.1
¶
S3 Connector
¶
PR-33
- Separate JSON records using line separator instead of single white space.
PR-34
- Reduce the default s3.part.size to 25MB to avoid OOM exceptions with the current default java heap size settings for Connect.
PR-32
- Add s3.region property to quickstart config and docs.
PR-25
- flush.size doc fixes.
Version 3.2.0
¶
S3 Connector
¶
Initial Version
On this page: