Get Confluent | Sign up for
Confluent Cloud
or download
Confluent Platform
Home
Platform
Cloud
Connectors
Tools
Clients
Download Confluent
Back to home
HDFS 2 SINK CONNECTOR
HDFS 2 Sink Connector Configuration Properties
HDFS
Security
Connector
Storage
Partitioner
Hive
Schema
Changelog
Version 10.0.0
Version 5.5.3
Version 5.5.2
Version 5.5.1
Version 5.5.0
Version 5.4.3
Version 5.4.2
Version 5.4.1
Version 5.4.0
Version 5.3.4
Version 5.3.3
Version 5.3.2
Version 5.3.1
Version 5.3.0
Version 5.2.4
Version 5.2.3
Version 5.2.2
Version 5.2.1
Version 5.2.0
Version 5.1.4
Version 5.1.3
Version 5.1.2
Version 5.1.1
Version 5.1.0
Version 5.0.4
Version 5.0.3
Version 5.0.2
Version 5.0.1
Version 5.0.0
Version 4.1.4
Version 4.1.3
Version 4.1.2
Version 4.1.1
Version 4.1.0
Version 4.0.3
Version 4.0.2
Version 4.0.1
Version 4.0.0
Version 3.3.3
Version 3.3.2
Version 3.3.1
Version 3.3.0
Version 3.2.4
Version 3.2.3
Version 3.2.2
Version 3.2.1
Version 3.2.0
Version 3.1.2
Version 3.1.1
Version 3.1.0
Version 3.0.1
HDFS Connector
Version 3.0.0
HDFS Connector
Version 2.0.1
Version 2.0.0
Changelog
¶
Version 10.0.0
¶
PR-529
- Import CVE-free htrace-core4 and avatica from kafka-connect-storage-common
PR-527
- Update transitive dependencies to fix CVEs
PR-526
- CC-12567: Fix duplicate columns in Hive for Parquet and Avro formats
PR-522
- Properly initialize AvroData by using config’s avroDataConfig method
PR-510
- Resolving netty CVEs
PR-507
- Upgrade hadoop version to address beanutils CVE
PR-501
- CC-7360: Recover gracefully if closing a temporary file fails
PR-506
- CC-9755: Validate timezone configuration with rotate.schedule.interval.ms
PR-503
- CC-9710: Retry effectively on exceptions from time-based commits
PR-496
- CC-8809: Replace assignment with topicPartitionWriters.keySet
PR-497
- MINOR: add orc format to configs
PR-484
- CC-7648: add ORC format with Hive
PR-162
- CC-1360: Enable enhanced avro schema support by default in storage sink connectors
PR-151
- Correct SR schemas cache size property name for AvroDataConfig
PR-156
- Update commons codec to 1.15 to resolve CVE and exclude zookeeper test jar
PR-155
- Upgrade hadoop to 2.10.1 and hive to 2.3.7
PR-138
- CC-12005: Exclude vulnerable dependency jackson mapper asl
PR-149
- Upgrade parquet.version to 1.11.1
PR-139
- CC-8704: Exclude jetty from hcatalog - CVE
PR-136
- CC-9779: PartitionerConfig should perform basic timezone validation
PR-132
- CC-8816: Document more accurate timezone restrictions
Version 5.5.3
¶
PR-530
- Fix typo in Javadoc
PR-522
- Fix #326 by using config’s
avroDataConfig
method
Version 5.5.2
¶
PR-510
- Resolving netty CVEs
PR-507
- Upgrade hadoop version to address beanutils CVE
PR-501
- CC-7360: Recover gracefully if closing a temporary file fails
PR-506
- CC-9755: Validate timezone configuration with rotate.schedule.interval.ms
PR-503
- CC-9710: Retry effectively on exceptions from time-based commits
Version 5.5.1
¶
PR-496
- CC-8809: Replace assignment with topicPartitionWriters.keySet
Version 5.5.0
¶
PR-491
- CC-8229: HDFS2 format options are not showing all available options
PR-477
- CC-7247: allow topics.dir config to be more configurable
Version 5.4.3
¶
PR-510
- Resolving netty CVEs
PR-507
- Upgrade hadoop version to address beanutils CVE
PR-501
- CC-7360: Recover gracefully if closing a temporary file fails
PR-506
- CC-9755: Validate timezone configuration with rotate.schedule.interval.ms
PR-503
- CC-9710: Retry effectively on exceptions from time-based commits
Version 5.4.2
¶
PR-496
- CC-8809: Replace assignment with topicPartitionWriters.keySet
PR-491
- CC-8229: HDFS2 format options are not showing all available options
Version 5.4.1
¶
No changes
Version 5.4.0
¶
PR-460
- CC-6423: Disable FileSystem Object Caching
PR-456
- CC-6196: fix NPE for store.url config
PR-451
- CC-6089: Improve fault tolerance of HDFS sink connector
PR-450
- CC-5969: Pass extra connector configs to the partitioner
PR-442
- Adding debug logs for WAL recovery
PR-440
- Remove jackson databind version override
PR-114
- CC-147: logical types support for HiveSchemaConverter
PR-116
- CC-6565: Add parquet.codec property to enable compression in storage sink connectors
PR-110
- CC-5963: Improved ability to handle schema evolution with different names
Version 5.3.4
¶
PR-501
- CC-7360: Recover gracefully if closing a temporary file fails
PR-506
- CC-9755: Validate timezone configuration with rotate.schedule.interval.ms
PR-503
- CC-9710: Retry effectively on exceptions from time-based commits
PR-496
- CC-8809: Replace assignment with topicPartitionWriters.keySet
PR-491
- CC-8229: HDFS2 format options are not showing all available options
Version 5.3.3
¶
No changes
Version 5.3.2
¶
PR-460
- CC-6423: Disable FileSystem Object Caching
PR-456
- CC-6196: fix NPE for store.url config
Version 5.3.1
¶
PR-451
- CC-6089: Improve fault tolerance of HDFS sink connector
Version 5.3.0
¶
No changes
Version 5.2.4
¶
PR-491
- CC-8229: HDFS2 format options are not showing all available options
PR-460
- CC-6423: Disable FileSystem Object Caching
PR-456
- CC-6196: fix NPE for store.url config
PR-451
- CC-6089: Improve fault tolerance of HDFS sink connector
Version 5.2.3
¶
No changes
Version 5.2.2
¶
No changes
Version 5.2.1
¶
No changes
Version 5.2.0
¶
PR-422
- CC-4124: Clear partition assignment on close
PR-417
- CC-3950: Change the HDFS sink task to return to Connect offsets for records committed to HDFS
Version 5.1.4
¶
PR-491
- CC-8229: HDFS2 format options are not showing all available options
PR-460
- CC-6423: Disable FileSystem Object Caching
PR-456
- CC-6196: fix NPE for store.url config
PR-451
- CC-6089: Improve fault tolerance of HDFS sink connector
Version 5.1.3
¶
No changes
Version 5.1.2
¶
No changes
Version 5.1.1
¶
No changes
Version 5.1.0
¶
No changes
Version 5.0.4
¶
PR-496
- CC-8809: Replace assignment with topicPartitionWriters.keySet
PR-491
- CC-8229: HDFS2 format options are not showing all available options
PR-460
- CC-6423: Disable FileSystem Object Caching
PR-456
- CC-6196: fix NPE for store.url config
PR-451
- CC-6089: Improve fault tolerance of HDFS sink connector
Version 5.0.3
¶
No changes
Version 5.0.2
¶
No changes
Version 5.0.1
¶
No changes
Version 5.0.0
¶
PR-63
- DataUtils::getField returns field
Version 4.1.4
¶
PR-496
- CC-8809: Replace assignment with topicPartitionWriters.keySet
PR-456
- CC-6196: fix NPE for store.url config
Version 4.1.3
¶
No changes
Version 4.1.2
¶
PR-335
- MINOR: Fix the secure HDFS test
Version 4.1.1
¶
PR-307
- CC-918 Add Kafka Connect Maven plugin to build
Version 4.1.0
¶
PR-299
- CC-1669: Set no offset when no offset was found for topic partition
PR-293
- Explicitly specify Jackson version instead of relying on transitive dependency version.
PR-283
- Adding StringFormat to available formatters
PR-286
- Support for multiple fields in field partitioner
PR-269
- Fix or suppress findbugs errors
PR-255
- CC-1299: Add configuration setting for Avro compression codec
PR-48
- RecordField nested timestamp extraction
PR-57
- Issue-53. Multiple field support for partitioning.
PR-55
- CC-1489: NPE on records with null timestamp.
PR-45
- HOTFIX: Consider all the properties in AvroDataConfig.
PR-52
- CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
PR-51
- CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
PR-50
- CC-1333: Remove dependency on hive-exec and instead use hive-exec:core
PR-46
- CC-1299: Create AVRO_CODEC ConfigKey
Version 4.0.3
¶
No changes
Version 4.0.2
¶
No changes
Version 4.0.1
¶
PR-305
- CC-1669: Disable commit of consumer offsets
PR-299
- CC-1669: Set no offset when no offset was found for topic partition
PR-293
- Explicitly specify Jackson version instead of relying on transitive dependency version.
PR-55
- CC-1489: NPE on records with null timestamp.
PR-45
- HOTFIX: Consider all the properties in AvroDataConfig.
Version 4.0.0
¶
PR-251
- CC-1270: Reinstantiate HiveConfig properties in getConfig.
PR-247
- MINOR: Exclude storage common dependencies.
PR-246
- CC-1213: Enable record based timebased partitioners in HDFS connector
PR-245
- CC-1212: Accept partitioners implementing the new partitioner interface
PR-243
- CC-1168: Recommenders for HDFS connector’s class type properties
PR-237
- Remove unused imports
PR-196
- CC-492: Add JsonFormat to HDFS Connector
PR-142
- not trying to hold leases on WAL files if we are holding them already.
PR-220
- HOTFIX: Enable default for format and storage classes in hdfs connector
PR-219
- Move to common pom
PR-188
- CC-397: Refactoring on top of storage-common
PR-214
- fix TimeBasedPartitionerTest by defining the expected locale
PR-212
- Remove invalid offset check.
PR-44
- HOTFIX: Remove unused parameter in newConfigDef for PartitionerConfig.
PR-41
- HOTFIX: Update dependencies
PR-40
- HOTFIX: Make specific dependencies explicit.
PR-37
- Remove unused imports
PR-35
- Add missing modules to the dependencyManagement pom section so downstream projects will inherit the right version automatically.
PR-31
- Remove schema.generator.class config and have Formats specify their own SchemaGenerator internally
Version 3.3.3
¶
No changes
Version 3.3.2
¶
No changes
Version 3.3.1
¶
PR-235
- CC-1172: Fix memory leak in TopicPartitionerWriter
PR-227
- Specify a nodeLabel for Jenkins that doesn’t die during the HDFS connector tests.
PR-226
- Add upstream project so builds will be triggered automatically
PR-217
- Update quickstart to use Confluent CLI
Version 3.3.0
¶
PR-187
- CC-491: Consolidate and simplify unit tests of HDFS connector.
PR-205
- Upgrade avro to 1.8.2.
Version 3.2.4
¶
No changes
Version 3.2.3
¶
No changes
Version 3.2.2
¶
PR-194
- Fix HdfsSinkConnector to extend from SinkConnector instead of Connector.
PR-200
- Fix incorrect licensing and webpage info.
Version 3.2.1
¶
No changes
Version 3.2.0
¶
PR-135
- Fix typos
PR-164
- Issue 136 - Support topic with dots in hive.
PR-170
- MINOR: Upgrade Hadoop version to 2.7.3 and joda-time to 2.9.7
Version 3.1.2
¶
No changes
Version 3.1.1
¶
No changes
Version 3.1.0
¶
PR-134
- Flush the last partial file when incoming stream is paused.
PR-133
- CC-331: Update config options docs
PR-126
- Fix TimeBasedPartitioner config validation
PR-112
- Lint change to avoid compiler error in Oracle JDK 1.7 using jenv.
PR-94
- Fix lint annoyances
PR-108
- Revert “support multi partition fields.”
PR-105
- support multi partition fields.
PR-101
- Added link to Confluent documentation for the connector.
PR-92
- Start a new WAL file after
truncate
instead of appending to log.1
PR-87
- Scheduled rotation implementation
PR-90
- Use configured Hadoop configuration object for Parquet writer
PR-91
- Upgrade to Hadoop 2.6.1
PR-70
- Fix handling of topics with periods
PR-68
- prints details of HDFS exceptions
PR-67
- clean up hive metastore artifacts from testing
PR-64
- cleaned up .gitignore. Now ignores Eclipse files
Version 3.0.1
¶
HDFS Connector
¶
PR-82
- add version.txt to share/doc
Version 3.0.0
¶
HDFS Connector
¶
PR-62
- Update doc for Confluent Platform 3.0.
PR-60
- Remove HDFS connectivity check.
PR-55
- Removing retry logic from HiveMetaStore to fix the metastore connection bloat.
PR-50
- Remove close of topic partition writers in DataWriter close.
PR-42
- Using new config validation.
PR-41
- Bump version to 3.0.0-SNAPSHOT and Apache Kafka® dependency to 0.10.0.0-SNAPSHOT.
PR-35
- Minor doc typo fix TimeBasedPartitioner.
PR-33
- Minor doc fix.
Version 2.0.1
¶
No changes
Version 2.0.0
¶
Initial version
On this page: