From 2a77c6f3a0e4ccc32988a41fb434ef59e90df2d1 Mon Sep 17 00:00:00 2001 From: Lars Francke Date: Thu, 29 Oct 2026 11:23:00 +0100 Subject: [PATCH 1/3] fix: Use the correct key for the data transfer cipher suite Hadoop reads dfs.encrypt.data.transfer.cipher.suites, plural. The singular key was ignored, so the client never offered AES/CTR during the SASL handshake, the DataNode never accepted it, and block data was only protected by SASL DIGEST-MD5 with the privacy QOP. Both sides have to set the key: the client only offers a cipher option when it is set in its own configuration (SaslDataTransferClient), and the DataNode only accepts one when it is set in its own (DataTransferSaslUtil.negotiateCipherOption). The discovery ConfigMap already carries the same wire encryption settings as the Pods, so downstream clients pick the corrected key up from there. Config overrides do not reach the discovery ConfigMap, which is why setting the right key through an override was no workaround for clients. --- CHANGELOG.md | 3 +++ docs/modules/hdfs/pages/usage-guide/security.adoc | 4 +++- rust/operator-binary/src/controller/build/kerberos.rs | 2 +- 3 files changed, 7 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d3a631ef..d620a5b6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -60,6 +60,8 @@ All notable changes to this project will be documented in this file. - The warning that `dfsReplication` is greater than the number of datanodes is now raised for datanodes. It was previously raised for journalnodes, even though its text refers to datanode replicas, and never for datanodes ([#830]). +- Set `dfs.encrypt.data.transfer.cipher.suites` instead of the misspelt `dfs.encrypt.data.transfer.cipher.suite`. + Hadoop ignored the misspelt key, so block data was never encrypted with AES/CTR and was only protected by SASL DIGEST-MD5 ([#XXX]). ### Removed @@ -86,6 +88,7 @@ All notable changes to this project will be documented in this file. [#847]: https://github.com/stackabletech/hdfs-operator/pull/847 [#848]: https://github.com/stackabletech/hdfs-operator/pull/848 [#850]: https://github.com/stackabletech/hdfs-operator/pull/850 +[#XXX]: https://github.com/stackabletech/hdfs-operator/pull/XXX ## [26.7.0] - 2026-07-21 diff --git a/docs/modules/hdfs/pages/usage-guide/security.adoc b/docs/modules/hdfs/pages/usage-guide/security.adoc index e9c3aa56..1708e2fc 100644 --- a/docs/modules/hdfs/pages/usage-guide/security.adoc +++ b/docs/modules/hdfs/pages/usage-guide/security.adoc @@ -33,7 +33,9 @@ The `kerberos.secretClass` is used to give HDFS the possibility to request keyta The `tlsSecretClass` is needed to request TLS certificates, used e.g. for the Web UIs. -NOTE: The hdfs-operator defaults to `AES/CTR/NoPadding` for `dfs.encrypt.data.transfer.cipher.suite`. This can be changed using config overrides. +NOTE: The hdfs-operator sets `dfs.encrypt.data.transfer.cipher.suites` to `AES/CTR/NoPadding`, in the HDFS Pods and in the discovery ConfigMap. +Block data is only encrypted with AES if both the client and the DataNode set it, so a client that does not use the discovery ConfigMap has to set it itself. +It can be changed using config overrides, which only apply to the HDFS Pods. === 4. Verify that Kerberos authentication is required Use `stackablectl stacklet list` to get the endpoints where the HDFS namenodes are reachable. diff --git a/rust/operator-binary/src/controller/build/kerberos.rs b/rust/operator-binary/src/controller/build/kerberos.rs index df0de2a3..6719d3d5 100644 --- a/rust/operator-binary/src/controller/build/kerberos.rs +++ b/rust/operator-binary/src/controller/build/kerberos.rs @@ -55,7 +55,7 @@ impl HdfsSiteConfigBuilder { self.add("dfs.data.transfer.protection", WIRE_ENCRYPTION_PRIVACY); self.add("dfs.encrypt.data.transfer", "true"); self.add( - "dfs.encrypt.data.transfer.cipher.suite", + "dfs.encrypt.data.transfer.cipher.suites", "AES/CTR/NoPadding", ); self From 4a5335625b68367001ae3622d673edb69f844891 Mon Sep 17 00:00:00 2001 From: Lars Francke Date: Thu, 29 Oct 2026 11:55:08 +0100 Subject: [PATCH 2/3] chore: Add the pull request number to the changelog entry --- CHANGELOG.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d620a5b6..845a1cf6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -61,7 +61,7 @@ All notable changes to this project will be documented in this file. datanodes. It was previously raised for journalnodes, even though its text refers to datanode replicas, and never for datanodes ([#830]). - Set `dfs.encrypt.data.transfer.cipher.suites` instead of the misspelt `dfs.encrypt.data.transfer.cipher.suite`. - Hadoop ignored the misspelt key, so block data was never encrypted with AES/CTR and was only protected by SASL DIGEST-MD5 ([#XXX]). + Hadoop ignored the misspelt key, so block data was never encrypted with AES/CTR and was only protected by SASL DIGEST-MD5 ([#854]). ### Removed @@ -88,7 +88,7 @@ All notable changes to this project will be documented in this file. [#847]: https://github.com/stackabletech/hdfs-operator/pull/847 [#848]: https://github.com/stackabletech/hdfs-operator/pull/848 [#850]: https://github.com/stackabletech/hdfs-operator/pull/850 -[#XXX]: https://github.com/stackabletech/hdfs-operator/pull/XXX +[#854]: https://github.com/stackabletech/hdfs-operator/pull/854 ## [26.7.0] - 2026-07-21 From b5906f0266c1e180e38ce9ffd75f70743cc5b8c1 Mon Sep 17 00:00:00 2001 From: Lars Francke Date: Thu, 29 Oct 2026 12:03:10 +0100 Subject: [PATCH 3/3] docs: Drop the note on overrides and the discovery ConfigMap --- docs/modules/hdfs/pages/usage-guide/security.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/modules/hdfs/pages/usage-guide/security.adoc b/docs/modules/hdfs/pages/usage-guide/security.adoc index 1708e2fc..3b86e5e7 100644 --- a/docs/modules/hdfs/pages/usage-guide/security.adoc +++ b/docs/modules/hdfs/pages/usage-guide/security.adoc @@ -35,7 +35,7 @@ The `tlsSecretClass` is needed to request TLS certificates, used e.g. for the We NOTE: The hdfs-operator sets `dfs.encrypt.data.transfer.cipher.suites` to `AES/CTR/NoPadding`, in the HDFS Pods and in the discovery ConfigMap. Block data is only encrypted with AES if both the client and the DataNode set it, so a client that does not use the discovery ConfigMap has to set it itself. -It can be changed using config overrides, which only apply to the HDFS Pods. +This can be changed using config overrides. === 4. Verify that Kerberos authentication is required Use `stackablectl stacklet list` to get the endpoints where the HDFS namenodes are reachable.