mirror of
https://github.com/keycloak/keycloak.git
synced 2025-12-21 14:30:05 -06:00
Document network ports for Keycloak clustering
Also switch the default to jdbc-ping as this should be a drop-in replacement looking at the networking behavior of udp. Closes #34658 Signed-off-by: Pedro Ruivo <pruivo@redhat.com> Signed-off-by: Alexander Schwartz <aschwart@redhat.com> Co-authored-by: Alexander Schwartz <aschwart@redhat.com>
This commit is contained in:
Pedro Ruivo
@@ -18,3 +18,11 @@ If you are using Microsoft AD and creating users through the administrative inte
|
||||
In previous versions, it was only possible to update the user status after setting a (non-temporary) password to the user.
|
||||
This behavior was not consistent with other built-in user storages as well as not consistent with others LDAP vendors supported
|
||||
by the LDAP provider.
|
||||
|
||||
= Transport stack `jdbc-ping` as new default, and recommended for cloud setups
|
||||
|
||||
{project_name} now uses by default its database to discover other nodes of the same cluster, which removes the need of additional network related configurations. It is also a default that will work out-of-the-box in cloud environments.
|
||||
|
||||
For information on how to migrate, see the link:{upgradingguide_link}[{upgradingguide_name}], and the https://www.keycloak.org/server/caching[Configuring distributed caches] guide.
|
||||
|
||||
|
||||
|
||||
@@ -29,22 +29,23 @@ To disable the virtual threads, add one of the Java system properties combinatio
|
||||
* `-Djgroups.thread.virtual=false`: disables virtual threads only in JGroups.
|
||||
* `-Dorg.infinispan.threads.virtual=false -Djgroups.thread.virtual=true`: disables virtual threads only in Infinispan.
|
||||
|
||||
= Default transport stack changed to JDBC_PING2 for distributed caches
|
||||
= Default transport stack changed to `jdbc-ping` for distributed caches
|
||||
|
||||
Previous versions of {project_name} used as a default UDP multicast to discover other nodes to form a cluster and to synchronize the replicated caches of {project_name}.
|
||||
This required multicast to be available and to be configured correctly, which is usually not the case in cloud environments.
|
||||
|
||||
Starting with this version, the default changes to a configuration of JDBC_PING2 which uses {project_name}'s database to discover other nodes.
|
||||
As this removes the need for multicast network capabilities, this is a simplification and a drop-in replacement.
|
||||
|
||||
To enable the previous behavior, choose the transport stack `udp`.
|
||||
Starting with this version, the default changes to the `jdbc-ping` configuration which uses {project_name}'s database to discover other nodes.
|
||||
As this removes the need for multicast network capabilities and UDP and no longer using dynamic ports for the TCP-based failure detection, this is a simplification and a drop-in replacement for environments which used the previous default.
|
||||
To enable the previous behavior, choose the transport stack `udp` which is now deprecated.
|
||||
|
||||
The {project_name} Operator will continue to configure `kubernetes` as a transport stack.
|
||||
|
||||
See the https://www.keycloak.org/server/caching[Configuring distributed caches] guide for more information.
|
||||
|
||||
= Deprecated transport stacks for distributed caches
|
||||
|
||||
The `azure`, `ec2` and `google` transport stacks have been deprecated. Users should use the TCP based `jdbc-ping`
|
||||
stack as a direct replacement.
|
||||
The `udp`, `jdbc-ping-udp`, `tcp`, `azure`, `ec2` and `google` transport stacks have been deprecated.
|
||||
Users should use the TCP based `jdbc-ping` stack as a direct replacement.
|
||||
|
||||
= Defining dependencies between provider factories
|
||||
|
||||
|
||||
@@ -16,6 +16,8 @@
|
||||
:jdgserver_name: Infinispan
|
||||
:infinispan-operator-docs: https://infinispan.org/docs/infinispan-operator/main/operator.html
|
||||
:infinispan-xsite-docs: https://infinispan.org/docs/stable/titles/xsite/xsite.html
|
||||
:infinispan_embedding_docs: https://infinispan.org/docs/stable/titles/embedding/embedding.html
|
||||
:infinispan_configuring_docs: https://infinispan.org/docs/stable/titles/configuring/configuring.html
|
||||
:containerlabel: latest
|
||||
:project_product: false
|
||||
:project_community: true
|
||||
|
||||
@@ -14,7 +14,7 @@ The current distributed cache implementation is built on top of https://infinisp
|
||||
== Enable distributed caching
|
||||
When you start {project_name} in production mode, by using the `start` command, caching is enabled and all {project_name} nodes in your network are discovered.
|
||||
|
||||
By default, caches use the `jdbc-ping-udp` stack which is based upon a UDP transport and uses the configured database to track nodes joining the cluster.
|
||||
By default, caches use the `jdbc-ping` stack which is based upon a TCP transport and uses the configured database to track nodes joining the cluster.
|
||||
{project_name} allows you to either choose from a set of pre-defined default transport stacks, or to define your own custom stack, as you will see later in this {section}.
|
||||
|
||||
To explicitly enable distributed infinispan caching, enter this command:
|
||||
@@ -27,7 +27,7 @@ The `local` cache mode is intended only for development and testing purposes.
|
||||
== Configuring caches
|
||||
{project_name} provides a cache configuration file with sensible defaults located at `conf/cache-ispn.xml`.
|
||||
|
||||
The cache configuration is a regular https://infinispan.org/docs/stable/titles/configuring/configuring.html[Infinispan configuration file].
|
||||
The cache configuration is a regular {infinispan_configuring_docs}[Infinispan configuration file].
|
||||
|
||||
The following table gives an overview of the specific caches {project_name} uses.
|
||||
You configure these caches in `conf/cache-ispn.xml`:
|
||||
@@ -197,7 +197,7 @@ The configuration file is relative to the `conf/` directory.
|
||||
=== CLI options for remote server
|
||||
|
||||
For configuration of {project_name} server for high availability and multi-node clustered setup there was introduced following CLI options `cache-remote-host`, `cache-remote-port`, `cache-remote-username` and `cache-remote-password` simplifying configuration within the XML file.
|
||||
Once any of declared CLI parameters are present, it is expected there is no configuration related to remote store present in the XML file.
|
||||
Once any of the declared CLI parameters are present, it is expected there is no configuration related to remote store present in the XML file.
|
||||
|
||||
==== Connecting to an insecure Infinispan server
|
||||
|
||||
@@ -220,63 +220,44 @@ To apply a specific cache stack, enter this command:
|
||||
|
||||
<@kc.start parameters="--cache-stack=<stack>"/>
|
||||
|
||||
The default stack is set to `jdbc-ping-udp` when distributed caches are enabled.
|
||||
The default stack is set to `jdbc-ping` when distributed caches are enabled, which is backwards compatible with the 26.x release stream of {project_name}.
|
||||
|
||||
=== Available transport stacks
|
||||
|
||||
The following table shows transport stacks that are available without any further configuration than using the `--cache-stack` build option:
|
||||
The following table shows transport stacks that are available without any further configuration than using the `--cache-stack` runtime option:
|
||||
|
||||
[%autowidth]
|
||||
|===
|
||||
|Stack name|Transport protocol|Discovery
|
||||
|
||||
|`tcp`|TCP|MPING (uses UDP multicast).
|
||||
|`udp`|UDP|UDP multicast
|
||||
|`jdbc-ping-udp`|UDP|JDBC_PING2
|
||||
|`jdbc-ping`|TCP|JDBC_PING2
|
||||
|`jdbc-ping`|TCP|Database registry - `JDBC_PING2` protocol.
|
||||
|`jdbc-ping-udp` (deprecated)|UDP|Database registry - `JDBC_PING2` protocol.
|
||||
|===
|
||||
|
||||
|
||||
The following table shows transport stacks that are available using the `--cache-stack` runtime option and a minimum configuration:
|
||||
|
||||
[%autowidth]
|
||||
|===
|
||||
|Stack name|Transport protocol|Discovery
|
||||
|
||||
|`kubernetes`|TCP|DNS_PING (requires `-Djgroups.dns.query=<headless-service-FQDN>` to be added to JAVA_OPTS or JAVA_OPTS_APPEND environment variable).
|
||||
|`kubernetes`|TCP|Dns resolution - `DNS_PING` protocol. It requires `-Djgroups.dns.query=<headless-service-FQDN>` to be added to your `JAVA_OPTS_APPEND` environment variable.
|
||||
|`tcp` (deprecated)|TCP|IP multicast - `MPING` protocol. See below on how to configure a unique `jgroups.mcast_addr` or `jgroups.mcast_port` for each cluster.
|
||||
|`udp` (deprecated)|UDP|IP multicast - `PING` protocol. See below on how to configure a unique `jgroups.mcast_addr` or `jgroups.mcast_port` for each cluster.
|
||||
|===
|
||||
|
||||
=== Additional transport stacks
|
||||
====
|
||||
*Additional Stacks*
|
||||
|
||||
IMPORTANT: The following stacks are deprecated. We recommend that you utilise the `jdbc-ping` stack in such environments
|
||||
as it does not require additional configuration or dependencies.
|
||||
|
||||
The following table shows transport stacks that are supported by {project_name}, but need some extra steps to work.
|
||||
Note that _none_ of these stacks are Kubernetes / OpenShift stacks, so no need exists to enable the `google` stack if you want to run {project_name} on top of the Google Kubernetes engine.
|
||||
In that case, use the `kubernetes` stack.
|
||||
Instead, when you have a distributed cache setup running on AWS EC2 instances, you would need to set the stack to `ec2`, because ec2 does not support a default discovery mechanism such as UDP.
|
||||
|
||||
[%autowidth]
|
||||
|===
|
||||
|Stack name|Transport protocol|Discovery
|
||||
|
||||
|`ec2`|TCP|NATIVE_S3_PING
|
||||
|`google`|TCP|GOOGLE_PING2
|
||||
|`azure`|TCP|AZURE_PING
|
||||
|===
|
||||
|
||||
Cloud vendor specific stacks have additional dependencies for {project_name}.
|
||||
For more information and links to repositories with these dependencies, see the https://infinispan.org/docs/dev/titles/embedding/embedding.html#jgroups-cloud-discovery-protocols_cluster-transport[Infinispan documentation].
|
||||
|
||||
To provide the dependencies to {project_name}, put the respective JAR in the `providers` directory and build {project_name} by entering this command:
|
||||
|
||||
<@kc.start parameters="--cache-stack=<ec2|google|azure>"/>
|
||||
It is recommended to use one of the stacks available above.
|
||||
Additional stacks are provided by Infinispan, but it is outside the scope of this guide how to configure them.
|
||||
Please refer to {infinispan_embedding_docs}#cluster-transport[Setting up Infinispan cluster transport] for further documentation.
|
||||
====
|
||||
|
||||
=== Custom transport stacks
|
||||
If none of the available transport stacks are enough for your deployment, you are able to change your cache configuration file
|
||||
and define your own transport stack.
|
||||
|
||||
For more details, see https://infinispan.org/docs/stable/titles/server/server.html#using-inline-jgroups-stacks_cluster-transport[Using inline JGroups stacks].
|
||||
For more details, see {infinispan_embedding_docs}#customizing-jgroups-stacks_cluster-transport[Customizing JGroups stacks].
|
||||
|
||||
.defining a custom transport stack
|
||||
[source]
|
||||
@@ -319,7 +300,29 @@ To restrict unauthorized access, use a self-signed certificate for each {project
|
||||
|
||||
For JGroups stacks with `UDP` or `TCP_NIO2`, see the http://jgroups.org/manual5/index.html#ENCRYPT[JGroups Encryption documentation] on how to set up the protocol stack.
|
||||
|
||||
For more information about securing cache communication, see the https://infinispan.org/docs/stable/titles/security/security.html#[Infinispan security guide].
|
||||
For more information about securing cache communication, see the {infinispan_embedding_docs}#secure-cluster-transport[Encrypting cluster transport] documentation.
|
||||
|
||||
== Network Ports
|
||||
|
||||
To ensure a healthy {project_name} clustering, some network ports need to be open.
|
||||
The table below shows the TCP ports that need to be open for the `jdbc-ping` stack, and a description of the traffic that goes through it.
|
||||
|
||||
|===
|
||||
|Port |Property | Description
|
||||
|
||||
m|7800
|
||||
m|jgroups.bind.address
|
||||
|Unicast data transmission.
|
||||
|
||||
m|57800
|
||||
m|jgroups.fd.port-offset
|
||||
|Failure detection by protocol `FD_SOCK2`.
|
||||
It listens to the abrupt closing of a socket to suspect a {project_name} server failure.
|
||||
The `jgroups.fd.port-offset` property defines the offset from the `jgroups.bind.address`.
|
||||
|
||||
|===
|
||||
|
||||
NOTE: Use `-D<property>=<value` to modify the ports above in your `JAVA_OPTS_APPEND` environment variable or in your CLI command.
|
||||
|
||||
== Exposing metrics from caches
|
||||
|
||||
|
||||
@@ -57,6 +57,17 @@ To ensure that users can continue to log in when a {project_name} instance goes
|
||||
|
||||
To find out more about using multiple nodes, the different caches and an appropriate stack for your environment, see <@links.server id="caching"/>.
|
||||
|
||||
=== Secure network communication
|
||||
|
||||
JGroups supports Java SSL sockets for TCP communication.
|
||||
Check <@links.server id="caching" anchor="securing-cache-communication"/> for more information on how to configure TLS and the alternatives available for UDP communication.
|
||||
|
||||
=== Configure Firewall ports
|
||||
|
||||
A set of network ports must be open to allow a healthy network communication between {project_name} servers.
|
||||
See <@links.server id="caching" anchor="network-ports"/>.
|
||||
It describes what ports need to be open and their usage.
|
||||
|
||||
== Configure {project_name} Server with IPv4 or IPv6
|
||||
The system properties `java.net.preferIPv4Stack` and `java.net.preferIPv6Addresses` are used to configure the JVM for use with IPv4 or IPv6 addresses.
|
||||
|
||||
|
||||
@@ -66,8 +66,8 @@ public class CachingOptions {
|
||||
.category(OptionCategory.CACHE)
|
||||
.expectedValues(false)
|
||||
.description("Define the default stack to use for cluster communication and node discovery.")
|
||||
.defaultValue(Stack.jdbc_ping_udp)
|
||||
.deprecatedValues(Stream.of(Stack.azure, Stack.ec2, Stack.google).map(Object::toString).collect(Collectors.toSet()), "Use 'jdbc-ping' instead")
|
||||
.defaultValue(Stack.jdbc_ping)
|
||||
.deprecatedValues(Stream.of(Stack.azure, Stack.ec2, Stack.google, Stack.tcp, Stack.udp, Stack.jdbc_ping_udp).map(Object::toString).collect(Collectors.toSet()), "Use 'jdbc-ping' instead")
|
||||
.build();
|
||||
|
||||
public static final Option<File> CACHE_CONFIG_FILE = new OptionBuilder<>(CACHE_CONFIG_FILE_PROPERTY, File.class)
|
||||
|
||||
@@ -419,21 +419,23 @@ public class CacheManagerFactory {
|
||||
|
||||
String transportStack = Configuration.getRawValue("kc.cache-stack");
|
||||
if (transportStack != null && !transportStack.isBlank() && !isJdbcPingStack(transportStack)) {
|
||||
warnDeprecatedStack(transportStack);
|
||||
transportConfig.defaultTransport().stack(transportStack);
|
||||
warnDeprecatedCloudStack(transportStack);
|
||||
return;
|
||||
}
|
||||
|
||||
var stackXmlAttribute = transportConfig.defaultTransport().attributes().attribute(STACK);
|
||||
// If the user has explicitly defined a transport stack that is not jdbc-ping or jdbc-ping-udp, return
|
||||
if (stackXmlAttribute.isModified() && !isJdbcPingStack(stackXmlAttribute.get())) {
|
||||
warnDeprecatedCloudStack(stackXmlAttribute.get());
|
||||
warnDeprecatedStack(stackXmlAttribute.get());
|
||||
return;
|
||||
}
|
||||
|
||||
var stackName = transportStack != null ?
|
||||
transportStack :
|
||||
stackXmlAttribute.isModified() ? stackXmlAttribute.get() : "jdbc-ping-udp";
|
||||
stackXmlAttribute.isModified() ? stackXmlAttribute.get() : "jdbc-ping";
|
||||
warnDeprecatedStack(stackName);
|
||||
|
||||
var udp = stackName.endsWith("udp");
|
||||
|
||||
var tableName = JpaUtils.getTableNameForNativeQuery("JGROUPS_PING", em);
|
||||
@@ -459,8 +461,11 @@ public class CacheManagerFactory {
|
||||
transportConfig.defaultTransport().stack(stackName);
|
||||
}
|
||||
|
||||
private static void warnDeprecatedCloudStack(String stackName) {
|
||||
private static void warnDeprecatedStack(String stackName) {
|
||||
switch (stackName) {
|
||||
case "jdbc-ping-udp":
|
||||
case "tcp":
|
||||
case "udp":
|
||||
case "azure":
|
||||
case "ec2":
|
||||
case "google":
|
||||
|
||||
@@ -47,15 +47,15 @@ public class ClusterConfigDistTest {
|
||||
void changeClusterSetting(LaunchResult result) {
|
||||
CLIResult cliResult = (CLIResult) result;
|
||||
cliResult.assertClusteredCache();
|
||||
assertThat(cliResult.getOutput(), Matchers.containsString("ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping-udp`"));
|
||||
assertThat(cliResult.getOutput(), Matchers.containsString("ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping`"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@Launch({ "start-dev", "--cache=ispn", "--cache-stack=jdbc-ping"})
|
||||
@Launch({ "start-dev", "--cache=ispn", "--cache-stack=jdbc-ping-udp"})
|
||||
void testJdbcPingTCP(LaunchResult result) {
|
||||
CLIResult cliResult = (CLIResult) result;
|
||||
cliResult.assertClusteredCache();
|
||||
assertThat(cliResult.getOutput(), Matchers.containsString("ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping`"));
|
||||
assertThat(cliResult.getOutput(), Matchers.containsString("ISPN000078: Starting JGroups channel `ISPN` with stack `jdbc-ping-udp`"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -97,7 +97,7 @@ public class ClusterConfigDistTest {
|
||||
CLIResult cliResult = (CLIResult) result;
|
||||
cliResult.assertStarted();
|
||||
cliResult.assertClusteredCache();
|
||||
assertTrue(cliResult.getOutput().contains("JGroups protocol stack: UDP"));
|
||||
assertTrue(cliResult.getOutput().contains("JGroups protocol stack: TCP"));
|
||||
}
|
||||
|
||||
@Test
|
||||
@@ -107,7 +107,7 @@ public class ClusterConfigDistTest {
|
||||
CLIResult cliResult = (CLIResult) result;
|
||||
cliResult.assertStarted();
|
||||
cliResult.assertClusteredCache();
|
||||
assertTrue(cliResult.getOutput().contains("JGroups protocol stack: UDP"));
|
||||
assertTrue(cliResult.getOutput().contains("JGroups protocol stack: TCP"));
|
||||
}
|
||||
|
||||
@Test
|
||||
|
||||
@@ -89,9 +89,10 @@ Cache:
|
||||
only when remote host is set.
|
||||
--cache-stack <stack>
|
||||
Define the default stack to use for cluster communication and node discovery.
|
||||
Possible values are: tcp, udp, jdbc-ping, jdbc-ping-udp, kubernetes, ec2
|
||||
(deprecated), azure (deprecated), google (deprecated), or a custom one.
|
||||
Default: jdbc-ping-udp. Available only when 'cache' type is set to 'ispn'.
|
||||
Possible values are: tcp (deprecated), udp (deprecated), jdbc-ping,
|
||||
jdbc-ping-udp (deprecated), kubernetes, ec2 (deprecated), azure
|
||||
(deprecated), google (deprecated), or a custom one. Default: jdbc-ping.
|
||||
Available only when 'cache' type is set to 'ispn'.
|
||||
|
||||
Config:
|
||||
|
||||
|
||||
@@ -70,9 +70,10 @@ Cache:
|
||||
Required when feature 'multi-site' or 'clusterless' is set.
|
||||
--cache-stack <stack>
|
||||
Define the default stack to use for cluster communication and node discovery.
|
||||
Possible values are: tcp, udp, jdbc-ping, jdbc-ping-udp, kubernetes, ec2
|
||||
(deprecated), azure (deprecated), google (deprecated), or a custom one.
|
||||
Default: jdbc-ping-udp. Available only when 'cache' type is set to 'ispn'.
|
||||
Possible values are: tcp (deprecated), udp (deprecated), jdbc-ping,
|
||||
jdbc-ping-udp (deprecated), kubernetes, ec2 (deprecated), azure
|
||||
(deprecated), google (deprecated), or a custom one. Default: jdbc-ping.
|
||||
Available only when 'cache' type is set to 'ispn'.
|
||||
|
||||
Config:
|
||||
|
||||
|
||||
@@ -90,9 +90,10 @@ Cache:
|
||||
only when remote host is set.
|
||||
--cache-stack <stack>
|
||||
Define the default stack to use for cluster communication and node discovery.
|
||||
Possible values are: tcp, udp, jdbc-ping, jdbc-ping-udp, kubernetes, ec2
|
||||
(deprecated), azure (deprecated), google (deprecated), or a custom one.
|
||||
Default: jdbc-ping-udp. Available only when 'cache' type is set to 'ispn'.
|
||||
Possible values are: tcp (deprecated), udp (deprecated), jdbc-ping,
|
||||
jdbc-ping-udp (deprecated), kubernetes, ec2 (deprecated), azure
|
||||
(deprecated), google (deprecated), or a custom one. Default: jdbc-ping.
|
||||
Available only when 'cache' type is set to 'ispn'.
|
||||
|
||||
Config:
|
||||
|
||||
|
||||
@@ -70,9 +70,10 @@ Cache:
|
||||
Required when feature 'multi-site' or 'clusterless' is set.
|
||||
--cache-stack <stack>
|
||||
Define the default stack to use for cluster communication and node discovery.
|
||||
Possible values are: tcp, udp, jdbc-ping, jdbc-ping-udp, kubernetes, ec2
|
||||
(deprecated), azure (deprecated), google (deprecated), or a custom one.
|
||||
Default: jdbc-ping-udp. Available only when 'cache' type is set to 'ispn'.
|
||||
Possible values are: tcp (deprecated), udp (deprecated), jdbc-ping,
|
||||
jdbc-ping-udp (deprecated), kubernetes, ec2 (deprecated), azure
|
||||
(deprecated), google (deprecated), or a custom one. Default: jdbc-ping.
|
||||
Available only when 'cache' type is set to 'ispn'.
|
||||
|
||||
Config:
|
||||
|
||||
|
||||
@@ -90,9 +90,10 @@ Cache:
|
||||
only when remote host is set.
|
||||
--cache-stack <stack>
|
||||
Define the default stack to use for cluster communication and node discovery.
|
||||
Possible values are: tcp, udp, jdbc-ping, jdbc-ping-udp, kubernetes, ec2
|
||||
(deprecated), azure (deprecated), google (deprecated), or a custom one.
|
||||
Default: jdbc-ping-udp. Available only when 'cache' type is set to 'ispn'.
|
||||
Possible values are: tcp (deprecated), udp (deprecated), jdbc-ping,
|
||||
jdbc-ping-udp (deprecated), kubernetes, ec2 (deprecated), azure
|
||||
(deprecated), google (deprecated), or a custom one. Default: jdbc-ping.
|
||||
Available only when 'cache' type is set to 'ispn'.
|
||||
|
||||
Config:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user