mirror of
https://github.com/keycloak/keycloak.git
synced 2026-01-07 07:19:55 -06:00
Address QE comments on HA guide (#34902)
Closes #34887
Signed-off-by: AndyMunro <amunro@redhat.com>
(cherry picked from commit 2201241949)
This commit is contained in:
@@ -23,7 +23,7 @@ Ensures that synchronous replication is available for both the database and the
|
||||
*Suggested setup:* Two AWS Availability Zones within the same AWS Region.
|
||||
|
||||
*Not considered:* Two regions on the same or different continents, as it would increase the latency and the likelihood of network failures.
|
||||
Synchronous replication of databases as a services with Aurora Regional Deployments on AWS is only available within the same region.
|
||||
Synchronous replication of databases as services with Aurora Regional Deployments on AWS is only available within the same region.
|
||||
|
||||
== Environment for {project_name} and {jdgserver_name}
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ For every 100 login/logout/refresh requests per second:
|
||||
|
||||
- Allocate between 0.35 and 0.7 vCPU.
|
||||
|
||||
The vCPU requirement is given as a range, as with an increased CPU saturation on the database host the CPU usage per request decreased while the response times increase. A lower CPU quota on the database can lead to slower response times during peak loads. Choose a larger CPU quota if fast response times during peak loads are critical. See below for an example.
|
||||
The vCPU requirement is given as a range, as with an increased CPU saturation on the database host the CPU usage per request decreases while the response times increase. A lower CPU quota on the database can lead to slower response times during peak loads. Choose a larger CPU quota if fast response times during peak loads are critical. See below for an example.
|
||||
|
||||
=== Calculation example (single site)
|
||||
|
||||
@@ -90,7 +90,7 @@ Limits calculated:
|
||||
This sums up to 410 requests per second.
|
||||
This expected DB usage is 1.4 to 2.8 vCPU, with a DB idle load of 0.3 vCPU.
|
||||
This indicates either a 2 vCPU `db.t4g.large` instance or a 4 vCPU `db.t4g.xlarge` instance.
|
||||
A 2 vCPU `db.t4g.large` would be more cost-effective if the response times are allowed be higher during peak usage.
|
||||
A 2 vCPU `db.t4g.large` would be more cost-effective if the response times are allowed to be higher during peak usage.
|
||||
In our tests, the median response time for a login and a token refresh increased by up to 120 ms once the CPU saturation reached 90% on a 2 vCPU `db.t4g.large` instance given this scenario.
|
||||
For faster response times during peak usage, consider a 4 vCPU `db.t4g.xlarge` instance for this scenario.)
|
||||
|
||||
@@ -127,16 +127,16 @@ The benefit of this setup is that the number of Pods does not need to scale duri
|
||||
The following setup was used to retrieve the settings above to run tests of about 10 minutes for different scenarios:
|
||||
|
||||
* OpenShift 4.16.x deployed on AWS via ROSA.
|
||||
* Machinepool with `m5.2xlarge` instances.
|
||||
* Machine pool with `m5.2xlarge` instances.
|
||||
* {project_name} deployed with the Operator and 3 pods in a high-availability setup with two sites in active/active mode.
|
||||
* OpenShift's reverse proxy running in passthrough mode were the TLS connection of the client is terminated at the Pod.
|
||||
* OpenShift's reverse proxy runs in the passthrough mode where the TLS connection of the client is terminated at the Pod.
|
||||
* Database Amazon Aurora PostgreSQL in a multi-AZ setup.
|
||||
* Default user password hashing with Argon2 and 5 hash iterations and minimum memory size 7 MiB https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html#argon2id[as recommended by OWASP] (which is the default).
|
||||
* Client credential grants do not use refresh tokens (which is the default).
|
||||
* Database seeded with 20,000 users and 20,000 clients.
|
||||
* Infinispan local caches at default of 10,000 entries, so not all clients and users fit into the cache, and some requests will need to fetch the data from the database.
|
||||
* All authentication sessions in distributed caches as per default, with two owners per entries, allowing one failing Pod without losing data.
|
||||
* All user and client sessions are stored in the database and are not cached in-memory as this was tested a multi-site setup.
|
||||
* All user and client sessions are stored in the database and are not cached in-memory as this was tested in a multi-site setup.
|
||||
Expect a slightly higher performance for single-site setups as a fixed number of user and client sessions will be cached.
|
||||
* OpenJDK 21
|
||||
|
||||
|
||||
@@ -46,7 +46,7 @@ Monitoring is necessary to detect degraded setups.
|
||||
| Seconds to minutes (depending on the database)
|
||||
|
||||
| {project_name} node
|
||||
| Multiple {project_name} instances run in each site. If one instance fails some incoming requests might receive an error message or are delayed for some seconds.
|
||||
| Multiple {project_name} instances run on each site. If one instance fails some incoming requests might receive an error message or are delayed for some seconds.
|
||||
| No data loss
|
||||
| Less than 30 seconds
|
||||
|
||||
|
||||
@@ -65,7 +65,7 @@ When this setting is active, requests that exceed the number of queued requests
|
||||
{project_name}'s liveness probe is non-blocking to avoid a restart of a Pod under a high load.
|
||||
|
||||
// Developer's note: See KeycloakReadyHealthCheck for the details of the blocking/non-blocking behavior
|
||||
The overall health probe and the readiness can probe in some cases block to check the connection to the database, so they might fail under a high load.
|
||||
The overall health probe and the readiness probe can in some cases block to check the connection to the database, so they might fail under a high load.
|
||||
Due to this, a Pod can become non-ready under a high load.
|
||||
|
||||
=== OS Resources
|
||||
|
||||
@@ -3,10 +3,10 @@
|
||||
|
||||
<@tmpl.guide
|
||||
title="Deploy an AWS Lambda to disable a non-responding site"
|
||||
summary="Building block for loadbalancer resilience"
|
||||
summary="Building block for load balancer resilience"
|
||||
tileVisible="false" >
|
||||
|
||||
This {section} explains how to resolve a split-brain scenarios between two sites in a multi-site deployment.
|
||||
This {section} explains how to resolve split-brain scenarios between two sites in a multi-site deployment.
|
||||
It also disables replication if one site fails, so the other site can continue to serve requests.
|
||||
|
||||
This deployment is intended to be used with the setup described in the <@links.ha id="concepts-multi-site"/> {section}.
|
||||
@@ -16,14 +16,14 @@ include::partials/blueprint-disclaimer.adoc[]
|
||||
|
||||
== Architecture
|
||||
|
||||
In the event of a network communication failure between sites in a multi-site deployment, it is no longer possible for the two sites to continue to replicate data between them.
|
||||
In the event of a network communication failure between sites in a multi-site deployment, it is no longer possible for the two sites to continue to replicate the data between them.
|
||||
The {jdgserver_name} is configured with a `FAIL` failure policy, which ensures consistency over availability. Consequently, all user requests are served with an error message until the failure is resolved, either by restoring the network connection or by disabling cross-site replication.
|
||||
|
||||
In such scenarios, a quorum is commonly used to determine which sites are marked as online or offline.
|
||||
However, as multi-site deployments only consist of two sites, this is not possible.
|
||||
Instead, we leverage "`fencing`" to ensure that when one of the sites is unable to connect to the other site, only one site remains in the loadbalancer configuration, and hence only this site is able to serve subsequent users requests.
|
||||
Instead, we leverage "`fencing`" to ensure that when one of the sites is unable to connect to the other site, only one site remains in the load balancer configuration, and hence only this site is able to serve subsequent users requests.
|
||||
|
||||
In addition to the loadbalancer configuration, the fencing procedure disables replication between the two {jdgserver_name} clusters to allow serving user requests from the site that remains in the loadbalancer configuration.
|
||||
In addition to the load balancer configuration, the fencing procedure disables replication between the two {jdgserver_name} clusters to allow serving user requests from the site that remains in the load balancer configuration.
|
||||
As a result, the sites will be out-of-sync once the replication has been disabled.
|
||||
|
||||
To recover from the out-of-sync state, a manual re-sync is necessary as described in <@links.ha id="operate-synchronize" />.
|
||||
@@ -36,13 +36,13 @@ The triggered Lambda function inspects the current Global Accelerator configurat
|
||||
|
||||
In a true split-brain scenario, where both sites are still up but network communication is down, it is possible that both sites will trigger the webhook simultaneously.
|
||||
We guard against this by ensuring that only a single Lambda instance can be executed at a given time.
|
||||
The logic in the AWS Lambda ensures that always one site entry remains in the loadbalancer configuration.
|
||||
The logic in the AWS Lambda ensures that always one site entry remains in the load balancer configuration.
|
||||
|
||||
== Prerequisites
|
||||
|
||||
* ROSA HCP based multi-site Keycloak deployment
|
||||
* AWS CLI Installed
|
||||
* AWS Global Accelerator loadbalancer
|
||||
* AWS Global Accelerator load balancer
|
||||
* `jq` tool installed
|
||||
|
||||
== Procedure
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
<#import "/templates/links.adoc" as links>
|
||||
|
||||
<@tmpl.guide
|
||||
title="Deploy an AWS Global Accelerator loadbalancer"
|
||||
summary="Building block for a loadbalancer"
|
||||
title="Deploy an AWS Global Accelerator load balancer"
|
||||
summary="Building block for a load balancer"
|
||||
tileVisible="false" >
|
||||
|
||||
This topic describes the procedure required to deploy an AWS Global Accelerator to route traffic between multi-site {project_name} deployments.
|
||||
@@ -20,11 +20,11 @@ availability-zone {project_name} deployments.
|
||||
|
||||
== Architecture
|
||||
|
||||
To ensure user requests are routed to each {project_name} site we need to utilise a loadbalancer. To prevent issues with
|
||||
To ensure user requests are routed to each {project_name} site we need to utilise a load balancer. To prevent issues with
|
||||
DNS caching on the client-side, the implementation should use a static IP address that remains the same
|
||||
when routing clients to both availability-zones.
|
||||
|
||||
In this {section} we describe how to route all {project_name} client requests via an AWS Global Accelerator loadbalancer.
|
||||
In this {section} we describe how to route all {project_name} client requests via an AWS Global Accelerator load balancer.
|
||||
In the event of a {project_name} site failing, the Accelerator ensures that all client requests are routed to the remaining
|
||||
healthy site. If both sites are marked as unhealthy, then the Accelerator will "`fail-open`" and forward requests to a site
|
||||
chosen at random.
|
||||
@@ -48,7 +48,7 @@ Perform the following on each of the {project_name} clusters:
|
||||
+
|
||||
.. Login to the ROSA cluster
|
||||
+
|
||||
.. Create a Kubernetes loadbalancer service
|
||||
.. Create a Kubernetes load balancer service
|
||||
+
|
||||
.Command:
|
||||
[source,bash]
|
||||
@@ -261,7 +261,7 @@ aws globalaccelerator create-endpoint-group \
|
||||
----
|
||||
. Optional: Configure your custom domain
|
||||
+
|
||||
If you are using a custom domain, pointed your custom domain to the AWS Global Loadbalancer by configuring an Alias or CNAME in your custom domain.
|
||||
If you are using a custom domain, pointed your custom domain to the AWS Global Load Balancer by configuring an Alias or CNAME in your custom domain.
|
||||
+
|
||||
. Create or update the {project_name} Deployment
|
||||
+
|
||||
@@ -290,7 +290,7 @@ spec:
|
||||
+
|
||||
To ensure that request forwarding works as expected, it is necessary for the Keycloak CR to specify the hostname through
|
||||
which clients will access the {project_name} instances. This can either be the `DualStackDnsName` or `DnsName` hostname associated
|
||||
with the Global Accelerator. If you are using a custom domain and pointed your custom domain to the AWS Global Loadbalancer, use your custom domain here.
|
||||
with the Global Accelerator. If you are using a custom domain, point your custom domain to the AWS Globa Accelerator, and use your custom domain here.
|
||||
|
||||
== Verify
|
||||
To verify that the Global Accelerator is correctly configured to connect to the clusters, navigate to hostname configured above, and you should be presented with the {project_name} admin console.
|
||||
|
||||
@@ -224,7 +224,7 @@ It will throw an error that allows the transaction to be safely rolled back.
|
||||
When this occurs, {project_name} will attempt a retry.
|
||||
|
||||
The `transaction.locking: PESSIMISTIC` is the only supported locking mode; `OPTIMISTIC` is not recommended due to its network costs.
|
||||
The same settings also prevent that one site it updated while the other site is unreachable.
|
||||
The same settings also prevent that one site is updated while the other site is unreachable.
|
||||
|
||||
The `backup.strategy: SYNC` ensures the data is visible and stored in the other site when the {project_name} request is completed.
|
||||
|
||||
@@ -259,7 +259,7 @@ kubectl wait --for condition=CrossSiteViewFormed --timeout=300s infinispans.infi
|
||||
[#connecting-infinispan-to-keycloak]
|
||||
== Connecting {jdgserver_name} with {project_name}
|
||||
|
||||
Now that an {jdgserver_name} server is running, here are the relevant {project_name} CR changes necessary to connect it to {project_name}. These changes will be required in the <@links.ha id="deploy-keycloak-kubernetes" /> {section}.
|
||||
Now that the {jdgserver_name} server is running, here are the relevant {project_name} CR changes necessary to connect it to {project_name}. These changes will be required in the <@links.ha id="deploy-keycloak-kubernetes" /> {section}.
|
||||
|
||||
. Create a Secret with the username and password to connect to the external {jdgserver_name} deployment:
|
||||
+
|
||||
@@ -272,7 +272,7 @@ include::examples/generated/keycloak-ispn.yaml[tag=keycloak-ispn-secret]
|
||||
+
|
||||
[NOTE]
|
||||
====
|
||||
All the memory, resource and database configurations are skipped from the CR below as they have been described in <@links.ha id="deploy-keycloak-kubernetes" /> {section} already.
|
||||
All the memory, resource and database configurations are skipped from the CR below as they have been described in the <@links.ha id="deploy-keycloak-kubernetes" /> {section} already.
|
||||
Administrators should leave those configurations untouched.
|
||||
====
|
||||
+
|
||||
@@ -282,7 +282,7 @@ include::examples/generated/keycloak-ispn.yaml[tag=keycloak-ispn]
|
||||
----
|
||||
<1> The hostname of the remote {jdgserver_name} cluster.
|
||||
<2> The port of the remote {jdgserver_name} cluster.
|
||||
This is optional and it default to `11222`.
|
||||
This is optional and it defaults to `11222`.
|
||||
<3> The Secret `name` and `key` with the {jdgserver_name} username credential.
|
||||
<4> The Secret `name` and `key` with the {jdgserver_name} password credential.
|
||||
<5> The `spi-connections-infinispan-quarkus-site-name` is an arbitrary {jdgserver_name} site name which {project_name} needs for its Infinispan caches deployment when a remote store is used.
|
||||
|
||||
@@ -9,11 +9,11 @@ summary="This describes how to take a site offline so that it no longer processe
|
||||
|
||||
During the deployment lifecycle it might be required that one of the sites is temporarily taken offline
|
||||
for maintenance or to allow for software upgrades. To ensure that no user requests are routed to the site requiring
|
||||
maintenance, it is necessary for the site to be removed from your loadbalancer configuration.
|
||||
maintenance, it is necessary for the site to be removed from your load balancer configuration.
|
||||
|
||||
== Procedure
|
||||
|
||||
Follow these steps to remove a site from the loadbalancer so that no traffic can be routed to it.
|
||||
Follow these steps to remove a site from the load balancer so that no traffic can be routed to it.
|
||||
|
||||
=== Global Accelerator
|
||||
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
The number of JGroup threads is `200` by default.
|
||||
While it can be configured using the property Java system property `jgroups.thread_pool.max_threads`, we advise keeping it at this value.
|
||||
While it can be configured using the Java system property `jgroups.thread_pool.max_threads`, we advise keeping it at this value.
|
||||
As shown in experiments, the total number of Quarkus worker threads in the cluster must not exceed the number of threads in the JGroup thread pool of 200 in each node to avoid deadlocks in the JGroups communication.
|
||||
Given a {project_name} cluster with four Pods, each Pod should then have 50 Quarkus worker threads.
|
||||
Use the {project_name} configuration option `http-pool-max-threads` to configure the maximum number of Quarkus worker threads.
|
||||
|
||||
Reference in New Issue
Block a user