[!WARNING] Working in progress

RedHat Keycloak-26 Performance Test

We want to test the performance of Red Hat Keycloak 26 on a specific requirements and configurations:

45 nodes
50k users in total
100 db connections per node

install keycloak operator

Create a namespace for Keycloak:

oc new-project demo-keycloak

And install the Keycloak Operator from web UI.

build customer keycloak image

In keycloak configuration options, there are build options and configuration. For the build options, we can only apply such options during container image creation.

Here is the offical documentation, tell us how to create a customized and optimized container image.

To create a customized keycloak image, we can run the keycloak with --optimize option, which will skip the first build stage, and run the keycloak app directly, which will boost the startup times.

for job_id in 0...9; do
  oc delete -n demo-keycloak -f ${BASE_DIR}/data/install/keycloak-script-create-users-${job_id}.yaml
done

mkdir -p ${BASE_DIR}/data/base-images/keycloak
cd ${BASE_DIR}/data/base-images/keycloak

cat << EOF > ${BASE_DIR}/data/base-images/keycloak/Dockerfile
FROM registry.redhat.io/rhbk/keycloak-rhel9:26.0-11 as builder

# Enable health and metrics support
ENV KC_HEALTH_ENABLED=true
ENV KC_METRICS_ENABLED=true

# Enable tracing support
# ENV KC_TRACING_ENABLED=true

# Enable features for tracing/opentelementry
ENV KC_FEATURES=opentelemetry

# Configure a database vendor
ENV KC_DB=postgres

WORKDIR /opt/keycloak
# for demonstration purposes only, please make sure to use proper certificates in production instead
# RUN keytool -genkeypair -storepass password -storetype PKCS12 -keyalg RSA -keysize 2048 -dname "CN=server" -alias server -ext "SAN:c=DNS:localhost,IP:127.0.0.1" -keystore conf/server.keystore
RUN /opt/keycloak/bin/kc.sh build

FROM registry.redhat.io/rhbk/keycloak-rhel9:26.0-11
COPY --from=builder /opt/keycloak/ /opt/keycloak/

# change these values to point to a running postgres instance
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]
EOF

podman build . -t quay.io/wangzheng422/qimgs:keycloak-26-2025.03.25-v01
podman push quay.io/wangzheng422/qimgs:keycloak-26-2025.03.25-v01

create a keycloak instance with basic settings

To create a Keycloak instance with basic settings, first you need to create a backend DB. Then, you can create the Keycloak instance based on this DB.

mkdir -p ${BASE_DIR}/data/install/

oc delete -f ${BASE_DIR}/data/install/keycloak-db-pvc.yaml -n demo-keycloak

cat << EOF > ${BASE_DIR}/data/install/keycloak-db-pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgresql-db-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
EOF

oc create -f ${BASE_DIR}/data/install/keycloak-db-pvc.yaml -n demo-keycloak


oc delete -f ${BASE_DIR}/data/install/keycloak-db.yaml -n demo-keycloak

cat << EOF > ${BASE_DIR}/data/install/keycloak-db.yaml
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgresql-db
spec:
  serviceName: postgresql-db-service
  selector:
    matchLabels:
      app: postgresql-db
  replicas: 1
  template:
    metadata:
      labels:
        app: postgresql-db
    spec:
      containers:
        - name: postgresql-db
          image: postgres:15
          args: ["-c", "max_connections=1000"]
          volumeMounts:
            - mountPath: /data
              name: cache-volume
          env:
            - name: POSTGRES_USER
              value: testuser
            - name: POSTGRES_PASSWORD
              value: testpassword
            - name: PGDATA
              value: /data/pgdata
            - name: POSTGRES_DB
              value: keycloak
      volumes:
        - name: cache-volume
          persistentVolumeClaim:
            claimName: postgresql-db-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: postgres-db
spec:
  selector:
    app: postgresql-db
  type: LoadBalancer
  ports:
  - port: 5432
    targetPort: 5432

EOF

oc create -f ${BASE_DIR}/data/install/keycloak-db.yaml -n demo-keycloak

Now, we have a Keycloak database running in our OpenShift cluster. Next, we need to configure Keycloak to use this database.


# create secret needed by keycloak
# the host name here, we use '*' to limit the length of the hostname in the certificate
RHSSO_HOST="*.apps.cluster-b2cpj.b2cpj.sandbox75.opentlc.com"

cd ${BASE_DIR}/data/install/

openssl req -subj "/CN=$RHSSO_HOST/O=Test Keycloak./C=US" -newkey rsa:2048 -nodes -keyout key.pem -x509 -days 365 -out certificate.pem

oc delete secret example-tls-secret -n demo-keycloak
oc create secret tls example-tls-secret --cert certificate.pem --key key.pem -n demo-keycloak


oc delete secret keycloak-db-secret -n demo-keycloak
oc create secret generic keycloak-db-secret -n demo-keycloak \
  --from-literal=username=testuser \
  --from-literal=password=testpassword


# here we create keycloak instance with postgres db and tls secret
# here we change back the host name to actual hostname
oc delete -f ${BASE_DIR}/data/install/keycloak.yaml -n demo-keycloak

# RHSSO_HOST="keycloak-demo-keycloak.apps.demo-01-rhsys.wzhlab.top"
RHSSO_HOST="example-kc-demo-keycloak.apps.cluster-b2cpj.b2cpj.sandbox75.opentlc.com"

RHSSO_IMAGE="quay.io/wangzheng422/qimgs:keycloak-26-2025.03.25-v01"

cat << EOF > ${BASE_DIR}/data/install/keycloak.yaml
apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: example-kc
spec:
  instances: 1
  db:
    vendor: postgres
    host: postgres-db
    usernameSecret:
      name: keycloak-db-secret
      key: username
    passwordSecret:
      name: keycloak-db-secret
      key: password
  http:
    tlsSecret: example-tls-secret
    httpEnabled: true
  # ingress:
  #   className: openshift-default
  hostname:
    hostname: $RHSSO_HOST
  proxy:
    headers: xforwarded
  image: $RHSSO_IMAGE
  startOptimized: true
EOF

oc create -f ${BASE_DIR}/data/install/keycloak.yaml -n demo-keycloak

# get the keycloak initial admin user and password
oc get secret example-kc-initial-admin -n demo-keycloak -o jsonpath='{.data.username}' | base64 --decode && echo
# temp-admin

oc get secret example-kc-initial-admin -n demo-keycloak -o jsonpath='{.data.password}' | base64 --decode && echo
# 0xxxxxxxxxxxxxxxxxxxxxxxxe


# in postgresql pod terminal
# we can see the current value of max_connections is 1000. We can change it to a higher value if needed.
psql -U testuser -d keycloak
# Type "help" for help.

# keycloak=# SHOW max_connections;
#  max_connections 
# -----------------
#  1000
# (1 row)

check the keycloak instance on web interface

Goto https://example-kc-demo-keycloak.apps.cluster-r9m7r.r9m7r.sandbox2453.opentlc.com and login with admin credentials. Then you can see the rhbk default admin console.

For testing purposes, we will extend the session timeout to several days.

[!WARNING] For testing only, do not use this configuration in production.

get current keycloak config

Let us check the config of keycloak instance from web console. Current version is 26.0.10-opr.1.

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: example-kc
  namespace: demo-keycloak
spec:
  db:
    host: postgres-db
    passwordSecret:
      key: password
      name: keycloak-db-secret
    usernameSecret:
      key: username
      name: keycloak-db-secret
    vendor: postgres
  hostname:
    hostname: example-kc-demo-keycloak.apps.cluster-b2cpj.b2cpj.sandbox75.opentlc.com
  http:
    httpEnabled: true
    tlsSecret: example-tls-secret
  # image: 'quay.io/wangzheng422/qimgs:keycloak-26-2025.03.25-v01'
  instances: 1
  proxy:
    headers: xforwarded
  # startOptimized: true

Next, we will check the contents of the /opt/keycloak/conf directory inside the example-kc-0 pod in the demo-keycloak namespace. This will help us learn more about how the Keycloak server is configured.


oc exec -it example-kc-0 -n demo-keycloak -- ls /opt/keycloak/conf
# cache-ispn.xml  keycloak.conf  README.md  truststores

# oc exec -it example-kc-0 -n demo-keycloak -- ls -R /opt/keycloak

oc exec -it example-kc-0 -n demo-keycloak -- cat /opt/keycloak/conf/keycloak.conf

content of /opt/keycloak/conf/keycloak.conf

# Basic settings for running in production. Change accordingly before deploying the server.

# Database

# The database vendor.
#db=postgres

# The username of the database user.
#db-username=keycloak

# The password of the database user.
#db-password=password

# The full database JDBC URL. If not provided, a default URL is set based on the selected database vendor.
#db-url=jdbc:postgresql://localhost/keycloak

# Observability

# If the server should expose healthcheck endpoints.
#health-enabled=true

# If the server should expose metrics endpoints.
#metrics-enabled=true

# HTTP

# The file path to a server certificate or certificate chain in PEM format.
#https-certificate-file=${kc.home.dir}conf/server.crt.pem

# The file path to a private key in PEM format.
#https-certificate-key-file=${kc.home.dir}conf/server.key.pem

# The proxy address forwarding mode if the server is behind a reverse proxy.
#proxy=reencrypt

# Do not attach route to cookies and rely on the session affinity capabilities from reverse proxy
#spi-sticky-session-encoder-infinispan-should-attach-route=false

# Hostname for the Keycloak server.
#hostname=myhostname

Next, let’s check the cache-ispn.xml file. We can do this by running the following command:

oc exec -it example-kc-0 -n demo-keycloak -- cat /opt/keycloak/conf/cache-ispn.xml

And the content of cache-ispn.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  ~ Copyright 2019 Red Hat, Inc. and/or its affiliates
  ~ and other contributors as indicated by the @author tags.
  ~
  ~ Licensed under the Apache License, Version 2.0 (the "License");
  ~ you may not use this file except in compliance with the License.
  ~ You may obtain a copy of the License at
  ~
  ~ http://www.apache.org/licenses/LICENSE-2.0
  ~
  ~ Unless required by applicable law or agreed to in writing, software
  ~ distributed under the License is distributed on an "AS IS" BASIS,
  ~ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  ~ See the License for the specific language governing permissions and
  ~ limitations under the License.
  -->

<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:15.0 http://www.infinispan.org/schemas/infinispan-config-15.0.xsd"
        xmlns="urn:infinispan:config:15.0">

    <cache-container name="keycloak">
        <transport lock-timeout="60000" stack="udp"/>
        <local-cache name="realms" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="1">
            <expiration lifespan="-1"/>
            <memory max-count="10000"/>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="1">
            <expiration lifespan="-1"/>
            <memory max-count="10000"/>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="1">
            <expiration lifespan="-1"/>
            <memory max-count="10000"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="1">
            <expiration lifespan="-1"/>
            <memory max-count="10000"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2">
            <expiration lifespan="-1"/>
        </distributed-cache>
        <local-cache name="authorization" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <replicated-cache name="work">
            <expiration lifespan="-1"/>
        </replicated-cache>
        <local-cache name="keys" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="1000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="-1" lifespan="-1" interval="300000"/>
            <memory max-count="-1"/>
        </distributed-cache>
    </cache-container>
</infinispan>

It is interesting to see how the cache-ispn.xml file has been modified since rhbk-24, espeically in the sessions section and the <memory max-count="10000"/> configuration.

monitor the keycloak instance

Now, we need to configure openshift-monitoring to monitor the keycloak instance. We can do this by creating a configmap and applying it to the cluster. Here is an example of how to create the configmap and monitoring settings.

# create a configmap for openshift-monitoring to enable monitoring for customer workload
cat << EOF > ${BASE_DIR}/data/install/enable-monitor.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-monitoring-config
  namespace: openshift-monitoring
data:
  config.yaml: |
    enableUserWorkload: true 
    # alertmanagerMain:
    #   enableUserAlertmanagerConfig: true 
EOF

oc apply -f ${BASE_DIR}/data/install/enable-monitor.yaml

oc -n openshift-user-workload-monitoring get pod

# monitor keycloak
oc delete -n demo-keycloak -f ${BASE_DIR}/data/install/keycloak-monitor.yaml

cat << EOF > ${BASE_DIR}/data/install/keycloak-monitor.yaml
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: keycloak
  namespace: demo-keycloak
spec:
  endpoints:
    - interval: 5s
      path: /metrics
      port: management
      scheme: https
      tlsConfig:
        insecureSkipVerify: true
  namespaceSelector:
    matchNames:
      - demo-keycloak
  selector:
    matchLabels:
      app: keycloak
EOF

oc apply -f ${BASE_DIR}/data/install/keycloak-monitor.yaml -n demo-keycloak

check the metric on ocp web console

By login to the ocp web console, we can check the metric on the dashboard, using jboss_infinispan_number_of_entries_in_memory as metrics, we can see internal statics of keycloak/infinispan.

create testing data in keycloak

We will create some (50k) users under a new realm in Keycloak to test the performance of keycloak.

create keycloak-tool container image

We need a pod to run testing scripts, and the script will call keycloak API to perform operations. So we will create a dedicated container image for the pod.

We will use redhat official container image for keycloak. You can find the image on Red Hat Container Catalog:

https://catalog.redhat.com/software/containers/rhbk/keycloak-rhel9/64f0add883a29ec473d40906?container-tabs=dockerfile

# as root

mkdir -p ${BASE_DIR}/data/keycloak.tool
cd ${BASE_DIR}/data/keycloak.tool

cat << 'EOF' > bashrc
alias ls='ls --color=auto'
export PATH=/opt/keycloak/bin:$PATH
EOF


cat << EOF > Dockerfile
FROM registry.redhat.io/ubi9/ubi AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs  --releasever 9 --setopt install_weak_deps=false --nodocs -y /usr/bin/ps bash-completion coreutils /usr/bin/curl jq python3 /usr/bin/tar /usr/bin/sha256sum vim nano && \
    dnf --installroot /mnt/rootfs clean all && \
    rpm --root /mnt/rootfs -e --nodeps setup

FROM registry.redhat.io/rhbk/keycloak-rhel9:26.0-11
COPY --from=ubi-micro-build /mnt/rootfs /
COPY bashrc /opt/keycloak/.bashrc
EOF

podman build -t quay.io/wangzheng422/qimgs:keycloak-26.tool-2025-03-21-v01 .

podman push quay.io/wangzheng422/qimgs:keycloak-26.tool-2025-03-21-v01

podman run -it --entrypoint /bin/bash quay.io/wangzheng422/qimgs:keycloak-26.tool-2025-03-21-v01

deploy keycloak tool on ocp


oc delete -n demo-keycloak -f ${BASE_DIR}/data/install/keycloak.tool.yaml

cat << EOF > ${BASE_DIR}/data/install/keycloak.tool.yaml
apiVersion: v1
kind: Pod
metadata:
  name: keycloak-tool
spec:
  containers:
  - name: keycloak-tool-container
    image: quay.io/wangzheng422/qimgs:keycloak-26.tool-2025-03-21-v01
    command: ["tail", "-f", "/dev/null"]
EOF

oc apply -f ${BASE_DIR}/data/install/keycloak.tool.yaml -n demo-keycloak

# start the shell
oc exec -it keycloak-tool -n demo-keycloak -- bash

# copy something out
oc cp -n demo-keycloak keycloak-tool:/opt/keycloak/metrics ./metrics

init demo users

First, we need to create 50k user in keycloak.

Lets do it by using keycloak admin cli.

This is the script framework, and we’ve written the entire process here. However, we are only creating realms. The script for creating users only demonstrates the creation logic; there are dedicated chapters later that will use more techniques to create users.


ADMIN_PWD='0xxxxxxxxxxxxxxxxxxxxxxxxxxe'

# after enable http in keycloak, you can use http endpoint
# it is better to set session timeout for admin for 1 day :)
kcadm.sh config credentials --server http://example-kc-service:8080/ --realm master --user temp-admin --password $ADMIN_PWD

# create a realm
kcadm.sh create realms -s realm=performance -s enabled=true

# Set SSO Session Max and SSO Session Idle to 1 day (1440 minutes)
kcadm.sh update realms/performance -s 'ssoSessionMaxLifespan=86400' -s 'ssoSessionIdleTimeout=86400'

# delete the realm
kcadm.sh delete realms/performance

# create a client
kcadm.sh create clients -r performance -s clientId=performance -s enabled=true -s 'directAccessGrantsEnabled=true'

# delete the client
CLIENT_ID=$(kcadm.sh get clients -r performance -q clientId=performance | jq -r '.[0].id')
if [ -n "$CLIENT_ID" ]; then
  echo "Deleting client performance"
  kcadm.sh delete clients/$CLIENT_ID -r performance
else
  echo "Client performance not found"
fi

# create 50k user, from user-00001 to user-50000, and set password for each user
for i in {1..50000}; do
  echo "Creating user user-$(printf "%05d" $i)"
  kcadm.sh create users -r performance -s username=user-$(printf "%05d" $i) -s enabled=true -s email=user-$(printf "%05d" $i)@wzhlab.top -s firstName=First-$(printf "%05d" $i) -s lastName=Last-$(printf "%05d" $i)
  kcadm.sh set-password -r performance --username user-$(printf "%05d" $i) --new-password password
done

# Delete users
for i in {1..50000}; do
  USER_ID=$(kcadm.sh get users -r performance -q username=user-$(printf "%05d" $i) | jq -r '.[0].id')
  if [ -n "$USER_ID" ]; then
    echo "Deleting user user-$(printf "%05d" $i)"
    kcadm.sh delete users/$USER_ID -r performance
  else
    echo "User user-$(printf "%05d" $i) not found"
  fi
done

create user using job

Now we try to create users using jobs.

First, we create a service account, and assign priviledge to it, so it can run the script in keycloak-tool.


oc delete -n demo-keycloak -f ${BASE_DIR}/data/install/keycloak-script-create-users.yaml

cat << EOF > ${BASE_DIR}/data/install/keycloak-script-sa.yaml
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: keycloak-sa
  namespace: demo-keycloak
---
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
  name: keycloak-scc
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: false
allowedCapabilities: []
defaultAddCapabilities: []
fsGroup:
  type: RunAsAny
groups: []
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities: []
runAsUser:
  type: MustRunAs
  uid: 1000
seLinuxContext:
  type: RunAsAny
seccompProfiles:
- '*'
supplementalGroups:
  type: RunAsAny
users:
- system:serviceaccount:demo-keycloak:keycloak-sa
volumes:
- configMap
- emptyDir
- projected
- secret
- downwardAPI
EOF

oc apply -f ${BASE_DIR}/data/install/keycloak-script-sa.yaml -n demo-keycloak

oc adm policy add-scc-to-user keycloak-scc -z keycloak-sa -n demo-keycloak

create user use multiple jobs


TOTAL_USERS=50000
NUM_JOBS=10
USERS_PER_JOB=$((TOTAL_USERS / NUM_JOBS))

for job_id in $(seq 1 $NUM_JOBS); do
  START_USER=$(( (job_id - 1) * USERS_PER_JOB + 1 ))
  END_USER=$(( job_id * USERS_PER_JOB ))

  cat << EOF > ${BASE_DIR}/data/install/keycloak-script-create-users-${job_id}.yaml

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: keycloak-script-config-${job_id}
data:
  create-users.sh: |
    kcadm.sh config credentials --server http://example-kc-service:8080/ --realm master --user temp-admin --password $ADMIN_PWD 

    for i in {$START_USER..$END_USER}; do
      echo "Creating user user-\$(printf "%05d" \$i)"
      kcadm.sh create users -r performance -s username=user-\$(printf "%05d" \$i) -s enabled=true -s email=user-\$(printf "%05d" \$i)@wzhlab.top -s firstName=First-\$(printf "%05d" \$i) -s lastName=Last-\$(printf "%05d" \$i)
      kcadm.sh set-password -r performance --username user-\$(printf "%05d" \$i) --new-password password
    done

---
apiVersion: batch/v1
kind: Job
metadata:
  name: keycloak-create-users-job-${job_id}
spec:
  template:
    spec:
      serviceAccountName: keycloak-sa
      containers:
      - name: keycloak-tool
        image: quay.io/wangzheng422/qimgs:keycloak-26.tool-2025-03-21-v01
        command: ["/bin/bash", "-c"]
        args: ["source /opt/keycloak/.bashrc && cp /scripts/create-users.sh /tmp/create-users.sh && chmod +x /tmp/create-users.sh && bash /tmp/create-users.sh"]
        securityContext:
          runAsUser: 1000
        volumeMounts:
        - name: script-volume
          mountPath: /scripts
      restartPolicy: Never
      volumes:
      - name: script-volume
        configMap:
          name: keycloak-script-config-${job_id}
  backoffLimit: 4
EOF
  oc delete -n demo-keycloak -f ${BASE_DIR}/data/install/keycloak-script-create-users-${job_id}.yaml
  oc apply -f ${BASE_DIR}/data/install/keycloak-script-create-users-${job_id}.yaml -n demo-keycloak
done


# if you want to remove the jobs
for job_id in {1..10}; do
  oc delete -n demo-keycloak -f ${BASE_DIR}/data/install/keycloak-script-create-users-${job_id}.yaml
done

If everything ok, you can see those users in web console.

test performance using curl

Now, we have 50k user in keycloak. Lets test the performance of keycloak.

Get the client secert from keycloak’s web console.

And test the login rest api using curl.

# test the performance of keycloak, by login with each user
CLIENT_SECRET="hxxxxxxxxxxxxxxxxxxxxq"

curl -X POST 'http://example-kc-service:8080/realms/performance/protocol/openid-connect/token' \
-H "Content-Type: application/x-www-form-urlencoded" \
-d "client_id=performance" \
-d "client_secret=$CLIENT_SECRET" \
-d "username=user-00001" \
-d "password=password" \
-d "grant_type=password" | jq .

# {
#   "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJick9pa2tPX3l2dmtoVzlLc05zTEVUMWctSWhfZ0g2WExZZnE5U1ZfeXZFIn0.eyJleHAiOjE3MjgyMjY5NTgsImlhdCI6MTcyODIyNjY1OCwianRpIjoiMzQ5ZGZjZTctNzY1Zi00Yjc0LTgyNjMtMzlmZmQ2NDA3ZjYwIiwiaXNzIjoiaHR0cHM6Ly9rZXljbG9hay1kZW1vLWtleWNsb2FrLmFwcHMuZGVtby0wMS1yaHN5cy53emhsYWIudG9wL3JlYWxtcy9wZXJmb3JtYW5jZSIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiIxZWMxMmRhZC0wMWMwLTQ5N2YtOTkzMS0xZjIyMGJiMmI5OTMiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJwZXJmb3JtYW5jZSIsInNlc3Npb25fc3RhdGUiOiIyOWQzYTUyZC0zNjExLTQ4YzktOWM5MC0yOTE2YmMxY2Q2ODciLCJhY3IiOiIxIiwicmVhbG1fYWNjZXNzIjp7InJvbGVzIjpbImRlZmF1bHQtcm9sZXMtcGVyZm9ybWFuY2UiLCJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiYWNjb3VudCI6eyJyb2xlcyI6WyJtYW5hZ2UtYWNjb3VudCIsIm1hbmFnZS1hY2NvdW50LWxpbmtzIiwidmlldy1wcm9maWxlIl19fSwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwic2lkIjoiMjlkM2E1MmQtMzYxMS00OGM5LTljOTAtMjkxNmJjMWNkNjg3IiwiZW1haWxfdmVyaWZpZWQiOmZhbHNlLCJuYW1lIjoiRmlyc3QtMDAwMDEgTGFzdC0wMDAwMSIsInByZWZlcnJlZF91c2VybmFtZSI6InVzZXItMDAwMDEiLCJnaXZlbl9uYW1lIjoiRmlyc3QtMDAwMDEiLCJmYW1pbHlfbmFtZSI6Ikxhc3QtMDAwMDEiLCJlbWFpbCI6InVzZXItMDAwMDFAd3pobGFiLnRvcCJ9.ioqCjbSuolrhGDPW8SF_Ls0NTOn9mJM8QO7btRo7N24lLZrNaKNrv7R5Mvcs4Bu5xDuB5KHEDh-IU-c3iT8TRK8hc5DHhWYwe7_WICp_O7DQEVIP-9wgeqSY4qmdwBkXvwYN0q8AIOjRwYOYqTP6rLcWiPEhdWDqkCL-S9tyhYBwRt44-k455zi1JOFSBd_vWVXp68TJ5b8TWResz3L-cT02Fk0y9_RZBXang1I3tZUOqpHBCVBhRlDwAvst2QtE3tG-rnIXBR4l1vVn1TXlfoRiDwXE5ski9B1KhHuRNZEqbPdkFpWIfb01h9qwtygv4yNKJEW_knw5t_7iaOwRhA",
#   "expires_in": 300,
#   "refresh_expires_in": 86400,
#   "refresh_token": "eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICI2YWY4Mjc5Mi02NmQ3LTQ0OWItODI4MS0wY2M0NWU4ZjU0ZTkifQ.eyJleHAiOjE3MjgzMTMwNTgsImlhdCI6MTcyODIyNjY1OCwianRpIjoiYjczYWYxODktZDQzZi00MjZiLWJhZGYtNjc0NTI3MGIzZWIzIiwiaXNzIjoiaHR0cHM6Ly9rZXljbG9hay1kZW1vLWtleWNsb2FrLmFwcHMuZGVtby0wMS1yaHN5cy53emhsYWIudG9wL3JlYWxtcy9wZXJmb3JtYW5jZSIsImF1ZCI6Imh0dHBzOi8va2V5Y2xvYWstZGVtby1rZXljbG9hay5hcHBzLmRlbW8tMDEtcmhzeXMud3pobGFiLnRvcC9yZWFsbXMvcGVyZm9ybWFuY2UiLCJzdWIiOiIxZWMxMmRhZC0wMWMwLTQ5N2YtOTkzMS0xZjIyMGJiMmI5OTMiLCJ0eXAiOiJSZWZyZXNoIiwiYXpwIjoicGVyZm9ybWFuY2UiLCJzZXNzaW9uX3N0YXRlIjoiMjlkM2E1MmQtMzYxMS00OGM5LTljOTAtMjkxNmJjMWNkNjg3Iiwic2NvcGUiOiJlbWFpbCBwcm9maWxlIiwic2lkIjoiMjlkM2E1MmQtMzYxMS00OGM5LTljOTAtMjkxNmJjMWNkNjg3In0.un_vmkLIo8elfXAwrgYAnCd6xMHtPkER1j7xuxaDn_lbdmFJSBYJld4YdB6Rxezv7auOmEdd9y1GiFGd3SOGUw",
#   "token_type": "Bearer",
#   "not-before-policy": 0,
#   "session_state": "29d3a52d-3611-48c9-9c90-2916bc1cd687",
#   "scope": "email profile"
# }

We can see it is ok to call rest api to login, so we can carry out out testing by using the login rest api.

test with python job

We also need a benchmark tool to test the rhsso, we can use python to write a simple script to do this. The python has a build-in prometheus client, so we can monitor the python pod with prometheus.

Make changes to performance_test_keycloak.py, espeically the client secert, then copy file performance_test_keycloak.py from ./files to bastation. And create jobs using the python script.

URL='http://example-kc-service:8080/realms/performance/protocol/openid-connect/token'

VAR_PROJECT=demo-keycloak

# copy file performance_test_keycloak.py from ./files
oc delete -n demo-keycloak configmap performance-test-script
oc create configmap performance-test-script -n demo-keycloak --from-file=${BASE_DIR}/data/install/performance_test_keycloak.py


oc delete -n $VAR_PROJECT -f ${BASE_DIR}/data/install/performance-test-deployment.yaml

cat << EOF > ${BASE_DIR}/data/install/performance-test-deployment.yaml
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: performance-test-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: performance-test
  template:
    metadata:
      labels:
        app: performance-test
    spec:
      containers:
      - name: performance-test
        image: quay.io/wangzheng422/qimgs:rocky9-test-2024.10.14.v01 
        command: ["/usr/bin/python3", "/scripts/performance_test_keycloak.py"]
        env:
        - name: CLIENT_SECRET
          value: $CLIENT_SECRET  # Replace this with your actual client key here.
        - name: NUM_USERS
          value: "50000"
        - name: NUM_THREADS
          value: "100"
        volumeMounts:
        - name: script-volume
          mountPath: /scripts
      restartPolicy: Always
      volumes:
      - name: script-volume
        configMap:
          name: performance-test-script
---
apiVersion: v1
kind: Service
metadata:
  name: performance-test-service
  labels:
    app: performance-test
spec:
  selector:
    app: performance-test
  ports:
    - name: http
      protocol: TCP
      port: 8000
      targetPort: 8000
EOF


oc apply -f ${BASE_DIR}/data/install/performance-test-deployment.yaml -n $VAR_PROJECT

monitor metrics using ocp monitoring

Our performance testing script has promethus build-in, so we can monitor the performance test result using openshift monitoring subsystem.

VAR_PROJECT=demo-keycloak

oc delete -n $VAR_PROJECT -f ${BASE_DIR}/data/install/performance-monitor.yaml

cat << EOF > ${BASE_DIR}/data/install/performance-monitor.yaml
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: performance-test
  namespace: $VAR_PROJECT
spec:
  endpoints:
    - interval: 5s
      path: /metrics
      port: http
      scheme: http
  namespaceSelector:
    matchNames:
      - $VAR_PROJECT
  selector:
    matchLabels:
      app: performance-test
EOF

oc apply -f ${BASE_DIR}/data/install/performance-monitor.yaml -n $VAR_PROJECT

You can checkout the metrics by search the name begins with wzh, like wzh_avg_time_sec and wzh_success_rate_sec

patch to the keycloak instance config using 2 owner 2 instance.

https://www.keycloak.org/server/all-config
https://www.keycloak.org/server/caching

Update cache-ispn.xml with the following content, I update the memory max-count to 50kx100 = 5000k = 5m, which is 10 times the target user number, and update lifespan to 1 day, which is based on use case, also enable statistics, and set owners to 2.


cat << EOF >  ${BASE_DIR}/data/install/keycloak.cache-ispn.xml
<infinispan
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="urn:infinispan:config:15.0 http://www.infinispan.org/schemas/infinispan-config-15.0.xsd"
        xmlns="urn:infinispan:config:15.0">

    <cache-container name="keycloak" statistics="true">
        <transport lock-timeout="60000" stack="udp"/>
        <metrics names-as-tags="true" />
        <local-cache name="realms" simple-cache="true" statistics="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="10000"/>
        </local-cache>
        <local-cache name="users" simple-cache="true" statistics="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="5000000"/>
        </local-cache>
        <distributed-cache name="sessions" owners="2" statistics="true">
            <expiration lifespan="86400"/>
        </distributed-cache>
        <distributed-cache name="authenticationSessions" owners="2" statistics="true">
            <expiration lifespan="86400"/>
            <memory max-count="5000000"/>
        </distributed-cache>
        <distributed-cache name="offlineSessions" owners="2" statistics="true">
            <expiration lifespan="86400"/>
            <memory max-count="500000"/>
        </distributed-cache>
        <distributed-cache name="clientSessions" owners="2" statistics="true">
            <expiration lifespan="86400"/>
            <memory max-count="500000"/>
        </distributed-cache>
        <distributed-cache name="offlineClientSessions" owners="2" statistics="true">
            <expiration lifespan="86400"/>
            <memory max-count="5000000"/>
        </distributed-cache>
        <distributed-cache name="loginFailures" owners="2" statistics="true">
            <expiration lifespan="86400"/>
            <memory max-count="5000000"/>
        </distributed-cache>
        <local-cache name="authorization" simple-cache="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <memory max-count="5000000"/>
        </local-cache>
        <replicated-cache name="work" statistics="true">
            <expiration lifespan="86400"/>
        </replicated-cache>
        <local-cache name="keys" simple-cache="true" statistics="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="3600000"/>
            <memory max-count="5000000"/>
        </local-cache>
        <distributed-cache name="actionTokens" owners="2" statistics="true">
            <encoding>
                <key media-type="application/x-java-object"/>
                <value media-type="application/x-java-object"/>
            </encoding>
            <expiration max-idle="86400" lifespan="86400" interval="300000"/>
            <memory max-count="5000000"/>
        </distributed-cache>
    </cache-container>
</infinispan>
EOF

# create configmap
oc delete configmap keycloak-cache-ispn -n demo-keycloak
oc create configmap keycloak-cache-ispn --from-file=${BASE_DIR}/data/install/keycloak.cache-ispn.xml -n demo-keycloak

Then, we update the config of the keycloak instance to enable HTTP and configure the cache, enable metrics.

apiVersion: k8s.keycloak.org/v2alpha1
kind: Keycloak
metadata:
  name: example-kc
  namespace: demo-keycloak
spec:
  ......
  http:
    httpEnabled: true
  cache:
    configMapFile:
      key: keycloak.cache-ispn.xml
      name: keycloak-cache-ispn
  db:
    poolMaxSize: 100
  resources:
    requests:
      memory: "2Gi"
    limits:
      memory: "64Gi"
  additionalOptions:
    - name: metrics-enabled
      value: 'true'
    # - name: cache-metrics-histograms-enabled
    #   value: 'true'
    - name: features-disabled
      value: 'persistent-user-sessions'
    - name: log-console-level
      value: 'error'
    # - name: log-level
    #   value: debug
  instances: 2
  ......

It seems the config is enabled using env. Here is part of the keycload pod yaml file:

kind: Pod
apiVersion: v1
metadata:
  name: example-kc-0
  namespace: demo-keycloak
  ......
spec:
  ......
  containers:
    - name: keycloak
      ......
      env:
        - name: KC_HOSTNAME
          value: example-kc-demo-keycloak.apps.cluster-r9m7r.r9m7r.sandbox2453.opentlc.com
        - name: KC_HTTP_ENABLED
          value: 'true'
        - name: KC_HTTP_PORT
          value: '8080'
        - name: KC_HTTPS_PORT
          value: '8443'
        - name: KC_HTTPS_CERTIFICATE_FILE
          value: /mnt/certificates/tls.crt
        - name: KC_HTTPS_CERTIFICATE_KEY_FILE
          value: /mnt/certificates/tls.key
        - name: KC_DB
          value: postgres
        - name: KC_DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: keycloak-db-secret
              key: username
        - name: KC_DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: keycloak-db-secret
              key: password
        - name: KC_DB_URL_HOST
          value: postgres-db
        - name: KC_DB_POOL_MAX_SIZE
          value: '100'
        - name: KC_CACHE_CONFIG_FILE
          value: cache/keycloak.cache-ispn.xml
        - name: KC_PROXY_HEADERS
          value: xforwarded
        - name: KC_BOOTSTRAP_ADMIN_USERNAME
          valueFrom:
            secretKeyRef:
              name: example-kc-initial-admin
              key: username
        - name: KC_BOOTSTRAP_ADMIN_PASSWORD
          valueFrom:
            secretKeyRef:
              name: example-kc-initial-admin
              key: password
        - name: KC_HEALTH_ENABLED
          value: 'true'
        - name: KC_CACHE
          value: ispn
        - name: KC_CACHE_STACK
          value: kubernetes
        - name: KC_TRUSTSTORE_PATHS
          value: '/var/run/secrets/kubernetes.io/serviceaccount/ca.crt,/var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt'
        - name: KC_TRACING_SERVICE_NAME
          value: example-kc
        - name: KC_TRACING_RESOURCE_ATTRIBUTES
          value: k8s.namespace.name=demo-keycloak
  ......

monitoring and reporting

We will use 2 keycloak instances, and 2 owner as baseline. We will use 100 testing pod, which is 100x10=1000 simulated users.

wzh_success_rate_sec
wzh_avg_time_sec
sum(rate(wzh_success_count_total[5m]))
sum by (state) (jvm_threads_states_threads)
rate(base_gc_time_total[5m])
sum by (cache) (rate(vendor_statistics_miss_primary_owner_total[5m]))
avg by (uri) (rate(http_server_requests_seconds_sum[1m]))
http_server_active_requests
jvm_threads_peak_threads
sum by (cache) (vendor_statistics_approximate_entries_in_memory)
sum by (cache) (vendor_statistics_approximate_entries)
vendor_statistics_approximate_entries
vendor_statistics_number_of_entries
sum by (cache) (vendor_statistics_number_of_entries)

keycloak’s cpu and memory usage looks like cpu: 24cpu, and memory: 16GB (with session persistence enabled)

And, it seems there is a increase for http request times, and the reason maybe

user session is storged in database (persistent)

If without user session storage in database, it will increase to 30cpu, 50GB

And it seems no increase for http request time.

Another interseting thing is that the number of logins/second, without user session persistence, seems to be much faster than with user session persistence, from ~400 to ~1100 logins/second.

The keycloak response time is relative stable at 200ms, and success rate is 100%.

conclusion

Based on the performance test results, the Keycloak system demonstrates stable response times and a 100% success rate. These results suggest that the Keycloak system is effectively handling requests and maintaining high performance levels. Additional testing and optimization may be required to ensure consistent performance across different use cases.