High availability and clustering

What you will do: choose a cluster shape, wire the nodes so they share one configuration, put an entry point in front of the UI, and know what happens when a node fails.

A cluster is the recommended model for production. It scales by adding nodes or processes, separates user-facing work (UI, queries, dashboards) from back-end work (ingestion, indexing, analytics, monitoring), and keeps running when a node or process fails because every node can take over core tasks and in-flight work is redistributed. Below about 10 GB/day a single process may do; above that, run processor and UI on dedicated instances — see Sizing and capacity planning. The per-host install steps are on Docker (X8) and Virtual machine and bare metal (V7); this page is the cluster layer on top of them.

Topology

Rules that apply to both: every node NTP-synced; one data centre or region per cluster; Windows sources are read from a Windows node only (V7) or from an Agent Mode instance on Windows.

X8: Compose cluster

1. Pick the layout. One server: COMPOSE_PROFILES=master,server. Two servers: master,server on the first, server-additional (processor and processor-1, no ui) on the second. Six servers: one role each (master, ui, listener, processor, processor-1). The same compose files go on every server; only .env differs.
2. Write the shared part of .env once and copy it to every server: the seed map XPLG_SEED_1_IP … XPLG_SEED_6_IP, IGNITE_CMG_NODES and IGNITE_META_STORAGE_NODES (a fixed odd-numbered list of <alias>-<role> names), XPLG_HAZELCAST_MEMBERS (every member as alias:port — a port-less entry leaves that member standalone), XPLG_IGNITE_DISCOVERY_SEEDS and XPLG_JOBS_MASTER_CONN=http://xplg-seed-1:30308/logeye (without it processors never run jobs). Ignite initialisation is not a variable: the master role does it.
3. Set the per-server part: SELF_ALIAS (xplg-seed-1, xplg-seed-2, …; a multi-server cluster must set it or two servers collide on node names and the join is refused) and COMPOSE_PROFILES.
4. Storage. XPLG_BASE_HOST_PATH on local NVMe or SSD on each server; XPLG_SHARED_HOST_PATH on one NFSv4 export mounted rw,hard,noatime,_netdev and exported with anonuid=30303,anongid=0, pre-owned 30303:0 with mode 2775.
5. Preflight and ports on every server: sudo ./xplg-preflight.sh ./.env (exit 0 = READY), then sudo ./xplg-open-ports.sh ./.env --source <cluster cidr>. Ignite discovery (13344–13348) and Hazelcast (11473–11477) must be reachable between all servers; see Networking and ports.
6. Start with docker compose -f docker-compose.yml [-f docker-compose.traefik.yml] up -d on each server and wait one to eight minutes; init-perms runs first and refuses to start the roles if the tree is not writable.
7. Edge. Run Traefik on one server, or on two or three as independent, equal entry points — never chained. Either list the servers per pool (XPLG_TRAEFIK_UI_SERVERS, XPLG_TRAEFIK_MASTER_SERVERS, XPLG_TRAEFIK_LISTENER_SERVERS, XPLG_TRAEFIK_PROCESSOR_SERVERS) or let nodes register themselves: each server writes one <alias>-<role>.node file into conf/registry/, traefik-init re-reads the folder every ten seconds and regenerates conf/dynamic/xplg-routes.yml (never hand-edit it). Point XPLG_TRAEFIK_REGISTRY_PATH at the shared mount so every server's nodes join the pools; the lists and the registry are unioned. Add or remove a node by hand with a one-line .node file (ui xplg-seed-7 10.0.0.77:30304).
8. Verify with ./verify.sh: every role port answers, every Traefik entry answers 200 or 302, and repeated requests through the ui pool return the same sticky cookie.

A second cluster can share the same hosts: shift every port with the {ROLE}_{KIND}_PORT variables, give it its own COMPOSE_PROJECT_NAME and XPLG_BASE_HOST_PATH, and write the shifted ports into its XPLG_IGNITE_DISCOVERY_SEEDS and XPLG_HAZELCAST_MEMBERS so the two clusters never probe each other.

V7: HA cluster on hosts you manage

Preparation: decide Windows or Linux (Windows when Windows machines must be read directly); size the UI and processor machines from the sizing page; NTP everywhere; a shared storage device every node can read and write, with the fastest connectivity to the UI node. The wiki recommends consulting support before building a cluster.

1. Install XPLG on every node with the installer and open http://<node>:30303 on each to confirm it runs.
2. Shared configuration. Create the EXTERNAL_CONFIGURATION_DIRECTORY on the shared storage. On every node open Settings > General, select Use external configuration directory, enter the full path, select Cluster Mode, save — and do not restart yet. On the Mail tab set SMTP and the administrator address: the cluster mails an alert when the active processor changes.
3. Name and size each node. In XpoLog.sh.lax (Linux) or XpoLog.lax (Windows), on the lax.nl.java.option.additional line, set -Xmx per the memory rules and append -Dxpolog.uid.structure=<NODE_NAME> (PROCESSOR1, UI01, …). Restart, processor node first.
4. Cluster properties. On the processor node open http://<processor>:30303/logeye/support, choose Advanced Settings and set: cluster.allowedMasters (the processor node names, comma-separated), htmlUtil.BaseUrl (http://<processor>:<port>/logeye/), htmlUtil.ui.BaseUrl and mail.link.baseUrl (the UI node, or the load balancer when there is one). With support only: cluster.shouldCheckForMaster=false when UI nodes must never take over the processor role, and cluster.takeOverAttempts.notAllowedMaster (minutes, above 5) to let the processor stay down longer before a UI node takes over.
5. Dedicated listener node (optional): give it 2 GB, switch it to Agent Mode, add an empty recycleCronExpression= line to its xpologInit.prop so it is never recycled, restart it, then on each listener account set Listening node to it and Indexing node to the master or a processor.
6. Verify. Settings > General on every node shows the external directory and Cluster Mode active, and EXTERNAL_CONFIGURATION_DIRECTORY/conf/general/cluster/ holds <PROCESSOR_NODE_NAME>.masterNode within two minutes.

Two processes on one machine: install into XpoLog_Center_UI and XpoLog_Center_PROCESSOR, rename server.xml.processor to server.xml in the processor's ServletContainer/conf/ so it listens on 30304 and 30444, then follow the configuration steps above. On Windows give each instance its own service name in the installer (or sc create <name> binpath= "<install dir>\XpoLog.exe -zglaxservice XpoLogCenter" start= auto); on Linux pin each process to a CPU range with cpus= in its runXpoLog.prop.

Load balancing the UI

• X8. The Traefik ui pool on :80 and :443 is sticky and health-checked; :30303 and :30443 reach the master pool, :8304–:8308 the per-role full-tree consoles. Traefik answers 503 until a member is healthy.
• V7. The wiki's recipe is Apache httpd with mod_jk: a workers.properties with worker.balancer.type=lb, balance_workers=ui1,ui2, sticky_session=True and one ajp13 worker per UI node; JkMount /* balancer in httpd.conf; on each UI node the AJP connector port in ServletContainer/conf/server.xml set to the worker's port and jvmRoute="ui1" on the Engine element; restart the UI nodes and httpd; users open the httpd address. On AWS, an Application Load Balancer with a target group on port 30303 does the same.
• A load balancer, proxy or firewall is your infrastructure: XPLG support may ask you to bypass it to isolate a problem — see Support terms.

Failover and continuity

• Active processor (V7). One processor runs the system tasks; when it is down or loses the storage, a UI node takes over after five minutes (unless disabled, see step 4) and System Health raises "processor node has changed to …", then "changed back" when it recovers. Workloads are redistributed and in-flight ingestion, indexing, alerts and reports complete.
• Spot instances (AWS). Processors and UI nodes can run on persistent spot instances with stop-on-interruption and a disk that survives the stop; only the master is a reserved instance, and it can be small (4 CPUs, 8 GB) because it orchestrates and does not process; the external configuration sits on EFS mounted nfs4,hard,_netdev from /etc/fstab.
• Geo-redundancy. A secondary site receives the full configuration through the Inbox/Outbox synchronisation and takes over on a regional outage; raw data is replicated by your own storage tooling. The procedure is on Backup and restore.
• Licensing. One license covers the whole cluster, generated from any node; in dock, high availability is an add-on below the 100 GB/day tier and included from it — see How licensing works.

When a node will not join or the active processor keeps changing, see Clusters and performance.

Next: Networking and ports

Source: XPLG wiki, pages 1602584512, 1602584510, 1594132401, 1594133322, 1683849221, 1594753047, 3161587738, 1599570438, 1594132938, 3150774276, 3150544899, 3151659011, 3180101636, 1594132392, 1598685161 (https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/1602584512). Pending from XPLG engineering: the X8 behaviour when the master server is lost (not documented).