Clusters and performance

A node does not join or stops reporting, the active processor changes, searches and dashboards are slow, system tasks run long, storage response time alerts, scan directory warnings, X8 quorum problems.

What you will do: tell a configuration problem (a node, a path, a clock) from a capacity problem (volume, storage, memory) and fix the right one.

A node does not join or does not report

  • Shared configuration. Every V7 node mounts the same external configuration directory with full read and write; Use external configuration directory and Cluster Mode are on under Settings > General; each node has a unique -Dxpolog.uid.structure=<NODE> in its lax file.
  • Clock. NTP on every node; a drifted clock makes a node look stale in System Health > Overview (each node reports every five minutes).
  • Version. "Conflict between installation and configuration versions" in System Health means a node missed the last patch; publish it to that node (Upgrades).
  • Allowed masters. cluster.allowedMasters lists the processor nodes; a UI node has cluster.shouldCheckForMaster=false so it never takes over. The active master writes conf/general/cluster/<PROCESSOR>.masterNode.
  • X8. Ignite needs a fixed, odd-numbered quorum of masters and meta-storage; Hazelcast members are listed as alias:port in .env; roles run on host networking, so the seed IPs must be reachable on 30304–30308. verify.sh reports the member list.

The active processor keeps changing

Failover is automatic: when the active processor stops reporting, another takes over and the cluster alerts by email (SMTP must be configured). Repeated changes point at memory pressure or storage latency on the processor, not at the election. Check System Health > Memory and Storage Response Time on that node.

Searches and dashboards are slow

  1. Scope. Narrow the time range and the log, folder or app scope; a search over all logs for a week reads everything hot.
  2. Index. The log is still indexing (Support Portal > Data Information > Indexing); re-index only when the index is reported broken.
  3. Storage. System Health > Storage Response Time above threshold means the data disk is the limit: NVMe or SSD, never RAID 5, local disk for X8 /node/data.
  4. Memory. A heap near its limit spends time in garbage collection; size it per System requirements.
  5. Concurrency. More than a handful of users or heavy dashboards need a UI node per the profile table, behind a load balancer with sticky sessions.
  6. Monitors and Analytics. Many scheduled searches compete with interactive ones; Support Portal > Activity Information > Processes shows what is running and lets an administrator stop a runaway job.

System task alerts

  • "The system task … is running slowly" and "Performance issue on …" come from System Health > System Tasks. Contact support with the System Report if they repeat; one-off alerts during an upgrade or a large archive job are expected.
  • "Scan directory performance issue on directory …": the scanned path is slow or huge; add include / exclude patterns and a subdirectory limit in the scan wizard's advanced settings.

Collection and forwarding backlog

  • collectionstat and portxstat logs (under log/) show per-log and per-forwarder throughput; a forwarder in SUSPENDED or PENDING status is waiting on its destination.
  • A listener node in Agent Mode runs with 2 GB and stores nothing; it must point at a processor for indexing.

Capacity, not configuration

When every check above passes and the symptoms scale with the day's volume, the deployment is at its size: add a processing node per the profile table, move the UI to its own node, or raise the heap. Sizing and capacity planning has the method; sizing above 500 GB/day is planned with support@xplg.com.

Next: License and updates

Source: XPLG wiki, pages 1594132401, 1602584512, 1594132938, 1594132940, 1594132942, 3150544899, 3151659011, 1594132078 (https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/1594132401).