Performance tuning

Make a Flux task move more data in less time: parallel readers, queue and block sizes per node, scan and checksum settings that cost time, and the metric log fields that show where a transfer waits.

What you will do: measure a task, turn on parallel readers where the data is large or fast, and size the queue and blocks from what the metrics show.

Each node uses a single, non-parallel sync flow by default. For large data sets or high-paced streams a node can run several readers in parallel, with a bounded queue between the agent and the repository. Tuning is per <SyncLogsNode>; measurement is in the two metric logs every Flux node writes.

Parameters

Attribute on `SyncLogsNode`	Meaning	Default
`multiReaders`	use a parallel sync flow	`false`
`numberOfReaders`	how many parallel readers	`-1`
`maxQueueSize`	bytes allowed in the queue at once	`52428800` (50 MB)
`dataBlocksSize`	block size for transfer	`1048576` (1 MB)
`readDataThroughput`	present in the wiki example (`80000000`); no documented default	—

<SyncLogsNode repositoryDirectory="data-server-1" multiReaders="true" numberOfReaders="10"
              maxQueueSize="5000000000" dataBlocksSize="40000000" readDataThroughput="80000000">

Other settings that cost or save time:

Checksum on large files affects performance; lengthen interval or scope it to the nodes that need it; see Security and integrity.
Scanning: subdirsScanLevel, directoriesToInclude / Exclude, filesToInclude / Exclude and timeInterval limit what each run looks at.
Repository growth: timeToKeep on the repository bounds the data kept.
Topology: a proxy instance adds a hop; place proxies close to their agents.

Measure first

metricinfo.log (JSON, one record per metric, with task.id, task.name, job.id, nodes.source.name) is the instrument. Ingest both metric logs into XPLG and put them on a dashboard.

Metric	Side	Tells you
`process.time`	hub and agent	full duration of the sync task — the number to improve
`process.wait.time`	hub and agent	time waiting in queue before execution; high values mean backpressure
`system.queue.cpu.waitingtime`	agent	time waited on the internal queue
`system.network.bytes.read`, `system.network.bytes.write`	hub	bytes received and sent per session
`system.network.read.cpu.time`, `system.network.write.cpu.time`	hub	time waiting for and writing packets
`system.io.bytes.read`, `system.io.read.cpu.time`	agent	bytes and time per block read from disk
`system.io.bytes.write`, `system.io.write.cpu.time`	hub	bytes and time per block written; the sum of `system.io.bytes.write` is the volume stored

metricevent.log marks event.data-sync.start, event.data-sync.end and the state of every file (NEW, NOT_CHANGE, UPDATE, DELETE, FAIL, CHECKSUM, COMPLETED).

Steps

Run the task as is and record process.time per node and the share of process.wait.time.
Where wait time dominates, set multiReaders="true" and a numberOfReaders in line with the agent's cores and disk.
Where network write time dominates, raise dataBlocksSize; where the agent's read time dominates, leave blocks and raise maxQueueSize so readers are not blocked.
Re-run and compare process.time; stop when the hub's system.io.write.cpu.time becomes the limit — that is the repository disk.
Tighten the filters so unchanged directories are not rescanned every run.

What is not published

The throughput figures quoted for Flux on xplg.com are pending confirmation from XPLG engineering and are not repeated here; use your own process.time and bytes written as the baseline.

Next: Security and integrity

Source: XPLG wiki, pages 2198994949, 2353528833, 2199191556, 1594131316 (https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/2198994949).