Sync task reference
Every element and attribute of a Flux sync task: the XML file (repository, checksum, nodes, accounts, directories, proxies, performance and notification tags), the Data Sync console fields and the task actions.
What you will do: write or read a task XML with every option in front of you, and know what each console action does.
A task is an XML file on the hub that the Data Sync console runs on a schedule. The console holds the name, description, the absolute Path to the file and the schedule; everything else is in the XML. Flux walks through a first task; this page is the reference.
Console
Flux > Data > Data Sync: New Task (name, description, path, schedule), then per task Execute (run now), Edit, Duplicate, Delete, Force Log Sync (All) (re-sync files deleted from the repository that still exist at the source, on the next run) and Force Log Sync (Checksum mismatch) (re-sync only files whose checksum differs, on the next run).
Structure
<SyncLogsConfiguration>
<SyncLogsRepository repositoryDirectory="/mnt/FluxRepo/" timeToKeep="30" />
<Checksum interval="30m" algorithm="MD5" mailRecipients="ops@example.com" enabled="true" retryAttempts="3" />
<ProxyRemote> … </ProxyRemote> <!-- optional, global or per node -->
<NotificationsService …> … </NotificationsService> <!-- optional -->
<SyncLogsNode repositoryDirectory="data-server-1" key="…" multiReaders="true" numberOfReaders="10">
<Remote>
<Account classKey="xpolog.eye.media.remote.RemoteXplgAccount" name="data-server-1"/>
</Remote>
<SyncLogsDirectory syncDirectory="/opt/app1/data/" repositoryDirectory="logs"
filesToInclude="*.log,*.zip" filesToExclude="*.gz" subdirsScanLevel="10"
timeInterval="1" timeIntervalUnit="days" markCompletedTime="10000" />
</SyncLogsNode>
</SyncLogsConfiguration>
Elements and attributes
| Element and attribute | Required | Meaning | Values and default |
|---|---|---|---|
SyncLogsRepository repositoryDirectory | yes | root directory of the repository | absolute path |
SyncLogsRepository timeToKeep | no | days to retain synced data | number |
Checksum algorithm | yes when present | checksum algorithm | MD5, SHA-1 |
Checksum enabled | no | activates validation | true / false |
Checksum interval | yes when present | how often checksums are calculated | 30m, 2h, 1d |
Checksum retryAttempts | yes when present | retries before alerting | number |
Checksum mailRecipients | no | alert recipients | semicolon-separated |
SyncLogsNode repositoryDirectory | yes | directory inside the repository for this node | string |
SyncLogsNode key | no | unique node key | string |
SyncLogsNode multiReaders, numberOfReaders, maxQueueSize, dataBlocksSize | no | parallel sync; see Performance tuning | false, -1, 52428800, 1048576 |
Remote > Account classKey | yes | account class | xpolog.eye.media.remote.RemoteXplgAccount |
Remote > Account name | yes | the Remote XpoLog account on the hub (the Flux pages call it the Remote Flux/PortX account) | string, case-sensitive |
ProxyRemote > Account … | no | proxy instances; global outside a node, or inside one node | as above |
SyncLogsDirectory syncDirectory | yes | source directory on the agent | path |
SyncLogsDirectory repositoryDirectory | no | subdirectory inside the node's repository directory | string |
directoriesToInclude, directoriesToExclude | no | directory filters, wildcards (Apps*) or regex; exclusions in lower case | string |
filesToInclude, filesToExclude | no | file filters, wildcards (*.log) or regex | string |
subdirsScanLevel | no | depth of sub-folder scanning | number |
timeInterval, timeIntervalUnit | no | only files updated within the interval | number; years, months, weeks, days, hours, mins |
markCompletedTime | with notifications | milliseconds after which a sync cycle counts as complete | number, for example 10000 |
A <Checksum> can sit at the configuration level or inside a <SyncLogsNode>; checksums on large files cost performance.
Notifications
<NotificationsService failureNumberOfRetries="2" failureRetriesTime="-1" failureNotificationKeepSendTime="100000">
<Notifications supportedTypes="FILE_COMPLETED" sendForEach="true" sendInBackground="false" sendAtEnd="false">
<Tasks>
<xpoTask exePath="cmd /c echo [NOTIFICATION_TIME] [FILE_NOTIFICATION_TYPE] >> log.txt" name="Log Notification" />
<xpoTask MailAccountToId="ops@example.com" MailSubject="Flux Alert" classKey="xpolog.eye.automation.tasks.mail.MailTask" name="Email Notification" />
<xpoTask classKey="xpolog.eye.automation.tasks.restapi.RestApiTask" method="POST" url="https://<xplg-host>/logeye/api/logger.jsp?token=<token>" name="API Notification">
<Param key="data" value="{"a":"b"}"/>
</xpoTask>
</Tasks>
</Notifications>
<Notifications supportedTypes="FILE_READ_FAILURE,FILE_WRITE_FAILURE,TASK_FAILED" sendForEach="true" sendInBackground="false" sendAtEnd="false">
<Tasks> … </Tasks>
</Notifications>
</NotificationsService>
| Attribute | Meaning | Default |
|---|---|---|
NotificationsService failureNumberOfRetries | failures before an alert; -1 disables the check | 3 |
NotificationsService failureRetriesTime | milliseconds from first to last failure, per the retry count, before alerting | -1 |
NotificationsService failureNotificationKeepSendTime | milliseconds to wait before alerting again while the issue persists | -1 (every failure) |
Notifications supportedTypes | FILE_COMPLETED, FILE_DELETE, FILE_READ_FAILURE, FILE_WRITE_FAILURE, TASK_FAILED, comma-separated | FILE_COMPLETED |
Notifications sendForEach | one alert per notification (true) or one global alert | false |
Notifications sendInBackground | send asynchronously | false |
Notifications sendAtEnd | send at the end of the task; the wiki says it is required when sendForEach is true, yet its own example (above) sets it false | true |
xpoTask | a script (exePath), an email (MailTask, MailAccountToId, MailSubject) or a REST call (RestApiTask, method, url, Param) |
Placeholders [NOTIFICATION_TIME] and [FILE_NOTIFICATION_TYPE] are available in script commands. Every SyncLogsDirectory needs markCompletedTime when notifications are used.
Checks
- The account names in the XML match the hub's Remote accounts exactly.
repositoryDirectorypaths exist and have space fortimeToKeepdays of data.- Run once with Execute and read
metricevent.logforevent.data-sync.end; see Performance tuning.
Next: Performance tuning
Source: XPLG wiki, pages 2199191556, 2198994949, 2199224325, 2199093250 (https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/2199191556).
