Parse
How PortX turns raw records into fields: automatic pattern detection, the XPLG Patterns Language, field types (ftypes), templates, and the handlers and mappings that set log path, name and host.
What you will do: parse a log into named columns, give the columns field types your searches and Apps can rely on, and reuse the result as a template.
Parsing happens when data enters PortX, not at search time. A pattern describes the structure of a record as a sequence of typed fields; XPLG applies it to every incoming record and stores the result as columns. Collect explains how data arrives; this page covers what happens to it next. The forwarder-side transformers (JSON, CEF, LEEF, masking, replace) are on Control and route.
How a pattern is chosen
- Automatically — when you save a log in the Add Log wizard, XPLG applies an automated pattern. In the Add Logs Directory wizard the scan method decides how: match existing patterns and templates by file name and content, then generate a pattern for anything unmatched (the default and slowest), or match existing configuration only, or match templates by file name only.
- From a template — a template is a complete configuration set for one log type: pattern, filters and metadata. XPLG ships templates for known systems (the Patterns Knowledgebase); you add your own by right-clicking a log and choosing Save as Template. A template can be matched during the directory wizard by file name alone or by file name plus content validation, and can ignore the file name expression altogether.
- By hand — the Patterns Administration page (the Next button of the Add Log or Edit Log wizard) offers three editors: Wizard (a builder that adds fields and separators one at a time), Manual (the Patterns Language) and Automatic (suggestions, only while adding a log). Deep parsing is available for local, Windows network, SSH and HDFS logs.
A log can carry several patterns (multi-pattern) when it holds records of different shapes, kept as numbered tabs that you can move, duplicate and remove. Always use Verify pattern before saving: a wrong pattern shows wrong data or none.
The Patterns Language
A pattern is the record with each variable part replaced by a tag. Every tag can be named with a colon ({string:Title}), and the name becomes the column header.
| Tag | Parses | Notes |
|---|---|---|
{string} | any text, including multi-line | {string:Message} |
{text} | a single-line string | |
{date,FORMAT} | a date | {date:Date,dd/MMM/yyyy:HH:mm:ss}; ;DISPLAY_FORMAT changes the view, timeDiff= offsets in ms, locale= / localeView= set locales |
{timestamp} | epoch milliseconds | product=1000 multiplies seconds into milliseconds |
{number} | a numeric value | |
{choice,a;b;c} | one of a fixed set | |
{priority,DEBUG;INFO;ERROR} | a severity from a fixed set | drives the priority column |
{ip} | an IP address | |
{geoip,type=country:region:city} | an IP resolved to country, region and city from the bundled database | any combination of the three |
{properties,keysSep==;propSep=,;,k1;k2} | key-value pairs | |
{json,key1;key2->nested} | JSON keys, nested with -> | a key can carry #_#ftype= and #_#name=; a date key takes #_#date::dateFormat=… |
{regexp,refName=Col;multiLine=true,(expr)} | a regular expression applied to another column | extracts a group into a new column; columnType=date or timestamp with dateFormat= for dates |
{map,v1=a;v2=b} | value mapping, inline, from a properties file (file:) or by regex (columnType=map;mapping=) | 404 becomes "Page Not Found" |
{expression,Num2-Num1} | a computed column from other columns | js("scripts/file.js",column.X) calls an external JavaScript function |
{term,TEXT} / {freetext,TEXT} | a constant that must appear / a label to display | used to tell multi-pattern records apart |
{block,start,emptiness=true}…{block,end,emptiness=true} | an optional part | |
{tab}, {eol}, {eoe} | a tab, an end of line inside a record, the end of a record | {eoe} speeds up multi-line parsing |
Attributes every tag accepts: uiMessageLength (wrap long values), charsLength (fixed width), stopPattern (a regex delimiter when there is no natural one) and masker (a regex whose match is shown as asterisks; see Secure). Curly braces in the data are written \u007B and \u007D; a literal quote is doubled.
Date formats use the usual identifiers (MM, MMM, dd, EEE, yyyy, HH, hh a, mm, ss, SSS, z, Z, XXX, quoted literals). Example, an Apache access log:
{string} - - [{date,dd/MMM/yyyy:HH:mm:ss Z} +0200] "{string}" {number} {number}
Field types (ftypes)
A field type is a semantic label on a column — transactionid, virtualMemory, args — that is the same across vendors and formats, so one search or App works on all of them. Set it in the builder (Field Types), in a JSON tag (data->row->virtualMemory#_#ftype=virtualMemory) or inline ({text:Entity_ID, ftype=transactionid}), then query it as ftype.<name>:
* in logtype.transaction | count | group by ftype.transactionid
The Start and End Event Correlation package on the Marketplace is built on ftypes and Environment Tables for exactly this reason; see Correlation.
Handlers and mappings
For data that arrives through a listener, PortX handlers and mappings turn the raw payload into the log path, log name, host, message, template and related metadata XpoLog expects, and strip listener framing so patterns see a clean payload. In the console this is the listener account's dynamic log creation configuration: Parent Folder, Log Name Prefix, Split by Source Device, the HTTP/S Message JSON Key, Split by JSON Columns and the reserved keys xpologPath, xpologName and xpologType (which applies a matching template), plus Select Log Template on syslog listeners. The per-listener options are on Listeners reference.
File name patterns
Rotated files join one log through a name pattern: log.log{string} captures log.log, log.log.1, …; log-{date,dd-MM-yyyy}.log captures dated files. A name pattern must capture one series only — never more than one file that is still being written — so three series in one directory become three logs. The directory wizard can unite similar names, unite by suffix or capture each file separately.
Checks
- Verify pattern shows every column filled for the sample records, including copied-in edge cases.
- The date column is parsed in the right time zone (log regional settings, or User Time Zone Mode).
- Columns you will filter, forward or correlate on carry an ftype.
- Export the template (Data > Patterns, a zip file) when another deployment needs the same parsing.
Next: Filter and enrich
Source: XPLG wiki, pages 1594131792, 1594132149, 1594132598, 1594133005, 1594133011, 1594133024, 1594133026, 1594131787, 1602191926, 1596391403, 1595637714, 1594131316, 1598685520, 2736095240, 1636827137 (https://xpolog.atlassian.net/wiki/spaces/XPOL/pages/1594131792).
