Improvements to MixMode's PQL: Packetsled Query Language

Packetsled Query Language (PQL) addresses the need of analysts to ask questions from large amounts of data without being forced to learn a complicated query language. It is a simple, intuitive expression language that lets users focus on finding what they need quickly.

Recently we have built upon PQL to include more powerful constructs, such as aggregations, data shaping, streaming results, and bulk query execution.

One of the primary strengths of PQL is it’s ability to understand simple date language, so analysts do not have to continually reach for a date selection widget in the user interface, or convert dates to epoch seconds or rigid formats. The following are examples of date range queries that PQL can understand:

yesterday between 2pm and 5pm http and geo=ni
last wednesday after 5pm /.*mixmode.*/
last 10 days | count by @day, dest_port
from "2019-09-01T07:00:00.000Z" to "2019-09-02T07:00:00.000Z"

A new introduction to PQL is the ability to perform aggregate queries, permitting analysts to summarize billions of data points in seconds.

For example, to find out the top senders in your network:

today | countby src_ip

In a multi-tenancy environment, you may wish to see total counts by tenant:

today | countby tenantName, sensorName

If you wanted to see these by hour (or day, week, etc…):

today | countby tenantName, sensorName, @hour

The MixMode api supports a streaming query capability that enables clients to extract large amounts of data efficiently, for example sending to a SIEM or other data system.

In an automation scenario, connecting data from the MixMode platform to an external system is as simple as:

curl -G -ks -X GET -H "x-mixmode-api-token: ${MIXMODE_TOKEN}" \
--data-urlencode "query=today severity>5" \
--data-urlencode "timezone=America/Los_Angeles" \
--data-urlencode "searchType=O_INDICATORS" \
"https://${MIXMODE_ENDPOINT}/v1/search/stream" > results.json

which will stream a custom query result to a local file that can be sent to any system that accepts JSON.

Another recent improvement made to PQL is the ability for PQL query authors to specify directly in their queries which attributes to return in the response. This can result in significant efficiency gains when extracting large amounts of data.

For example, to extract only the md5 attribute from a weeks worth of bro sensor logs, which can total several hundreds of gigabytes or more, one can execute:

last 7 days md5 like /.+/ | include md5

Or it’s streaming api equivalent

curl -G -ks -X GET -H "x-mixmode-api-token: ${MIXMODE_TOKEN}" \
--data-urlencode "query=last 7 days md5 like /.+/ | include md5" \
--data-urlencode "timezone=America/Los_Angeles" \
--data-urlencode "searchType=O_INDICATORS" \
"https://${MIXMODE_ENDPOINT}/v1/search/stream"

And the resulting JSON is typically less than a few Kilobytes in size.

Packetsled Query Language makes it simple to ask questions and get answers.