The osquery daemon uses a default filesystem logger plugin. Like the config, output from the filesystem plugin is written as JSON. Results from the query schedule are written to /var/log/osquery/osqueryd.results.log.

There are two types of logs:

  • Status logs (info, warning, error, and fatal)
  • Query schedule results logs, including logs from snapshot queries

If you run osqueryd in a verbose mode then peek at /var/log/osquery/:

$ ls -l /var/log/osquery/
total 24
lrwxr-xr-x   1 root  wheel    77 Sep 30 17:37 osqueryd.INFO -> osqueryd.INFO.20140930
-rw-------   1 root  wheel  1226 Sep 30 17:37 osqueryd.INFO.20140930
-rw-------   1 root  wheel   388 Sep 30 17:37 osqueryd.results.log

On Windows this directory defaults to C:\ProgramData\osquery\log.

Logger plugins

osquery includes logger plugins that support configurable logging to a variety of interfaces. The built in logger plugins are filesystem (default), tls and syslog. Multiple logger plugins may be used simultaneously, effectively copying logs to each interface. To enable multiple loggers set the --logger_plugin option to a comma separated list of the requested plugins.

For information on configuring logger plugins, see logging/results flags. Developing new logger plugins is explored in the development docs.

Status logs

Status logs are generated by the Glog logging framework. The default filesystem logger plugin writes these logs to disk the same way Glog would. Logger plugins may intercept these status logs and write them to system or otherwise.

As the above directory listing reveals, osqueryd.INFO is a symlink to the most recent execution's INFO log. The same is true for the WARNING, ERROR and FATAL logs. For more information on the format of Glog logs, please refer to the Glog documentation.

Results logs

Differential logs

The results of your scheduled queries are logged to the "results log". These are differential changes between the last (most recent) query execution and the current execution. Each log line is a JSON string that indicates what data has been added/removed by which query. The first time the query is executed (there is no "last" run), the last run is treated as having null results, so the differential consists entirely of log lines with the added indication. There are two format options, single, or event, and batched. Some queries do not make sense to log "removed" events like:

SELECT i.*, p.resident_size, p.user_time, p.system_time, t.minutes AS c
  FROM osquery_info i, processes p, time t
  WHERE p.pid = i.pid;

By adding an outer join of time and using time.minutes as a counter this query will always log a single "added" and a single "removed" line. The purpose is to create a continuous monitor of osquery's performance. For these cases add a "removed": false to the scheduled query.

{
  "schedule": {
    "osquery_monitor": {
      "query": "SELECT ... t.minutes AS c FROM time t WHERE ...",
      "interval": 60,
      "removed": false
    }
  }
}

Snapshot logs

Snapshot logs are an alternate form of query result logging. A snapshot is an 'exact point in time' set of results, no differentials. If you always want a list of mounts, not the added and removed mounts, use a snapshot. In the mounts case, where differential results are seldom emitted (assuming hosts do not often mount and unmount), a complete snapshot will log after every query execution. This will be a lot of data amortized across your fleet.

Data snapshots may generate a large amount of output. For log collection safety, output is written to a dedicated sink. The filesystem logger plugin writes snapshot results to /var/log/osquery/osqueryd.snapshots.log.

To schedule a snapshot query, use:

{
  "schedule": {
    "mounts": {
      "query": "SELECT * FROM mounts;",
      "interval": 3600,
      "snapshot": true
    }
  }
}

Schedule results

Event format

Event is the default result format. Each log line represents a state change. This format works best for log aggregation systems like Logstash or Splunk.

Example output of SELECT name, path, pid FROM processes; (whitespace added for readability):

{
  "action": "added",
  "columns": {
    "name": "osqueryd",
    "path": "/usr/local/bin/osqueryd",
    "pid": "97830"
  },
  "name": "processes",
  "hostname": "hostname.local",
  "calendarTime": "Tue Sep 30 17:37:30 2014",
  "unixTime": "1412123850",
  "epoch": "314159265"
}
{
  "action": "removed",
  "columns": {
    "name": "osqueryd",
    "path": "/usr/local/bin/osqueryd",
    "pid": "97650"
  },
  "name": "processes",
  "hostname": "hostname.local",
  "calendarTime": "Tue Sep 30 17:37:30 2014",
  "unixTime": "1412123850",
  "epoch": "314159265"
}

This tells us that a binary called "osqueryd" was stopped and a new binary with the same name was started (note the different pids). The data is generated by keeping a cache of previous query results and only logging when the cache changes. If no new processes are started or stopped, the query won't log any results.

Snapshot format

Snapshot queries attempt to mimic the differential event format, instead of emitting "columns", the snapshot data is stored using "snapshot". An action is included as, you guessed it, "snapshot"!

Consider the following example:

{
  "action": "snapshot",
  "snapshot": [
    {
      "parent": "0",
      "path": "/sbin/launchd",
      "pid": "1"
    },
    {
      "parent": "1",
      "path": "/usr/sbin/syslogd",
      "pid": "51"
    },
    {
      "parent": "1",
      "path": "/usr/libexec/UserEventAgent",
      "pid": "52"
    },
    {
      "parent": "1",
      "path": "/usr/libexec/kextd",
      "pid": "54"
    }
  ],
  "name": "process_snapshot",
  "hostIdentifier": "hostname.local",
  "calendarTime": "Mon May  2 22:27:32 2016 UTC",
  "unixTime": "1462228052",
  "epoch": "314159265"
}

Batch format

If a query identifies multiple state changes, the batched format will include all results in a single log line. If you're programmatically parsing lines and loading them into a backend datastore, this is probably the best solution.

To enable batch log lines, launch osqueryd with the --log_result_events=false argument.

Example output of SELECT name, path, pid FROM processes; (whitespace added for readability):

{
  "diffResults": {
    "added": [
      {
        "name": "osqueryd",
        "path": "/usr/local/bin/osqueryd",
        "pid": "97830"
      }
    ],
    "removed": [
      {
        "name": "osqueryd",
        "path": "/usr/local/bin/osqueryd",
        "pid": "97650"
      }
    ]
  },
  "name": "processes",
  "hostname": "hostname.local",
  "calendarTime": "Tue Sep 30 17:37:30 2014",
  "unixTime": "1412123850",
  "epoch": "314159265"
}

Most of the time the Event format is the most appropriate. The next section in the deployment guide describes log aggregation methods. The aggregation methods describe collecting, searching, and alerting on the results from a query schedule.

Schedule epoch

When differential logs were described above, we mentioned that after the initial execution of a scheduled query, only differential results are logged. While this is very efficient from a size-of-logs perspective, it introduces some challenges. To begin with, if the logs are stored in a log management system of some kind, it becomes difficult or impossible to identify which log results are from the initial run of the query, and which ones are differentials to the initial results. In some situations, this becomes problematic - for example, for some tables like the users table that don't change very often at all and so don't generate differential results very often, one would have to search far into historical logs to find the last results returned by osquery; conversely, for some tables like processes that change frequently, one would have to do a fair amount of logic applying the effects of added and removed rows to reconstruct the current state of running processes.

To aid with this, osquery maintains an epoch marker along with each scheduled query execution, and calculates differentials only if the epoch of the last run matches the current epoch. If it doesn't, then it treats the current execution of the query as an initial run. You can set the epoch marker by starting osquery with the --schedule_epoch= flag or by updating the schedule_epoch flag remotely from a TLS backend. The epoch is transmitted with each log result, so that it is easy to identify which results belong to which execution of the scheduled query.

Unique host identification

If you need a way to uniquely identify hosts embedded into osqueryd's results log, then the --host_identifier flag is what you're looking for. By default, host_identifier is set to "hostname". The host's hostname will be used as the host identifier in results logs. If hostnames are not unique or consistent in your environment, you can launch osqueryd with --host_identifier=uuid.