File Integrity Monitoring with osquery
File integrity monitoring (FIM) is available for Linux (in file_events
, using the inotify subsystem, and in process_file_events
using the Audit subsystem), Windows (in ntfs_journal_events
, using NTFS Journaling) and macOS (in file_events
, using FSEvents).
FIM basics in osquery
Collecting file events in osquery requires that you first specify a list of files/directories to monitor from the osquery configuration. The events that relate to those selected files will then populate the corresponding tables on each platform.
FIM is also disabled by default in osquery. To enable it, first ensure that events are enabled in osquery (--disable_events=false
), then ensure that the desired FIM table is enabled with the corresponding CLI flag (--enable_file_events=true
for file_events
, --disable_audit=false
for process_file_events
, --enable_ntfs_event_publisher=true
for ntfs_journal_events
).
To specify which files and directories you wish to monitor, you must use fnmatch-style, or filesystem globbing, patterns to represent the target paths. You may use standard wildcards *
/**
or SQL-style wildcards *%*
, as shown below.
Matching wildcard rules
%
: Match all files and folders for one level.%%
: Match all files and folders recursively.%abc
: Match all within-level ending in "abc".abc%
: Match all within-level starting with "abc".
Matching examples
The three elements of a FIM config in osquery are (a) the scheduled query against file_events
, (b) the added file_paths
section, and (c) the exclude_paths
sections.
The file_events
query is scheduled to collect all of the FIM events that have occurred on any files within the paths specified within file_paths
but excluding the paths specified within exclude_paths
on a five minute interval. At a high level, this means events are buffered within osquery and sent to the configured logger every five minutes. That is, the events are always recorded in real time; the interval is just how often the already recorded events will be logged.
After you identify the files and directories you wish to monitor, add their match rules to the file_paths
section in the osquery FIM config.
/Users/%/Library
: Monitor for changes to every user's Library folder, but not the contents within./Users/%/Library/
: Monitor for changes to files within each Library folder, but not the contents of their subdirectories./Users/%/Library/%
: Same, changes to files within each Library folder./Users/%/Library/%%
: Monitor changes recursively within each Library./bin/%sh
: Monitor thebin
directory for changes ending insh
.
Note: You cannot match recursively inside a path. For example /Users/%%/Configuration.conf
is not a valid wildcard.
Note: Many applications may replace a file instead of editing it in place. If you monitor the file directly, osquery will need to be restarted in order to monitor the replacement. You can avoid this by monitoring the containing directory instead.
Note: Remember to specify home paths as, for instance, /Users/%
, instead of ~/%
which will not work.
Example FIM Config
{
"schedule": {
"crontab": {
"query": "SELECT * FROM crontab;",
"interval": 300
},
"file_events": {
"query": "SELECT * FROM file_events;",
"removed": false,
"interval": 300
}
},
"file_paths": {
"homes": [
"/root/.ssh/%%",
"/home/%/.ssh/%%"
],
"etc": [
"/etc/%%"
],
"tmp": [
"/tmp/%%"
]
},
"exclude_paths": {
"homes": [
"/home/not_to_monitor/.ssh/%%"
],
"tmp": [
"/tmp/too_many_events/"
]
}
}
Do not use arbitrary category names under the exclude_paths
node; only valid names are allowed.
- Valid categories - Categories referenced under the
file_paths
node. In the above example config,homes
,etc
andtmp
are valid categories. - Invalid categories - Any name not referenced under
file_paths
. In the above example, any name besideshomes
,etc
andtmp
is invalid. Invalid categories get dropped silently, i.e., they don't have any effect on the events generated.
In addition to file_paths
, you can use file_paths_query
to specify the file paths to monitor as the path
column of the results of the given query. For example:
{
"file_paths_query": {
"category_name": [
"SELECT DISTINCT '/home/' || username || '/.gitconfig' as path FROM last WHERE username != '' AND username != 'root';"
]
}
}
Sample Event Output
As file changes happen, events will appear in the file_events table. During a file change event, the md5, sha1, and sha256 for the file will be calculated if possible. A sample event looks like this:
{
"action":"ATTRIBUTES_MODIFIED",
"category":"homes",
"md5":"bf3c734e1e161d739d5bf436572c32bf",
"sha1":"9773cf934440b7f121344c253a25ae6eac3e3182",
"sha256":"d0d3bf53d6ae228122136f11414baabcdc3d52a7db9736dd256ad81229c8bfac",
"target_path":"\/root\/.ssh\/authorized_keys",
"time":"1429208712",
"transaction_id":"0"
}
Tuning Linux inotify limits
On Linux, the file_events
table in osquery uses inotify to subscribe to file changes. There are inherently some limitations on the number of files that can be monitored, since each inotify watch takes up a certain amount of memory in kernel space (non-swappable memory). Adjusting your limits accordingly can help increase the file limit, at a cost of kernel memory.
Example sysctl.conf modifications
#/proc/sys/fs/inotify/max_user_watches = 8192
fs.inotify.max_user_watches = 524288
#/proc/sys/fs/inotify/max_user_instances = 128
fs.inotify.max_user_instances = 256
#/proc/sys/fs/inotify/max_queued_events = 16384
fs.inotify.max_queued_events = 32768
File Accesses (Linux only)
In addition to FIM, which generates events if a file is created/modified/deleted, osquery also supports file access monitoring which can generate events if a file is accessed.
Monitoring file accesses on Linux uses inotify and may incur unexpected and unwanted performance overhead. To prevent 'flooding' of access events alongside FIM, enabling access events for file_path
categories is an explicit opt-in. You may add categories that were defined in your file_paths
stanza:
{
"file_paths": {
"homes": [
"/root/.ssh/%%",
"/home/%/.ssh/%%"
],
"etc": [
"/etc/%%"
],
"tmp": [
"/tmp/%%"
]
},
"file_accesses": ["homes", "etc"]
}
The above configuration snippet will enable file integrity monitoring for homes
, etc
, and tmp
but only enable access monitoring for the homes
and etc
directories.
NOTICE: The hashes of files will not be calculated, to avoid generating additional access events.
Troubleshooting FIM
Sometimes, despite a correct osquery configuration, the file events tables don't receive any events.
First, make sure you are launching osquery as root/Administrator.
In some cases, the problem might be interference from additional security permissions settings:
- On CentOS-like systems, check that the SElinux settings are not preventing osquery from performing FIM.
- On Debian-like systems, check that AppArmor is not blocking osquery.
- On macOS, the
osqueryd
agent (orTerminal.app
, if usingosqueryi
) may need Full Disk Access permissions, in Security and Privacy settings.
Also, remember you can run osqueryd
with the --verbose
flag to see if any helpful warnings appear.
If you're attempting to use FIM on Linux via Audit and you see an error like the following, your system is probably configured to use auditd
; unfortunately osquery cannot share access to the Audit subsystem with auditd
. You can use osquery with one of the other FIM sources, or discontinue the use of auditd
.
osquery> I1107 17:49:42.229321 8235 auditdnetlink.cpp:601] Failed to set the netlink owner
Last but not least, see the troubleshooting guidance in process auditing with osquery, which similarly covers the topic of event-based tables in osquery and its use of the Audit subsystem on Linux.
Known Issues
Implementing FIM across all platforms and using multiple sources means that there are a few problematic corner cases. This is not an exhaustive list, but rather, some long-standing issues to be aware of. If you encounter one that is not in our tracked issues, please submit it.
- On some platforms, it may not be possible to monitor a given path, until a file or directory already exists at that path. Issue 3212
- If a watched file is deleted, inotify stops watching that path. Issue 6495
- With inotify, moving a directory of directories into a watched directory does not immediately add all of those subdirectories to the watched set. Issue 1969
- With inotify, you'll get a "modify" event on every occurrence of an open-file-with-write-permission action. Issue 3920
- If you have a directory with an extremely large number of subdirectories, setting a watch on it using inotify will exhaust the available inotify handles and result in receiving no events. Setting an
exclude_path
on the subdirectories will not help here; the workaround is to be more specific with thefile_paths
. Unfortunately, this means not being able to watch for new files/directories getting created in a directory that already has many subdirectories. Issue 4296 - inotify may not track events done via hard links Issue 5704