We include a make deps command to make it easier for developers to get started with the osquery project. make deps uses Homebrew for OS X and Linuxbrew for Linux. The following basic dependencies are need before running make deps:

  • sudo
  • make (that is, GNU make)
  • python
  • ruby
  • git
  • bash

WARNING: This will install or build various dependencies on the build host that are not required to "use" osquery, only build osquery binaries and packages.

For our build hosts (CentOS, Ubuntu 12, 14, 16, macOS 10.12, Windows 2016) we use a sysprep target to update the host and install these basic dependencies.

make sysprep

Building on OS X

make deps will take care of installing the appropriate library dependencies, but it's recommended to take a look at the Makefile, just in case something conflicts with your environment.

The complete installation/build steps are as follows:

$ git clone
$ # make sysprep
$ cd osquery
$ make deps
$ make

Once the project is built, try running the project's unit tests:

$ make test

And the binaries are built in:

$ ls -la ./build/darwin/osquery/

Building on Linux

osquery supports several distributions of Linux.

For each supported distro, we supply vagrant infrastructure for creating native operating system packages. To create a package (e.g. a deb on Ubuntu or an rpm on CentOS), simply spin up a vagrant instance.

For example:

$ vagrant up ubuntu14
$ vagrant ssh ubuntu14

By default vagrant will allocate 2 virtual CPUs to the virtual machine instance. You can override this by setting OSQUERY_BUILD_CPUS environment variable before spinning up an instance. To allocate the maximum number of CPUs, OSQUERY_BUILD_CPUS can be set as:

OSQUERY_BUILD_CPUS=`nproc`             # for Linux
OSQUERY_BUILD_CPUS=`sysctl -n hw.ncpu` # for OS X

Once you have logged into the vagrant box, run the following to create a package:

$ cd /vagrant
$ make sysprep
$ make deps
$ make
$ make test

The binaries are built to a distro-specific folder within build and symlinked in:

$ ls -la ./build/linux/osquery/

Submitting Pull Requests

Once you have made changes you'll want to submit them to Github as a Pull Request. There are tons of wonderful guides and documentation around Pull Requests, and that is just out of scope for this wiki-- but consider the following workflow:

$ git checkout -b new_feature1
$ # write some code!
$ make -j 4
$ git commit -m "New Feature: do something wonderful"
$ git push

This assumes your remote origin is your osquery fork, and that you receive updates from an upstream. It is also common to use origin as facebook/osquery then add your fork as a target named after your Github username.

In that case the final push becomes git push USERNAME.

Testing changes

Our Jenkins CI will test your changes in three steps.

  1. A code audit is run using ./tools/
  2. The code is rebuilt, built again for release, then a package is generated using ./tools/ on various Linux and OS X versions.
  3. The same step is run on Windows 10.

The audit step attempts to build the documentation, run code formatting checks, and a brief static code analysis. The formatting check is performed with clang-format (installed with make deps to your osquery dependencies directory). Your changes are compared against the local master branch. Within the build host this is always the TIP of facebook/osquery, but locally the branch may be behind.

To speed up the format auditing process please configure your code editor to run clang-format on files changed. Or periodically during your development run make format. Running make check is also helpful, but it will use cppcheck which is not installed by default.

Dependencies and build internals

The make deps command is fairly intense and serves two purposes: (1) to communicate a standard set of environment setup instructions for our build and test nodes, (2) to provide an environment for reproducing errors. The are wonderful auxiliary benefits such as controlling the compiler and compile flags for almost all of our dependencies, controlling security-related features for dependencies, allowing a "mostly" universal build for Linux that makes deployment simple. To read more about the motivation and FAQ for our dependencies environment see the Github Refererence #2253.

When using make deps the environment the resultant binaries will have a minimum set of requirements to run:

  • glibc version 2.13
  • libgcc_s
  • libz

All other dependencies are built, compiled, and linked statically. This makes for a rather large set of output binaries (15M on Linux and 9M on OS X) but the trade-off for deployment simplicity is very worthwhile.

Under the hood the make deps script is calling ./tools/, which performs the simplified set of steps:

  • Create a "runtime" directory or dependency home: /usr/local/osquery.
  • Clone a pinned version of Homebrew or Linuxbrew into that home.
  • Install a local Tap using ./tools/provision/formulas into that home.
  • Run optional distro-specific setup scripts from ./tools/provision/
  • Install a list of packages from the local Tap defined in ./tools/

We use a minimum set of packages from Homebrew and Linuxbrew, mostly just tools. The remaining tools and C/C++ library dependencies as well as C and C++ runtimes are built from source. If we need to change compile options or variables we can bump these formula's bottle revisions.

Adding or changing dependencies

If you need to bump a dependency version, change the way it is built, or add a new dependency-- use the formulas in ./tools/provision/formula/. Let's consider a simple example:

require File.expand_path("../Abstract/abstract-osquery-formula", __FILE__)

class Libaudit < AbstractOsqueryFormula
  desc "Linux auditing framework"
  url ""
  sha256 "63020c88b0f37a93438894e67e63ccede23d658277ecc6afb9d40e4043147d3f"

  def install
    system "./"
    system "./configure", "--prefix=#{prefix}"
    cd "lib" do
      system "make"
      system "make", "install"

This looks A LOT like normal *brew formulas. For a new dependency do not add a bottle section. For help with writing formulas see Homebrew's Formula Cookbook. Note that we use an Abstract to control the environment variables and control relocation on Linux.

If you want to make build changes see the Cookbook for revision edits. Note that committing new or edited formulas will invalidate package caches this will cause the package to be built from source on the test/build hosts.

If this is a new dependency then you need to add a line to ./tools/ for Linux and or OS X at the order/time it should be installed.

When a dependency is updated by a maintainer or contributor the flow should follow: Update the target formula in ./tools/provision/formula/. Run make deps and the dependency change should cause a rebuild from source. * Build and run the osquery tests, and submit the change in a pull request.

After the change is merged, a maintainer can provide a bottle/binary version: Run ./tools/ uninstall TARGET to remove the from-source build. Run make build_deps to build a bottle version from source. Run ./tools/ bottle TARGET to generate the bottle. Update the formula again with the SHA256 printed to stdout. Upload the /usr/local/osquery/TARGET-VERSION.tar.gz to the S3 bottles folder. Create a pull request with the updated SHA256.

AWS EC2 Backed Vagrant Targets

The osquery vagrant infrastructure supports leveraging AWS EC2 to run virtual machines. This capability is provided by the vagrant-aws plugin, which is installed as follows:

$ vagrant plugin install vagrant-aws

Next, add a vagrant dummy box for AWS:

$ vagrant box add andytson/aws-dummy

Before launching an AWS-backed virtual machine, a few environment variables must be set:

# Required. Credentials for AWS API. vagrant-aws will error if these are unset.
export AWS_ACCESS_KEY_ID=...
# Name of AWS keypair for launching and accessing the EC2 instance.
export AWS_KEYPAIR_NAME=my-osquery-vagrant-security-group
export AWS_SSH_PRIVATE_KEY_PATH=/path/to/keypair.pem
# Name of AWS security group that allows TCP/22 from vagrant host.
# If using a non-default VPC use the security group ID instead.
export AWS_SECURITY_GROUP=my-osquery-vagrant-security-group
# Set this to the AWS region, "us-east-1" (default) or "us-west-1".
# Set this to the AWS instance type. If unset, m3.medium is used.
export AWS_INSTANCE_TYPE=m3.large
# (Optional) Set this to the VPC subnet ID.
# (Optional) Make sure your subnet assigns public IPs and there is a route.
export AWS_SUBNET_ID=...

Spin up a VM in EC2 and SSH in (remember to suspect/destroy when finished):

$ vagrant up aws-amazon2015.03 --provider=aws
$ vagrant ssh aws-amazon2015.03

Debug Builds, formatting, and more

To generate a non-optimized debug build use make debug.

CMake regenerates build information every time make [all] is run. To avoid the "configure" and project setup use make build.

make building and testing macros:

make # Default optimized configure/build
make test # Test the default build
make debug # Configure and build non-optimized with debuginfo
make test_debug # Test the debug build
make build # Skip the CMake project configure (compile only)
make debug_build # Same as build, but applied to the debug target
make test_debug_build # Take a guess ;)
make clean # Clean the CMake project configuration
make distclean # Clean all cached information and build dependencies
make deps # Install the osquery dependency environment into /usr/local/osquery
make depsclean # Remove the dependency environment
make docs # Build the Doxygen and mkdocs wiki

There are several additional code testing and formatting macros:

make format # Apply clang-format using osquery's format spec*
make format-all # Not recommended but formats the entire code base
make format_master # Format everything changed from the local master branch
make analyze # Run clean first, then rebuild with the LLVM static analyzer
make sanitize # Run clean first, then rebuild with sanitations
make fuzz # Run basic fuzz tests, as defined in each table spec

Generating the osquery SDK or sync:

make sdk # Build only the osquery SDK (libosquery.a)
make sync # Create a tarball for building the SDK externally

Finally, subtle changes to the build are mostly controlled through environment variables. When making these changes it is best to removed your build cache by removing the ./build/ or ./build/{platform}/ directory.

OSQUERY_BUILD_LINK_SHARED=True # Set CMake library discovery to prefer shared libraries
OSQUERY_BUILD_SHARED=True # Build libosquery* as shared objects and link appropriately
OSQUERY_BUILD_DEPS=True # Install dependencies from source when using make deps
OSQUERY_BUILD_BOTTLES=True # Create Homebrew bottles from installed dependencies

OSQUERY_BUILD_VERSION=9.9.9 # Set a wacky version string
OSQUERY_PLATFORM=custom_linux;1.0 # Set a wacky platform/distro name
SDK_VERSION=9.9.9 # Set a wacky SDK-version string
OSX_VERSION_MIN=10.11 # Override the native minimum OS X version ABI
OSQUERY_DEPS=/path/to/dependencies # Use or create a custom dependency environment

FAST=True # Build and link as quick as possible
SANITIZE_THREAD=True # Add -fsanitize=thread when using "make sanitize"
OPTIMIZED=True # Enable specific CPU optimizations (not recommended)
SKIP_TESTS=True # Skip unit test building (very very not recommended!)
SKIP_INTEGRATION_TESTS=True # Skip python tests when using "make test"
SKIP_BENCHMARKS=True # Build unit tests but skip building benchmark targets
SKIP_TABLES=True # Build platform without any table implementations or specs
SKIP_DISTRO_MAIN=False # Run the sysprep update/install within make deps
SQLITE_DEBUG=True # Enable SQLite query debugging (very verbose!)

Custom Packages

Building osquery on OS X or Linux requires a significant number of dependencies, which are not needed when deploying. It does not make sense to install osquery on your build hosts. See the Custom Packages guide for generating pkgs, debs or rpms.

Debug and from-source package building is not recommended but supported. You may generate several packages including devel, debuginfo, and additional sets of tools:

make package # Generate an osquery package for the build target
make packages # Generate optional/additional packages

These targets set the environment variable PACKAGE=1 and the CMake variable OSQUERY_BUILD_RELEASE that enables allows build and CMake logic to make compile and linking decisions. This may enable additional compile defines or remove sources from targets.

The osquery package build hosts run a series of additional unit and integration tests. This involves building targets with debug routines, testing, building the same targets ready for packaging, testing, and running a final set of "sanity" and deployment tests.

To mimic and follow the same build/release testing workflow use:


Pay attention to the environment variable RUN_RELEASE_TESTS=1, which enables the deployment sanity tests. If you are building an optimized or distribution package manager target this will most likely fail. The RUN_BUILD_DEPS variable tells the build to begin with a make deps.

Notes and FAQ

When trying to make, if you encounter:

Requested dependencies may have changed, run: make deps

You must run make deps to make sure you are pulling in the most-recent dependency assumptions. This error typically means a new virtual table has been added that includes new third-party development libraries.

make deps will take care of installing everything you need to compile osquery. However, to properly develop and contribute code, you'll need to install some additional programs. If you write C++ often, you likely already have these programs installed. We don't bundle these tools with osquery because many programmers are quite fond of their personal installations of LLVM utilities, debuggers, etc.

  • clang-format: we use clang-format to format all code in osquery. After staging your commit changes, run make format (requires clang-format).
  • valgrind: performance is a top priority for osquery, so all code should be thoroughly tested with valgrind or instruments. After building your code use ./tools/analysis/ --leaks to run all queries and test for memory leaks.

Build Performance

Generating a virtual table should NOT impact system performance. This is easier said than done as some tables may seem inherently latent such as SELECT * from suid_bin; if your expectation is a complete filesystem traversal looking for binaries with suid permissions. Please read the osquery features and guide on performance safety.

Some quick features include:

  • Performance regression and leak detection CI guards.
  • Blacklisting performance-impacting virtual tables.
  • Scheduled query optimization and profiling.
  • Query implementation isolation options.