CompileBot.jl

CompileBot automatically generates precompilation data for your Julia packages, which results in reducing the time it takes for runtime compilation, loading, and startup.

Installation

using Pkg
Pkg.add("CompileBot")

using CompileBot

Usage

As you change the code in your package, the precompile statements likely need to be updated too. You can use SnoopCompile bot to automatically and continuously create precompile files. This bot can be used offline or online.

Follow these steps to set up SnoopCompile bot for your package.

1 - Add Julia to your system PATH (if you haven't done that already)

The CompileBot spawns a new Julia process when running the snoop_bot function. Therefore, you need to make sure that Julia is added to your system PATH. See the official documentation on how to do that: https://julialang.org/downloads/platform/. To test whether Julia has been added successfully, simply open a terminal and type in julia. If everything has been configured correctly, the Julia REPL should be invoked now.

2 - Setting up the SnoopCompile bot configuration folder

Here, we will configure the bot in a directory deps/SnoopCompile/ that should be added to your repository. All configuration files for the SnoopCompile bot should go in this directory. If you choose a different name for this directory, be sure to change the path in the configuration steps below.

3 - Create the precompile script

You will need a precompile script, here called example_script.jl, that "exercises" the functionality you'd like to precompile. If you write a dedicated precompile script, place it in the bot configuration folder.

Alternatively, you may use your package's "runtests.jl" file. While less precise about which functionality is worthy of precompilation, this can slightly simplify configuration as described below.

4 - Create the script that runs snoop_bot

The snoop_bot function generates precompile statements and writes them to a file that will be incorporated into your package. snoop_bot requires a few settings, which can be most easily generated by BotConfig. The script that runs snoop_bot should be saved in your configuration directory, with a name like snoop_bot.jl.

The example below (from here) supports multiple operating systems, multiple Julia versions, and excludes a function whose precompilation would be problematic:

using CompileBot

botconfig = BotConfig(
  "Zygote";                            # package name (the one this configuration lives in)
  yml_path = "SnoopCompile.yml"        # parse `os` and `version` from `SnoopCompile.yml`
  exclusions = ["SqEuclidean"],        # exclude functions (by name) that would be problematic if precompiled
)

snoop_bot(
  botconfig,
  "$(@__DIR__)/example_script.jl",
)

If you choose to use your "runtests.jl" file as your precompile script, configuration can be as simple as specifying just the name of the package:

using CompileBot

snoop_bot(BotConfig("MyPackage"))

if !isdefined(Main, :SnoopCompile_ENV) || SnoopCompile_ENV == false
    # Tests that you want to skip when snooping
end

Finally, you could use package loading as the only operation, with snoop_bot(config, :(using MyPackage)).

snoop_bot uses different strategies depending on the Julia version:

• On Julia 1.2 or higher, it identifies methods for precompilation based on @snoopi;
• On Julia 1.0 or 1.1 (which do not support @snoopi), it identifies methods for precompilation based on @snoopc.

You can override this default behavior with a keyword argument, see snoop_bot for details.

5 - Optionally test the impact of your precompiles with snoop_bench

Call snoop_bench to measure the effect of adding precompile files. It takes the same parameters as snoop_bot above. You can run this manually, or choose to run it during automatic precompile file generation. To perform it automatically, create a snoop_bench.jl script in the bot configuration directory. This should be nearly identical to your snoop_bot.jl file, but calling snoop_bench instead. Note that benchmarking has the option of different performance metrics, snoop_mode=:snoopi or snoop_mode=:run_time depending on whether you want to measure inference time or the run time of your precompile script.

6 - Configure the bot to run with a GitHub Action file

You can create the precompile files automatically when you merge pull requests to master by adding a workflow file under .github/workflows/SnoopCompile.yml. This file should have content such as the example below. Lines marked with NOTE deserve special attention as likely places you may need to adjust the configuration.

name: SnoopCompile

on:
  push:
    branches:
    #  - 'master'  # NOTE: uncomment to run the bot only on pushes to master

defaults:
  run:
    shell: bash

jobs:
  SnoopCompile:
    if: "!contains(github.event.head_commit.message, '[skip ci]')"
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false

      matrix:
        # NOTE: only keep the versions you want to support
        # NOTE: if not using `yml_path`, these should match the version in `BotConfig`
        version:
          - 'nightly'
          - '1.5.3'
          - '1.4.2'
          - '1.3.1'
          - '1.2.0'
          - '1.1.1'
          - '1.0.5'
        os:        # NOTE: if not using `yml_path`, these should match the os in `BotConfig`
          - ubuntu-latest
          - windows-latest
          - macos-latest
        arch:
          - x64

    steps:
      - uses: actions/checkout@v2
      - uses: julia-actions/setup-julia@latest
        with:
          version: ${{ matrix.version }}

      - name: Install dependencies
        run: |
          julia --project -e 'using Pkg; Pkg.instantiate();'
          julia -e 'using Pkg; Pkg.add( PackageSpec(name="CompileBot", version = "1") );
                    Pkg.develop(PackageSpec(; path=pwd()));
                    using CompileBot; CompileBot.addtestdep();'

      - name: Generating precompile files
        run: julia --project -e 'include("deps/SnoopCompile/snoop_bot.jl")'   # NOTE: notice the path

      - name: Running Benchmark
        run: julia --project -e 'include("deps/SnoopCompile/snoop_bench.jl")' # NOTE: optional, if have benchmark file

      - name: Upload all
        continue-on-error: true # due to connection issues
        uses: actions/upload-artifact@v2.0.1
        with:
          path: ./

  Create_PR:
    if: "!contains(github.event.head_commit.message, '[skip ci]')"
    needs: SnoopCompile
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v2
      - name: Download all
        uses: actions/download-artifact@v2

      - name: CompileBot postprocess
        run: julia -e 'using Pkg; Pkg.add( PackageSpec(name="CompileBot", version = "1") );
                       using CompileBot; CompileBot.postprocess();'

      - name: Create Pull Request
        uses: peter-evans/create-pull-request@v3
        with:
          token: ${{ secrets.GITHUB_TOKEN }}
          commit-message: Update precompile_*.jl file
          title: "[AUTO] Update precompiles"
          labels: SnoopCompile
          branch: "SnoopCompile_AutoPR_${{ github.ref }}"


  Skip:
    if: "contains(github.event.head_commit.message, '[skip ci]')"
    runs-on: ubuntu-latest
    steps:
      - name: Skip CI 🚫
        run: echo skip ci

You can learn more about these files and the workflow process in the documentation. Examples of such files in projects can be found in other packages, for example Zygote.