CompileBot.jl

CompileBot automatically generates precompilation data for your Julia packages, which results in reducing the time it takes for runtime compilation, loading, and startup.

Installation

using Pkg
Pkg.add("CompileBot")
using CompileBot

Usage

As you change the code in your package, the precompile statements likely need to be updated too. You can use SnoopCompile bot to automatically and continuously create precompile files. This bot can be used offline or online.

The CompileBot spawns a new Julia process when running the snoop_bot function. Therefore, you need to make sure that Julia is added to your system PATH. See the official documentation on how to do that: https://julialang.org/downloads/platform/. To test whether Julia has been added successfully, simply open a terminal and type in julia. If everything has been configured correctly, the Julia REPL should be invoked now.

2 - Setting up the SnoopCompile bot configuration folder

Here, we will configure the bot in a directory deps/SnoopCompile/ that should be added to your repository. All configuration files for the SnoopCompile bot should go in this directory. If you choose a different name for this directory, be sure to change the path in the configuration steps below.

3 - Create the precompile script

You will need a precompile script, here called example_script.jl, that "exercises" the functionality you'd like to precompile. If you write a dedicated precompile script, place it in the bot configuration folder.

Alternatively, you may use your package's "runtests.jl" file. While less precise about which functionality is worthy of precompilation, this can slightly simplify configuration as described below.

4 - Create the script that runs snoop_bot

The snoop_bot function generates precompile statements and writes them to a file that will be incorporated into your package. snoop_bot requires a few settings, which can be most easily generated by BotConfig. The script that runs snoop_bot should be saved in your configuration directory, with a name like snoop_bot.jl.

The example below (from here) supports multiple operating systems, multiple Julia versions, and excludes a function whose precompilation would be problematic:

using CompileBot

botconfig = BotConfig(
"Zygote";                            # package name (the one this configuration lives in)
yml_path = "SnoopCompile.yml"        # parse os and version from SnoopCompile.yml
exclusions = ["SqEuclidean"],        # exclude functions (by name) that would be problematic if precompiled
)

snoop_bot(
botconfig,
"$(@__DIR__)/example_script.jl", ) If you choose to use your "runtests.jl" file as your precompile script, configuration can be as simple as specifying just the name of the package: using CompileBot snoop_bot(BotConfig("MyPackage")) Note Some of your regular tests may not be appropriate for snoop_bot. snoop_bot sets a global variable SnoopCompile_ENV to true during snooping, and sets it to false when finished. You can exploit this in your tests to determine whether snooping is on: if !isdefined(Main, :SnoopCompile_ENV) || SnoopCompile_ENV == false # Tests that you want to skip when snooping end Finally, you could use package loading as the only operation, with snoop_bot(config, :(using MyPackage)). snoop_bot uses different strategies depending on the Julia version: • On Julia 1.2 or higher, it identifies methods for precompilation based on @snoopi; • On Julia 1.0 or 1.1 (which do not support @snoopi), it identifies methods for precompilation based on @snoopc. You can override this default behavior with a keyword argument, see snoop_bot for details. 5 - Optionally test the impact of your precompiles with snoop_bench Call snoop_bench to measure the effect of adding precompile files. It takes the same parameters as snoop_bot above. You can run this manually, or choose to run it during automatic precompile file generation. To perform it automatically, create a snoop_bench.jl script in the bot configuration directory. This should be nearly identical to your snoop_bot.jl file, but calling snoop_bench instead. Note that benchmarking has the option of different performance metrics, snoop_mode=:snoopi or snoop_mode=:run_time depending on whether you want to measure inference time or the run time of your precompile script. 6 - Configure the bot to run with a GitHub Action file You can create the precompile files automatically when you merge pull requests to master by adding a workflow file under .github/workflows/SnoopCompile.yml. This file should have content such as the example below. Lines marked with NOTE deserve special attention as likely places you may need to adjust the configuration. name: SnoopCompile on: push: branches: # - 'master' # NOTE: uncomment to run the bot only on pushes to master defaults: run: shell: bash jobs: SnoopCompile: if: "!contains(github.event.head_commit.message, '[skip ci]')" runs-on:${{ matrix.os }}
strategy:
fail-fast: false

matrix:
# NOTE: only keep the versions you want to support
# NOTE: if not using yml_path, these should match the version in BotConfig
version:
- 'nightly'
- '1.5.3'
- '1.4.2'
- '1.3.1'
- '1.2.0'
- '1.1.1'
- '1.0.5'
os:        # NOTE: if not using yml_path, these should match the os in BotConfig
- ubuntu-latest
- windows-latest
- macos-latest
arch:
- x64

steps:
- uses: actions/checkout@v2
- uses: julia-actions/setup-julia@latest
with:
version: ${{ matrix.version }} - name: Install dependencies run: | julia --project -e 'using Pkg; Pkg.instantiate();' julia -e 'using Pkg; Pkg.add( PackageSpec(name="CompileBot", version = "1") ); Pkg.develop(PackageSpec(; path=pwd())); using CompileBot; CompileBot.addtestdep();' - name: Generating precompile files run: julia --project -e 'include("deps/SnoopCompile/snoop_bot.jl")' # NOTE: notice the path - name: Running Benchmark run: julia --project -e 'include("deps/SnoopCompile/snoop_bench.jl")' # NOTE: optional, if have benchmark file - name: Upload all continue-on-error: true # due to connection issues uses: actions/upload-artifact@v2.0.1 with: path: ./ Create_PR: if: "!contains(github.event.head_commit.message, '[skip ci]')" needs: SnoopCompile runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - name: Download all uses: actions/download-artifact@v2 - name: CompileBot postprocess run: julia -e 'using Pkg; Pkg.add( PackageSpec(name="CompileBot", version = "1") ); using CompileBot; CompileBot.postprocess();' - name: Create Pull Request uses: peter-evans/create-pull-request@v3 with: token:${{ secrets.GITHUB_TOKEN }}
commit-message: Update precompile_*.jl file
title: "[AUTO] Update precompiles"
labels: SnoopCompile
branch: "SnoopCompile_AutoPR_\${{ github.ref }}"

Skip:
runs-on: ubuntu-latest
steps:
- name: Skip CI 🚫
run: echo skip ci

You can learn more about these files and the workflow process in the documentation. Examples of such files in projects can be found in other packages, for example Zygote.

Note

Upgrading from an old SnoopCompile bot:

CompileBot is now in a separate repository, and the API is changed because of that. Call using CompileBot directly in your snoop scripts and update your workflow based on this guide: Configure the bot to run with a GitHub Action file

In addition to the previous steps, you should also remove _precompile_() and any other code that includes a _precompile_() function. In the new version, SnoopCompile automatically creates the appropriate code.