Edit on GitHub

AcuteBenchmark

AcuteBenchmark allows you to benchmark functions that get Arrays as their input.

It is used inside IntelVectorMath for benchmarking its functions. A fully working example available here: https://github.com/JuliaMath/VML.jl/blob/AcuteBenchmark/benchmark/benchmark.jl

Creates random inputs for a function based on limits, types, and dims specified.

config = Funb(
    fun = sin,
    limits = [(-1,1)],
    types = [Float32, Float64],
    dims = [100],
)

or just in a compact form:

config = Funb( sin, [(-1,1)], [Float32, Float64], [10])

use benchmark! to run the benchmark:

using AcuteBenchmark

configs = FunbArray([
    Funb( sin, [(-1,1)],[Float32, Float64], [10] );
    Funb( atan, [(-1,1), (-1,1)],[Float32, Float64],[10, 10] );
    Funb( *, [(-1, 1), (-1, 1), (-1, 1)], [Float32, Float64], [(10,10), (10,10)] );
    ])

benchmark!(configs)

Plot the benchmark result using:

bar(configs)

bench-dims-set1

To have a same color for the same types use:

bar(configs, uniqueType = true, dimAnnotation = true)

bench-dims-set1-unique

To plot the relative speed, pass a pair of configs:

bar(configsRealBase => configsRealIVM, uniqueType = true, dimAnnotation = false, uniqueDim = true, "Base" => "IntelVectorMath")

IntelVectorMath Performance Comparison

To plot how the function acts over different dimension sets:

configs2 = Funb( sin, [(-1,1)],[Float32, Float64], [10 20 30 40] );
benchmark!(configs2)
dimplot(configs2)

bench-sin

To compare different sets pass an array of configs:

dimplot([configsRealBase,configsRealIVM],["Base", "IntelVectorMath"])

Performance over dimensions

AcuteBenchmark.Funb — Type
Funb(;fun, limits, types, dims)

Creates random inputs for a function based on limits, types, and dims specified.

Arguments

  • fun: the function :fun or :(Module.fun)
  • limits: min and max of possible values
  • types : type of elements
  • dims: Array of dimensions of the input vectors for each argument. Each column is for a new set of sizes, and each row is for different input arguments. So:
    • each element gives the size of the input, and it is a:
      • Number (for 1D)
      • Tuple (for N-D)
    • each row for each function argument
    • each column for each dimension set

Examples

config = Funb(
    fun = sin,
    limits = [(-1,1)],
    types = [Float32, Float64],
    dims = [100],
)

or just in a compact form:

config = Funb( sin, [(-1,1)], [Float32, Float64], [100])
AcuteBenchmark.FunbArray — Method
FunbArray

Array of Funb configs for different functions.

Examples

using AcuteBenchmark

configs = FunbArray([
    Funb( sin, [(-1,1)],[Float32, Float64], [100] );
    Funb( atan, [(-1,1), (-1,1)],[Float32, Float64],[100, 100] );
    Funb( *, [(-1, 1), (-1, 1), (-1, 1)], [Float32, Float64], [(100,100), (100,100)] );
    ])

You can also directly give the configs in vectors:

configs = FunbArray(
    fun =   [sin,
             atan,
             *],
    limits = [[(-1,1)],
             [(-1,1), (-1,1)],
             [(-1, 1), (-1, 1), (-1, 1)]],
    types = fill([Float32, Float64], (3)),
    dims = [ [100],
             [100, 100],
             [(100,100), (100,100)] ],
)
AcuteBenchmark.bar — Method
bar(config::StructArray{Funb}; uniqueType::Bool = false, dimAnnotation::Bool = true, uniqueDim::Bool = false)

Plots bars for each dimension set.

It is assumed that number of dimension sets are the same.

  • To have a same color for the same types, set true as the 2nd argument.
  • To turn off dimension annotations pass false as the 3rd argument.
  • In case of unique dimensions pass true as 4th argument to print dimension in title instead.

Examples

bar(configs)
bar(configs, uniqueType = true, dimAnnotation = false, uniqueDim = true)
AcuteBenchmark.bar — Method
bar(config::Pair{StructArray{Funb}}; uniqueType::Bool = false, dimAnnotation::Bool = false, uniqueDim::Bool =false, configName::Pair{String,String} = "1" => "2")

Gets a pair of StructArrays and calculates relative speed of coresponding elements and plots them.

Uses the first element of the pair for the configurations and only uses runtimes from the 2nd element of the pair. The relative speed is calculated like this:

1st element runtimes / 2nd element runtimes

Give the configsets names as a pair for the ylabel.

bar(configs => configs, uniqueType = true, dimAnnotation = false, uniqueDim =true, "group 1" => "group 2")
AcuteBenchmark.benchmark! — Method
benchmark!(config::StructArray{Funb}) # FunbArray{Funb}
benchmark!(config::Array{Funb})

Performs the benchmarking on a given Funb.

Examples

using AcuteBenchmark

configs = FunbArray([
    Funb( sin, [(-1,1)],[Float32, Float64], [100] );
    Funb( atan, [(-1,1), (-1,1)],[Float32, Float64],[100, 100] );
    Funb( *, [(-1, 1), (-1, 1), (-1, 1)], [Float32, Float64], [(100,100), (100,100)] );
    ])

benchmark!(configs)
AcuteBenchmark.dimplot — Method
dimplot(config::StructArray{Funb})

Shows runtime of a function for different dimension sets. The axes are logarithmic.

Examples

configs2 = Funb( sin, [(-1,1)],[Float32, Float64], [10 20 30 40] );

benchmark!(configs2)

dimplot(configs2)
AcuteBenchmark.dimplot — Method
dimplot(config::Vector{StructArray{Funb}}, labels::Vector{String})

By passing a vector different benchmark sets can be grouped together to be shown in a single plot. The axes are logarithmic.

Examples

dimplot([config1,config2, config3], ["group 1", "group 2", "group3"])
AcuteBenchmark.load — Method
load(file_name)

load benchmark data

Examples

configs = AcuteBenchmark.load("benchmarkdata.jld2")
AcuteBenchmark.numArgsDims — Method
numArgsDims(in)

Finding number of arguments and number of dimension sets

Examples

numArgs, numDimsSets = numArgsDims(in)
AcuteBenchmark.save — Method
save(file_name, config)

Save benchmark data

Examples

AcuteBenchmark.save("benchmarkdata.jld2", config)