The included benchmark suite measures series of 1000 rate limit checks. It serves as an illustration of available options. Run it like so:
mix run bench/fixed_cost_limiter.exs
mix run bench/variable_cost_limiter.exs
mix run bench/multi_rate_limiter.exs
There are 2 bucket sizes in the benchmarks: small and big, depending on the required atomic integer. For fixed and variable cost limiters small size means numbers typical for rate limiters, and big size is called "monster" because it has extreme parameters, unlikely to be used by anyone. On the other hand, multi-rate limiter can easily reach big atomic size with 3 or more sub-buckets.
Big buckets take a significant performance hit because of the way big integers are implemented in BEAM. 32bit architectures take a similar hit with all types of buckets - for our purposes all integers on 32bit are "big".
In general, such benchmarks should be taken with a grain of salt:
they often use artificial conditions different from real life, thus devaluing the results
above certain point a rate limiter is "fast enough" for many purposes. For BEAM this probably means "ETS or faster". Any atomics-based library should be much faster than ETS and is likely the fastest you can get.
comparing results between differently implemented or completely different algorithms is neither 100% valid nor very useful. It's very easy to make a pointless benchmark measuring wrong things, simply because it's hard to make different solutions perform same work achieving same results.
Tradeoffs are important. It's better to look at the overall picture, rather than raw performance.
This benchmark suite, in particular, is heavily biased towards DoS attack scenarios, where the vast majority of requests are denied, and doesn't model realistic concurrent access.
Example results
The following results were collected on Intel i7-13700K, Elixir 1.20, Erlang 29.
###############################################################################################################
# FixedCostLimiter #
###############################################################################################################
Benchmarking request (literals, default opts) with input Monster bucket (big atomic) ...
Benchmarking request (literals, default opts) with input Normal bucket (small atomic) ...
Benchmarking request (literals, persistent) with input Monster bucket (big atomic) ...
Benchmarking request (literals, persistent) with input Normal bucket (small atomic) ...
Benchmarking request (literals, reusing ref) with input Monster bucket (big atomic) ...
Benchmarking request (literals, reusing ref) with input Normal bucket (small atomic) ...
Benchmarking request (non-literals, default opts) with input Monster bucket (big atomic) ...
Benchmarking request (non-literals, default opts) with input Normal bucket (small atomic) ...
Benchmarking request (non-literals, persistent) with input Monster bucket (big atomic) ...
Benchmarking request (non-literals, persistent) with input Normal bucket (small atomic) ...
Benchmarking request (non-literals, reusing ref) with input Monster bucket (big atomic) ...
Benchmarking request (non-literals, reusing ref) with input Normal bucket (small atomic) ...
Calculating statistics...
Formatting results...
##### With input Monster bucket (big atomic) #####
Name ips average deviation median 99th %
request (literals, reusing ref) 7.48 K 133.69 μs ±1.90% 133.19 μs 141.23 μs
request (literals, persistent) 6.69 K 149.45 μs ±2.17% 148.63 μs 158.53 μs
request (non-literals, reusing ref) 5.98 K 167.21 μs ±2.72% 166.32 μs 178.75 μs
request (non-literals, persistent) 5.42 K 184.39 μs ±2.75% 182.83 μs 197.15 μs
request (literals, default opts) 4.51 K 221.58 μs ±3.59% 218.52 μs 242.85 μs
request (non-literals, default opts) 3.86 K 259.06 μs ±2.69% 258.82 μs 275.77 μs
Comparison:
request (literals, reusing ref) 7.48 K
request (literals, persistent) 6.69 K - 1.12x slower +15.76 μs
request (non-literals, reusing ref) 5.98 K - 1.25x slower +33.52 μs
request (non-literals, persistent) 5.42 K - 1.38x slower +50.71 μs
request (literals, default opts) 4.51 K - 1.66x slower +87.89 μs
request (non-literals, default opts) 3.86 K - 1.94x slower +125.37 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 128.00 μs 141.69 μs 33.80 K 131.34 μs, 131.57 μs
request (literals, persistent) 144.72 μs 159.02 μs 30.99 K 146.44 μs
request (non-literals, reusing ref) 157.87 μs 180.37 μs 28.74 K 164.96 μs, 164.76 μs
request (non-literals, persistent) 177.19 μs 200.86 μs 26.11 K 182.63 μs, 178.91 μs
request (literals, default opts) 206.57 μs 250.15 μs 21.16 K213.92 μs, 213.24 μs, 213
request (non-literals, default opts) 241.01 μs 278.14 μs 17.80 K259.24 μs, 261 μs, 259.15
##### With input Normal bucket (small atomic) #####
Name ips average deviation median 99th %
request (literals, reusing ref) 15.89 K 62.93 μs ±3.74% 62.93 μs 69.37 μs
request (literals, persistent) 13.33 K 75.02 μs ±3.08% 74.84 μs 81.70 μs
request (non-literals, reusing ref) 9.98 K 100.19 μs ±2.37% 99.51 μs 106.26 μs
request (non-literals, persistent) 8.82 K 113.40 μs ±1.79% 113.53 μs 118.13 μs
request (literals, default opts) 6.75 K 148.23 μs ±2.27% 147.49 μs 158.40 μs
request (non-literals, default opts) 5.44 K 183.83 μs ±4.23% 180.46 μs 204.94 μs
Comparison:
request (literals, reusing ref) 15.89 K
request (literals, persistent) 13.33 K - 1.19x slower +12.09 μs
request (non-literals, reusing ref) 9.98 K - 1.59x slower +37.25 μs
request (non-literals, persistent) 8.82 K - 1.80x slower +50.46 μs
request (literals, default opts) 6.75 K - 2.36x slower +85.30 μs
request (non-literals, default opts) 5.44 K - 2.92x slower +120.90 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 58.45 μs 70.06 μs 74.62 K 63.17 μs, 59.81 μs
request (literals, persistent) 70.37 μs 82.31 μs 61.56 K 74.30 μs
request (non-literals, reusing ref) 94.97 μs 107.00 μs 46.35 K 99.26 μs
request (non-literals, persistent) 108.13 μs 118.74 μs 40.69 K 113.44 μs
request (literals, default opts) 138.26 μs 159.26 μs 28.98 K 145.28 μs
request (non-literals, default opts) 165.51 μs 210.99 μs 25.01 K174.91 μs, 178.74 μs, 175
###############################################################################################################
# VariableCostLimiter #
###############################################################################################################
Benchmarking request (literals, default opts) with input Monster bucket (big atomic) ...
Benchmarking request (literals, default opts) with input Normal bucket (small atomic) ...
Benchmarking request (literals, persistent) with input Monster bucket (big atomic) ...
Benchmarking request (literals, persistent) with input Normal bucket (small atomic) ...
Benchmarking request (literals, reusing ref) with input Monster bucket (big atomic) ...
Benchmarking request (literals, reusing ref) with input Normal bucket (small atomic) ...
Benchmarking request (non-literals, default opts) with input Monster bucket (big atomic) ...
Benchmarking request (non-literals, default opts) with input Normal bucket (small atomic) ...
Benchmarking request (non-literals, persistent) with input Monster bucket (big atomic) ...
Benchmarking request (non-literals, persistent) with input Normal bucket (small atomic) ...
Benchmarking request (non-literals, reusing ref) with input Monster bucket (big atomic) ...
Benchmarking request (non-literals, reusing ref) with input Normal bucket (small atomic) ...
Calculating statistics...
Formatting results...
##### With input Monster bucket (big atomic) #####
Name ips average deviation median 99th %
request (literals, reusing ref) 7.47 K 133.89 μs ±1.79% 133.50 μs 140.67 μs
request (non-literals, reusing ref) 7.16 K 139.62 μs ±1.71% 138.97 μs 146.77 μs
request (literals, persistent) 6.63 K 150.73 μs ±2.78% 149.50 μs 162.17 μs
request (non-literals, persistent) 6.30 K 158.70 μs ±1.84% 158.23 μs 166.77 μs
request (literals, default opts) 4.49 K 222.53 μs ±3.60% 219.38 μs 243.27 μs
request (non-literals, default opts) 4.24 K 236.12 μs ±3.63% 236.75 μs 254.86 μs
Comparison:
request (literals, reusing ref) 7.47 K
request (non-literals, reusing ref) 7.16 K - 1.04x slower +5.73 μs
request (literals, persistent) 6.63 K - 1.13x slower +16.83 μs
request (non-literals, persistent) 6.30 K - 1.19x slower +24.81 μs
request (literals, default opts) 4.49 K - 1.66x slower +88.63 μs
request (non-literals, default opts) 4.24 K - 1.76x slower +102.23 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 127.61 μs 141.28 μs 32.84 K 133.40 μs
request (non-literals, reusing ref) 133.51 μs 147.22 μs 31.66 K 138.51 μs, 139.09 μs
request (literals, persistent) 143.88 μs 163.27 μs 31.54 K 147.17 μs
request (non-literals, persistent) 151.78 μs 167.29 μs 28.41 K 158.24 μs, 158.06 μs
request (literals, default opts) 206.73 μs 251.31 μs 21.11 K 214.08 μs
request (non-literals, default opts) 216.29 μs 260.30 μs 19.38 K 231.01 μs
##### With input Normal bucket (small atomic) #####
Name ips average deviation median 99th %
request (literals, reusing ref) 16.86 K 59.32 μs ±3.88% 59.27 μs 65.67 μs
request (non-literals, reusing ref) 16.01 K 62.47 μs ±3.77% 62.29 μs 68.82 μs
request (non-literals, persistent) 13.36 K 74.86 μs ±3.27% 74.56 μs 81.59 μs
request (literals, persistent) 12.88 K 77.64 μs ±3.24% 77.34 μs 84.64 μs
request (literals, default opts) 6.91 K 144.73 μs ±2.24% 144.25 μs 153.78 μs
request (non-literals, default opts) 6.75 K 148.24 μs ±2.05% 147.80 μs 157.07 μs
Comparison:
request (literals, reusing ref) 16.86 K
request (non-literals, reusing ref) 16.01 K - 1.05x slower +3.14 μs
request (non-literals, persistent) 13.36 K - 1.26x slower +15.54 μs
request (literals, persistent) 12.88 K - 1.31x slower +18.32 μs
request (literals, default opts) 6.91 K - 2.44x slower +85.41 μs
request (non-literals, default opts) 6.75 K - 2.50x slower +88.92 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 55.60 μs 66.16 μs 78.56 K 56.31 μs, 59.73 μs
request (non-literals, reusing ref) 58.19 μs 69.36 μs 74.82 K 59.78 μs
request (non-literals, persistent) 70.36 μs 82.26 μs 62.95 K 72.60 μs
request (literals, persistent) 72.97 μs 85.09 μs 59.28 K 78.47 μs
request (literals, default opts) 135.15 μs 154.40 μs 30.50 K 141.85 μs
request (non-literals, default opts) 139.75 μs 157.64 μs 29.15 K 148.42 μs, 145.52 μs
###############################################################################################################
# MultiRateLimiter #
###############################################################################################################
Benchmarking request (literals, default opts) with input 2 buckets ...
Benchmarking request (literals, default opts) with input 3 buckets (big atomic) ...
Benchmarking request (literals, default opts) with input 3 buckets (small atomic) ...
Benchmarking request (literals, persistent) with input 2 buckets ...
Benchmarking request (literals, persistent) with input 3 buckets (big atomic) ...
Benchmarking request (literals, persistent) with input 3 buckets (small atomic) ...
Benchmarking request (literals, reusing ref) with input 2 buckets ...
Benchmarking request (literals, reusing ref) with input 3 buckets (big atomic) ...
Benchmarking request (literals, reusing ref) with input 3 buckets (small atomic) ...
Benchmarking request (non-literals, default opts) with input 2 buckets ...
Benchmarking request (non-literals, default opts) with input 3 buckets (big atomic) ...
Benchmarking request (non-literals, default opts) with input 3 buckets (small atomic) ...
Benchmarking request (non-literals, persistent) with input 2 buckets ...
Benchmarking request (non-literals, persistent) with input 3 buckets (big atomic) ...
Benchmarking request (non-literals, persistent) with input 3 buckets (small atomic) ...
Benchmarking request (non-literals, reusing ref) with input 2 buckets ...
Benchmarking request (non-literals, reusing ref) with input 3 buckets (big atomic) ...
Benchmarking request (non-literals, reusing ref) with input 3 buckets (small atomic) ...
Benchmarking request_details (literals, default opts) with input 2 buckets ...
Benchmarking request_details (literals, default opts) with input 3 buckets (big atomic) ...
Benchmarking request_details (literals, default opts) with input 3 buckets (small atomic) ...
Benchmarking request_details (non-literals, default opts) with input 2 buckets ...
Benchmarking request_details (non-literals, default opts) with input 3 buckets (big atomic) ...
Benchmarking request_details (non-literals, default opts) with input 3 buckets (small atomic) ...
Calculating statistics...
Formatting results...
##### With input 2 buckets #####
Name ips average deviation median 99th %
request (literals, reusing ref) 10.13 K 98.73 μs ±3.56% 98.94 μs 106.48 μs
request (literals, persistent) 8.66 K 115.50 μs ±3.00% 115.42 μs 123.63 μs
request (literals, default opts) 5.24 K 190.90 μs ±3.17% 191.05 μs 204.70 μs
request_details (literals, default opts) 4.72 K 211.71 μs ±1.31% 211.72 μs 217.99 μs
request (non-literals, reusing ref) 3.68 K 271.94 μs ±2.00% 271.62 μs 285.95 μs
request (non-literals, persistent) 3.54 K 282.57 μs ±1.60% 282.68 μs 293.71 μs
request (non-literals, default opts) 2.74 K 364.64 μs ±1.86% 364.88 μs 384.85 μs
request_details (non-literals, default opts) 2.27 K 439.92 μs ±12.74% 410.95 μs 590.53 μs
Comparison:
request (literals, reusing ref) 10.13 K
request (literals, persistent) 8.66 K - 1.17x slower +16.77 μs
request (literals, default opts) 5.24 K - 1.93x slower +92.17 μs
request_details (literals, default opts) 4.72 K - 2.14x slower +112.98 μs
request (non-literals, reusing ref) 3.68 K - 2.75x slower +173.21 μs
request (non-literals, persistent) 3.54 K - 2.86x slower +183.84 μs
request (non-literals, default opts) 2.74 K - 3.69x slower +265.91 μs
request_details (non-literals, default opts) 2.27 K - 4.46x slower +341.19 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 89.19 μs 108.60 μs 47.93 K 99.16 μs
request (literals, persistent) 105.83 μs 125.53 μs 40.73 K 114.59 μs
request (literals, default opts) 173.71 μs 210.04 μs 23.19 K 191.94 μs, 190.85 μs
request_details (literals, default opts) 203.65 μs 220.29 μs 22.32 K210.12 μs, 212.57 μs, 212
request (non-literals, reusing ref) 257.70 μs 287.66 μs 16.58 K272.37 μs, 272.82 μs, 270
request (non-literals, persistent) 270.45 μs 295.75 μs 15.54 K 283.34 μs, 282.98 μs
request (non-literals, default opts) 345.13 μs 389.50 μs 10.96 K 363.90 μs
request_details (non-literals, default opts) 382.33 μs 607.88 μs 11.04 K406.54 μs, 398.36 μs, 401
##### With input 3 buckets (big atomic) #####
Name ips average deviation median 99th %
request (literals, reusing ref) 6.96 K 143.71 μs ±3.15% 142.81 μs 155.42 μs
request (literals, persistent) 6.22 K 160.65 μs ±2.73% 159.76 μs 172.33 μs
request (literals, default opts) 4.08 K 244.89 μs ±3.00% 243.11 μs 265.19 μs
request_details (literals, default opts) 2.99 K 334.95 μs ±14.72% 328.44 μs 468.60 μs
request (non-literals, reusing ref) 2.55 K 392.92 μs ±3.02% 391.39 μs 424.02 μs
request (non-literals, persistent) 2.48 K 403.89 μs ±2.85% 402.35 μs 433.64 μs
request (non-literals, default opts) 2.03 K 492.48 μs ±3.37% 490.76 μs 542.97 μs
request_details (non-literals, default opts) 1.75 K 570.05 μs ±4.42% 565.42 μs 654.42 μs
Comparison:
request (literals, reusing ref) 6.96 K
request (literals, persistent) 6.22 K - 1.12x slower +16.94 μs
request (literals, default opts) 4.08 K - 1.70x slower +101.17 μs
request_details (literals, default opts) 2.99 K - 2.33x slower +191.23 μs
request (non-literals, reusing ref) 2.55 K - 2.73x slower +249.21 μs
request (non-literals, persistent) 2.48 K - 2.81x slower +260.18 μs
request (non-literals, default opts) 2.03 K - 3.43x slower +348.76 μs
request_details (non-literals, default opts) 1.75 K - 3.97x slower +426.33 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 132.97 μs 156.16 μs 32.68 K 144.04 μs, 141.67 μs
request (literals, persistent) 150.20 μs 172.70 μs 27.82 K 157.99 μs
request (literals, default opts) 222.62 μs 271.12 μs 18.60 K 238.80 μs
request_details (literals, default opts) 245.25 μs 501.48 μs 14.64 K338.19 μs, 320.63 μs, 332
request (non-literals, reusing ref) 371.14 μs 432.43 μs 12.10 K 379.48 μs
request (non-literals, persistent) 374.42 μs 442.10 μs 11.78 K 391.42 μs
request (non-literals, default opts) 454.32 μs 545.49 μs 8.63 K 474.32 μs
request_details (non-literals, default opts) 522.89 μs 662.82 μs 7.12 K575.70 μs, 570.44 μs, 554
##### With input 3 buckets (small atomic) #####
Name ips average deviation median 99th %
request (literals, reusing ref) 10.14 K 98.66 μs ±3.38% 98.36 μs 107.20 μs
request (literals, persistent) 8.75 K 114.25 μs ±2.82% 113.95 μs 122.49 μs
request (literals, default opts) 5.15 K 194.30 μs ±2.53% 194.22 μs 206.70 μs
request_details (literals, default opts) 3.66 K 273.56 μs ±10.29% 265.62 μs 351.89 μs
request (non-literals, reusing ref) 3.06 K 326.94 μs ±2.41% 325.36 μs 349.47 μs
request (non-literals, persistent) 2.96 K 338.16 μs ±2.02% 337.33 μs 356.89 μs
request (non-literals, default opts) 2.38 K 420.62 μs ±2.05% 419.75 μs 451.75 μs
request_details (non-literals, default opts) 2.00 K 500.29 μs ±5.34% 492.64 μs 594.23 μs
Comparison:
request (literals, reusing ref) 10.14 K
request (literals, persistent) 8.75 K - 1.16x slower +15.59 μs
request (literals, default opts) 5.15 K - 1.97x slower +95.63 μs
request_details (literals, default opts) 3.66 K - 2.77x slower +174.89 μs
request (non-literals, reusing ref) 3.06 K - 3.31x slower +228.28 μs
request (non-literals, persistent) 2.96 K - 3.43x slower +239.49 μs
request (non-literals, default opts) 2.38 K - 4.26x slower +321.95 μs
request_details (non-literals, default opts) 2.00 K - 5.07x slower +401.63 μs
Extended statistics:
Name minimum maximum sample size mode
request (literals, reusing ref) 89.91 μs 108.54 μs 47.85 K 98.04 μs
request (literals, persistent) 105.48 μs 123.77 μs 41.34 K 113.86 μs
request (literals, default opts) 179.36 μs 210.80 μs 22.23 K193.13 μs, 190.99 μs, 191
request_details (literals, default opts) 194.48 μs 359.87 μs 15.87 K 260.56 μs
request (non-literals, reusing ref) 313.44 μs 352.47 μs 14.06 K321.29 μs, 322.14 μs, 324
request (non-literals, persistent) 318.99 μs 358.70 μs 13.44 K333.64 μs, 331.60 μs, 338
request (non-literals, default opts) 396.66 μs 455.43 μs 9.42 K 417.69 μs
request_details (non-literals, default opts) 459.04 μs 600.01 μs 8.12 K 489.16 μs, 486.15 μs