-
Notifications
You must be signed in to change notification settings - Fork 0
Add massiv vs accelarate blog code #11
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
99375bb
d461fbd
0abab78
cc02a8c
a5d016c
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| # Revision history for massiv-vs-accelerate | ||
|
|
||
| ## 0.1.0.0 -- YYYY-mm-dd | ||
|
|
||
| * First version. Released on an unsuspecting world. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| # We use GHC 9.6 as our base | ||
| FROM haskell:9.6-slim | ||
|
|
||
| # Install wget and gnupg to add the LLVM repository | ||
| RUN apt-get update && apt-get install -y \ | ||
| wget \ | ||
| gnupg \ | ||
| lsb-release \ | ||
| software-properties-common \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # Use the official LLVM setup script to install LLVM 15 | ||
| # This handles the repository addition for any Debian version (Buster/Bullseye/Bookworm) | ||
| RUN wget https://apt.llvm.org/llvm.sh \ | ||
| && chmod +x llvm.sh \ | ||
| && ./llvm.sh 15 \ | ||
| && apt-get install -y \ | ||
| llvm \ | ||
| clang \ | ||
| llvm-15-dev \ | ||
| libffi-dev \ | ||
| libedit-dev \ | ||
| libfftw3-dev \ | ||
| pkg-config \ | ||
| zlib1g-dev \ | ||
| g++ \ | ||
| make \ | ||
| && rm -rf /var/lib/apt/lists/* | ||
|
|
||
| # THE FIXES: | ||
| # 1. Create the symlinks so GHC and Accelerate tools exist cleanly in the PATH | ||
| # We use -sf to force overwrite existing default paths safely | ||
| RUN ln -sf /usr/bin/clang-15 /usr/bin/clang && \ | ||
| ln -sf /usr/bin/opt-15 /usr/bin/opt && \ | ||
| ln -sf /usr/bin/llc-15 /usr/bin/llc && \ | ||
| ln -sf /usr/bin/llvm-config-15 /usr/bin/llvm-config | ||
|
|
||
| # 2. Set the environment variables globally for the container | ||
| ENV LLVM_CONFIG=/usr/bin/llvm-config | ||
| ENV ACCELERATE_LLVM_CLANG_PATH=/usr/bin/clang | ||
|
|
||
| # Set up the working directory inside the container | ||
| WORKDIR /app | ||
|
|
||
| # Update the package list | ||
| RUN cabal update | ||
|
|
||
| # Copy the cabal configuration first to cache dependencies (optimization step) | ||
| COPY *.cabal cabal.project /app/ | ||
|
|
||
| # Copy the rest of the application source files | ||
| COPY . /app | ||
|
|
||
| # Pre-compile the project inside the container image | ||
| RUN cabal build | ||
|
|
||
| # Set the default entry point to execute your compiled benchmark suite | ||
| ENTRYPOINT ["cabal", "run", "massiv-vs-accelerate", "--"] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,97 @@ | ||
| # Massiv vs Accelerate Benchmark Suite | ||
|
|
||
| This project benchmarks and compares the performance of [Massiv](https://github.com/lehins/massiv) and [Accelerate](https://github.com/AccelerateHS/accelerate) Haskell array libraries using a variety of operations (map, dot product, stencil/mean blur). | ||
|
|
||
| ## Project Structure | ||
|
|
||
| - `app/Main.hs` — Main benchmarking entry point (Criterion-based) | ||
| - `src/` — Library and dependency sources (submodules) | ||
| - `Dockerfile` — Containerized build and run environment matching target LLVM pipelines | ||
| - `cabal.project`, `massiv-vs-accelerate.cabal` — Cabal build configuration | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - [Docker](https://www.docker.com/get-started) (highly recommended for ARM64 toolchain compatibility and reproducibility) | ||
| - Or: [GHC 9.6+](https://www.haskell.org/ghc/), [Cabal](https://www.haskell.org/cabal/), and **LLVM 15** installed with `clang`/`opt`/`llc` binaries mapped to your system path. | ||
|
|
||
| ## Quick Start (Docker) | ||
|
|
||
| 1. **Build the containerized environment:** | ||
|
|
||
| ```sh | ||
| docker build -t massiv-vs-accelerate . | ||
| ``` | ||
|
|
||
| 2. **Run isolated library benchmarks:** | ||
| Pass the specific operation name directly to the container runner to evaluate both libraries back-to-back: | ||
|
|
||
| Massiv Stencil Only: | ||
| ```bash | ||
| docker run --rm massiv-vs-accelerate stencil-massiv | ||
| ``` | ||
| Accelerate Stencil Only: | ||
|
|
||
| ```bash | ||
| docker run --rm massiv-vs-accelerate stencil-accel | ||
| ``` | ||
|
|
||
| 3. **Passing Criterion and GHC Runtime (+RTS) Flags:** | ||
|
|
||
| Because the container features an automated entrypoint wrapper, you can append reporting flags or memory profiling switches straight to your commands: | ||
|
|
||
| ```bash | ||
| # Export an interactive HTML chart to your current host directory | ||
| docker run --rm -v $(pwd):/app/out massiv-vs-accelerate stencil --output out/results.html | ||
|
|
||
| # Inspect memory allocation and GC productivity profiles (+RTS -s) | ||
| docker run --rm massiv-vs-accelerate stencil-massiv +RTS -s | ||
|
|
||
| ``` | ||
|
|
||
| 4. **Manual Setup (Without Docker on Mac)** | ||
| Configure dependencies: | ||
|
|
||
| Ensure llvm-15 toolchains are [globally](https://www.acceleratehs.org/get-started.html) exposed on your machine. | ||
|
|
||
| 1. Install llvm dependencies | ||
|
|
||
| brew install llvm@15 libffi pkg-config | ||
|
|
||
| 2. Expose the New Binaries to Your Local Terminal | ||
| ```bash | ||
| # 1. Force your active terminal context to prioritize LLVM 15 | ||
| export PATH="/opt/homebrew/opt/llvm@15/bin:$PATH" | ||
|
|
||
| # 2. Bind development header search flags | ||
| export LDFLAGS="-L/opt/homebrew/opt/llvm@15/lib" | ||
| export CPPFLAGS="-I/opt/homebrew/opt/llvm@15/include" | ||
|
|
||
| # 3. Explicitly link tool locations for the GHC and Accelerate compiler pipelines | ||
| export LLVM_CONFIG="/opt/homebrew/opt/llvm@15/bin/llvm-config" | ||
| export ACCELERATE_LLVM_CLANG_PATH="/opt/homebrew/opt/llvm@15/bin/clang" | ||
| ``` | ||
|
|
||
| Trigger the native build | ||
| ``` | ||
| cabal clean && cabal build | ||
| ``` | ||
| Run a benchmark: | ||
|
|
||
| ```Bash | ||
| cabal run massiv-vs-accelerate -- [map|dot|stencil|stencil-massiv|stencil-accel] [criterion flags] | ||
| ``` | ||
| Example: | ||
|
|
||
| ```Bash | ||
| cabal run massiv-vs-accelerate -- stencil | ||
| ``` | ||
| 5. **Benchmark Types** | ||
| - map — Map (+1) over a large array | ||
|
|
||
| - dot — Dot product of two large arrays | ||
|
|
||
| - stencil — 3x3 mean blur matrix operations comparison | ||
|
|
||
| - stencil-massiv — Isolated Massiv stencil loop | ||
|
|
||
| - stencil-accel — Isolated Accelerate LLVM backend stencil |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| module BasicComputation ( | ||
| benchBasicComputation, | ||
| BenchArgs(..) | ||
| ) where | ||
|
|
||
| import Criterion.Main (Benchmark, bench, nf, whnf) | ||
| import qualified Data.Massiv.Array as M | ||
| import qualified Data.Array.Accelerate as A | ||
| import qualified Data.Array.Accelerate.LLVM.Native as Native | ||
|
|
||
| -- | Arguments for benchmarking basic computation | ||
| -- Add more fields as needed for future extensibility | ||
| -- For now, only size is used | ||
|
|
||
| data BenchArgs = BenchArgs | ||
| { size :: Int | ||
| } | ||
|
|
||
| benchBasicComputation :: BenchArgs -> [Benchmark] | ||
| benchBasicComputation args = | ||
| let n = size args | ||
| -- Massiv Setup (Parallel Array) | ||
| massivArr = M.makeArray M.Par (M.Sz1 n) id :: M.Array M.P M.Ix1 Int | ||
| -- Accelerate Setup (Expression Graph) | ||
| accelArr = A.fromList (A.Z A.:. n) [0..n-1] :: A.Vector Int | ||
| in | ||
| [ bench "Massiv (CPU Parallel)" $ | ||
| nf (M.computeAs M.P . M.map (+1)) massivArr | ||
| , bench "Accelerate (LLVM Native)" $ | ||
| whnf (Native.run . A.map (+1) . A.use) accelArr | ||
| ] |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,129 @@ | ||
| module Main where | ||
|
|
||
| import Criterion.Main | ||
| import System.Environment (getArgs, withArgs) | ||
| import qualified Data.Massiv.Array as M | ||
| import qualified Data.Array.Accelerate as A | ||
| import qualified Data.Array.Accelerate.LLVM.Native as Native | ||
|
|
||
|
|
||
| -- | 3x3 mean (average) stencil for 2D arrays in Massiv. | ||
| -- | ||
| -- This stencil computes the average of a 3x3 neighborhood around each element. | ||
| -- It is used for mean blurring (smoothing) operations on 2D arrays. | ||
| -- | ||
| -- The stencil uses zero-padding (see usage with 'M.Fill 0.0') at the boundaries. | ||
| -- | ||
| -- Example usage: | ||
| -- | ||
| -- > M.mapStencil (M.Fill 0.0) massivMeanStencil arr | ||
| -- | ||
| -- Where 'arr' is a 2D Massiv array of type 'M.Array M.P M.Ix2 Double'. | ||
| massivMeanStencil :: M.Stencil M.Ix2 Double Double | ||
| massivMeanStencil = M.makeStencil (M.Sz2 3 3) (1 M.:. 1) $ \get -> | ||
| ( get (-1 M.:. -1) + get (-1 M.:. 0) + get (-1 M.:. 1) | ||
| + get ( 0 M.:. -1) + get ( 0 M.:. 0) + get ( 0 M.:. 1) | ||
| + get ( 1 M.:. -1) + get ( 1 M.:. 0) + get ( 1 M.:. 1) | ||
| ) / 9 | ||
|
|
||
| -- | 3x3 mean (average) stencil function for Accelerate. | ||
| -- | ||
| -- This function computes the average of a 3x3 neighborhood for use with Accelerate's 'stencil' operation. | ||
| -- It is typically used for mean blurring (smoothing) on 2D arrays. | ||
| -- | ||
| -- The input is a 3x3 tuple of values (top-left, top-center, top-right, etc.). | ||
| -- | ||
| -- Example usage: | ||
| -- | ||
| -- > A.stencil accelMeanStencil A.clamp arr | ||
| -- | ||
| -- Where 'arr' is an Accelerate 2D array of type 'A.Array A.DIM2 Double'. | ||
| accelMeanStencil :: A.Stencil3x3 Double -> A.Exp Double | ||
| accelMeanStencil ((tl, tc, tr), | ||
| (ml, mc, mr), | ||
| (bl, bc, br)) = (tl + tc + tr + ml + mc + mr + bl + bc + br) / 9 | ||
|
|
||
| size :: Int | ||
| size = 10000000 | ||
|
|
||
| main :: IO () | ||
| main = do | ||
| allArgs <- getArgs | ||
| case allArgs of | ||
| (testType:rest) -> | ||
| -- 'withArgs' replaces the global command-line arguments | ||
| -- for the duration of the provided IO action. | ||
| withArgs rest $ case testType of | ||
| "map" -> runMapBenchmarks | ||
| "dot" -> runDotBenchmarks | ||
| "stencil-massiv" -> runMassivOnly -- Run only the Massiv stencil benchmark with +RTS -s flag | ||
| "stencil-accel" -> runAccelOnly -- Run only the Accelerate stencil benchmark with +RTS -s flag | ||
| "stencil" -> runStencilBenchmarks | ||
| _ -> putStrLn "Unknown test type. Use 'map', 'dot', or 'stencil'." | ||
| _ -> putStrLn "Usage: cabal run massiv-vs-accelerate -- [map|dot|stencil] [criterion flags]" | ||
|
|
||
| runMapBenchmarks :: IO () | ||
| runMapBenchmarks = do | ||
| let massivArr = M.makeArray M.Par (M.Sz1 size) id :: M.Array M.P M.Ix1 Int | ||
| let accelArr = A.fromList (A.Z A.:. size) [0..size-1] :: A.Vector Int | ||
|
|
||
| defaultMain [ | ||
| bgroup "Map (+1)" [ | ||
| bench "Massiv (CPU Parallel)" $ | ||
| nf (M.computeAs M.P . M.map (+1)) massivArr, | ||
| bench "Accelerate (LLVM Native)" $ | ||
| whnf (Native.run . A.map (+1) . A.use) accelArr | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. For lines 75, 92,113:
Native.run . A.map (+1) . A.use rebuilds the expression graph and does a cache lookup on every Criterion iteration. Accelerate's own guidance is to hoist that out with runN/run1: let mapAccel = Native.runN (A.map (+1)) -- compiled once, outside the timed loop
...
bench "Accelerate (LLVM Native)" $ nf mapAccel accelArrAs written, the Accelerate numbers include front-end overhead that has no Massiv counterpart. For a post whose whole point is a head-to-head comparison, this is the finding I'd fix first.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. A.map (+1) was just a simple example, the blog post is based on the stencil operation |
||
| ] | ||
| ] | ||
|
|
||
| runDotBenchmarks :: IO () | ||
| runDotBenchmarks = do | ||
| let m1 = M.makeArray M.Par (M.Sz1 size) (const 2) :: M.Array M.P M.Ix1 Int | ||
| let m2 = M.makeArray M.Par (M.Sz1 size) (const 3) :: M.Array M.P M.Ix1 Int | ||
|
|
||
| let a1 = A.fromList (A.Z A.:. size) (replicate size 2) :: A.Vector Int | ||
| let a2 = A.fromList (A.Z A.:. size) (replicate size 3) :: A.Vector Int | ||
|
|
||
| defaultMain [ | ||
| bgroup "Dot Product" [ | ||
| bench "Massiv (CPU Parallel)" $ | ||
| nf (\(x, y) -> M.sum (M.zipWith (*) x y)) (m1, m2), | ||
| bench "Accelerate (LLVM Native)" $ | ||
| whnf (\(x, y) -> Native.run (A.fold (+) 0 (A.zipWith (*) (A.use x) (A.use y)))) (a1, a2) | ||
| ] | ||
| ] | ||
|
|
||
| runStencilBenchmarks :: IO () | ||
| runStencilBenchmarks = do | ||
| let side = 3162 | ||
| let sz = M.Sz2 side side | ||
|
|
||
| -- Massiv Setup | ||
| let mArr = M.makeArray M.Par sz (\(i M.:. j) -> fromIntegral (i + j)) :: M.Array M.P M.Ix2 Double | ||
|
|
||
| -- Accelerate Setup | ||
| let aArr = A.fromList (A.Z A.:. side A.:. side) [0..(fromIntegral (side*side - 1))] :: A.Array A.DIM2 Double | ||
|
|
||
| defaultMain [ | ||
| bgroup "3x3 Mean Blur" [ | ||
| bench "Massiv (Stencil)" $ | ||
| nf (M.computeAs M.P . M.mapStencil (M.Fill 0.0) massivMeanStencil) mArr, | ||
|
|
||
| bench "Accelerate (LLVM Stencil)" $ | ||
| nf (Native.run . A.stencil accelMeanStencil A.clamp . A.use) aArr | ||
|
Comment on lines
+102
to
+113
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Two mismatches in the same benchmark: Input data differs: Massiv gets fromIntegral (i + j), Accelerate gets [0 .. side*side-1].
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We care more about how each library uses the arrays not the implementations. |
||
| ] | ||
| ] | ||
|
|
||
| runMassivOnly :: IO () | ||
| runMassivOnly = do | ||
| let side = 3162 | ||
| let sz = M.Sz2 side side | ||
| let mArr = M.makeArray M.Par sz (\(i M.:. j) -> fromIntegral (i + j)) :: M.Array M.P M.Ix2 Double | ||
| defaultMain [ bench "Massiv (Isolated)" $ nf (M.computeAs M.P . M.mapStencil (M.Fill 0.0) massivMeanStencil) mArr ] | ||
|
|
||
| runAccelOnly :: IO () | ||
| runAccelOnly = do | ||
| let side = 3162 | ||
| let aArr = A.fromList (A.Z A.:. side A.:. side) [0..(fromIntegral (side*side - 1))] :: A.Array A.DIM2 Double | ||
| defaultMain [ bench "Accelerate (Isolated)" $ whnf (Native.run . A.stencil accelMeanStencil A.clamp . A.use) aArr ] | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. [Main.hs:113][Main.hs:128] uses The stencil-accel mode exists so you can profile it in isolation and compare against the paired run — but it measures a different amount of work, so the two aren't comparable. Same whnf/nf split appears at [Main.hs:72-75] and [Main.hs:89-92] between the two libraries. Standardize on
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Accelerate doesn't compile the arrays with Haskell it compiles them with target architecture (CPU/GPU), so nf doesn't apply here but whnf does because it evaluates their constructor. |
||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| packages: . | ||
|
|
||
| source-repository-package | ||
| type: git | ||
| location: https://github.com/AccelerateHS/accelerate.git | ||
| tag: 981db031c7573bd38decbf3c123769a051ce9f32 | ||
|
|
||
| source-repository-package | ||
| type: git | ||
| location: https://github.com/AccelerateHS/accelerate-llvm.git | ||
| tag: 1b3acb629cf90b0fbcecb985e4349c3055d70ee2 | ||
| -- This tells Cabal to look inside these specific folders for the .cabal files | ||
| subdir: | ||
| accelerate-llvm | ||
| accelerate-llvm-native | ||
|
|
||
| allow-newer: | ||
| *:bytestring, | ||
| *:base, | ||
| *:template-haskell, | ||
| llvm-hs:llvm-hs-pure, | ||
| llvm-hs-pure:base |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Setup work leaks into the first timed iteration
[Main.hs:67-68][Main.hs:81-85],[Main.hs:102-105]
These are lazy let bindings, so building the 10M-element arrays (including replicate size 2 as a lazy list) happens inside the benchmark loop on first evaluation. Use Criterion's env, which forces setup to NF before timing starts: