Liquid-LLM: Breaking the Low-to-Moderate Sparsity Barrier for Efficient LLM Inference

1. Clone this project.

git clone --recursive https://github.com/ParCIS/Liquid-LLM.git

• Requirements:

• Ubuntu 16.04+
• cmake >= 3.29
• CUDA >= 11.8
• one H100 PCIe GPU and one NVIDIA RTX5090 GPU.

2. Environment Setup.

Conda environments need to be set up on machines with H100 PCIe and RTX5090 GPUs following the steps below.

2.1 Install via Conda.

• 2.1.1 Install conda on system. (Toturial).
• 2.1.2 Create a conda environment:

conda create -n env_name python=3.9

• 2.1.3 Install PyTorch (Toturial):

conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia 

3. Install Liquid-LLM in kernel benchmark.

cd $LiquidLLM_HOME/kernel_benchmark/
source test_env

• Build Sputnik.

cd $LiquidLLM_HOME/third_party/
source build_sputnik.sh

• Build SparTA.

cd $LiquidLLM_HOME/third_party/
source preparse_cusparselt.sh

The libSpMM_API.so and SpMM_API.cuh will be available for easy integration after:

cd $LiquidLLM_HOME/kernel_benchmark/
# Choose the target GPU explicitly:
#   h100    : build with the H100 source path and enable WGMMA/v7 kernels.
#   rtx5090 : build with the RTX 5090 source path and disable WGMMA/v7 kernels.
source myinstall.sh h100
# or
source myinstall.sh rtx5090

4. Running Liquid-LLM in kernel benchmark.

cd $LiquidLLM_HOME/kernel_benchmark
source test_env

# Run the kernel benchmark and baselines from the test directory.
cd test
# Running all N values can take about 8 hours.
# For a quick check, edit launch.py and uncomment the N filter, for example:
# for row in rows:
#   if row["N"] != 8:
#       continue
# Change 8 to 16 or 32 if you only want to benchmark that N value.
python launch.py
python launch_sparta.py # About 2 hours
python launch_cusparse.py # About 1 hours
python launch_sputnik.py # About 1 hours

# Merge the raw throughput CSV files and compute speedups.
cd ../result/kernel
python all_process.py

4.1 Reproduce Figure 11

Check the raw throughput CSV files in $LiquidLLM_HOME/kernel_benchmark/result/kernel/. all_process.py produces merged and speedup CSV files, and plot.py reproduces Figure 11.

cd $LiquidLLM_HOME/kernel_benchmark/result/kernel
python plot.py

4.2 Reproduce Figure 12

cd $LiquidLLM_HOME/kernel_benchmark/result/ablation_study
python plot.py

4.3 Reproduce Table 1

Profiling of micro-architectural metrics for sparse kernels. Check the profile_Qwen.ncu-rep using Nsight Compute.

cd $LiquidLLM_HOME/kernel_benchmark
/usr/local/cuda-12.6/bin/ncu --export ./profile_Qwen ./spmm_test 5120 17408 8 60 5 0 #M,K,N,Sparsity,SplitK,CUDA_VISIBLE_DEVICES

5. Running End-to-end model.

5.1 Building

Build FasterTransformer with the Liquid-LLM integration. Start from a clean FasterTransformer-main tree, apply the Liquid-LLM patch, and then build it with the H100 architecture flag.

cd $LiquidLLM_HOME/third_party/FasterTransformer-main

# Apply the Liquid-LLM changes to the clean FasterTransformer source tree.
git apply ../ft_liquidllm_final.patch

# Build FasterTransformer with Liquid-LLM enabled.
mkdir -p build
cd build
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DLiquid_LLM=ON -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

For other end-to-end baselines, keep the same build directory workflow and replace the CMake command with one of the following:

# Standard FasterTransformer: use cuBLAS for all MatMuls.
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DFLASH_LLM=OFF -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

# FasterTransformer with Flash-LLM.
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DFLASH_LLM=ON -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

# FasterTransformer with SpInfer.
cmake -DSM=90a -DCMAKE_BUILD_TYPE=Release -DBUILD_MULTI_GPU=ON -DSpInfer=ON -DCMAKE_CXX_COMPILER=mpicxx ..
make -j

5.2 Downloading & Converting OPT models

The following commands use OPT-13B as an example. Replace opt-13b with the target OPT model name if you evaluate a different OPT model size.

Download the Hugging Face checkpoint:

cd $LiquidLLM_HOME/end2end_inference/models
git lfs install
git clone https://huggingface.co/facebook/opt-13b
cd opt-13b
git lfs pull --include="pytorch_model*"

Convert the PyTorch checkpoint to the FasterTransformer format:

cd $LiquidLLM_HOME/end2end_inference/ft_tools
python huggingface_opt_convert_Phase1.py \
    -i $LiquidLLM_HOME/end2end_inference/models/opt-13b \
    -o $LiquidLLM_HOME/end2end_inference/models/opt-13b/c-model \
    -i_g 1 \
    -weight_data_type fp16 \
    -p 64

Here, -i_g is the tensor-parallel GPU number used for inference, and -p is the number of CPU threads used during conversion. Keep -i_g consistent with the GPU count used later by mpirun.

Run the Phase 2 preprocessing script to generate the sparse model files. The scripts below correspond to 1, 2, and 4 GPU tensor-parallel settings:

cd $LiquidLLM_HOME/end2end_inference/ft_tools

# 1 GPU
bash prepare.sh h100

# 2 GPUs
bash prepare-2.sh h100

# 4 GPUs
bash prepare-4.sh h100

To change the sparsity ratio, modify p in the corresponding Phase 2 script, for example huggingface_opt_convert_Phase2_liquidllm.py, huggingface_opt_convert_Phase2_flashllm.py, or huggingface_opt_convert_Phase2_spinfer.py. Setting p=0.3 keeps 30% of the weights and produces 70% sparsity.

5.3 Configuration

Initialize the environment.

source init_env
echo $LiquidLLM_HOME

Before running inference, update the FasterTransformer config file under:

$LiquidLLM_HOME/third_party/FasterTransformer-main/examples/cpp/multi_gpu_gpt/

The provided config files are named by request batch size, for example gpt_config_8.ini, gpt_config_16.ini, gpt_config_32.ini, and gpt_config_64.ini. You can edit one of them directly, or copy it to a new file before changing the fields below.

The key fields are:

model_name=opt_13B
tensor_para_size=1
model_dir=$LiquidLLM_HOME/end2end_inference/models/opt-13b/c-model/1-gpu-liquidllm
request_batch_size=8

Please keep the following settings consistent:

• model_name: use the model size you converted, such as opt_13B.
• tensor_para_size: set this to the same GPU number used by -i_g during conversion and by mpirun -n during inference.
• model_dir: point this to the converted model directory generated in Section 5.2.
• request_batch_size: match the config file or the batch size you want to evaluate.

Choose the model_dir suffix according to the backend:

# Liquid-LLM
model_dir=$LiquidLLM_HOME/end2end_inference/models/opt-13b/c-model/1-gpu-liquidllm

# Flash-LLM
model_dir=$LiquidLLM_HOME/end2end_inference/models/opt-13b/c-model/1-gpu-flashllm

# SpInfer
model_dir=$LiquidLLM_HOME/end2end_inference/models/opt-13b/c-model/1-gpu-spinfer

# Standard FasterTransformer
model_dir=$LiquidLLM_HOME/end2end_inference/models/opt-13b/c-model/1-gpu

For 2-GPU or 4-GPU inference, replace 1-gpu-* with 2-gpu-* or 4-gpu-*, and set tensor_para_size to 2 or 4.

5.4 Running Inference

Run the FasterTransformer end-to-end example from the patched source tree:

cd $LiquidLLM_HOME/third_party/FasterTransformer-main
bash run_all.sh

run_all.sh runs the configured batch sizes in sequence. By default, it uses gpt_config_8.ini, gpt_config_16.ini, and gpt_config_32.ini, together with examples/cpp/multi_gpu_gpt/start_ids_64.csv. It also exports LD_LIBRARY_PATH=$LiquidLLM_HOME/build:$LD_LIBRARY_PATH, so the executable can find libSpMM_API.so.

To run a single config manually, use:

cd $LiquidLLM_HOME/third_party/FasterTransformer-main
export LD_LIBRARY_PATH=$LiquidLLM_HOME/build:$LD_LIBRARY_PATH

mpirun -n 1 --allow-run-as-root -x CUDA_VISIBLE_DEVICES -x LD_LIBRARY_PATH \
    ./build/bin/multi_gpu_gpt_example \
    ./examples/cpp/multi_gpu_gpt/gpt_config_8.ini \
    ./examples/cpp/multi_gpu_gpt/start_ids_64.csv

The -n value in mpirun should match tensor_para_size in the selected config file. For 2-GPU or 4-GPU inference, use mpirun -n 2 or mpirun -n 4 and the corresponding model directory generated in Section 5.2.

For profiling with Nsight Systems or Nsight Compute, keep the same executable, config file, and LD_LIBRARY_PATH settings. The SpInfer, Flash-LLM, Liquid-LLM, and standard FasterTransformer baselines are run in the same way; only the compiled backend and model_dir in the config file change.

5.5 Reproduce Figure 14

The result directory is:

$LiquidLLM_HOME/end2end_inference/result/opt-13b

Each baseline should provide one CSV file with the same format:

batch_size,config,time_ms
8,./examples/cpp/multi_gpu_gpt/gpt_config_8.ini,7316.60
16,./examples/cpp/multi_gpu_gpt/gpt_config_16.ini,8277.65
32,./examples/cpp/multi_gpu_gpt/gpt_config_32.ini,10191.47

The expected file names are:

end2end_times_liquid.csv
end2end_times_spinfer.csv
end2end_times_flashllm.csv
end2end_times_deepspeed.csv
end2end_times_fastertransformer.csv

For Liquid-LLM, run:

cd $LiquidLLM_HOME/third_party/FasterTransformer-main
RESULT_CSV=$LiquidLLM_HOME/end2end_inference/result/opt-13b/end2end_times_liquid.csv \
    bash run_all.sh

For SpInfer, Flash-LLM, and standard FasterTransformer, use the same workflow: build the corresponding backend, update model_dir in the selected gpt_config_*.ini, and set RESULT_CSV to the matching output file. For example:

# SpInfer
RESULT_CSV=$LiquidLLM_HOME/end2end_inference/result/opt-13b/end2end_times_spinfer.csv \
    bash run_all.sh

# Flash-LLM
RESULT_CSV=$LiquidLLM_HOME/end2end_inference/result/opt-13b/end2end_times_flashllm.csv \
    bash run_all.sh

# Standard FasterTransformer
RESULT_CSV=$LiquidLLM_HOME/end2end_inference/result/opt-13b/end2end_times_fastertransformer.csv \
    bash run_all.sh

For DeepSpeed, record its measured latency with the same CSV schema and save it as end2end_times_deepspeed.csv.

cd $LiquidLLM_HOME/end2end_inference/ds_scripts
pip install -r requirements.txt
deepspeed --num_gpus 1 $LiquidLLM_HOME/end2end_inference/ds_scripts/inference-test.py --ds_inference --greedy --use_kernel --name $LiquidLLM_HOME/end2end_inference/models/opt-13b --batch_size 8

Tokens/s = batch_size * (64 + 512) / (time_ms / 1000)

Here, 64 is the prefill length and 512 is the decode length.

Generate Figure 14:

cd $LiquidLLM_HOME/end2end_inference/result/opt-13b
python plot_13b.py

The output figure is:

$LiquidLLM_HOME/end2end_inference/result/opt-13b/opt_13b_1_50.png