Multi-effector decoding

This repository contains supplementary model and figure-generation code accompanying Brosler*, Liu*, Silva*, et al. "Multi-effector neuroprosthesis for restoring gestural and speech communication in severe paralysis", 2026.

The included notebooks demonstrate the core offline pipelines used to train and evaluate the speech and gesture decoders.

It includes two example modeling notebooks:

1. notebooks/offline_eval_on_simultaneous.ipynb trains speech and gesture decoders on isolated plus simultaneous data, then evaluates performance on held-out simultaneous trials using nested cross-validation and model ensembling.

2. notebooks/rest_cross_modality_training_cleaned.ipynb shows how to build a rest class using both true rest trials and opposite-modality trials, then evaluates false positive rates separately for true rest and opposite-modality test trials.

These notebooks are intended to provide runnable examples of the main modeling pipelines that support the manuscript’s decoder analyses. The decoding analyses in Figs. 3 and 4 of the manuscript use the same nested cross-validation and model ensembling approach demonstrated in offline_eval_on_simultaneous.ipynb, although the exact training data, test data, metrics, and plotting code differ depending on the figure. The rest_cross_modality_training_cleaned.ipynb notebook demonstrates the cross-modality rest-class training strategy used for the false-positive-rate analyses in Fig. 4, where true rest trials and opposite-modality trials are both labeled as rest to reduce false positives when speech and gesture decoders are run in parallel during real-time use. The real-time experiments in Fig. 5 used models that were trained using the same architecture and training code described in this repository.

Repository structure

configs/                    Model and analysis configuration files
data/                       Data used by the analysis and figure notebooks
figures/                    Figure-generation notebooks
multi_effector_decoding/    Core preprocessing, model, training, evaluation, plotting, and utility code
notebooks/                  Example analysis notebooks
scripts/                    Higher-level offline evaluation functions
requirements.txt            Python environment specification

Setup

Create and activate a Python environment (e.g. through conda), then install the required Python packages from requirements.txt:

conda create -n multi-effector-decoding python=3.8
conda activate multi-effector-decoding
pip install -r requirements.txt

Example data

The example notebooks expect data files in a data/ directory, which should appear alongside folders in this repository:

data/bravo6_speech.pkl
data/bravo6_gesture.pkl
data/bravo6_simultaneous.pkl
data/bravo6_rest.pkl
data/bravo1r_speech.pkl
data/bravo1r_gesture.pkl
data/bravo1r_simultaneous.pkl
data/bravo1r_rest.pkl

Each pickle file contains arrays stored with keys such as X, y, and y_text, where X has shape:

trials x time x channels

y contains the integer class labels for each trial, and y_text contains the corresponding text labels for those same trials.

Large data files for model training are not tracked in this repository, only smaller data files used for figure generation. Data required by the notebooks can be downloaded from the Zenodo repository linked here and placed in the data/ directory. More information about the example data is included in the notebooks in notebooks/.

Running the notebooks

After setting up the environment and placing the example data files in data/, notebooks can be run using Jupyter, an IDE, or similar method:

notebooks/offline_eval_on_simultaneous.ipynb
notebooks/rest_cross_modality_training_cleaned.ipynb

Generating figures

Figures from the manuscript can be generated by running the corresponding figure notebooks in figures/ using Jupyter, an IDE, or a similar method. These notebooks are typically organized by loading and processing the source data in the data/ directory, defining functions for individual figure panels, and then assembling the overall figure. Source data required by the figure notebooks are already provided in the data/ directory.

Configuration

Model and analysis parameters are stored in JSON files under configs/:

configs/bravo6_speech_config.json
configs/bravo6_gesture_config.json
configs/bravo6_speech_config_no_rest.json
configs/bravo6_gesture_config_no_rest.json

Important fields include:

• device: cuda or cpu

• test_numfolds and val_numfolds: outer and inner cross-validation folds

• model_params: CNN-RNN architecture parameters

• inds: time window used for training/evaluation

• jits: time-jitter augmentation values

• fig_save_path: optional directory for saved figures

• model_save_path: optional directory for saved model weights

• label_mapping: mapping from simultaneous labels to speech or gesture labels

Outputs

If fig_save_path is set, the code saves confusion matrices, ensemble confusion matrices, accuracy boxplots, and false-positive-rate plots. If model_save_path is set, trained model state dictionaries are saved by test fold.