Fast correlations with turbocor

We often want to calculate Pearson correlation between different datasets, for example, we have used it to identify the hosts of different phages. Often, we want to calculate Pearson on really large matrices, and so our usual solution is to use crappy code and be patient!

However, recently Daniel Jones released turbocor, a fast, rust-based implementation, of pairwise Pearson correlations, and so we are intrigued to work with it. Here is a brief guide to making correlations using turbocor.

Installing turbocor

Of course, we want to install it using conda but there are a couple of simple gotchas that are easy to overcome. Here are some step by step instructions

Specifically, hdf5-sys requires hdf5=1.10.1 at the moment, and balks at hdf5=1.12.1 (the current version in anaconda).

Step 1: Create the conda environment and install hdf5

mamba create -n turbocor rust hdf5=1.10.1

Step 2: Activate the environment

conda activate turbocor

Step 3: Set your environment variables

You will need to set this to the base of your conda environment. Change the path as appropriate for your conda installation. See the hdf5 conda installation for more details.

export HDF5_DIR=$HOME/miniconda3/envs/turbocor
export RUSTFLAGS="-C link-args=-Wl,-rpath,$HOME/miniconda3/envs/turbocor/lib"

Step 4: Build the release

cargo build --release

and now the executable will be in the target/release directory, so you can either add this to your path (e.g. PATH=$PATH:$PWD/target/release) or remember the location (e.g. TURBOCOR=$PWD//target/release).

Using turbocor

You need your data conceptually in a matrix. Turbocor will read an hdf5 format file with a dataset tag that points to a two-dimensional matrix.

Step 1: Convert your data to hdf5 format

If you have your data in a tab-separated (or even comma-separated) text file, you can use matrix_to_h5.py to convert that data into the hdf5 format file that you need.

For example, here is how to convert the matrix to an hdf5 file. In this example, matrix.tsv is a tab-separated matrix file that has a header row. It outputs the data into matrix.h5 with a dataset tag mydata and outputs a separate file with the column names (taken from the first column) into indices.tsv.

python3 /home/edwa0468/GitHubs/EdwardsLab/h5py/matrix_to_h5.py --file matrix.tsv --output matrix.h5 --header --dataset mydata --indexfile indices.tsv

Step 2: Run turbocor

Next, we run turbocor on the matrix.h5 file. Here we use the $TURBOCOR path we set earlier, and output the correlations to the matrix.cor file.

$TURBOCOR/turbor compute --dataset mydata matrix.h5 matrix.cor

We convert that to a comma-separated list of row indices and their correlation coefficients:

Step 3: Extract the coefficients

$TURBOCOR/turbor topk 1000 matrix.cor

You can use the indices.tsv file we created in step 1 to identify the row names of things that correlate with each other.

Enjoy!