Upload folder using huggingface_hub

LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2022 ai4nucleome
+Copyright (c) 2024 ai4nucleome
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

README.md

@@ -19,101 +19,12 @@ tags:
    <!-- <img src="https://img.shields.io/badge/dependencies-tested-green"> -->
 </a>  
 
+### See https://github.com/ai4nucleome/Polaris for more details.
 
 🌟 **Polaris** is a versatile and efficient command line tool tailored for rapid and accurate chromatin loop detectionfrom from contact maps generated by various assays, including bulk Hi-C, scHi-C, Micro-C, and DNA SPRITE. Polaris is particularly well-suited for analyzing **sparse scHi-C data and low-coverage datasets**.
 
-<div style="text-align: center;">
-    <img src="./doc/Polaris.png" alt="Polaris Model" title="Polaris Model" width="600">
-</div>
-
-
-- Using examples for single cell Hi-C and bulk cell Hi-C loop annotations are under [**example folder**](https://github.com/ai4nucleome/Polaris/tree/master/example).
-- The scripts and data to **reproduce our analysis** can be found at: [**Polaris Reproducibility**](https://zenodo.org/records/14294273).
-
-> ❗️<b>NOTE❗️:</b> We suggest users run Polaris on <b>GPU</b>. 
-> You can run Polaris on CPU for loop annotations, but it is much slower than on GPU. 
-
-> ❗️**NOTE❗️:** If you encounter a `CUDA OUT OF MEMORY` error, please:
-> - Check your GPU's status and available memory.
-> - Reduce the --batchsize parameter. (The default value of 128 requires approximately 36GB of CUDA memory. Setting it to 24 will reduce the requirement to less than 10GB.)
-
-## Documentation
-📝 **Extensive documentation** can be found at: [Polaris Doc](https://nucleome-polaris.readthedocs.io/en/latest/).
-
-## Installation
-Polaris is developed and tested on Linux machines with python3.9 and relies on several libraries including pytorch, scipy, etc. 
-We **strongly recommend** that you install Polaris in a virtual environment.
-
-We suggest users using [conda](https://anaconda.org/) to create a virtual environment for it (It should also work without using conda, i.e. with pip). You can run the command snippets below to install Polaris:
-
-```bash
-git clone https://github.com/ai4nucleome/Polaris.git
-cd Polaris
-conda create -n polaris python=3.9
-conda activate polaris
-```
--------
-### ❗️Important Note❗️: Downloading Polaris Network  Weights
-
-The Polaris repository utilizes Git Large File Storage (Git-LFS) to host its pre-trained model weight files. Standard `git clone` operations **will not** automatically download these large files unless Git-LFS is installed and configured. 
-
-To resolve this, please follow one of the methods below:
-
-#### Method 1: Manual Download via Browser
-
-1. Directly download the pre-trained model weights (`sft_loop.pt`) from the [Polaris model directory](https://github.com/ai4nucleome/Polaris/blob/master/polaris/model/sft_loop.pt).
-2. Save the file to the directory:
-    ```bash
-    Polaris/polaris/model/
-    ```
-#### Method 2: Install Git-LFS
-1. Install Git-LFS by following the official instructions: [Git-LFS Installation Guide](https://git-lfs.com/).
-
-2. After installation, either:
-
-    Re-clone the repository:
-
-    ```bash
-    git clone https://github.com/ai4nucleome/Polaris.git
-    ```
-    OR, if the repository is already cloned, run:
-
-    ```bash
-    git lfs pull
-    ```
-    This ensures all large files, including model weights, are retrieved.
-----------
-
-Install [PyTorch](https://pytorch.org/get-started/locally/) as described on their website. It might be the following command depending on your cuda version:
-
-```bash
-pip install torch==2.2.2 torchvision==0.17.2 torchaudio==2.2.2 --index-url https://download.pytorch.org/whl/cu121
-```
-Install Polaris:
-```bash
-pip install --use-pep517 --editable .
-```
-If fail, please try `python setup build` and `python setup install` first.
-
-The installation requires network access to download libraries. Usually, the installation will finish within 5 minutes. The installation time is longer if network access is slow and/or unstable.
-
-## Quick Start for Loop Annotation
-```bash
-polaris loop pred -i [input mcool file] -o [output path of annotated loops]
-```
-It outputs predicted loops from the input contact map at 5kb resolution.
-### output format
-It contains tab separated fields as follows:
-```
-Chr1    Start1    End1    Chr2    Start2    End2    Score
-```
-|     Field     |                                  Detail                                 |
-|:-------------:|:-----------------------------------------------------------------------:|
-|   Chr1/Chr2   | chromosome names                                                        |
-| Start1/Start2 | start genomic coordinates                                               |
-|   End1/End2   | end genomic coordinates (i.e. End1=Start1+resol)                        |
-|     Score     | Polaris's loop score [0~1]                                              | 
-
+## [📝Documentation](https://nucleome-polaris.readthedocs.io/en/latest/)
+ **Detailed documentation** can be found at: [Polaris Doc](https://nucleome-polaris.readthedocs.io/en/latest/).
 
 ## Citation:
 Yusen Hou, Audrey Baguette, Mathieu Blanchette*, & Yanlin Zhang*. __A versatile tool for chromatin loop annotation in bulk and single-cell Hi-C data__. _bioRxiv_, 2024. [Paper](https://doi.org/10.1101/2024.12.24.630215)
@@ -128,6 +39,6 @@ Yusen Hou, Audrey Baguette, Mathieu Blanchette*, & Yanlin Zhang*. __A versatile
 ```
 
 ## 📩 Contact
-A GitHub issue is preferable for all problems related to using Polaris. 
+A [GitHub issue](https://github.com/ai4nucleome/Polaris/issues) is preferable for all problems related to using Polaris. 
 
 For other concerns, please email Yusen Hou or Yanlin Zhang ([email protected],  [email protected]).

example/CLI_walkthrough.ipynb

@@ -27,8 +27,8 @@
       "\n",
       "  Polaris\n",
       "\n",
-      "  A Versatile Tool for Chromatin Loop Annotation in Bulk and Single-cell Hi-C\n",
-      "  Data\n",
+      "  A Versatile Framework for Chromatin Loop Annotation in Bulk and Single-cell\n",
+      "  Hi-C Data\n",
       "\n",
       "Options:\n",
       "  --help  Show this message and exit.\n",
@@ -80,10 +80,10 @@
       "  --help  Show this message and exit.\n",
       "\n",
       "Commands:\n",
-      "  dev    *development function* Coming soon...\n",
-      "  pool   Call loops from loop candidates by clustering\n",
-      "  pred   Predict loops from input contact map directly\n",
-      "  score  Predict loop score for each pixel in the input contact map\n"
+      "  pool     Call loops from loop candidates by clustering\n",
+      "  pred     Predict loops from input contact map directly\n",
+      "  score    Predict loop score for each pixel in the input contact map\n",
+      "  scorelf  *development* Score Pixels for Very Large mcool (>30GB) ...\n"
      ]
     }
    ],
@@ -102,7 +102,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 6,
+   "execution_count": 3,
    "metadata": {},
    "outputs": [
     {
@@ -120,6 +120,7 @@
       "\n",
       "Commands:\n",
       "  cool2bcool  covert a .mcool file to a .bcool file\n",
+      "  depth       Calculate intra-chromosomal contacts with bin distance >=...\n",
       "  pileup      2D pileup contact maps around given foci\n"
      ]
     }

example/README.md

@@ -10,6 +10,11 @@ You can re-run **Polaris** to reproduce these results by following the commands
 
 ## Loop Prediction on GM12878 (250M Valid Read Pairs)
 
+You can download example data from the [Hugging Face repo of Polaris](https://huggingface.co/rr-ss/Polaris/resolve/main/example/loop_annotation/GM12878_250M.bcool?download=true) by runing:
+```bash
+wget https://huggingface.co/rr-ss/Polaris/resolve/main/example/loop_annotation/GM12878_250M.bcool?download=true -O "./loop_annotation/GM12878_250M.bcool"
+```
+And run following code to annotate loops from the example data:
 ```bash
 polaris loop pred --chrom chr15,chr16,chr17 -i ./loop_annotation/GM12878_250M.bcool -o ./loop_annotation/GM12878_250M_chr151617_loops.bedpe
 ```

polaris/loop.py

@@ -36,7 +36,7 @@ def rhoDelta(data,resol,dc,radius):
         except ValueError as e:
             if "Found array with 0 sample(s)" in str(e):
                 print("#"*88,'\n#')
-                print("#\033[91m Error!!! The data is too sparse. Please increase the value of: [t]\033[0m\n#")
+                print("#\033[91m Error!!! The data is too sparse. Please decrease the value of: [t]\033[0m\n#")
                 print("#"*88,'\n')
                 sys.exit(1)
             else:
@@ -76,12 +76,11 @@ def rhoDelta(data,resol,dc,radius):
     else:
         data['rhos']=[]
         data['deltas']=[]
-
     return data
 
 def pool(data,dc,resol,mindelta,t,output,radius,refine=True):
     ccs = set(data.iloc[:,0])
-    
+
     if data.shape[0] == 0:
         print("#"*88,'\n#')
         print("#\033[91m Error!!! The file is empty. Please check your file.\033[0m\n#")

polaris/loopLF.py

@@ -0,0 +1,218 @@
+import torch
+import click
+import cooler
+import warnings
+import numpy as np
+from torch import nn
+from tqdm import tqdm
+from torch.cuda.amp import autocast
+from importlib_resources import files
+from polaris.utils.util_loop import bedpewriter
+from polaris.model.polarisnet import polarisnet
+from scipy.sparse import coo_matrix
+from scipy.sparse import SparseEfficiencyWarning
+warnings.filterwarnings("ignore", category=SparseEfficiencyWarning)
+
+def getLocal(mat, i, jj, w, N):
+    if i >= 0 and jj >= 0 and i+w <= N and jj+w <= N:
+        mat = mat[i:i+w,jj:jj+w].toarray()
+        # print(f"global: {mat.shape}")
+        return mat[None,...]
+    # pad_width = ((up, down), (left, right))
+    slice_pos = [[i, i+w], [jj, jj+w]]
+    pad_width = [[0, 0], [0, 0]]
+    if i < 0:
+        pad_width[0][0] = -i
+        slice_pos[0][0] = 0
+    if jj < 0:
+        pad_width[1][0] = -jj
+        slice_pos[1][0] = 0
+    if i+w > N:
+        pad_width[0][1] = i+w-N
+        slice_pos[0][1] = N
+    if jj+w > N:
+        pad_width[1][1] = jj+w-N
+        slice_pos[1][1] = N
+    _mat = mat[slice_pos[0][0]:slice_pos[0][1],slice_pos[1][0]:slice_pos[1][1]].toarray()
+    padded_mat = np.pad(_mat, pad_width, mode='constant', constant_values=0)
+    # print(f"global: {padded_mat.shape}",slice_pos, pad_width)
+    return padded_mat[None,...]
+
+def upperCoo2symm(row,col,data,N=None):
+    # print(np.max(row),np.max(col),N)
+    if N:
+        shape=(N,N)
+    else:
+        shape=(row.max() + 1,col.max() + 1)
+
+    sparse_matrix = coo_matrix((data, (row, col)), shape=shape)
+    symm = sparse_matrix + sparse_matrix.T
+    diagVal = symm.diagonal(0)/2
+    symm = symm.tocsr()
+    symm.setdiag(diagVal)
+    return symm
+
+def processCoolFile(coolfile, cchrom, raw=False):
+    extent = coolfile.extent(cchrom)
+    N = extent[1] - extent[0]
+    if raw:
+        ccdata = coolfile.matrix(balance=False, sparse=True, as_pixels=True).fetch(cchrom)
+        v='count'
+    else:
+        ccdata = coolfile.matrix(balance=True, sparse=True, as_pixels=True).fetch(cchrom)
+        v='balanced'
+    ccdata['bin1_id'] -= extent[0]
+    ccdata['bin2_id'] -= extent[0]
+        
+    ccdata['distance'] = ccdata['bin2_id'] - ccdata['bin1_id']
+    d_means = ccdata.groupby('distance')[v].transform('mean')
+    ccdata[v] = ccdata[v].fillna(0)
+    
+    ccdata['oe'] = ccdata[v] / d_means
+    ccdata['oe'] = ccdata['oe'].fillna(0)
+    ccdata['oe'] = ccdata['oe'] / ccdata['oe'].max()
+    oeMat = upperCoo2symm(ccdata['bin1_id'].ravel(), ccdata['bin2_id'].ravel(), ccdata['oe'].ravel(), N)
+           
+    return oeMat, N
+
+@click.command()
+@click.option('-b','--batchsize', type=int, default=128, help='Batch size [128]')
+@click.option('-C','--cpu', type=bool, default=False, help='Use CPU [False]')
+@click.option('-G','--gpu', type=str, default=None, help='Comma-separated GPU indices [auto select]')
+@click.option('-c','--chrom', type=str, default=None, help='Comma separated chroms [all autosomes]')
+@click.option('-t','--threshold', type=float, default=0.5, help='Loop Score Threshold [0.5]')
+@click.option('-s','--sparsity', type=float, default=0.9, help='Allowed sparsity of submatrices [0.9]')
+@click.option('-md','--max_distance', type=int, default=3000000, help='Max distance (bp) between contact pairs [3000000]')
+@click.option('-r','--resol',type=int,default=5000,help ='Resolution [5000]')
+@click.option('--raw',type=bool,default=False,help ='Raw matrix or balanced matrix')
+@click.option('-i','--input', type=str,required=True,help='Hi-C contact map path')
+@click.option('-o','--output', type=str,required=True,help='.bedpe file path to save loop candidates')
+def scorelf(batchsize, cpu, gpu, chrom, threshold, sparsity, max_distance, resol, input, output, raw, image=224):
+    """ *development* Score Pixels for Very Large mcool (>30GB) ...
+    """
+    print('\npolaris loop scorelf START :) ')
+
+    center_size = image // 2
+    start_idx = (image - center_size) // 2
+    end_idx = (image + center_size) // 2
+    slice_obj_pred = (slice(None), slice(None), slice(start_idx, end_idx), slice(start_idx, end_idx))
+    slice_obj_coord = (slice(None), slice(start_idx, end_idx), slice(start_idx, end_idx))
+
+    loopwriter = bedpewriter(output,resol,max_distance)
+    
+    if cpu:
+        assert gpu is None, "\033[91m QAQ The CPU and GPU modes cannot be used simultaneously. Please check the command. \033[0m\n"
+        gpu = ['None']
+        device = torch.device("cpu")
+        print('Using CPU mode... (This may take significantly longer than using GPU mode.)')
+    else:
+        if torch.cuda.is_available():
+            if gpu is not None:
+                print("Using the specified GPU: " + gpu)
+                gpu=[int(i) for i in gpu.split(',')]
+                device = torch.device(f"cuda:{gpu[0]}")
+            else:
+                gpuIdx = torch.cuda.current_device()
+                device = torch.device(gpuIdx)
+                print("Automatically selected GPU: " + str(gpuIdx))
+                gpu=[gpu]
+        else:
+            device = torch.device("cpu")
+            gpu = ['None']
+            cpu = True
+            print('GPU is not available!')
+            print('Using CPU mode... (This may take significantly longer than using GPU mode.)')
+           
+
+    coolfile = cooler.Cooler(input + '::/resolutions/' + str(resol))
+    modelstate = str(files('polaris').joinpath('model/sft_loop.pt'))
+    _modelstate = torch.load(modelstate, map_location=device.type)
+    parameters = _modelstate['parameters']
+
+    if chrom is None:
+        chrom =coolfile.chromnames
+    else:
+        chrom = chrom.split(',')
+        
+    # for rmchr in ['chrMT','MT','chrM','M','Y','chrY','X','chrX','chrW','W','chrZ','Z']: # 'Y','chrY','X','chrX'
+    #     if rmchr in chrom:
+    #         chrom.remove(rmchr)    
+
+    print(f"Analysing chroms: {chrom}")
+    
+    model = polarisnet(
+            image_size=parameters['image_size'], 
+            in_channels=parameters['in_channels'], 
+            out_channels=parameters['out_channels'],
+            embed_dim=parameters['embed_dim'], 
+            depths=parameters['depths'],
+            channels=parameters['channels'], 
+            num_heads=parameters['num_heads'], 
+            drop=parameters['drop'], 
+            drop_path=parameters['drop_path'], 
+            pos_embed=parameters['pos_embed']
+    ).to(device)
+    model.load_state_dict(_modelstate['model_state_dict'])
+    if not cpu and len(gpu) > 1:
+        model = nn.DataParallel(model, device_ids=gpu) 
+    model.eval()
+        
+    badc=[]
+    chrom_ = tqdm(chrom, dynamic_ncols=True)
+    for _chrom in chrom_:
+        chrom_.desc = f"[Analyzing {_chrom}]"
+
+        oeMat, N = processCoolFile(coolfile, _chrom, raw)
+        start_point = -(image - center_size) // 2
+        joffset = np.repeat(np.linspace(0, image, image, endpoint=False, dtype=int)[np.newaxis, :], image, axis=0)
+        ioffset = np.repeat(np.linspace(0, image, image, endpoint=False, dtype=int)[:, np.newaxis], image, axis=1)
+        data, i_list, j_list = [], [], []
+        count=0
+        for i in range(start_point, N - image - start_point, center_size):
+            for j in range(0, max_distance//resol, center_size):
+                jj = j + i
+                # if jj + w <= N and i + w <= N:
+                _oeMat = getLocal(oeMat, i, jj, image, N)
+                if np.sum(_oeMat == 0) <= (image*image*sparsity):
+                    data.append(_oeMat)
+                    i_list.append(i + ioffset)
+                    j_list.append(jj + joffset)
+
+            while len(data) >= batchsize or (i + center_size > N - image - start_point and len(data) > 0):
+                count += len(data)
+                
+                bin_i = torch.tensor(np.stack(i_list[:batchsize], axis=0)).to(device)
+                bin_j = torch.tensor(np.stack(j_list[:batchsize], axis=0)).to(device)
+                targetX = torch.tensor(np.stack(data[:batchsize], axis=0)).to(device)
+                bin_i = bin_i*resol
+                bin_j = bin_j*resol
+
+                data = data[batchsize:]
+                i_list = i_list[batchsize:]
+                j_list = j_list[batchsize:]
+
+                # print(targetX.shape)
+                # print(bin_i.shape)
+                # print(bin_j.shape)
+
+                with torch.no_grad():
+                    with autocast():
+                        pred = torch.sigmoid(model(targetX.float().to(device)))[slice_obj_pred].flatten()
+                        loop = torch.nonzero(pred>threshold).flatten().cpu()
+                        prob = pred[loop].cpu().numpy().flatten().tolist()
+                        frag1 = bin_i[slice_obj_coord].flatten().cpu().numpy()[loop].flatten().tolist()
+                        frag2 = bin_j[slice_obj_coord].flatten().cpu().numpy()[loop].flatten().tolist()
+
+                    loopwriter.write(_chrom,frag1,frag2,prob)   
+        if count == 0:
+            badc.append(_chrom)
+    
+    if len(badc)==len(chrom):
+        raise ValueError("polaris loop scorelf FAILED :( \nThe '-s' value needs to be increased for more sparse data.")
+    else:
+        print(f'\npolaris loop scorelf FINISHED :)\nLoopscore file saved at {output}')  
+        if len(badc)>0:
+            print(f"But the size of {badc} are too small or their contact matrix are too sparse.\nYou may need to check the data or run these chr respectively by increasing -s.")         
+    
+if __name__ == '__main__':
+    scorelf()

polaris/model/sft_loops.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:cae9e9a28e5c3ff0d328934c066d275371d5301db084a914431198134f66ada2
+size 547572280

polaris/polaris.py

@@ -7,7 +7,7 @@
 
 import click
 from polaris.loopScore import score
-from polaris.loopDev import dev
+from polaris.loopLF import scorelf
 from polaris.loopPool import pool
 from polaris.loop import pred
 from polaris.utils.util_cool2bcool import cool2bcool
@@ -42,7 +42,7 @@ def util():
 
 loop.add_command(pred)
 loop.add_command(score)
-loop.add_command(dev)
+loop.add_command(scorelf)
 loop.add_command(pool)
 
 util.add_command(depth)

polaris/version.py

@@ -1 +1 @@
-__version__ = '1.0.0'
+__version__ = '1.1.0'

setup.py

@@ -10,14 +10,14 @@ Setup script for Polaris.
 A Versatile Framework for Chromatin Loop Annotation in Bulk and Single-cell Hi-C Data.
 """
 
-from setuptools import setup, find_packages
+from setuptools import setup
 
 with open("README.md", "r") as readme:
     long_des = readme.read()
 
 setup(
     name='polaris',
-    version='1.0.1',
+    version='1.1.0',
     author="Yusen HOU, Audrey Baguette, Mathieu Blanchette*, Yanlin Zhang*",
     author_email="[email protected]",
     description="A Versatile Framework for Chromatin Loop Annotation in Bulk and Single-cell Hi-C Data",

setup.sh

@@ -0,0 +1,43 @@
+#!/bin/bash
+
+# Configuration: Model file path and expected SHA-256 checksum
+MODEL_PATH="polaris/model/sft_loop.pt"
+EXPECTED_HASH="cae9e9a28e5c3ff0d328934c066d275371d5301db084a914431198134f66ada2"
+
+# Pre-check: Verify if the model file exists with valid checksum
+if [ -f "$MODEL_PATH" ]; then
+    # Calculate current file hash
+    ACTUAL_HASH=$(sha256sum "$MODEL_PATH" | awk '{print $1}')
+    
+    # Hash validation logic
+    if [ "$ACTUAL_HASH" = "$EXPECTED_HASH" ]; then
+        echo "✅ Valid model file detected, skipping download"
+        pip install --use-pep517 --editable .
+        echo "✅ Polaris installation completed"
+        exit 0
+    else
+        # Security measure: Remove corrupted/invalid file
+        echo "⚠️ Invalid file hash detected, triggering re-download"
+        rm -f "$MODEL_PATH"
+    fi
+fi
+
+# Model download process
+echo "⏳ Downloading model from Hugging Face..."
+wget -O "$MODEL_PATH" "https://huggingface.co/rr-ss/Polaris/resolve/main/polaris/model/sft_loop.pt?download=true"
+
+# Post-download verification
+ACTUAL_HASH=$(sha256sum "$MODEL_PATH" | awk '{print $1}')
+if [ "$ACTUAL_HASH" != "$EXPECTED_HASH" ]; then
+    # Error handling for failed verification
+    rm -f "$MODEL_PATH"
+    echo "❌ Download failed: Checksum mismatch (Actual: $ACTUAL_HASH)"
+    echo "Manual download required:"
+    echo "wget -O polaris/model/sft_loop.pt \"https://huggingface.co/rr-ss/Polaris/resolve/main/polaris/model/sft_loop.pt?download=true\""
+    exit 1
+else
+    # Success workflow
+    pip install --use-pep517 --editable .
+    echo "✅ Model saved to: $MODEL_PATH"
+    echo "✅ Polaris installed successfully"
+fi