-
Notifications
You must be signed in to change notification settings - Fork 7
/
README.txt
1417 lines (1063 loc) · 59.4 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
---
title: "UniverSC: Single-cell processing across technologies"
author: "S. Thomas Kelly^1†^, Kai Battenberg^1,2†^, Makoto Hayashi^2^, Aki Minoda^1^ <br> ^1^ RIKEN Center for Integrative Medical Sciences, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama <br> ^2^ RIKEN Center for Sustainable Resource Sciences, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama <br> † These authors contributed equally to this work"
affiliations:
- name: "RIKEN Center for Integrative Medical Sciences, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Kanagawa 230-0045, Japan"
index: 1
- name: "RIKEN Center for Sustainable Resource Sciences, Suehiro-cho-1-7-22, Tsurumi Ward, Yokohama, Kanagawa 230-0045, Japan"
index: 2
date: "Thursday 25 May 2023"
output:
prettydoc::html_pretty:
theme: cayman
number_sections: true
toc: true
toc_depth: 4
keep_html: true
keep_md: true
toc-title: "Table of Contents"
tags:
- single-cell
- next-generation-sequencing
- UMI-tools
- genomics
- gene-expression
- scRNA-Seq
- bioinformatics
- data-processing
---
![MicroBadger Layers (latest)](https://img.shields.io/microbadger/layers/tomkellygenetics/universc/latest?label="[email protected]")
![Docker Image Size (v1.2.6)](https://img.shields.io/docker/image-size/tomkellygenetics/universc/1.2.6?label="image%[email protected])
[](http://hits.dwyl.com/minoda-lab/universc)
[](http://hits.dwyl.com/tomkellygenetics/universc)
[](https://doi.org/10.5281/zenodo.7116956)
# UniverSC
**Single-cell processing across technologies**
------------------------------------------
**Summary**
Single-cell RNA-sequencing analysis to quantify RNA molecules in individual cells has become popular owing to
the large amount of information one can obtain from each experiment. UniverSC is a universal single-cell processing
tool that supports any UMI-based platform. Our command-line tool enables consistent and comprehensive integration,
comparison, and evaluation across data generated from a wide range of platforms. Here we provide a guide to install
and use this tool to process single-cell RNA-Seq data from FASTQ format.
**Package**
UniverSC version 1.2.6
**Maintainers**
Tom Kelly^†^ (RIKEN IMS) and Kai Battenberg^†^ (RIKEN CSRS/IMS)
† These authors contributed equally to this work
Contact: <first name>.<family name>[at]riken.jp
------------------------------------------
**Disclaimer**: we are third party developers not affiliated with 10X Genomics or any other vendor of single-cell technologies.
We are releasing this code on an open-source [license](#licensing) which calls Cell Ranger™ as an external dependency.
------------------------------------------
## Getting Started
### Advanced users
If you have `cellranger` already installed, then all you need to do is clone or download this git repository. You can then run the script in this directory or add it your `PATH`. See the [Quick Start](#quick-start) guide below.
If you wish to install `cellranger` and configure this script to run on a Linux environment, we provide details on [installation](#installation) below. Note that `launch_universc.sh` requires write-access a Cell Ranger installation so it needs to be installed in a user's "home" directory on a server. No admin powers needed!
Note that `cellranger` installations that are pre-compiled on Linux will not run on Mac or Windows. Note that Mac OS and some Linux distributions also have different version of sed and rename. It is possible to compile an open-source version of Cell Ranger but it is tricky to install the dependencies so we recommend using our docker [image](#Docker) if you wish to do this.
### Command-line Beginners
If you are a beginner bioinformatician or wish to run this on a local computer (Mac or Windows), no problem! We provide a "docker" image containing everything needed to run it without installing the software needed. All you need to do is install [docker](https://docs.docker.com/desktop/) and follow our guide to use the [image](#Docker). This comes bundled with all the compatible versions needed to run it.
Note that you need to run the shell commands given in a unix-like command-line interface (the "Terminal" application on Mac or Linux systems). Many shells are supported but we recommend the "bash" shell for beginners (this is the default on most systems). Windows 10 includes a [subsystem](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to run `bash`. If this is too complicated, you can open a Linux environment (Ubuntu) in docker by following our instructions. Then you can enter bash commands into the terminal opened by docker.
If you run into problems installing or running `launch_universc.sh` please don't hesistate to contact us via email or GitHub.
### Graphical application Users
We also provide a graphical user interface (GUI) based application to run the Docker [image](#Docker). Please install Docker as described above and pull our `tomkellygenetics/universc:latest` image using either the docker command-line interface (CLI) or the docker graphical application.
Once you have a docker image installed on your system, you can run the applicable binary available here:
[https://genomec.gsc.riken.jp/gerg/UniverSC/UniverSC_app_release/](https://genomec.gsc.riken.jp/gerg/UniverSC/UniverSC_app_release/)
### Nextflow users
A [nextflow](https://www.nextflow.io/) custom [module](https://nf-co.re/modules) is has been developed for the [nf-core](https://nf-co.re) community.
It will be release open-source to use in nf-core [pipelines](https://nf-co.re/pipelines).
See the issue and pull request for more details:
[https://github.com/nf-core/modules/issues/1644](https://github.com/nf-core/modules/issues/1644)
[https://github.com/nf-core/modules/pull/1706](https://github.com/nf-core/modules/pull/1706)
We welcome feedback on these tools. If you are interested in using it or contributing to
development of nextflow pipelines, please contact the maintainers. Thank you!
## Purpose
We've developed a bash script that will run Cell Ranger on FASTQ files for these technologies. See below for details on how to use it.
If you use this tool, please [cite](#Citation) to acknowledge the efforts of the authors. You can report problems and request
new features to the maintainers with and [issue](#Issues) on GitHub. Details on how to [install](#Install) and [run](#Usage) are provided
below. Please see the [help](#Help) and [examples](#Examples) to try solve your problem before submitting an issue.
Details on the [Docker image](#Docker) are given below. We recommend using Docker unless you have a server environment with Cell Ranger installed already.
### Supported Technologies
In principle, any technology with a cell barcode and unique molecular identifier (UMI) can be supported.
The following technologies have been tested to ensure that they give the expected results: 10x Genomics, Nadia (DropSeq), ICELL8 version 3
We provide the following preset configurations for convenience based on published data and configurations used by other pipelines
(e.g, DropSeqPipe and Kallisto/Bustools). To add further support for other technologies or troubleshoot problems, please submit an Issue
to the GitHub repository: [minoda-lab/universc](https://github.com/minoda-lab/universc/issues)
as described in [Bug Reports](#Issues) below.
Some changes to the Cell Ranger install are required to run other technologies. Therefore we provide settings for 10x Genomics
which restores settings for the Chromium instrument. We therefore recommend using UniverSC for processing all data from different
technologies as the tool manages these changes. Please note that on a single install of Cell Ranger, multiple technologies or multiple samples
of the same technology with different whitelist barcodes cannot be run cannot be run simultaneousely (the tool will also check for this to
avoid causing problems with existing runs). Multiple samples of the same technology with the same barcode whitelist can be run simultaneously.
If you are using `UniverSC` you should also do so to run 10x Genomics data. If you wish to restore Cell Ranger to
default settings, see the [installation](#Uninstalling) or [troubleshooting](#Debugging) sections below.
#### Pre-set configurations
- 10x Genomics (version automatically detected): 10x, chromium
- 10x Genomics version 2 (16 bp barcode, 10 bp UMI): 10x-v2, chromium-v2
- 10x Genomics version 3 (16 bp barcode, 12 bp UMI): 10x-v3, chromium-v3
- Aligent Bravo B (16 bp barcode, No UMI): aligent, bravo
- BD Rhapsody
- BD Rhapsody v1 (27 bp barcode, 8 bp UMI): bd-rhapsody
- BD Rhapsody v2 enhanced beads (27 bp barcode, 8 bp UMI): bd-rhapsody-v2
- C1
- C1 Fluidigm (16 bp barcode, No UMI): c1, fluidgm-c1
- C1 CAGE (16 bp, No UMI): c1-cage
- C1 RamDA-Seq (16 bp, No UMI): c1-ramda-seq
- CEL-Seq
- CEL-Seq (8 bp barcode, 4 bp UMI): celseq
- CEL-Seq2 (6 bp UMI, 6 bp barcode): celseq2
- Drop-Seq (12 bp barcode, 8 bp UMI): dropseq
- ICELL8
- ICELL8 version 2 (11 bp barcode, No UMI): icell8-non-umi, icell8-v2
- ICELL8 version 3 (11 bp barcode, 14 bp UMI): icell8 or custom
- ICELL8 5′ scRNA with TCR OR kit (10bp barcode, NO bp UMI): icell8-5-prime
- ICELL8 full-length scRNA with Smart-Seq (16 bp barcode, No UMI): icell8-full-length
- inDrops
- inDrops version 1 (19 bp barcode, 6 bp UMI): indrops-v1, 1cellbio-v1
- inDrops version 2 (19 bp barcode, 6 bp UMI): indrops-v2, 1cellbio-v2
- inDrops version 3 (16 bp barcode, 6 bp UMI): indrops-v3, 1cellbio-v3
- MARS-Seq
- MARS-Seq (6 bp barcode, 10 bp UMI): marsseq, marsseq-v1
- MARS-Seq2 (7 bp barcode, 8 bp UMI): marsseq2, marsseq-v2
- Microwell-Seq (18 bp barcode, 6 bp UMI): microwell
- Nadia (12 bp barcode, 8 bp UMI): nadia, dropseq
- PIP-Seq
- PIP-Seq version 1 (16 bp barcode, 6 bp UMI): pip-seq-v1
- PIP-Seq version 2 (24 bp barcode, 12 bp UMI): pip-seq-v2
- PIP-Seq version 3 (28 bp barcode, 12 bp UMI): pip-seq-v3
- PIP-Seq version 4 (28 bp barcode, 12 bp UMI): pip-seq-v4
- Quartz-Seq
- QuartzSeq (6 bp index, no UMI): quartz-seq
- Quartz-Seq2 (14 bp barcode, 8 bp UMI): quartzseq2-384
- Quartz-Seq2 (15 bp barcode, 8 bp UMI): quartzseq2-1536
- RamDA-Seq (6 bp index, no UMI): ramda-seq
- Single-cell combinatorial indexing (SCI-RNA-Seq)
- SCI-Seq 2-level indexing (30 bp barcode, 8 bp UMI): sciseq2
- SCI-Seq 3-level indexing (40 bp barcode, 8 bp UMI): sciseq3
- SCIFI-Seq (27 bp barcode, 8 bp UMI): scifiseq
- SCRB-Seq (6 bp barcode, 10 bp UMI): scrbseq, mcscrbseq
- SeqWell (12 bp barcode, 8 bp UMI): plexwell, seqwell, seqwells3
- Smart-seq
- Smart-Seq (16 bp barcode, No UMI): smartseq
- Smart-Seq2 (16 bp barcode, No UMI): smartseq2
- Smart-Seq2-UMI, Smart-seq3 (16 bp barcode, 8 bp UMI): smartseq3
- SPLiT-Seq
- SPLiT-Seq v1.0 (8 bp UMI, 24 bp barcode): splitseq
- SPLiT-Seq v2.1 (10 bp UMI, 24 bp barcode): splitseq2
- STRT-Seq
- STRT-Seq (6 bp barcode, no UMI): strt-seq
- STRT-Seq-C1 (8 bp barode, 5 bp UMI): strt-seq-c1
- STRT-Seq-2i (13 bp barcode, 6 bp UMI): strt-seq-2i
- STRT-Seq-2018 (8 bp barcode, 8 bp UMI): strt-seq-2018
- SureCell (18 bp barcode, 8 bp UMI): surecell, ddseq, biorad
- VASA-Seq
- VASA-plate (6 bp UMI, 6 bp barcode): vasa-plate
- VASA-drop (6 bp UMI, 16 bp barcode): vasa-drop
#### Chemistry settings available
All technologies support 3′ single-cell RNA-Seq. Barcode adjustments and
whitelists are changed automatically. For 5′ single-cell RNA-Seq, this
is only supported for 10x Genomics version 2 chemistry, ICELL8,
Smart-Seq, and STRT-Seq.
For 10x Genomics, this is detected automatically but can be
configured with the `--chemistry` argument.
For other technologies, the template switching oligonucleotide
is automatically converted to the match the 10x sequence.
#### Support for UMI-based and non-UMI technologies
By default, UMIs are supported where available so with the following
exceptions for non-UMI technologies:
ICELL8 v2, RamDA-Seq, Quartz-Seq, Smart-Seq, Smart-Seq2.
While using UMI is recommended we provide a mock UMI for counting
reads for these technologies (and data from previous versions).
Other techniques can be forced to replace the UMI with a mock sequence
for counting reads only with `--non-umi` or `--read-only` arguments.
Forcing non-UMI techniques is _not recommended_ unless you are
integrating non-UMI and UMI-based technologies. It is not necessary
to specific `--non-umi` for non-UMI techniques as these will be used
automatically when applicable. For ICELL8 and Smart-Seq where both
non-UMI (icell8-v2, smartseq2) and UMI-based (icell8-v3, smartseq3)
techniques are available it is possible to specify which to use.
#### Single and dual indexed technologies
Where needed the cell barcode can be detected in the index I1 or I2 file.
Single indexes are supported for STRT-Seq and Quartz-Seq.
Dual indexes are supported for Fluidigm C1, ICELL8 full-length,
inDrops-v3, RamDA-Seq, SCI-RNA-Seq, scifi-seq, and Smart-Seq.
Combinatorial indexing technologies have linkers between barcodes removed
automatically to match the barcode whitelist.
#### Demultiplexing for dual-indexing
For dual-indexed technologies such as Fluidigm C1, inDrops-v3, Sci-Seq,
SmartSeq3 it is advised to use "bcl2fastq" before calling UniverSC:
```
/usr/local/bin/bcl2fastq -v --runfolder-dir "/path/to/illumina/bcls" --output-dir "./Data/Intensities/BaseCalls"\
--sample-sheet "/path/to/SampleSheet.csv" --create-fastq-for-index-reads\
--use-bases-mask Y26n,I8n,I8n,Y50n --mask-short-adapter-reads 0\
--minimum-trimmed-read-length 0
```
Please adjust the lengths for `--use-bases-mask` accordingly for read 1, index 1 (i7), index 2 (i5), and read 2.
Ensure that `--create-fastq-for-index-read` is used where possible.
Using `--no-lane-splitting` is optional as UniverSC can process an arbirtary number of lanes.
There is no need to specify index sequences in the same sheet for cell barcodes, using "NNNNNNNN" will match all
samples and the cell barcodes will be distinguished by the single-cell processing pipeline. Index sequences should
only be used to demultiplex samples and replicates (not cells).
#### Missing index sequences
If a sequencing facility has demultiplexed the samples for you without this,
UniverSC will attempt to extract index sequences from FASTQ headers in read 1.
If index sequences are not stored in the file headers and samples have already
been demultiplexed, a dummy index file of the same number of reads as R1 and R2
will be required. As a workaround, you can generate this by copying the R1 and R2
files and replacing the sequences with the first barcode in the relevant whitelist.
For example:
```
index1="TAAGGCGA"
index2="AAGGAGTA"
# create new files
cp R1_file.fastq I1_file.fastq
cp R2_file.fastq I2_file.fastq
# replace sequences
sed -i "2~4s/^.*$/${index1}/g" I1_file.fastq
sed -i "2~4s/^.*$/${index2}/g" I2_file.fastq
# replace quality scores
sed -i "4~4s/^.*$/IIIIIIII/g" I1_file.fastq I2_file.fastq
```
This results in a new "sample index" for each demultiplexed sample.
To combine demultiplexed sampls for dual indexed techniques use the following:
```
# for fastq files
cat Sample1_R1_file.fastq Sample2_R1_file.fastq Sample3_R1_file.fastq > Combined_R1_file.fastq
cat Sample1_R2_file.fastq Sample2_R2_file.fastq Sample3_R2_file.fastq > Combined_R2_file.fastq
cat Sample1_I1_file.fastq Sample2_I1_file.fastq Sample3_I1_file.fastq > Combined_I1_file.fastq
cat Sample1_I2_file.fastq Sample2_I2_file.fastq Sample3_I2_file.fastq > Combined_I2_file.fastq
# for compressed files (not need to uncompress)
cat Sample1_R1_file.fastq.gz Sample2_R1_file.fastq.gz Sample3_R1_file.fastq.gz > Combined_R1_file.fastq.gz
cat Sample1_R2_file.fastq.gz Sample2_R2_file.fastq.gz Sample3_R2_file.fastq.gz > Combined_R2_file.fastq.gz
cat Sample1_I1_file.fastq.gz Sample2_I1_file.fastq.gz Sample3_I1_file.fastq.gz > Combined_I1_file.fastq.gz
cat Sample1_I2_file.fastq.gz Sample2_I2_file.fastq.gz Sample3_I2_file.fastq.gz > Combined_I2_file.fastq.gz
```
As this needs to done on a case-by-case basis it has not been implemented by the UniverSC core functions.
We provide this workaround for using published data and data already processed by sequencing facilities.
Please contact the maintainers or file an issue on GitHub if you are having problems with this case.
#### Custom inputs
Custom inputs are also supported by giving the name "custom" and length of barcode and UMI separated by a "_" character.
e.g., Custom (16bp barcode, 10bp UMI): `custom_16_10`
Custom barcode files are also supported for preset technologies. These are particularly useful for well-based
technologies to demutliplex based on the wells.
Note that custom inputs do not remove linker or adapter sequences for combinatorial indexng technologies.
These must be removed from the Read 1 file before running UniverSC. To request a preset technology
setting instead, please submit a feature request on GitHub as described below.
## Release
This tool will be released open-source (see [legal stuff](#licensing) below).
We welcome any feedback on it and any contributions to improve it.
Hopefully it will save people time by making it easier to compare technologies.
We have tested it on several technologies but we need users like you
to let us know how we can improve it. We hope that it will save you
time by handing tedious parts of data formatting so that you can
focus on the results.
### Citation <span id="Citation"><span>
Please cite our publication when you use our software as follows:
Battenberg, K., Kelly, S.T., Ras, R.A., Hetherington, N.A., Hayashi, K., and Minoda, A. (2022)
A flexible cross-platform single-cell data processing pipeline. _Nat Commun_ **13**(1): 1-7. https://doi.org/10.1038/s41467-022-34681-z
```
@Article{pmid36369450,
author="Battenberg, K. and Kelly, S. T. and Ras, R. A. and Hetherington, N. A. and Hayashi, M. and Minoda, A. ",
title="{{A} flexible cross-platform single-cell data processing pipeline}",
journal="Nat Commun",
year="2022",
volume="13",
number="1",
pages="1-7",
month="Nov",
note = {https://github.com/minoda-lab/universc package version 1.2.6},
URL = {https://doi.org/10.1038/s41467-022-34681-z}
}
```
The preprint can also be found here:
Kelly, S.T., Battenberg K., Hetherington, N.A., Hayashi, K., and Minoda, A. (2021)
UniverSC: a flexible cross-platform single-cell data processing pipeline.
bioRxiv 2021.01.19.427209; doi: [https://doi.org/10.1101/2021.01.19.427209](https://doi.org/10.1101/2021.01.19.427209)
package version 1.2.6. [https://github.com/minoda-lab/universc](https://github.com/minoda-lab/universc)
```
@article {Kelly2021.01.19.427209,
author = {Kelly, S. Thomas and Battenberg, Kai and Hetherington, Nicola A. and Hayashi, Makoto and Minoda, Aki},
title = {{UniverSC}: a flexible cross-platform single-cell data processing pipeline},
elocation-id = {2021.01.19.427209},
year = {2021},
doi = {10.1101/2021.01.19.427209},
publisher = {Cold Spring Harbor Laboratory},
abstract = {Single-cell RNA-sequencing analysis to quantify RNA molecules in individual cells has become popular owing to the large amount of information one can obtain from each experiment. We have developed UniverSC (https://github.com/minoda-lab/universc), a universal single-cell processing tool that supports any UMI-based platform. Our command-line tool enables consistent and comprehensive integration, comparison, and evaluation across data generated from a wide range of platforms.Competing Interest StatementThe authors have declared no competing interest.},
eprint = {https://www.biorxiv.org/content/early/2021/01/19/2021.01.19.427209.full.pdf},
journal = {{bioRxiv}},
note = {package version 1.2.6},
URL = {https://github.com/minoda-lab/universc},
}
```
The software can also be cited directly as a manual:
```
@Manual{,
title = {{UniverSC}: a flexible cross-platform single-cell data processing pipeline},
author = {S. Thomas Kelly, Kai Battenberg, Nicola A. Hetherington, Makoto Hayashi, and Aki Minoda},
year = {2023},
note = {package version 1.2.6},
url = {https://github.com/minoda-lab/universc},
}
```
### Bug Reports <span id="Issues"><span>
#### Reporting issues
To add further support for other technologies or troubleshoot problems, please submit an Issue
to the GitHub repository: https://github.com/minoda-lab/universc/issues
Please submit [issues](https://github.com/minoda-lab/universc/issues) on GitHub to report
problems or suggest features. [Pull requests](https://github.com/minoda-lab/universc/pulls)
to the `dev` branch on GitHub are also welcome to add features or correct problems. Please see
the [contributor guide](CONTRIBUTING.md) for more details.
### Requesting new technologies
Where possible, please provide an minimal example of the first few lines of each FASTQ file for testing purposes.
It is also helpful to describe the technology, such as:
- length of barcode
- length of UMI
- which reads they're on
- whether there is a known barcode whitelist available
- whether adapters or linkers are required
- whether a preprint, publication, or company specifications are available
Technologies that may be difficult to support are those with:
- barcodes longer than 16bp
- barcodes with phase blocks or varying length
- UMIs longer than 12bp
- technologies that do not have UMI
- combinatorial indexing
- dual indexing
Please bear this in mind when submitting requests. We will consider to add further technologies but
it could take significant resources to add support for techniques with these designs.
Note that updates to the tool have added support for several examples of these.
## Installation <span id="installation"><span>
This script requires Cell Ranger to be installed and exported to the PATH (version 3.0.0 or higher recommended).
The script itself is exectuable and does not require installation to run but you can put it in your PATH or
bin of your Cell Ranger install if you wish to do so. We provide scripts to do this for your convenience.
See the details below on how set up Cell Ranger and launch_universc.sh.
#### Download UniverSC
To download UniverSC open a terminal prompt and enter the following commands.
```
cd $HOME/Downloads
git clone https://github.com/minoda-lab/universc.git
cd universc
```
### Quick Start
If you already have Cell Ranger installed, then you can run the script without installing it.
```
bash launch_universc.sh
```
You can call it in another directory by giving the path to the script.
```
cd $/HOME/my_project
bash $HOME/Downloads/universc/launch_universc.sh
```
See the details below on how to install Cell Ranger and launch_universc.sh add them
to the PATH so that `launch_universc.sh` can be run from any directory.
### Runnning in a git repository
If you are running code in a git repository you can add UniverSC as a submodule.
```
cd $/HOME/my_git_repo
git submodule add https://github.com/minoda-lab/universc.git
bash universc/launch_universc.sh
```
### System Requirements
In principle, the script can run on any Unix systems with Cell Ranger installed. You can check whether
Cell Ranger is already availble by running:
```
whereis cellranger
```
You can see which Cell Ranger installation will run as follows:
```
which cellranger
cellranger count --version
```
If Cell Ranger is already installed on your system, you can add it to your $PATH as follows:
```
export PATH=/home/username/path/to/cellranger-x.x.x:$PATH
```
#### Installing dependencies
If Cell Ranger is not installed on your system, you must install it before running launch_universc.sh.
Please see the [manual for Cell Ranger](https://support.10xgenomics.com/single-cell-gene-expression/software/pipelines/latest/what-is-cell-ranger)
on the 10x Genomics website for more details on how to use it. We provide support for
passing various options to Cell Ranger and sensible defaults for each technology.
This script is compatible with the installation of Cell Ranger that you can
[download](https://support.10xgenomics.com/single-cell-gene-expression/software/downloads/latest)
from the 10x Genomics website and gives the same output formats.
However, we recommend to use the open-source release of Cell Ranger on GitHub. This is
release on an MIT License and is not subject to the 10x Genomics End User
License Agreement.
We provide open-source repositories with minor updates for compatibility
with current versions of dependencies.
The code is available here:
[https://github.com/minoda-lab/cellranger/releases](https://github.com/minoda-lab/cellranger/releases)
We also provide Docker images for Cell Ranger versions 2.0.2, 2.1.0, 2.1.1, and 3.0.2:
[https://github.com/minoda-lab/cellranger_clean/packages](https://github.com/minoda-lab/cellranger_clean/packages)
[https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags](https://hub.docker.com/r/tomkellygenetics/cellranger_clean/tags)
#### Cluster Mode configuration
#### Software Requirements
These have been pre-installed in the Docker image described above.
A full example of installation is available in the [GitHub repository](https://github.com/minoda-lab/cellranger)
and on [DockerHub](https://hub.docker.com/r/tomkellygenetics/cellranger_clean/dockerfile).
- Python 2.7.13
- rust 1.28.0
- clang 6.0
- go 1.11
- node 8.11.4
- Cython 0.28.0
- STAR 2.5.1b
- bcl2fastq 2.19.1.403
- tsne 0.15
The following additional shell utilities are required. Mac OS and
most Linux distributions come with these pre-installed.
- make 3.81
- git 2.20.1
- sed (GNU sed) 4.4 (gsed)
- tar 2.8.3
- rename 0.20 (perl-rename)
- perl 5.26.1
- rsync 2.6.9
Note that rename is installed by default on Mac, Ubuntu and Debian
but a different version must be used on other Linux distrubutions.
CentOS and Fedora:
```
sudo yum install prename
```
```
sudo dnf install prename
```
Red Hat Linux:
```
sudo rpm install prename
```
Arch Linux:
```
yay perl-rename
```
##### Recommended software
- git-lfs 2.10.0
#### Hardware requirements
- 8-core Intel or AMD processor (16 cores recommended)
- 64GB RAM (128GB recommended)
- 1TB free disk space
- 64-bit CentOS/RedHat 6.0 or Ubuntu 12.04
#### Ensuring write-access to Cell Ranger
The conversion process requires write-access to to the Cell Ranger install directory so
an install on your user directory is recommended.
You can check where Cell Ranger is installed with:
```
which cellranger
```
If calling the script gives the help menu, launch_universc.sh has sucessfully run
with access to the directories that it needs. It will give an error
message if the Cell Ranger directory is not writeable.
```
bash launch_universc.sh
```
This script requires Cell Ranger (version 3.0.0 or higher recommended) to be installed and have write-access
to the Cell Ranger install directory, so an install on your user directory is recommended.
This script also requires Cell Ranger to be exported to the PATH.
The script itself is exectuable and does not require installation to run but you can put it
in your PATH or bin of your Cell Ranger install if you wish to do so.
This script will run in bash on any OS (but it has only been tested on Linux Debian). Running Cell Ranger
with this configuration requires a lot of memory (40Gb) so running on server is recommended.
SGE job modes are supported to run Cell Ranger with multiple threads.
This is required because launch_universc.sh will make changes to the Cell Ranger install
to ensure compatibility with the technology running. A local install in
you user home directory is needed to make these changes. This ensures
that these changes do not affect jobs run by other users and allows
launch_universc.sh to change the whitelist and source code as needed.
These changes are reversible but mean that only one technology can be
run at the same time. You can restore original configurations with:
```
bash launch_universc.sh -t "10x" --setup
```
##### Local install
If Cell Ranger is not already installed we recommend installing it in a directory that
you have write access to such as `$HOME/local`.
##### Importing an installed version of Cell Ranger
If Cell Ranger has been installed by a system administrator, you will only have read-access
to that installation. You can still use rather than installing a new version but you
will need to copy it to your home directory and add this version to your PATH.
```
mkdir -p $HOME/local
cd ~/local
installed_version=$(echo $(which cellranger) | rev | cut -d"/" -f2- | rev)
cp -rv $installed_version .
installed_directory=$(echo $(which cellranger) | rev | cut -d"/" -f2 | rev)
cd $installed_directory
new_version=$(pwd)
#remove previous version from PATH
export PATH=$(echo $PATH | sed "s;$installed_version:;;g")
#add new version to PATH
eval $(echo export PATH=$new_version:\$PATH)
cd ..
```
You should be able to see that the locally installed version can be called as follows:
```
echo $PATH
which cellranger
bash launch_universc.sh
```
#### Installing launch_universc.sh to the PATH
##### Running the script
Adding the script to the PATH is not absolutely neccessary, it can
be called as follows from the directory that it is downloaded in.
```
cd $HOME/Downloads
git clone https://github.com/minoda-lab/universc.git
cd universc
bash launch_universc.sh
```
##### Automated configuration
We provide a Makefile with all necessary configurations to automatically
check whether launch_universc.sh is installed correctly.
You can specify any directory to install as a "prefix". This will
create a directory "$HOME/local/universc-0.3" where
the files needed will be stored.
```
cd $HOME/Downloads
git clone https://github.com/minoda-lab/universc.git
make
make install prefix=$HOME/local
```
It is also possible to add the current working directory as
the installed directory.
```
make install prefix="."
```
In this case *do not* delete the installed directory after you
install it or the script will fail to run.
By default it will be installed in a root directory with
read-only access. This requires administrator priviledges.
Note that the manual can onlly be installed with root
priviledges.
```
sudo make install
sudo make manual
```
You can verify that launch_universc.sh has been added to the PATH.
```
echo $PATH
which launch_universc.sh
```
You can then run `launch_universc.sh` from any working directory.
##### Updating
If launch_universc.sh is already installed and you wish to update it,
first you need to pull the changes from GitHub from the
universc directory.
```
cd universc
git pull origin master
make
```
Then you can update to the directory of your choice using
the same options for `--prefix` as to install.
```
make install prefix=$HOME/local
```
This is remove previous versions of launch_universc.sh and install
the latest version.
The manual can be updated with:
```
sudo make manual
```
##### Uninstalling
Before uninstalling UniverSC please ensure that any
versions of Cell Ranger used are restored to their default configuration:
```
export PATH=/Users/tom/Downloads/cellranger-x.y.z:$PATH
bash launch_universc.sh -t "10x" --setup
```
We provide an automated script to reverse the changes above.
```
make uninstall
```
This is will automatically detect the installation of launch_universc.sh.
If multiple versions of Cell Ranger are present, you can
specify which to remove with.
```
make remove prefix=$HOME/local
```
#### Custom shell
Make will install the script with `bash` by default but
alternative shells are supported. You will need to run the
install script or run it with your shell of choice.
```
sh launch_universc.sh
sh inst/INSTALL --prefix $HOME/local
launch_universc.sh
```
```
ksh launch_universc.sh
ksh inst/INSTALL --prefix $HOME/local
launch_universc.sh
```
```
zsh launch_universc.sh
zsh inst/INSTALL --prefix $HOME/local
launch_universc.sh
```
```
fish launch_universc.sh
fish inst/INSTALL --prefix $HOME/local
launch_universc.sh
```
The help menu should reflect the shell used to run it.
To update, similarly run the `inst/UPGRADE` script with
your chosen shell.
### Docker image <span id="Docker"><span>
We provide a docker image with all software needed
to run UniverSC.
This requires "docker" to be installed and a valid DockerHub account.
You can check whether docker is available by running:
```
which docker
docker run hello-world
```
This may require you to login to your account.
```
docker login -u "myusername"
```
If you cannot run docker on a remote server, contact
your systems administrator.
#### Pulling from remote DockerHub repository
We provide a docker image for UniverSC version 1.2.6.
You can import it if you have docker installed.
```
docker pull tomkellygenetics/universc:latest
```
#### Running processes in a docker container
Then you can run UniverSC with:
```
docker run -it tomkellygenetics/universc:latest launch_universc.sh
```
You can open a shell in the docker image with:
```
docker run -it tomkellygenetics/universc:latest /bin/bash
```
```
docker run -it tomkellygenetics/universc:latest /bin/zsh
```
Either of these shells are supported.
Note that Docker containers run with a default of 2 GB of memory.
It is recommended to use at least 8 GB of memory as this is often
insufficient for running UniverSC. Ideally, 16 GB of memory should
be used if it is available on your system.
```
docker run --memory 16g -it tomkellygenetics/universc:latest /bin/bash
```
##### Building the Docker image locally
The Dockerfile is provided in the repository so it can be built from
source. This will build a Docker image with the latest version of
universc provided that updates to dependencies on GitHub
are still compatible.
```
git clone https://github.com/minoda-lab/universc.git
docker build -t universc:latest .
```
Please bear mind that it can take considerable time to install
all necessary dependencies. A stable internet connection is required.
### Manual configuration
You can manually add the script here to the PATH, for example:
```
PATH=$HOME/Downloads/universc:$PATH
```
This means that the directory where the script is can be found
from the shell.
```
echo $PATH
cd ~
launch_universc.sh
```
Add the following line to the `~/.bashrc` file and use `source ~/.bashrc`
to load a new session. This means that you do not need to add the
sript to the PATH in future sessions.
```
export PATH=$HOME/Downloads/universc:$PATH
```
### Setting up Cell Ranger references
This repository comes with almost all necessary files to run test jobs. Test data and
Cell Ranger references are available with Git large file storage (LFS).
To install git LFS run:
```
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | sudo bash
sudo apt-get install git-lfs
git lfs install
```
To import large files from Github change to the "universc" directory and run:
```
git lfs pull origin
```
This provides almost all files required. The STAR index and reference need to be generated or
imported from an existing reference. The following code detects whether the references are
available in an existing cellranger installation.
```
cellrangerversion=`cellranger count --version | head -n 2 | tail -n 1 | cut -f2 -d'(' | cut -f1 -d')'`
cellrangerpath=`which cellranger`
# set up cellranger reference
if [[ ! -f test/cellranger_reference/cellranger-tiny-ref/3.0.0/star/SA ]] && [[ -f $(dirname $cellrangerpath)/cellranger-tiny-ref/3.0.0/star/SA ]]; then
rsync $(dirname $cellrangerpath)/cellranger-tiny-ref/3.0.0/star/SA test/cellranger_reference/cellranger-tiny-ref/3.0.0/star/SA
fi
if [[ ! -f test/cellranger_reference/cellranger-tiny-ref/1.2.0/star/SA ]] && [[ -f $(dirname $cellrangerpath)/cellranger-tiny-ref/1.2.0/star/SA ]]; then
rsync $(dirname $cellrangerpath)/cellranger-tiny-ref/1.2.0/star/SA test/cellranger_reference/cellranger-tiny-ref/1.2.0/star/SA
fi
```
This creates a reference for Cell Ranger here:
- test/cellranger_reference/cellranger-tiny-ref/1.2.0
- test/cellranger_reference/cellranger-tiny-ref/3.0.0