From 39df861585ee056ae092e65bbdac2143a04e97d3 Mon Sep 17 00:00:00 2001 From: John Kleint Date: Tue, 29 Dec 2015 23:55:04 -0500 Subject: [PATCH] Add GNU Parallel --- pages/common/parallel.md | 27 +++++++++++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 pages/common/parallel.md diff --git a/pages/common/parallel.md b/pages/common/parallel.md new file mode 100644 index 000000000..1338b8451 --- /dev/null +++ b/pages/common/parallel.md @@ -0,0 +1,27 @@ +# GNU Parallel + +> Run commands on multiple CPU cores + +- gzip several files at once, using all cores + +`parallel gzip ::: {{file1}} {{file2}} {{file3}}` + +- read arguments from stdin, run 4 jobs at once + +`ls *.txt | parallel -j4 gzip` + +- Convert JPG images to PNG using replacement strings + +`parallel convert {} {.}.png ::: *.jpg` + +- parallel xargs, cram as many args as possible onto one command + +`{{args}} | parallel -X {{command}}` + +- break stdin into ~1M blocks, feed each block to stdin of new command + +`cat {{bigfile.txt}} | parallel --pipe --block 1M {{command}}` + +- run on multiple machines via SSH + +`parallel -S {{machine1}},{{machine2}} {{command}} ::: {{arg1}} {{arg2}}`