Source code

Download source archive ↓

upstream-extended/community/tools/text_processing/text_processing/readme.rst

1Galaxy wrappers for common unix text-processing tools
2=====================================================
3
4The initial work was done by Assaf Gordon and Greg Hannon's lab ( http://hannonlab.cshl.edu )
5in Cold Spring Harbor Laboratory ( http://www.cshl.edu ). In late 2013 maintainence and
6further development was taken over by Bjoern Gruening. Feel free to contribute any general purpose
7text manipulation tool to this repository.
8
9
10Tools:
11------
12
13  * awk - The AWK programmning language ( http://www.gnu.org/software/gawk/ )
14  * sed - Stream Editor ( http://sed.sf.net )
15  * grep - Search files ( http://www.gnu.org/software/grep/ )
16  * sort_columns - Sorting every line according to there columns
17  * GNU Coreutils programs ( http://www.gnu.org/software/coreutils/ ):
18
19  * sort - sort files
20  * join - join two files, based on common key field.
21  * cut  - keep/discard fields from a file
22  * unsorted_uniq - keep unique/duplicated lines in a file
23  * sorted_uniq - keep unique/duplicated lines in a file
24  * head - keep the first X lines in a file.
25  * tail - keep the last X lines in a file.
26  * unfold_column - unfold a column with multiple entities into multiple lines
27
28
29Few improvements over the standard tools:
30-----------------------------------------
31
32  * EasyJoin - A Join tool that does not require pre-sorted the files ( https://github.com/agordon/filo/blob/scripts/src/scripts/easyjoin )
33  * Multi-Join - Join multiple (>2) files ( https://github.com/agordon/filo/blob/scripts/src/scripts/multijoin )
34  * Find_and_Replace - Find/Replace text in a line or specific column.
35  * Grep with Perl syntax - uses grep with Perl-Compatible regular expressions.
36  * HTML'd Grep - grep text in a file, and produced high-lighted HTML output, for easier viewing ( uses https://github.com/agordon/filo/blob/scripts/src/scripts/sort-header )
37
38
39Requirements:
40-------------
41
42    * Coreutils vesion 8.22 or later.
43    * AWK version 4.0.1 or later.
44    * SED version 4.2 *with* a special patch
45    * Grep with PCRE support
46
47All dependencies will be installed automatically with the Galaxy `Tool Shed`_ and the following repository: https://toolshed.g2.bx.psu.edu/view/bgruening/text_processing
48
49
50-------------------
51NOTE About Security
52-------------------
53
54The included tools are secure (barring unintentional bugs):
55The main concern might be executing system commands with awk's "system" and sed's "e" commands,
56or reading/writing arbitrary files with awk's redirection and sed's "r/w" commands.
57These commands are DISABLED using the "--sandbox" parameter to awk and sed.
58
59User trying to run an awk program similar to::
60
61  BEGIN { system("ls") }
62
63Will get an error (in Galaxy) saying::
64
65  fatal: 'system' function not allowed in sandbox mode.
66
67User trying to run a SED program similar to::
68
69  1els
70
71will get an error (in Galaxy) saying::
72
73  sed: -e expression #1, char 2: e/r/w commands disabled in sandbox mode
74
75That being said, if you do find some vulnerability in these tools, please let me know and I'll try fix them.
76
77------------
78Installation
79------------
80
81Should be done via the Galaxy `Tool Shed`_.
82Install the following repository: https://toolshed.g2.bx.psu.edu/view/bgruening/text_processing
83
84.. _`Tool Shed`: http://wiki.galaxyproject.org/Tool%20Shed
85
86
87----
88TODO
89----
90
91 * add shuf, we can remove the random feature from sort and use shuf instead
92 * move some advanced settings under a conditional, for example the cut tools offers to cut bytes
93 * cut wrapper has some output conditional magic for interval files, that needs to be checked
94 * comm wrapper, see the Galaxy default one
95 * evaluate the join wrappers against the Galaxy ones, maybe we should drop them
96
97
98-------
99License
100-------
101
102  * Copyright (c) 2009-2013   A. Gordon  (gordon <at> cshl dot edu)
103  * Copyright (c) 2013-2015   B. Gruening  (bjoern dot gruening <at> gmail dot com)
104
105
106Permission is hereby granted, free of charge, to any person obtaining
107a copy of this software and associated documentation files (the
108"Software"), to deal in the Software without restriction, including
109without limitation the rights to use, copy, modify, merge, publish,
110distribute, sublicense, and/or sell copies of the Software, and to
111permit persons to whom the Software is furnished to do so, subject to
112the following conditions:
113
114The above copyright notice and this permission notice shall be
115included in all copies or substantial portions of the Software.
116
117THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
118EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
119MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
120IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
121CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
122TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
123SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.