README.md
1# Galaxy text and tabular operators
2
3[中文](README_zh.md)
4
5Fifteen source-published Scala operators adapt the [Galaxy tools-devteam](https://github.com/galaxyproject/tools-devteam) text tools at revision `ddbe2a1e6eded54877d786fbcfc18e594fb12932`. The original scripts, XML definitions and golden fixtures are in `upstream/`; SHA-256 hashes are recorded in `upstream-manifest.json`. This set covers fifteen text/FASTA/SAM tools, not the complete Galaxy tool collection or a Galaxy workflow importer.
6
7## Choose an operator
8
9Operators use version `1.0` (`galaxy_tabular_to_fasta`: `1.0.1`), one `source` bytes input and one `result` bytes output. Text is UTF-8. They implement structure inference and bounded output samples using the JVM SDK. A `bytes` output describes tabular text; these tools do not claim an Arrow records schema. Connect a suitable text/TSV parser when records are required. The shared runtime is `jvm-arrow@2.1`; artifacts are thin JARs with no new runtime dependencies.
10
11| Operator | Parameters (defaults) | Behavior / upstream tool |
12| --- | --- | --- |
13| `galaxy_change_case` | `columns: c1`, `delimiter: tab`, `casing: upper` | Change selected fields, output TSV / `change_case/changeCase.pl` |
14| `galaxy_condense_characters` | `character: tab` | Collapse repeated characters, output text / `condense_characters/condense_characters.pl` |
15| `galaxy_convert_delimiters` | `character: tab` | Trim lines, replace delimiter runs with tabs, output TSV / `convert_characters/convert_characters.py` |
16| `galaxy_cut_columns` | `columns: c1`, `delimiter: tab` | Select/reorder/repeat fields, skip `#` lines, fill missing fields with `.`, output TSV / `cut_columns/cutWrapper.pl` |
17| `galaxy_merge_columns` | `columns: c1` | Append concatenated selected fields without a separator, skip missing fields; comments and blank lines remain, output TSV / `merge_cols/mergeCols.py` |
18| `galaxy_add_column` | `value: "1"`, `iterate: false` | Append a constant; with iteration a nonnegative integer increases from its value, other values become `value-0`, `value-1`, …; output TSV / `add_value/fixedValueColumn.pl` |
19| `galaxy_fasta_length` | `keep_first: 0`, `id_only: false` | Count characters per sequence, emit title/identifier and length as TSV / `fasta_compute_length/fasta_compute_length.py` |
20| `galaxy_fasta_filter_length` | `min_length: 0`, `max_length: 0` | Keep entries within inclusive length bounds; zero maximum removes the upper threshold / `fasta_filter_by_length/fasta_filter_by_length.py` |
21| `galaxy_fasta_to_tabular` | `keep_first: 0`, `description_columns: 1` | Emit title fields and concatenated sequence as TSV / `fasta_to_tabular/fasta_to_tabular.py` |
22| `galaxy_remove_beginning` | `lines: 10` | Discard first N lines, preserve remaining line endings / `remove_beginning/remove_beginning.pl` |
23| `galaxy_head` | `lines: 10` | Return first N lines and drain remaining input / `show_beginning/headWrapper.pl` |
24| `galaxy_tail` | `lines: 10` | Return last N lines with a bounded rolling buffer / `show_tail/tailWrapper.pl` |
25
26| `galaxy_tabular_to_fasta` | `title_columns: "1"`, `sequence_column: 2` | Join selected title fields with underscores and emit FASTA; skip missing columns / `tabular_to_fasta/tabular_to_fasta.py` |
27| `galaxy_sam_filter_flags` | `require_flags: 0`, `exclude_flags: 0` | Require all specified bits and reject all excluded bits; omit headers / `sam_bitwise_flag_filter/sam_bitwise_flag_filter.py` |
28| `galaxy_sam_to_interval` | `print_all: false` | Emit mapped alignments as zero-based half-open intervals / `sam2interval/sam2interval.py` |
29
30`columns` uses one-based selectors such as `c1,c4,c2-c3`; at most 1024 expanded columns, maximum index 999999. `delimiter` is `tab`, `comma`, `dash`, `underscore`, `pipe`, `dot` or `whitespace`. `character` additionally supports `space` and `colon`. Unknown parameters, unsupported enum values and malformed columns fail inference and execution. `value` cannot contain tabs or line breaks. Parameters are not shell commands or regular expressions. These are delimiter-based text tools, not quote-aware CSV tools; do not use them to parse quoted CSV fields.
31
32```mermaid
33flowchart LR
34 A[Item bytes view] --> B[Normalize delimiters]
35 B --> C[Change selected columns]
36 C --> D[Append row identifier]
37 D --> E[TSV bytes stream]
38```
39
40## Run a Job
41
42After publication, run from the XStream repository root:
43
44```bash
45xstream-engine/target/universal/stage/bin/xstream-runner \
46 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job.yaml
47```
48
49The example reads `context:/galaxy/input.txt`, converts comma runs to tabs, uppercases the first column, and appends sequential labels. Its output is:
50
51```text
52SAMPLE group row-0
53ALPHA a row-1
54BETA b row-2
55```
56
57Set `remote_operator_library_urls` to your repository and configure its transport trust as needed. Publication does not require modifying core registries or rebuilding the execution server.
58
59## Build and verify
60
61From this directory, with the XStream JVM SDK published locally:
62
63```bash
64sbt package 'Test / runMain xstream.galaxy.SafetyCheck'
65python3 make-catalog.py
66python3 verify.py --docker-prefix '' --report /tmp/galaxy-tests.json
67```
68
69The verification interpreter needs PyArrow and PyYAML. `verify.py` uses the registered runtime image locally in Docker, without network access. `--docker-prefix` defaults to `sudo -n -u cnic -g docker` for this development host; use `''` when your user already has Docker access. `--runtime IMAGE` selects another compatible image. Tests compare 28 Galaxy fixture cases (two interval expectations correct the upstream padding bug), 65 execution cases including empty input, two-byte Arrow chunks, Unicode, CRLF and missing final newline; they also verify 15 inference/sample contracts, invalid parameters, invalid UTF-8, oversized lines and a three-step operator pipeline. `SafetyCheck` verifies cancellation, I/O error propagation, parameter errors, output expansion limits, FASTA record limits and tail-buffer limits. `--artifact FILE --only NAME` verifies an individual repository-built JAR before review.
70
71Input is read across Arrow batches without buffering the entire Item. Lines are capped at 1,048,576 UTF-16 characters, selected/generated field joins at 4,194,304 characters, parameter strings at 4096 characters and selected columns at 1024. Working memory is bounded by these per-line limits plus SDK buffers. Output samples honor the SDK byte budget. Cancellation is checked during reading and per line; input and output failures propagate to the engine. No shell, subprocess, network or application file access is used by the business logic.
72
73## Publish sources
74
75```bash
76python3 publish.py --url http://localhost:8820 \
77 --token-file /secure/publisher.token --output /tmp/galaxy-publication
78```
79
80This submits fifteen source archives for the repository's isolated Scala build and leaves them pending review. To submit one tool, append its short name, such as `cut_columns`. An administrator can add `--approve-password-file /secure/admin-password`; this mode waits for the build, downloads and verifies each candidate artifact, assigns the `Galaxy`, `Text processing` and `Tabular data` categories (or `Galaxy`, `FASTA` and `Bioinformatics` for FASTA tools, and `Galaxy`, `SAM`, `Bioinformatics` for SAM tools) and a trust score of 4, approves it, and checks that the public download matches the verified candidate. Test failure stops publication. Credentials are read from separate files and never packaged or printed. HTTPS is required for remote publication. The trust score is an assessment, not a safety guarantee. Review records and artifact test reports are written under `--output`.
81
82Each archive contains the Scala implementation, upstream evidence, bilingual instructions, validation scripts and complete license texts; the published JAR includes `META-INF/LICENSE`, `META-INF/NOTICE` and its operator definition. Source visibility is public. See [source submission](../../../docs/operator-submission.md) for repository administration and build prerequisites.
83
84## Behavior and attribution
85
86The golden fixtures preserve upstream behavior. Deliberate differences: LF/CRLF/CR input is normalized to LF without dropping a final character when a trailing newline is absent; malformed UTF-8 is rejected; case conversion uses locale-independent Unicode; `condense_characters` maps whitespace runs to one space (the upstream `s` option has no valid replacement branch); dot compression emits a literal dot rather than the Perl script's escaped replacement; delimiter names are readable enumerations; malformed parameters fail instead of being silently ignored. Trailing empty fields follow the original Perl tools for change/cut and Python behavior for merge. Limits and cancellation are explicit.
87
88Galaxy Contributors retain copyright in the original tools and fixtures. Upstream licensing assigns pre-2024-12-01 work to AFL-3.0 and subsequent contributions to MIT. Both full texts are included unchanged in [LICENSE](LICENSE); mixed provenance is declared as `AFL-3.0 AND MIT`. See [NOTICE](NOTICE). These are independent XStream adaptations, not official Galaxy releases.
89
90## FASTA and line selection details
91
92`keep_first` is a Unicode character limit (zero retains the complete title); `id_only` selects the first title word. FASTA length is the count of sequence characters, not a biological quality/validation measure. For FASTA-to-tabular, `description_columns: 1` retains the full title; greater values split initial words, pad absent fields and retain the remaining description in the final title column. The sequence is emitted directly across lines without whole-sequence buffering. A sequence before a `>` header is rejected; comments and empty lines are skipped. Empty FASTA produces no entries, rather than a fictitious zero-length entry. FASTA output uses LF. Filtering retains at most one 8,388,608-character record; entries already over a nonzero maximum are discarded without retaining their sequence. As in upstream fixtures, the last retained entry has no trailing newline only if it is the final input entry. This does not flatten sequences into an inferred records schema.
93
94Head/remove-beginning accept 0–1,000,000 lines. Tail accepts 0–10,000 lines and at most 8,388,608 retained UTF-16 characters. These three tools preserve LF, CRLF, CR and the final missing newline. They drain the input to Arrow EOS, including head when it has already emitted enough lines. They do not invoke `head`, `tail` or a shell. Length counting strips outer whitespace from sequence lines as in the original tool; filtering and tabular conversion count/preserve other sequence characters. Parameter checks occur during inference as well as processing.
95
96Run the second complete example from the XStream root:
97
98```bash
99xstream-engine/target/universal/stage/bin/xstream-runner \
100 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-fasta.yaml
101```
102
103```text
104seq1 12 sequence-0
105seq3 20 sequence-1
106```
107
108Every catalog entry includes `documentation` describing all inputs, outputs and parameters, a schema-valid example value and a complete Job. The publisher sends it with the source submission; the repository validates it before accepting the submission. `catalog.json` also serves as the local reference for all fifteen operator definitions. See [verification results](verification.json) for the pinned runtime and tests.
109
110## SAM and tabular sequence examples
111
112`job-sam.yaml` reads `data/alignments.sam`, keeps properly paired mapped reads and calculates genomic intervals. Run it with the same engine configuration as the other examples:
113
114```bash
115xstream-engine/target/universal/stage/bin/xstream-runner \
116 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-sam.yaml
117xstream-engine/target/universal/stage/bin/xstream-runner \
118 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-tabular-to-fasta.yaml
119```
120 The expected output is:
121
122```text
123#chrom start end strand read_name
124chr1 9 13 + read1
125chr1 19 27 - read2
126```
127
128`galaxy_tabular_to_fasta` uses one-based integer column numbers (`1,2`), rather than the `c1,c2` syntax of the text-column tools. Its example reads `data/sequence-table.tsv`. A leading `>` is removed from the first selected title field; missing selected columns cause the row to be skipped, matching upstream. It does not validate the sequence alphabet.
129
130SAM tools require at least 11 fields and a 16-bit flag. Comments and SAM headers are omitted, matching the upstream flag filter; its output is alignment rows, not a header-complete SAM document. Required and excluded masks must not overlap. The interval tool skips unmapped flags and `*` references, requires positive positions and a complete valid CIGAR, and detects coordinate overflow. `M`, `D`, `N`, `=` and `X` consume reference positions; `I`, `S`, `H` and `P` do not. This deliberately corrects the upstream script, which counted `P` and ignored `=`/`X`. Two original interval expectations are adjusted for that documented correction; separate edge tests cover extended CIGAR and padding. `print_all: true` appends the original SAM row after chromosome, start, end and strand. Input rows are limited to 1 MiB characters and output rows to 2 MiB characters; processing is streaming and cancellation-aware.