README.md
1# Galaxy text and tabular operators
2
3[中文](README_zh.md)
4
5Fifty source-published Scala operators adapt reviewed Galaxy Devteam, Community and IUC tools for Arrow bytes streams. This page covers the original twenty adapters; the [complete extension reference](../../../docs/galaxy-operators.md) describes thirty additional text, table and biological adapters, their supported scope and limitations. Original scripts, XML definitions, golden fixtures and licenses are retained in `upstream/` and `upstream-extended/`; the corresponding manifests record pinned revisions and SHA-256 hashes. These are independent adaptations of selected tools, not the complete Galaxy tool collection or a Galaxy workflow importer.
6
7## Choose an operator
8
9For the additional thirty adapters, see [Galaxy streaming operator adapters](../../../docs/galaxy-operators.md).
10
11Operators use version `1.0` (`galaxy_tabular_to_fasta`: `1.0.1`), one `source` bytes input and one `result` bytes output. Text is UTF-8. They implement structure inference and bounded output samples using the JVM SDK. A `bytes` output describes tabular text; these tools do not claim an Arrow records schema. Connect a suitable text/TSV parser when records are required. The shared runtime is `jvm-arrow@2.1`; artifacts are thin JARs with no new runtime dependencies.
12
13| Operator | Parameters (defaults) | Behavior / upstream tool |
14| --- | --- | --- |
15| `galaxy_change_case` | `columns: c1`, `delimiter: tab`, `casing: upper` | Change selected fields, output TSV / `change_case/changeCase.pl` |
16| `galaxy_condense_characters` | `character: tab` | Collapse repeated characters, output text / `condense_characters/condense_characters.pl` |
17| `galaxy_convert_delimiters` | `character: tab` | Trim lines, replace delimiter runs with tabs, output TSV / `convert_characters/convert_characters.py` |
18| `galaxy_cut_columns` | `columns: c1`, `delimiter: tab` | Select/reorder/repeat fields, skip `#` lines, fill missing fields with `.`, output TSV / `cut_columns/cutWrapper.pl` |
19| `galaxy_merge_columns` | `columns: c1` | Append concatenated selected fields without a separator, skip missing fields; comments and blank lines remain, output TSV / `merge_cols/mergeCols.py` |
20| `galaxy_add_column` | `value: "1"`, `iterate: false` | Append a constant; with iteration a nonnegative integer increases from its value, other values become `value-0`, `value-1`, …; output TSV / `add_value/fixedValueColumn.pl` |
21| `galaxy_fasta_length` | `keep_first: 0`, `id_only: false` | Count characters per sequence, emit title/identifier and length as TSV / `fasta_compute_length/fasta_compute_length.py` |
22| `galaxy_fasta_filter_length` | `min_length: 0`, `max_length: 0` | Keep entries within inclusive length bounds; zero maximum removes the upper threshold / `fasta_filter_by_length/fasta_filter_by_length.py` |
23| `galaxy_fasta_to_tabular` | `keep_first: 0`, `description_columns: 1` | Emit title fields and concatenated sequence as TSV / `fasta_to_tabular/fasta_to_tabular.py` |
24| `galaxy_remove_beginning` | `lines: 10` | Discard first N lines, preserve remaining line endings / `remove_beginning/remove_beginning.pl` |
25| `galaxy_head` | `lines: 10` | Return first N lines and drain remaining input / `show_beginning/headWrapper.pl` |
26| `galaxy_tail` | `lines: 10` | Return last N lines with a bounded rolling buffer / `show_tail/tailWrapper.pl` |
27
28| `galaxy_tabular_to_fasta` | `title_columns: "1"`, `sequence_column: 2` | Join selected title fields with underscores and emit FASTA; skip missing columns / `tabular_to_fasta/tabular_to_fasta.py` |
29| `galaxy_sam_filter_flags` | `require_flags: 0`, `exclude_flags: 0` | Require all specified bits and reject all excluded bits; omit headers / `sam_bitwise_flag_filter/sam_bitwise_flag_filter.py` |
30| `galaxy_sam_to_interval` | `print_all: false` | Emit mapped alignments as zero-based half-open intervals / `sam2interval/sam2interval.py` |
31
32| `galaxy_trim` | `column: 0`, `start: 1`, `end: 0`, `ignore_prefixes: ""`, `fastq: false` | Trim whole lines or one field / `trimmer/trimmer.py` |
33| `galaxy_make_windows` | `window_size: 500`, `offset: 0`, `start_column: 2`, `end_column: 3` | Complete fixed or sliding interval windows / `windowsplitter/windowSplitter.py` |
34| `galaxy_vcf_extract` | Empty reference/region/tags, `quality: 0`, `quality_operator: none`, `pass_only: false` | Select variant rows, preserve headers / `vcf_extract/extract.py` |
35
36`columns` uses one-based selectors such as `c1,c4,c2-c3`; at most 1024 expanded columns, maximum index 999999. `delimiter` is `tab`, `comma`, `dash`, `underscore`, `pipe`, `dot` or `whitespace`. `character` additionally supports `space` and `colon`. Unknown parameters, unsupported enum values and malformed columns fail inference and execution. `value` cannot contain tabs or line breaks. Parameters are not shell commands or regular expressions. These are delimiter-based text tools, not quote-aware CSV tools; do not use them to parse quoted CSV fields.
37
38```mermaid
39flowchart LR
40 A[Item bytes view] --> B[Normalize delimiters]
41 B --> C[Change selected columns]
42 C --> D[Append row identifier]
43 D --> E[TSV bytes stream]
44```
45
46## Run a Job
47
48After publication, run from the XStream repository root:
49
50```bash
51xstream-engine/target/universal/stage/bin/xstream-runner \
52 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job.yaml
53```
54
55The example reads `context:/galaxy/input.txt`, converts comma runs to tabs, uppercases the first column, and appends sequential labels. Its output is:
56
57```text
58SAMPLE group row-0
59ALPHA a row-1
60BETA b row-2
61```
62
63Set `remote_operator_library_urls` to your repository and configure its transport trust as needed. Publication does not require modifying core registries or rebuilding the execution server.
64
65## Build and verify
66
67From this directory, with the XStream JVM SDK published locally:
68
69```bash
70sbt package 'Test / runMain xstream.galaxy.SafetyCheck'
71python3 make-catalog.py
72python3 verify.py --docker-prefix '' --report /tmp/galaxy-tests.json
73```
74
75The verification interpreter needs PyArrow and PyYAML. `verify.py` uses the registered runtime image locally in Docker, without network access. `--docker-prefix` defaults to `sudo -n -u cnic -g docker` for this development host; use `''` when your user already has Docker access. `--runtime IMAGE` selects another compatible image. Tests compare 28 Galaxy fixture cases (two interval expectations correct the upstream padding bug), 65 execution cases including empty input, two-byte Arrow chunks, Unicode, CRLF and missing final newline; they also verify 15 inference/sample contracts, invalid parameters, invalid UTF-8, oversized lines and a three-step operator pipeline. `SafetyCheck` verifies cancellation, I/O error propagation, parameter errors, output expansion limits, FASTA record limits and tail-buffer limits. `--artifact FILE --only NAME` verifies an individual repository-built JAR before review.
76
77The line-oriented tools in the table above read input across Arrow batches without buffering the entire Item. Lines are capped at 1,048,576 UTF-16 characters, selected/generated field joins at 4,194,304 characters, parameter strings at 4096 characters and selected columns at 1024. Working memory is bounded by these per-line limits plus SDK buffers. Output samples honor the SDK byte budget. Cancellation is checked during reading and per line; input and output failures propagate to the engine. No shell, subprocess, network or application file access is used by the business logic.
78
79## Publish sources
80
81```bash
82python3 publish.py --url http://localhost:8820 \
83 --token-file /secure/publisher.token --output /tmp/galaxy-publication
84```
85
86This submits fifty source archives for the repository's isolated Scala build and leaves them pending review. To submit one tool, append its short name, such as `cut_columns`. An administrator can add `--approve-password-file /secure/admin-password`; this mode waits for the build, downloads and verifies each candidate artifact, assigns the `Galaxy`, `Text processing` and `Tabular data` categories (or `Galaxy`, `FASTA` and `Bioinformatics` for FASTA tools, and `Galaxy`, `SAM`, `Bioinformatics` for SAM tools) and a trust score of 4, approves it, and checks that the public download matches the verified candidate. Test failure stops publication. Credentials are read from separate files and never packaged or printed. HTTPS is required for remote publication. The trust score is an assessment, not a safety guarantee. Review records and artifact test reports are written under `--output`.
87
88Each archive contains the Scala implementation, upstream evidence, bilingual instructions, validation scripts and complete license texts; the published JAR includes `META-INF/LICENSE`, `META-INF/NOTICE` and its operator definition. Source visibility is public. See [source submission](../../../docs/operator-submission.md) for repository administration and build prerequisites.
89
90## Behavior and attribution
91
92The golden fixtures preserve upstream behavior. Deliberate differences: LF/CRLF/CR input is normalized to LF without dropping a final character when a trailing newline is absent; malformed UTF-8 is rejected; case conversion uses locale-independent Unicode; `condense_characters` maps whitespace runs to one space (the upstream `s` option has no valid replacement branch); dot compression emits a literal dot rather than the Perl script's escaped replacement; delimiter names are readable enumerations; malformed parameters fail instead of being silently ignored. Trailing empty fields follow the original Perl tools for change/cut and Python behavior for merge. Limits and cancellation are explicit.
93
94Galaxy Contributors retain copyright in the original tools and fixtures. Community and IUC MIT license notices are also retained in `upstream-extended/` and the published artifact. Upstream licensing assigns pre-2024-12-01 work to AFL-3.0 and subsequent contributions to MIT. Both full texts are included unchanged in [LICENSE](LICENSE); mixed provenance is declared as `AFL-3.0 AND MIT`. See [NOTICE](NOTICE). These are independent XStream adaptations, not official Galaxy releases.
95
96## FASTA and line selection details
97
98`keep_first` is a Unicode character limit (zero retains the complete title); `id_only` selects the first title word. FASTA length is the count of sequence characters, not a biological quality/validation measure. For FASTA-to-tabular, `description_columns: 1` retains the full title; greater values split initial words, pad absent fields and retain the remaining description in the final title column. The sequence is emitted directly across lines without whole-sequence buffering. A sequence before a `>` header is rejected; comments and empty lines are skipped. Empty FASTA produces no entries, rather than a fictitious zero-length entry. FASTA output uses LF. Filtering retains at most one 8,388,608-character record; entries already over a nonzero maximum are discarded without retaining their sequence. As in upstream fixtures, the last retained entry has no trailing newline only if it is the final input entry. This does not flatten sequences into an inferred records schema.
99
100Head/remove-beginning accept 0–1,000,000 lines. Tail accepts 0–10,000 lines and at most 8,388,608 retained UTF-16 characters. These three tools preserve LF, CRLF, CR and the final missing newline. They drain the input to Arrow EOS, including head when it has already emitted enough lines. They do not invoke `head`, `tail` or a shell. Length counting strips outer whitespace from sequence lines as in the original tool; filtering and tabular conversion count/preserve other sequence characters. Parameter checks occur during inference as well as processing.
101
102Run the second complete example from the XStream root:
103
104```bash
105xstream-engine/target/universal/stage/bin/xstream-runner \
106 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-fasta.yaml
107```
108
109```text
110seq1 12 sequence-0
111seq3 20 sequence-1
112```
113
114Every catalog entry includes `documentation` describing all inputs, outputs and parameters, a schema-valid example value and a complete Job. The publisher sends it with the source submission; the repository validates it before accepting the submission. `catalog.json` also serves as the local reference for all fifty operator definitions. See [verification results](verification.json) for the pinned runtime and tests.
115
116## SAM and tabular sequence examples
117
118`job-sam.yaml` reads `data/alignments.sam`, keeps properly paired mapped reads and calculates genomic intervals. Run it with the same engine configuration as the other examples:
119
120```bash
121xstream-engine/target/universal/stage/bin/xstream-runner \
122 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-sam.yaml
123xstream-engine/target/universal/stage/bin/xstream-runner \
124 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-tabular-to-fasta.yaml
125```
126 The expected output is:
127
128```text
129#chrom start end strand read_name
130chr1 9 13 + read1
131chr1 19 27 - read2
132```
133
134`galaxy_tabular_to_fasta` uses one-based integer column numbers (`1,2`), rather than the `c1,c2` syntax of the text-column tools. Its example reads `data/sequence-table.tsv`. A leading `>` is removed from the first selected title field; missing selected columns cause the row to be skipped, matching upstream. It does not validate the sequence alphabet.
135
136SAM tools require at least 11 fields and a 16-bit flag. Comments and SAM headers are omitted, matching the upstream flag filter; its output is alignment rows, not a header-complete SAM document. Required and excluded masks must not overlap. The interval tool skips unmapped flags and `*` references, requires positive positions and a complete valid CIGAR, and detects coordinate overflow. `M`, `D`, `N`, `=` and `X` consume reference positions; `I`, `S`, `H` and `P` do not. This deliberately corrects the upstream script, which counted `P` and ignored `=`/`X`. Two original interval expectations are adjusted for that documented correction; separate edge tests cover extended CIGAR and padding. `print_all: true` appends the original SAM row after chromosome, start, end and strand. Input rows are limited to 1 MiB characters and output rows to 2 MiB characters; processing is streaming and cancellation-aware.
137
138Repository-built artifact checks and successful standalone runner Jobs for the tabular/SAM tools are recorded in [stream-tool verification](verification-stream-tools.json).
139
140## Trimming, interval windows and VCF selection
141
142`galaxy_trim` retains characters from `start` through `end` (one-based, inclusive); `end: 0` keeps the remainder. `column: 0` trims the whole line; a positive number trims one tab-separated field. `ignore_prefixes` contains literal leading characters, not ASCII codes or regular expressions. Empty lines are skipped and line endings become LF, matching upstream. Unicode positions count characters rather than UTF-8 bytes. `fastq: true` preserves alternating identifier/separator lines in four-line FASTQ; it is not a wrapped-FASTQ parser. Invalid positions and missing selected columns fail explicitly.
143
144`galaxy_make_windows` preserves other fields and emits zero-based half-open intervals. `offset: 0` means non-overlapping windows; positive offsets select a sliding stride. Only complete windows are emitted, including a first complete window when the stride exceeds interval length. Short intervals yield no windows. These rules correct two upstream sliding-window boundary errors. Invalid coordinates fail instead of silently dropping a row. One interval may expand to at most 100,000 windows and an estimated 64 MiB of UTF-16 character payload; every iteration checks cancellation. There is no whole-file buffering.
145
146`galaxy_vcf_extract` supports exact reference names, inclusive one-based `region: "20:80000..100000"`, quality comparisons, INFO tag presence and PASS status. `keep_info` requires any listed exact tag; `discard_info` excludes any listed tag. The two cannot be combined. Empty criteria retain all records. Missing quality (`.`) never satisfies a numeric comparison. Headers and original record fields remain unchanged except for LF normalization. Unlike the original Python tool, it does not reorder header declarations or inject a `vcfPytools` provenance line. Five upstream VCF fixtures compare the exact selected rows plus independent preservation of input headers. The original 65-case report is complemented by 19 execution cases for these tools in `verification-trim.json`, `verification-make-windows.json` and `verification-vcf-extract.json`. `GenomicSafetyCheck` verifies malformed inputs, expansion budgets, large coordinates, exact tags, missing quality, UTF-8/I/O errors and cancellation during window expansion.
147
148Run from the XStream repository root (with Docker access):
149
150```bash
151xstream-engine/target/universal/stage/bin/xstream-runner \
152 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-windows.yaml
153xstream-engine/target/universal/stage/bin/xstream-runner \
154 examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-vcf.yaml
155```
156
157The first Job reads `data/intervals.bed`, creates four-base windows every two bases, and shortens feature labels to `gene`. The second reads `data/variants.vcf` and retains PASS variants with quality at least 90 within the selected region. No external Galaxy installation or additional runtime libraries are required.
158
159Repository artifact checks and exact runner Job output checks for trimming, windows and VCF selection are in [genomic-tool verification](verification-genomic-tools.json).
160
161## Format plugins and bytes inputs
162
163| Source format | Existing structured support | These operators read |
164| --- | --- | --- |
165| CSV / TSV | Basic delimited-records plugin | Original `bytes` |
166| FASTA | `xstream-data-fasta` provides the `sequence` view | Original `bytes`, not the parsed sequence view |
167| FASTQ / SAM / VCF / BED | No dedicated structured-format plugin | Generic `bytes`; each operator parses its own content |
168
169A dedicated format plugin is not a prerequisite for these Jobs. It is useful when a catalog should expose format-specific schemas, structured views and common parsing to other operators. The format handler then owns decoding; an operator owns analysis. No file path or temporary intermediate file is required by these operator interfaces.
170
171## FASTQ conversion
172
173`galaxy_fastq_to_fasta@1.0` emits FASTA; `galaxy_fastq_to_qual@1.0` emits numeric QUAL bytes. Both accept only `encoding`: default `solexa64` (raw Solexa scores, no Phred conversion), `sanger33`, or `integers` (-5 through 93). No encoding guessing or offset-marker prefix is supported. Exactly four lines per record are required, with matching @/+ titles and sequence/quality counts. Lines are bounded to 1 MiB characters; only the current record is buffered. Integer separators are normalized to one space without trailing spaces. CRLF and an unterminated final line are supported; truncation, invalid UTF-8 and invalid quality values fail.
174
175`job-fastq.yaml` reads bytes from `context:/galaxy/reads.fastq`, converts FASTQ to FASTA and emits sequence lengths as TSV without intermediate files.
176
177Golden tests strip invalid trailing whitespace from upstream fixture `2.fastqsolexa` and normalize expected QUAL separators. Runtime FASTQ validation remains strict.
178
179Verify and run:
180
181```bash
182cd examples/operators/galaxy
183sbt package 'Test/runMain xstream.galaxy.FastqSafetyCheck'
184/path/to/python-with-pyarrow verify.py --only fastq_to_fasta --report verification-fastq-to-fasta.json
185/path/to/python-with-pyarrow verify.py --only fastq_to_qual --report verification-fastq-to-qual.json
186cd ../../..
187xstream-engine/target/universal/stage/bin/xstream-runner examples/operators/galaxy/engine.yaml examples/operators/galaxy/job-fastq.yaml
188```
189
190## Extended collection
191
192See [Galaxy operators](../../../docs/galaxy-operators.md) for 30 additional adapters, supported scope, memory bounds, provenance and run commands. Every tool has a complete Job, input fixture and regression expectation. Reviewed sources and license notices accompany the submission.