Source code

Download source archive ↓

README.md

1# Galaxy text and tabular operators
2
3[中文](README_zh.md)
4
5Six source-published Scala operators adapt the [Galaxy tools-devteam](https://github.com/galaxyproject/tools-devteam) text tools at revision `ddbe2a1e6eded54877d786fbcfc18e594fb12932`. The original scripts, XML definitions and golden fixtures are in `upstream/`; SHA-256 hashes are recorded in `upstream-manifest.json`. This set covers six general tools, not the complete Galaxy tool collection or a Galaxy workflow importer.
6
7## Choose an operator
8
9All operators use version `1.0`, one `source` bytes input and one `result` bytes output. Text is UTF-8. They implement structure inference and bounded output samples using the JVM SDK. A `bytes` output describes tabular text; these tools do not claim an Arrow records schema. Connect a suitable text/TSV parser when records are required. The shared runtime is `jvm-arrow@2.1`; artifacts are thin JARs with no new runtime dependencies.
10
11| Operator | Parameters (defaults) | Behavior / upstream tool |
12| --- | --- | --- |
13| `galaxy_change_case` | `columns: c1`, `delimiter: tab`, `casing: upper` | Change selected fields, output TSV / `change_case/changeCase.pl` |
14| `galaxy_condense_characters` | `character: tab` | Collapse repeated characters, output text / `condense_characters/condense_characters.pl` |
15| `galaxy_convert_delimiters` | `character: tab` | Trim lines, replace delimiter runs with tabs, output TSV / `convert_characters/convert_characters.py` |
16| `galaxy_cut_columns` | `columns: c1`, `delimiter: tab` | Select/reorder/repeat fields, skip `#` lines, fill missing fields with `.`, output TSV / `cut_columns/cutWrapper.pl` |
17| `galaxy_merge_columns` | `columns: c1` | Append concatenated selected fields without a separator, skip missing fields; comments and blank lines remain, output TSV / `merge_cols/mergeCols.py` |
18| `galaxy_add_column` | `value: "1"`, `iterate: false` | Append a constant; with iteration a nonnegative integer increases from its value, other values become `value-0`, `value-1`, …; output TSV / `add_value/fixedValueColumn.pl` |
19
20`columns` uses one-based selectors such as `c1,c4,c2-c3`; at most 1024 expanded columns, maximum index 999999. `delimiter` is `tab`, `comma`, `dash`, `underscore`, `pipe`, `dot` or `whitespace`. `character` additionally supports `space` and `colon`. Unknown parameters, unsupported enum values and malformed columns fail inference and execution. `value` cannot contain tabs or line breaks. Parameters are not shell commands or regular expressions. These are delimiter-based text tools, not quote-aware CSV tools; do not use them to parse quoted CSV fields.
21
22```mermaid
23flowchart LR
24    A[Item bytes view] --> B[Normalize delimiters]
25    B --> C[Change selected columns]
26    C --> D[Append row identifier]
27    D --> E[TSV bytes stream]
28```
29
30## Run a Job
31
32After publication, run from the XStream repository root:
33
34```bash
35xstream-engine/target/universal/stage/bin/xstream-runner \
36  examples/operators/galaxy/engine.yaml examples/operators/galaxy/job.yaml
37```
38
39The example reads `context:/galaxy/input.txt`, converts comma runs to tabs, uppercases the first column, and appends sequential labels. Its output is:
40
41```text
42SAMPLE	group	row-0
43ALPHA	a	row-1
44BETA	b	row-2
45```
46
47Set `remote_operator_library_urls` to your repository and configure its transport trust as needed. Publication does not require modifying core registries or rebuilding the execution server.
48
49## Build and verify
50
51From this directory, with the XStream JVM SDK published locally:
52
53```bash
54sbt package 'Test / runMain xstream.galaxy.SafetyCheck'
55python3 make-catalog.py
56python3 verify.py --docker-prefix '' --report /tmp/galaxy-tests.json
57```
58
59The verification interpreter needs PyArrow. `verify.py` uses the registered runtime image locally in Docker, without network access. `--docker-prefix` defaults to `sudo -n -u cnic -g docker` for this development host; use `''` when your user already has Docker access. `--runtime IMAGE` selects another compatible image. Tests compare nine original Galaxy golden fixtures, 24 execution cases including empty input, two-byte Arrow chunks, Unicode, CRLF and missing final newline; they also verify six inference/sample contracts, invalid parameters, invalid UTF-8, oversized lines and a three-step operator pipeline. `SafetyCheck` verifies cancellation, I/O error propagation, parameter errors and output expansion limits. `--artifact FILE --only NAME` verifies an individual repository-built JAR before review.
60
61Input is read across Arrow batches without buffering the entire Item. Lines are capped at 1,048,576 UTF-16 characters, selected/generated field joins at 4,194,304 characters, parameter strings at 4096 characters and selected columns at 1024. Working memory is bounded by these per-line limits plus SDK buffers. Output samples honor the SDK byte budget. Cancellation is checked during reading and per line; input and output failures propagate to the engine. No shell, subprocess, network or application file access is used by the business logic.
62
63## Publish sources
64
65```bash
66python3 publish.py --url http://localhost:8820 \
67  --token-file /secure/publisher.token --output /tmp/galaxy-publication
68```
69
70This submits six source archives for the repository's isolated Scala build and leaves them pending review. To submit one tool, append its short name, such as `cut_columns`. An administrator can add `--approve-password-file /secure/admin-password`; this mode waits for the build, downloads and verifies each candidate artifact, assigns the `Galaxy`, `Text processing` and `Tabular data` categories and a trust score of 4, approves it, and checks that the public download matches the verified candidate. Test failure stops publication. Credentials are read from separate files and never packaged or printed. HTTPS is required for remote publication. The trust score is an assessment, not a safety guarantee. Review records and artifact test reports are written under `--output`.
71
72Each archive contains the Scala implementation, upstream evidence, bilingual instructions, validation scripts and complete license texts; the published JAR includes `META-INF/LICENSE`, `META-INF/NOTICE` and its operator definition. Source visibility is public. See [source submission](../../../docs/operator-submission.md) for repository administration and build prerequisites.
73
74## Behavior and attribution
75
76The golden fixtures preserve upstream behavior. Deliberate differences: LF/CRLF/CR input is normalized to LF without dropping a final character when a trailing newline is absent; malformed UTF-8 is rejected; case conversion uses locale-independent Unicode; `condense_characters` maps whitespace runs to one space (the upstream `s` option has no valid replacement branch); dot compression emits a literal dot rather than the Perl script's escaped replacement; delimiter names are readable enumerations; malformed parameters fail instead of being silently ignored. Trailing empty fields follow the original Perl tools for change/cut and Python behavior for merge. Limits and cancellation are explicit.
77
78Galaxy Contributors retain copyright in the original tools and fixtures. Upstream licensing assigns pre-2024-12-01 work to AFL-3.0 and subsequent contributions to MIT. Both full texts are included unchanged in [LICENSE](LICENSE); mixed provenance is declared as `AFL-3.0 AND MIT`. See [NOTICE](NOTICE). These are independent XStream adaptations, not official Galaxy releases.