README.md
1# PiFlow EcoFlow operator adaptations
2
3Adapt operators from [PiFlow EcoFlow](https://github.com/cas-bigdatalab/piflow/tree/EcoFlow) to XStream. The review baseline is pinned to commit `84de10436a0d382e0d8d2ce1ab3a82f71ebafc29`; this branch contains 67 operators derived from `ConfigurableStop`. [inventory.json](inventory.json) records the complete inventory, upstream classes, source files, and SHA-256 digests. Only implementations passing their individual contract tests can be published; `pending` entries are not implemented capabilities.
4
5Original Scala / Java sources are retained under `upstream/`; see [LICENSE](LICENSE). Each operator's source archive includes its adaptation, change descriptions, and input/output contracts. This is not a compatibility executor for PiFlow workflow files; steps use XStream's port and parameter contracts.
6
7The current collaboration deployment publishes only CSV parser, filter, execute SQL, merge, join and CSV save at version `2.0`. See the [current catalog](../../../docs/operator-catalog.md) for runnable parameters and workflows. The sections below describe the full source project and adaptation scope, not a fully published online repository; `/v1/operators` is authoritative.
8
9
10The six primary examples use [Scala implementations](../scala-piflow/README.md); the Python implementations in this directory demonstrate the Python adapter.
11
12## Authoring and usage
13
14Each Python operator under `operators/` implements `process(ctx)` and uses the XStream SDK to read and write Arrow streams. Published names have a `piflow_` prefix. Ordinary single-input/output ports are `source` / `result`, and two-input operators generally use `left` / `right`; [specs.json](specs.json) is authoritative. It also records example parameters and execution modes.
15
16For example, `piflow_select_field@1.0` accepts:
17
18```json
19{"columnNames":"station_id,temperature"}
20```
21
22Its `records` input contains columns such as `station_id` and `temperature`; its `records` output contains only the selected columns, in the requested order. Processing is incremental by batch, and empty inputs retain an output schema. Column names must be unique and present. `piflow_drop_field@1.0` uses the same parameter structure to remove columns; like upstream, absent columns are ignored, but removing every column is rejected.
23
24Pure Python operators reuse the pinned image digest of `xstream/operator-python:1.0`. The SDK is installed in the shared Runtime; current repository packages contain only business code. Sources declare dependencies in `pyproject.toml` and select the Python dependency provider. The repository locks additional dependencies before execution and caches environments by content fingerprint. No dependencies are installed during Job execution. See the [dependency environment guide](../../../docs/dependency-environments.md).
25
26## Verify and publish individually
27
28Prepare a local directory containing `identity_python/1.1/manifest.json` and its runtime blob. The backup retained when clearing the live repository can serve as this directory. Run from the project root:
29
30```bash
31.venv/bin/python examples/operators/piflow/test.py select_field
32.venv/bin/python examples/operators/piflow/publish.py select_field \
33 --runtime-catalog /path/to/runtime-catalog \
34
35```
36
37The local packager runs the corresponding test and builds development artifacts under `examples/operators/piflow/target/catalog`. It does not publish to the repository; submit standardized sources using the source-only submission workflow below. Each invocation handles one operator.
38
39`mode: streaming` means incremental processing by Arrow batch. Implementations requiring global comparisons or state must declare an appropriate mode and resource limits. An Arrow streaming interface does not mean every algorithm executes in constant memory. Complete integration verification of external database, HDFS, Hive, and SSH operators additionally requires reachable test services and node authorization.
40
41## Scope and semantic differences
42
43The [port review](review.md) lists all 67 upstream operators, corresponding names, status, and limitations; the [Chinese review](review_zh.md) is maintained alongside it. States in `inventory.json` track adaptation work and do not replace the actual inventory of the current repository. Parameters, ports, executors, and execution modes are in `specs.json`. Build scientific Scala operators using [spark/README.md](spark/README.md).
44
45Generic SQL and chart-data operators use SQLite, so not every Spark SQL function is supported. Chart operators return records for plotting, without rendering images. `Fork` has fixed `left/right` outputs; `Route` has fixed `result/matched/unmatched` outputs. CSV and Excel readers return string columns without type inference; empty Excel cells become null. File-writing operators return byte streams rather than requiring server-side result files. Input/output examples are in the review.
46
47`MockData` declares `source` as a trigger input: its contents are consumed and discarded before generating records according to `schema/count`. Output is independent of input contents. `ReichsteinCheck` uses a direct Python implementation to avoid Spark jobs for repeated small-sample statistics. It explicitly fixes bin boundaries, deterministic threshold ordering, and invalid thresholds entering the median, and excludes missing values from estimation samples.
48
49### Excel dependencies
50
51Excel operators reuse the Python runtime and bundle pinned pure-Python dependencies and their licenses with the code. Execution never installs packages over the network. Download the wheels before building; publication verifies SHA-256 against `excel-dependencies.json`:
52
53```bash
54mkdir -p examples/operators/piflow/target/excel-deps
55.venv/bin/pip download --only-binary=:all: --no-deps \
56 --dest examples/operators/piflow/target/excel-deps \
57 openpyxl==3.1.5 et_xmlfile==2.0.0
58```
59
60Excel supports XLSX, not legacy XLS. Writing serializes a temporary workbook, and reading buffers the compressed file; neither is a constant-memory byte transformation. Input and expanded files are limited to 128 MiB, selected read ranges to one million cells, and writes to one million rows. Sheet/start-cell selection is supported; reads also support rectangular ranges. Dates become ISO text, formulas use existing cached results only, and text beginning with `=` remains text on export.
61
62```json
63{"header":true,"dataAddress":"'Weather Station'!B2:C100"}
64```
65
66This example is for `piflow_excel_read`; writing accepts only a start cell, such as `"'Weather Station'!B2"`. Excel's `header` parameter is a JSON boolean.
67
68## Verification and resources
69
70Python publication now runs SDK contract tests, builds the package, runs packaged-container tests, and uploads over HTTP in that order. Any failure stops publication. Built packages can also be tested independently:
71
72```bash
73.venv/bin/python examples/operators/piflow/container_test.py select_field
74.venv/bin/python examples/operators/piflow/spark/test.py add_day_id \
75 --spark-home /tmp/dacp-piflow-spark --docker
76```
77
78Fixtures cover cross-batch records, empty input, Unicode, quoted CSV/newlines, multiple inputs/outputs, invalid parameters, and scientific numeric examples; coverage varies by operator's test branch.
79
80Container tests use a read-only root, no network, 1 GiB memory, 128 processes/threads, and 512 MiB temporary space. Spark operators may create temporary shuffle files, and Excel export may create temporary workbook files; these are neither durable results nor checkpoints. Stateful Python algorithms typically limit materialized inputs or state to 128 MiB, with additional memory overhead for intermediate objects. Scientific JARs are approximately 230 KB, sharing a Spark runtime of approximately 413 MB. Python runtimes are shared independently; Excel code includes approximately 270 KB of compressed dependencies.
81
82The previous repository contents were moved into a timestamped directory under `.operator-backups/` before clearing the live catalog. The live repository contains only this individually published collection. Consult the running repository for its address and current publication status; operators can be downloaded as soon as each one is published.
83
84Run `.venv/bin/python examples/operators/piflow/audit.py` to audit upstream source hashes, live publication inventory, pages, and filenames, lengths, and SHA-256 of code, source, and license downloads.
85
86## Bounded output samples
87
88All Python adapters explicitly register `preview_outputs`. The six operators in the current repository use SDK `preview_process` to apply their pure transforms only to bounded upstream samples. CSV save returns sample bytes without writing files. The interface is required, and missing implementations fail explicitly. See [sample interfaces and limits](../../../docs/structure-inference.md#required-sample-interface-and-existing-operators). From the repository root, run `./.venv/bin/python scripts/test-operator-previews.py` to verify all six deployed operators and size limits.
89
90The repository accepts source only. Local builds in these examples are for development verification; use the [source submission and review workflow](../../../docs/operator-submission.md) for public publication.
91
92`piflow_merge@1.1` infers and combines left/right schemas by field name: retain left field order, append right-only fields, and fill absent fields with null while marking them nullable. Shared fields require identical types, except null types may adopt the other type; nested structures are preserved without implicit incompatible conversions. Execution appends left then right batches, and previews use the same logic; this is not a key-based join. If either input schema is unknown, the complete output schema remains unknown.
93
94For packaged checks use `DACP_PIFLOW_CATALOG=/path/to/catalog XSTREAM_PIFLOW_VERSION=1.2 python container_test.py merge`; the verifier loads the locked environment image when present.