README.md
1# PiFlow EcoFlow operator adaptations
2
3Adapt operators from [PiFlow EcoFlow](https://github.com/cas-bigdatalab/piflow/tree/EcoFlow) to XStream. The review baseline is pinned to commit `84de10436a0d382e0d8d2ce1ab3a82f71ebafc29`; this branch contains 67 operators derived from `ConfigurableStop`. [inventory.json](inventory.json) records the complete inventory, upstream classes, source files, and SHA-256 digests. Only implementations passing their individual contract tests can be published; `pending` entries are not implemented capabilities.
4
5Original Scala / Java sources are retained under `upstream/`; see [LICENSE](LICENSE). Each operator's source archive includes its adaptation, change descriptions, and input/output contracts. This is not a compatibility executor for PiFlow workflow files; steps use XStream's port and parameter contracts.
6
7The [public repository](https://repo.conox.cn/?q=piflow) provides the implemented EcoFlow collection. Six primary Scala operators use version `2.0`; the remaining adapters use the Python or shared Spark runtime. The inventory covers the 67 operators at the pinned upstream revision. The review records the scope of entries that depend on external services or an engine-specific contract.
8
9
10The six primary examples use [Scala implementations](../scala-piflow/README.md); the Python implementations in this directory demonstrate the Python adapter.
11
12## Authoring and usage
13
14Each Python operator under `operators/` implements `process(ctx)` and uses the XStream SDK to read and write Arrow streams. Published names have a `piflow_` prefix. Ordinary single-input/output ports are `source` / `result`, and two-input operators generally use `left` / `right`; [specs.json](specs.json) is authoritative. It also records example parameters and execution modes.
15
16For example, `piflow_select_field@1.0` accepts:
17
18```json
19{"columnNames":"station_id,temperature"}
20```
21
22Its `records` input contains columns such as `station_id` and `temperature`; its `records` output contains only the selected columns, in the requested order. Processing is incremental by batch, and empty inputs retain an output schema. Column names must be unique and present. `piflow_drop_field@1.0` uses the same parameter structure to remove columns; like upstream, absent columns are ignored, but removing every column is rejected.
23
24Pure Python operators reuse the pinned image digest of `xstream/operator-python:1.0`. The SDK is installed in the shared Runtime; current repository packages contain only business code. Sources declare dependencies in `pyproject.toml` and select the Python dependency provider. The repository locks additional dependencies before execution and caches environments by content fingerprint. No dependencies are installed during Job execution. See the [dependency environment guide](../../../docs/dependency-environments.md).
25
26## Verify and publish individually
27
28Prepare a local directory containing `identity_python/1.1/manifest.json` and its runtime blob. The backup retained when clearing the live repository can serve as this directory. Run from the project root:
29
30```bash
31.venv/bin/python examples/operators/piflow/test.py select_field
32.venv/bin/python examples/operators/piflow/publish.py select_field \
33 --runtime-catalog /path/to/runtime-catalog \
34
35```
36
37The local packager runs the corresponding test and builds development artifacts under `examples/operators/piflow/target/catalog`. It does not publish to the repository; submit standardized sources using the source-only submission workflow below. Each invocation handles one operator.
38
39`mode: streaming` means incremental processing by Arrow batch. Implementations requiring global comparisons or state must declare an appropriate mode and resource limits. An Arrow streaming interface does not mean every algorithm executes in constant memory. Complete integration verification of external database, HDFS, Hive, and SSH operators additionally requires reachable test services and node authorization.
40
41## Publish to the operator repository
42
43Use the pinned EcoFlow checkout to reproduce byte-exact upstream sources in the published archives. The publishing tools build through the repository API, verify the candidate against Arrow execution and inference fixtures, and check the public downloads after review. The six existing Scala versions are handled by their own project in `../scala-piflow`.
44
45```sh
46python examples/operators/piflow/repository_publish.py --kind python \
47 --upstream-checkout /path/to/piflow --password-file /secure/admin-password \
48 --output /tmp/piflow-publication
49python examples/operators/piflow/repository_publish.py --kind spark \
50 --upstream-checkout /path/to/piflow --password-file /secure/admin-password \
51 --output /tmp/piflow-publication
52```
53
54The publisher uses the registered `python-arrow@2.0` and `piflow-ecoflow-spark@1.0` runtimes. Published source bundles include the reviewed source and example parameters. For a runnable example, resolve each `item_source` URI to an Item exposing the requested View. Standard Parquet fixtures can be converted to newline-delimited JSON records before using the directory catalog. Select a suitable output of `fork` or `route` at `client_sink`.
55
56
57## Scope and semantic differences
58
59The [port review](review.md) lists all 67 upstream operators, corresponding names, status, and limitations; the [Chinese review](review_zh.md) is maintained alongside it. States in `inventory.json` track adaptation work and do not replace the actual inventory of the current repository. Parameters, ports, executors, and execution modes are in `specs.json`. Build scientific Scala operators using [spark/README.md](spark/README.md).
60
61Generic SQL and chart-data operators use SQLite, so not every Spark SQL function is supported. Chart operators return records for plotting, without rendering images. `Fork` has fixed `left/right` outputs; `Route` has fixed `result/matched/unmatched` outputs. CSV and Excel readers return string columns without type inference; empty Excel cells become null. File-writing operators return byte streams rather than requiring server-side result files. Input/output examples are in the review.
62
63`MockData` declares `source` as a trigger input: its contents are consumed and discarded before generating records according to `schema/count`. Output is independent of input contents. `ReichsteinCheck` uses a direct Python implementation to avoid Spark jobs for repeated small-sample statistics. It explicitly fixes bin boundaries, deterministic threshold ordering, and invalid thresholds entering the median, and excludes missing values from estimation samples.
64
65### Excel dependencies
66
67Excel operators reuse the Python runtime and bundle pinned pure-Python dependencies and their licenses with the code. Execution never installs packages over the network. Download the wheels before building; publication verifies SHA-256 against `excel-dependencies.json`:
68
69```bash
70mkdir -p examples/operators/piflow/target/excel-deps
71.venv/bin/pip download --only-binary=:all: --no-deps \
72 --dest examples/operators/piflow/target/excel-deps \
73 openpyxl==3.1.5 et_xmlfile==2.0.0
74```
75
76Excel supports XLSX, not legacy XLS. Writing serializes a temporary workbook, and reading buffers the compressed file; neither is a constant-memory byte transformation. Input and expanded files are limited to 128 MiB, selected read ranges to one million cells, and writes to one million rows. Sheet/start-cell selection is supported; reads also support rectangular ranges. Dates become ISO text, formulas use existing cached results only, and text beginning with `=` remains text on export.
77
78```json
79{"header":true,"dataAddress":"'Weather Station'!B2:C100"}
80```
81
82This example is for `piflow_excel_read`; writing accepts only a start cell, such as `"'Weather Station'!B2"`. Excel's `header` parameter is a JSON boolean.
83
84## Verification and resources
85
86Python publication now runs SDK contract tests, builds the package, runs packaged-container tests, and uploads over HTTP in that order. Any failure stops publication. Built packages can also be tested independently:
87
88```bash
89.venv/bin/python examples/operators/piflow/container_test.py select_field
90.venv/bin/python examples/operators/piflow/spark/test.py add_day_id \
91 --spark-home /tmp/dacp-piflow-spark --docker
92```
93
94Fixtures cover cross-batch records, empty input, Unicode, quoted CSV/newlines, multiple inputs/outputs, invalid parameters, and scientific numeric examples; coverage varies by operator's test branch.
95
96Container tests use a read-only root, no network, 1 GiB memory, 128 processes/threads, and 512 MiB temporary space. Spark operators may create temporary shuffle files, and Excel export may create temporary workbook files; these are neither durable results nor checkpoints. Stateful Python algorithms typically limit materialized inputs or state to 128 MiB, with additional memory overhead for intermediate objects. Scientific JARs are approximately 230 KB, sharing a Spark runtime of approximately 413 MB. Python runtimes are shared independently; Excel code includes approximately 270 KB of compressed dependencies.
97
98Run `python examples/operators/piflow/audit.py --upstream-checkout /path/to/piflow --reports /tmp/piflow-publication` to verify pinned upstream files, published metadata, source browsing and download hashes.
99
100## Bounded output samples
101
102All Python adapters explicitly register `preview_outputs`. The Python adaptations use SDK `preview_process` to apply their pure transforms only to bounded upstream samples. CSV save returns sample bytes without writing files. The interface is required, and missing implementations fail explicitly. See [sample interfaces and limits](../../../docs/structure-inference.md#required-sample-interface-and-existing-operators). From the repository root, run `./.venv/bin/python scripts/test-operator-previews.py` to verify all six deployed operators and size limits.
103
104The repository accepts source only. Local builds in these examples are for development verification; use the [source submission and review workflow](../../../docs/operator-submission.md) for public publication.
105
106`piflow_merge@1.1` infers and combines left/right schemas by field name: retain left field order, append right-only fields, and fill absent fields with null while marking them nullable. Shared fields require identical types, except null types may adopt the other type; nested structures are preserved without implicit incompatible conversions. Execution appends left then right batches, and previews use the same logic; this is not a key-based join. If either input schema is unknown, the complete output schema remains unknown.
107
108For packaged checks use `DACP_PIFLOW_CATALOG=/path/to/catalog XSTREAM_PIFLOW_VERSION=1.2 python container_test.py merge`; the verifier loads the locked environment image when present.