README.md
1# Scala Source and Sink examples
2
3[中文](README_zh.md)
4
5Four small operators demonstrate graph roles independently of implementation type. All four use `type: sdk` and the Scala JVM SDK. They process Arrow-backed bytes streams in bounded chunks; no file is written to the server and no external service is contacted.
6Each implementation uses the base `Operator` interface; the published manifest declares its Source or Sink role and port contract. The dedicated `SourceOperator` and `SinkOperator` interfaces are optional role markers.
7
8| Operator | Role | Behavior |
9|---|---|---|
10| `text_source@1.0` | Source | Emits UTF-8 `text` repeated `repeat` times, with an 8 MiB output bound |
11| `range_source@1.0` | Source | Emits `count` signed integers from `start`, one per line; at most 100,000 values |
12| `contains_text_sink@1.0` | Sink | Consumes bytes and fails the Job unless `text` occurs, including across chunk boundaries |
13| `min_lines_sink@1.0` | Sink | Consumes bytes and fails the Job unless at least `minimum` lines occur |
14
15Source steps have no inputs. Sink steps consume an input and have no output port. For example:
16
17```yaml
18name: Check generated lines
19steps:
20- id: generate
21 operator: range_source@1.0
22 inputs: {}
23 params: {start: 1, count: 100}
24- id: check
25 operator: min_lines_sink@1.0
26 inputs: {source: generate}
27 params: {minimum: 100}
28```
29
30To return a Source stream to a caller, connect it to `client_sink@1.0` instead of a validation Sink. Source and Sink examples do not replace `item_source@1.0`, which reads catalog Items, or dataset publication, which persists data products.
31
32Run contract tests with `sbt test` in this directory. Submit and approve on a **local** repository with:
33
34```bash
35python3 publish.py --url http://127.0.0.1:8820 \
36 --token-file /secure/submitter.token \
37 --approve-password-file /secure/admin-password
38```
39
40Omit `--approve-password-file` to leave submissions pending for administrator review. The script packages only Scala source, `pom.xml`, documentation and license; the repository builds the thin JAR. Runtime `jvm-arrow@2.1` must already be registered. See [operator development](../../../docs/operator-authoring.md) and [submission](../../../docs/operator-submission.md).