Source code

Download source archive ↓

README.md

1# Item collection operators
2
3[中文](README_zh.md)
4
5Scala SDK implementations of `folder_items_source`, `csv_items_source`, and `item_inventory`, `item_byte_count`. Sources emit Item references, not file contents. They use the execution host's private Item access service and require a runtime containing the matching SDK, execution API and data-core versions.
6
7| Operator | Role | Input | Output |
8|---|---|---|---|
9| folder_items_source | source | URI parameter, recursive flag | items collection |
10| csv_items_source | source | CSV URI parameter, uri_column | items collection |
11| item_inventory | transform | items | text bytes containing URI/kind/format |
12| item_byte_count | transform | items | URI and byte count, reading one member at a time |
13
14
15## Build and test
16
17```bash
18cd examples/operators/item-sources
19sbt test package
20```
21
22The JVM SDK must be available in your configured artifact repository. `*.json` files illustrate execution and port declarations; repository submission additionally requires runtime pins, source provenance, icon, parameter help/examples and complete usage documentation. Output ports are inferred by operator methods, never declared at manifest top level. See [operator submission](../../../docs/operator-submission.md).
23
24## Use locally
25
26Configure an engine `inline_runtimes.jvm_tools` using a locally installed matching JVM image. Add the following definition to a local Job and reference it as `operator: {inline: folder_items}`:
27
28```yaml
29operators:
30  folder_items:
31    type: sdk
32    role: source
33    runtime: jvm_tools
34    program:
35      path: ./target/scala-2.13/xstream-item-sources_2.13-1.0.jar
36      entrypoint: xstream.examples.items.FolderItemsSource
37    inputs: {}
38```
39
40The step parameters are `{uri: 'context:/science/labels', recursive: true}`. For CSV use `CsvItemsSource` and `{uri: 'context:/science/manifest.csv', uri_column: uri}`. CSV manifests preserve row order, duplicates and other columns as metadata; member metadata can be unknown until opening. Relative paths refer to the manifest's logical parent. Raw filesystem paths and URI credentials are not accepted.
41
42Collection consumers declare `representations: [items]` and use `ctx.input("source").foreachItem`. They may open a member with `ctx.openItem(item, "bytes") { stream => ... }`; consume incrementally and release it inside the callback. See [Item reference streams](../../../docs/items-stream.md). Remote Jobs require approved repository references; inline definitions are local only.
43
44For a single Item, use the `view` parameter of `item_source` to read its contents directly. Collection Sources emit references for explicit access by collection consumers.
45
46## Publish
47
48Register runtime `jvm-items@1.0` with the matching SDK, then run:
49
50```bash
51python3 publish.py --url http://localhost:8820 \
52  --token-file /path/to/submitter.token \
53  --approve-password-file /path/to/admin-password
54```
55Omit the approval password option to await administrator review. `item_byte_count` skips directories without bytes, preserves duplicates and ordering, and fails the Job if a member cannot be read.