Operator catalog

More tags45

All operators

Showing 81–114 of 114
OperatorVersionsExecutorTrustDownloadsActions
piflow_add_day_idAuthor dacp-examplesPiFlow · Data processing

Spark EcoFlow: sort year/month/day/hour; add id and day_id. Params: partitionNum string. Inputs materialized, max 128 MiB / one million rows.

jvm189.9 KiB6
piflow_add_day_stateAuthor dacp-examplesPiFlow · Data processing

EcoFlow sunrise/sunset day/night classifier using longitude, latitude and day_id (day-of-year); hour uses Beijing time. Params: longitude, latitude as strings. Spark batch, 128 MiB input limit.

jvm190.2 KiB5
piflow_add_uuid_stopAuthor dacp-examplesPiFlow · Data processing

Prepend a distinct 32-character hexadecimal UUID per row. Params: column (new column name).

python9.2 KiB5
piflow_batch_field_threshold_cleanAuthor dacp-examplesPiFlow · Data processing

EcoFlow inclusive threshold cleaning. left: records to clean; right: field,min_value,max_value,flag_value threshold records. No params. Fixes upstream Row-to-string field matching; field names must match exactly. Spark may coerce affected column types.

jvm189.6 KiB5
piflow_column_checkAuthor dacp-examplesPiFlow · Data processing

Validate full ordered column names against a reference and return left input. Params: none. Inputs: left, right. Left-input buffering limit: 128 MiB. Does not check physical units.

python8.6 KiB5
piflow_complement_by_memcacheAuthor dacp-examplesPiFlow · Data processing

Replace selected columns on a lookup key hit. Rows without a matching key keep their original values.

python35.7 KiB3
piflow_consistency_checkAuthor dacp-examplesPiFlow · Data processing

Left-join reference keys; prepend flagField=0 for matches, 1 otherwise. Params: flagField, correlationColumn (comma-separated). Duplicate reference keys retain join multiplicity. 128 MiB input limit each.

python9.7 KiB5
piflow_convert_schemaAuthor dacp-examplesPiFlow · Data processing

Rename columns; does not cast data types. Params: schema, e.g. station_id->station,temperature->air_c.

python9.1 KiB5
piflow_current_and_post_value_interpolationAuthor dacp-examplesPiFlow · Data processing

Interpolate two consecutive missing same-hour daily observations between valid endpoints. Requires year,month,day,hour. Params fields,qualityControlMarkerValue string. Fixes ignored marker and upstream absolute-difference error to support decreasing series.

jvm190.8 KiB5
piflow_data_sortAuthor dacp-examplesPiFlow · Data processing

Globally sort a bounded input on one or more fields. Ascending order places nulls first, and descending places them last.

python35.7 KiB3
piflow_distinctAuthor dacp-examplesPiFlow · Data processing

Deduplicate records across Arrow batches; retains the first row per key. Params: optional columnNames; empty means all columns. Key-state limit: 128 MiB.

python9.0 KiB5
piflow_dr_coordinate_rotationAuthor dacp-examplesPiFlow · Data processing

EcoFlow double coordinate rotation. Params map covariance/wind columns and output Fc,Le,Hs,u_star names. Nonzero horizontal wind required; outputs five-decimal strings. Original Breeze matrix algorithm.

jvm193.7 KiB5
piflow_drop_fieldAuthor dacp-examplesPiFlow · Data processing

Remove record columns incrementally. Params: columnNames (comma-separated string); absent names ignored.

python9.2 KiB5
piflow_execute_sql_stopAuthor dacp-examplesData transformation · SQL

Read-only SQLite SQL over a named input relation (not Spark SQL). Params: sql, viewName. Scalar inputs, max 128 MiB. Output types inferred; empty computed columns have null type.

jvm61.0 KiB6
piflow_filterAuthor dacp-examplesData transformation · SQL

Filter Arrow batches by a SQLite SQL predicate (not Spark SQL). Params: condition, e.g. quality = 0 AND temperature >= 12. Scalar columns only.

jvm61.0 KiB8
piflow_front_and_back_values_are_validAuthor dacp-examplesPiFlow · Data processing

EcoFlow same-hour interpolation using valid preceding and following days. Requires day_id,hour. Params fields and qualityControlMarkerValue integer string; fixes upstream ignored marker.

jvm190.8 KiB5
piflow_joinAuthor dacp-examplesData transformation · Relational

Key-based inner/left/right/full_outer record join. Params: joinMode, correlationColumn. Colliding right non-key columns receive _right suffix. 128 MiB per input.

jvm52.0 KiB7
piflow_mdv_gAuthor dacp-examplesPiFlow · Data processing

EcoFlow MDV sliding-window interpolation at the same hour on neighboring days, excluding current day. Requires day_id,hour. Params fields,mdv_num (positive odd string),qualityControlMarkerValue (integer string).

jvm190.5 KiB5
piflow_mdv_iAuthor dacp-examplesPiFlow · Data processing

EcoFlow fixed-window MDV interpolation grouped by ceiling(day_id/mdv_num),hour. Preserves original zero contribution of flagged values to the group mean. Adds zone; fixes multiple-field replacement. Params fields,mdv_num positive string,qualityControlMarkerValue integer string.

jvm190.5 KiB5
piflow_mergeAuthor dacp-examplesData transformation · Relational

Stream left records followed by right records, retaining duplicates. Merge schemas by field name: preserve left field order, append right-only fields and fill missing values with null. Conflicting types are rejected. No params. Fixed left/right ports.

jvm52.0 KiB10
piflow_mock_dataAuthor dacp-examplesPiFlow · Data processing

Random Arrow records. source is a trigger stream consumed and discarded (matching upstream unused default input); output independent of input. Params schema e.g. name:String,age:Int,optional:Double:true; count string0..1000000. Types String/Int/Long/Float/Double/Boolean. Batches1024.

python10.3 KiB5
piflow_pf_coordinate_rotationAuthor dacp-examplesPiFlow · Data processing

EcoFlow planar-fit coordinate rotation using OLS and Breeze matrices. Params covariance/wind column names and output names. Requires a full-rank wind regression sample. Fixes upstream repeated Fc output into Le,Hs,u_star.

jvm193.7 KiB5
piflow_rainfall_data_cullingAuthor dacp-examplesPiFlow · Data processing

EcoFlow rainfall quality control: flag selected flux columns at rainy timestamps and following 30 minutes. left=flux,right=meteorology; both require year,month,day,hour. Params fields,rainFallField,flagValue integer string. Upstream drops flux rows whose rainfall is null or unmatched.

jvm190.9 KiB5
piflow_reichstein_checkAuthor dacp-examplesPiFlow · Data processing

EcoFlow Reichstein u-star screening implemented without Spark: six equal-count temperature groups, twenty wind bins, Pearson<0.4, median valid threshold; flag nighttime low-wind FC. Params soilTemperatureField,frictionWindSpeedField,FC strings,flagValue int32,percentFlux numeric(0,1]. Excludes null/nonfinite/flagged samples from estimation; requires120 valid observations. Fixes upper-bin bound and deterministic threshold order. Bounded128MiB.

python11.4 KiB5
piflow_select_fieldAuthor dacp-examplesPiFlow · Data processing

Select named record columns in order, incrementally by Arrow batch. Params: columnNames (comma-separated string).

python2.6 KiB8
piflow_single_field_threshold_cleanAuthor dacp-examplesPiFlow · Data processing

EcoFlow inclusive field thresholds with an additional Spark SQL condition; replace invalid values with flagValue. Params: fields,maxValue,minValue,flagValue,additionalCondition (strings). Spark may coerce affected column types.

jvm191.0 KiB5
piflow_storage_item_ecAuthor dacp-examplesPiFlow · Data processing

EcoFlow EC CO2 storage correction for 30-minute observations sorted by day_id,hour. Params Fc,co2_Avg column names,para_H height metres string,flagValue numeric string. Adds id; retains upstream missing/large-jump condition.

jvm190.6 KiB5
piflow_storage_item_pm__hsAuthor dacp-examplesPiFlow · Data processing

EcoFlow seven-layer sensible heat profile storage correction. left flux,right meteorology; date columns year,month,day,hour. Params Hs,rho_a_Avg column names,equipmentHeightDifference seven metre heights,everyTaFields seven temperature columns,time_interval positive seconds string. Fixes ignored density column name.

jvm192.3 KiB5
piflow_storage_item_pm_leAuthor dacp-examplesPiFlow · Data processing

EcoFlow seven-layer latent heat profile storage correction. left flux,right meteorology; year,month,day,hour required. Params LE column,equipmentHeightDifference seven metre heights,everyTaFields seven Celsius columns,everyPvaporFields seven vapor-pressure columns,time_interval seconds string. Original vapor conversion216.5*p/(273.15+T).

jvm192.4 KiB5
piflow_subtractAuthor dacp-examplesPiFlow · Data processing

EXCEPT DISTINCT over whole rows: return unique left rows absent from right. Nulls compare equal for this set operation. No params. 128 MiB input limit each.

python8.9 KiB5
piflow_wild_spot_culling_mdv_gAuthor dacp-examplesPiFlow · Data processing

EcoFlow outlier removal from neighboring id windows, excluding current row. One field per step. Requires id; params fields,mdv_num positive odd string,multiple nonnegative string,statisticalType stddev/variance,flagValue numeric string. Fixes upstream SQL syntax and undefined contrast_flag.

jvm192.0 KiB5
piflow_wild_spot_culling_mdv_iAuthor dacp-examplesPiFlow · Data processing

EcoFlow outlier removal in fixed windows ceiling(id/mdv_num). Requires id. Params fields (one column),mdv_num positive string,multiple nonnegative string,statisticalType stddev/variance,flagValue numeric string. Fixes upstream ignored statistic.

jvm191.3 KiB5
piflow_wpl_correctionAuthor dacp-examplesPiFlow · Data processing

EcoFlow WPL flux correction, preserves original conversions and five-decimal string output Fc,Le,Hs. All params are column-name strings. co2_Avg in mg/m3,h2o_Avg in g/m3,rho_a_Avg in kg/m3,Ts_Avg in Celsius. Requires non-null numeric columns.

jvm191.0 KiB5
script_jsonl_projectAuthor xstream-script-examplesJSON · Data processing

Select top-level fields from JSONL objects without buffering the complete stream.

python9.4 KiB0