Operator catalog

More tags45

All operators

Showing 1–38 of 38
OperatorVersionsExecutorTrustDownloadsActions
piflow_add_day_idAuthor dacp-examplesPiFlow · Data processing

Spark EcoFlow: sort year/month/day/hour; add id and day_id. Params: partitionNum string. Inputs materialized, max 128 MiB / one million rows.

jvm189.9 KiB7
piflow_add_day_stateAuthor dacp-examplesPiFlow · Data processing

EcoFlow sunrise/sunset day/night classifier using longitude, latitude and day_id (day-of-year); hour uses Beijing time. Params: longitude, latitude as strings. Spark batch, 128 MiB input limit.

jvm190.2 KiB6
piflow_batch_field_threshold_cleanAuthor dacp-examplesPiFlow · Data processing

EcoFlow inclusive threshold cleaning. left: records to clean; right: field,min_value,max_value,flag_value threshold records. No params. Fixes upstream Row-to-string field matching; field names must match exactly. Spark may coerce affected column types.

jvm189.6 KiB6
piflow_bisecting_kmeans_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark bisecting k-means model to Arrow records and append prediction.

jvm22.6 KiB5
piflow_bisecting_kmeans_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark bisecting k-means model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.2 KiB4
piflow_current_and_post_value_interpolationAuthor dacp-examplesPiFlow · Data processing

Interpolate two consecutive missing same-hour daily observations between valid endpoints. Requires year,month,day,hour. Params fields,qualityControlMarkerValue string. Fixes ignored marker and upstream absolute-difference error to support decreasing series.

jvm190.8 KiB6
piflow_decision_tree_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark decision tree model to Arrow records and append prediction.

jvm22.5 KiB6
piflow_decision_tree_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark decision tree model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.2 KiB5
piflow_dr_coordinate_rotationAuthor dacp-examplesPiFlow · Data processing

EcoFlow double coordinate rotation. Params map covariance/wind columns and output Fc,Le,Hs,u_star names. Nonzero horizontal wind required; outputs five-decimal strings. Original Breeze matrix algorithm.

jvm193.7 KiB6
piflow_front_and_back_values_are_validAuthor dacp-examplesPiFlow · Data processing

EcoFlow same-hour interpolation using valid preceding and following days. Requires day_id,hour. Params fields and qualityControlMarkerValue integer string; fixes upstream ignored marker.

jvm190.8 KiB6
piflow_gaussian_mixture_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark Gaussian mixture model to Arrow records and append prediction.

jvm22.6 KiB5
piflow_gaussian_mixture_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark Gaussian mixture model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.2 KiB4
piflow_gbt_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark gradient-boosted trees model to Arrow records and append prediction.

jvm22.5 KiB5
piflow_gbt_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark gradient-boosted trees model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.1 KiB4
piflow_kmeans_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark k-means model to Arrow records and append prediction.

jvm22.5 KiB5
piflow_kmeans_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark k-means model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.2 KiB4
piflow_lda_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark online latent Dirichlet allocation model to Arrow records and append topicDistribution.

jvm22.5 KiB5
piflow_lda_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark online latent Dirichlet allocation model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.1 KiB4
piflow_logistic_regression_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark logistic regression model to Arrow records and append prediction.

jvm22.6 KiB5
piflow_logistic_regression_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark logistic regression model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.3 KiB4
piflow_mdv_gAuthor dacp-examplesPiFlow · Data processing

EcoFlow MDV sliding-window interpolation at the same hour on neighboring days, excluding current day. Requires day_id,hour. Params fields,mdv_num (positive odd string),qualityControlMarkerValue (integer string).

jvm190.5 KiB5
piflow_mdv_iAuthor dacp-examplesPiFlow · Data processing

EcoFlow fixed-window MDV interpolation grouped by ceiling(day_id/mdv_num),hour. Preserves original zero contribution of flagged values to the group mean. Adds zone; fixes multiple-field replacement. Params fields,mdv_num positive string,qualityControlMarkerValue integer string.

jvm190.5 KiB5
piflow_multilayer_perceptron_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark binary multilayer perceptron model to Arrow records and append prediction.

jvm22.6 KiB5
piflow_multilayer_perceptron_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark binary multilayer perceptron model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.3 KiB4
piflow_naive_bayes_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark multinomial naive Bayes model to Arrow records and append prediction.

jvm22.5 KiB5
piflow_naive_bayes_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark multinomial naive Bayes model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.2 KiB4
piflow_pf_coordinate_rotationAuthor dacp-examplesPiFlow · Data processing

EcoFlow planar-fit coordinate rotation using OLS and Breeze matrices. Params covariance/wind column names and output names. Requires a full-rank wind regression sample. Fixes upstream repeated Fc output into Le,Hs,u_star.

jvm193.7 KiB5
piflow_rainfall_data_cullingAuthor dacp-examplesPiFlow · Data processing

EcoFlow rainfall quality control: flag selected flux columns at rainy timestamps and following 30 minutes. left=flux,right=meteorology; both require year,month,day,hour. Params fields,rainFallField,flagValue integer string. Upstream drops flux rows whose rainfall is null or unmatched.

jvm190.9 KiB5
piflow_random_forest_predictionAuthor dacp-examplesPiFlow · Machine learning

Apply a trained Spark random forest model to Arrow records and append prediction.

jvm22.6 KiB5
piflow_random_forest_trainingAuthor dacp-examplesPiFlow · Machine learning

Train a Spark random forest model on Arrow records and emit the trained model as a ZIP byte stream.

jvm21.2 KiB4
piflow_single_field_threshold_cleanAuthor dacp-examplesPiFlow · Data processing

EcoFlow inclusive field thresholds with an additional Spark SQL condition; replace invalid values with flagValue. Params: fields,maxValue,minValue,flagValue,additionalCondition (strings). Spark may coerce affected column types.

jvm191.0 KiB5
piflow_storage_item_ecAuthor dacp-examplesPiFlow · Data processing

EcoFlow EC CO2 storage correction for 30-minute observations sorted by day_id,hour. Params Fc,co2_Avg column names,para_H height metres string,flagValue numeric string. Adds id; retains upstream missing/large-jump condition.

jvm190.6 KiB5
piflow_storage_item_pm__hsAuthor dacp-examplesPiFlow · Data processing

EcoFlow seven-layer sensible heat profile storage correction. left flux,right meteorology; date columns year,month,day,hour. Params Hs,rho_a_Avg column names,equipmentHeightDifference seven metre heights,everyTaFields seven temperature columns,time_interval positive seconds string. Fixes ignored density column name.

jvm192.3 KiB5
piflow_storage_item_pm_leAuthor dacp-examplesPiFlow · Data processing

EcoFlow seven-layer latent heat profile storage correction. left flux,right meteorology; year,month,day,hour required. Params LE column,equipmentHeightDifference seven metre heights,everyTaFields seven Celsius columns,everyPvaporFields seven vapor-pressure columns,time_interval seconds string. Original vapor conversion216.5*p/(273.15+T).

jvm192.4 KiB5
piflow_wild_spot_culling_mdv_gAuthor dacp-examplesPiFlow · Data processing

EcoFlow outlier removal from neighboring id windows, excluding current row. One field per step. Requires id; params fields,mdv_num positive odd string,multiple nonnegative string,statisticalType stddev/variance,flagValue numeric string. Fixes upstream SQL syntax and undefined contrast_flag.

jvm192.0 KiB5
piflow_wild_spot_culling_mdv_iAuthor dacp-examplesPiFlow · Data processing

EcoFlow outlier removal in fixed windows ceiling(id/mdv_num). Requires id. Params fields (one column),mdv_num positive string,multiple nonnegative string,statisticalType stddev/variance,flagValue numeric string. Fixes upstream ignored statistic.

jvm191.3 KiB5
piflow_word_to_vecAuthor dacp-examplesPiFlow · Machine learning

Train Spark Word2Vec on the input corpus and append the mean token embedding for each document.

jvm21.1 KiB4
piflow_wpl_correctionAuthor dacp-examplesPiFlow · Data processing

EcoFlow WPL flux correction, preserves original conversions and five-decimal string output Fc,Le,Hs. All params are column-name strings. co2_Avg in mg/m3,h2o_Avg in g/m3,rho_a_Avg in kg/m3,Ts_Avg in Celsius. Requires non-null numeric columns.

jvm191.0 KiB5