From Sensor to Insight: Factory Data Pipelines with DataFloem

Why sensor data goes unused

A typical production line has hundreds of sensors measuring temperature, vibration, pressure and power draw. The readings exist, but they are scattered across PLCs, gateways and CSV exports, in different units and with gaps and spikes. Before anyone can ask "which machine is drifting?", someone has to collect, clean and join that data by hand.

That manual step is where most initiatives stall. It is slow, it depends on one or two people, and it breaks every time a new line or sensor is added. DataFloem replaces it with a visual, repeatable pipeline that runs on a scalable Spark cluster.

The scenario

Our example plant runs several production lines. Each machine reports temperature, vibration and pressure every few seconds. The machines' PLCs expose these readings over OPC UA, the industrial standard for machine-to-machine communication. A collector writes them to Apache Kafka. From there, a streaming DataFloem pipeline reads the topic and writes the readings to a MinIO bucket (S3-compatible) in an open file or open table format. This page does not cover that streaming step. Here we pick up where the files land in MinIO and build the batch pipeline that turns them into hourly signals.

Machine master data lives in a PostgreSQL database: each machine's line, model and install date, and the safe operating limits for each of its sensors.

The team wants three things:

• Clean data: remove broken readings, fill in units and timestamps consistently.
• Context: know which line and model each reading comes from, and which limits apply to each sensor.
• Signals: hourly summaries per machine and sensor, and a flag when a sensor runs outside its limits.

Collecting the data: OPC UA to Kafka to MinIO

OPC UA gives every machine a standard, secure way to publish its data, whatever the PLC vendor. Kafka adds a durable buffer between the shop floor and the analytics layer, so a slow or paused pipeline never loses a reading. MinIO keeps the full history as low-cost files that DataFloem reads directly.

1. Expose the tags. Each PLC, or the line's SCADA system, runs an OPC UA server. Engineers map the relevant tags, such as spindle temperature, motor vibration and hydraulic pressure, into a clear node hierarchy: plant > line > machine > sensor.
2. Secure the connection. The collector connects to each OPC UA server with certificate-based authentication and the SignAndEncrypt security mode, so readings are signed and encrypted on the plant network.
3. Subscribe, don't poll. The collector creates OPC UA subscriptions with a sampling interval per tag, for example every second for vibration and every 10 seconds for temperature. The server sends a value only when it changes, which keeps network load low.
4. Normalize at the edge. Each reading becomes one small message: machine ID, sensor ID, value, unit, source timestamp and OPC UA status code. Readings with a Bad status code are kept but marked, so the pipeline can decide what to do with them.
5. Publish to Kafka. The collector, for example a Kafka Connect OPC UA source connector or a lightweight edge gateway, writes the messages to a topic such as plant.sensor-readings. The machine ID is the message key, so all readings of one machine stay in order within a partition.
6. Land the stream in MinIO. A streaming DataFloem pipeline reads the topic continuously and writes the readings to MinIO. How that pipeline is built is out of scope here. What matters is the result: JSON files with one reading per line, partitioned by date and hour, for example topics/plant.sensor-readings/date=2026-10-27/hour=10/. The folders follow each reading's source timestamp, so a late reading still lands in the right hour.
7. Keep a replay window. Topic retention of a few days lets the streaming pipeline re-write files after an outage, and the bucket keeps the long-term raw history without touching the machines again.

Building the pipeline in DataFloem

The shop floor, Kafka and a streaming DataFloem pipeline handle collection. A batch DataFloem flow does all the processing in one visual job and writes analysis-ready tables for the people who act on them.

The whole pipeline is one job on DataFloem's flow canvas, with no code required. Each step below is a node you add with a right-click and configure in a side panel.

1. Connect the sources. In System Architecture > Sources, add an Object Storage connection for the MinIO bucket and an RDBMS connection for the PostgreSQL database. Test Connection confirms both before you save. Then assign them to the project.
2. Read the sensor files. Add a Source node on the Object Storage connection and set its Directory to the folder the streaming pipeline writes, with flow parameters for the hour to process: /topics/plant.sensor-readings/date=$$run_date/hour=$$run_hour/. The parameters, for example run_date = 2026-10-27 and run_hour = 10, are set under Flow > Parameters, so each run reads one completed hour. Choose file type JSON and use Read Columns from File to take the schema straight from a file in that folder: machine, sensor, value, unit, timestamp and status.
3. Clean the readings. A Filter node drops readings with a Bad OPC UA status, missing values or physically impossible values. An Expression node standardizes units and derives an hourly time bucket from the source timestamp.
4. Summarize. An Aggregation node groups by machine, sensor and hour and computes the average, minimum and maximum of each. Grouping by sensor keeps temperature, pressure and vibration apart, since their values cannot be averaged together. The node outputs only its group-by keys and the computed columns, so context is added after it.
5. Add context and flag limits. A second Source node reads the limits table from PostgreSQL, with one row per machine and sensor. A Join on machine and sensor attaches line, model and safe operating limits to each hourly summary. An Expression node then flags hours where a sensor's minimum or maximum crossed its limits. Joining after the summary also means the join handles one row per sensor per hour instead of every raw reading. For more advanced detection, a Python Scripting node can apply your own model.
6. Write the results. A Target node writes the flagged summaries to an Iceberg table registered in a DataFloem catalog, or to a reporting database, with Store Strategy Append. A Pre Query on the target first clears the rows of the hour being processed, so a rerun after an outage replaces that hour instead of duplicating it.
7. Check, then run. A Partial Run with Data Preview shows the output of any node before the full run. After the run, Logs show every run's details, and Versions keeps each change to the flow.

Because the flow runs on a Spark cluster, the same job scales from one line to the whole plant by adding workers, not by rewriting logic.

What it means for the business

For plant and operations leaders, the value is not the pipeline itself but what it makes routine:

• Earlier warnings, less downtime. Machines that drift outside their limits show up in hourly data, not in the next breakdown report.
• One source of truth. Maintenance, quality and management read the same cleaned, contextualized tables instead of separate spreadsheets.
• Less dependence on individuals. The logic lives in a visible, versioned flow that anyone on the team can open, review and change.
• Room to grow. Adding a new line means connecting its OPC UA server to the collector; the pipeline stays the same. Adding a new question means adding a node, not starting a new project.
• Governance built in. User and group permissions on folders, projects and jobs, plus audit and change logs, keep production data under control.

The result is a shorter path from "the sensor measured it" to "someone acted on it."

See it in action

The video below walks through the same pipeline on the DataFloem canvas, from connecting the MinIO bucket to previewing the hourly summaries.