Built for the workdata teams actually do.
Data teams in banking, retail, automotive and manufacturing build and run their pipelines with DataFloem, on their own servers. These are the situations it is made for.
Replace ETL scripts nobody else can read.
Pipelines live in scripts and scheduler jobs that only their authors understand. Every change starts with reading code to find out what it touches.
How DataFloem handles it
- A job you can see. Sources, transformations and targets on one canvas, each node with typed columns.
- Rows at every step. Run up to any node and check its rows before the whole job runs.
- Versions and logs. Every version is a full copy of the job, and every run logs each step.
- Code where it fits. Write a step as SQL or Python when that is simpler, inside the same job.
Catch upstream changes before they break a report.
A source team adds, drops or retypes a column. Nothing fails right away; a downstream table goes wrong a few days later, and someone has to trace it back.
How DataFloem handles it
- Nodes remember their columns. Each node keeps the columns it was saved with and compares them with what arrives.
- Only the affected nodes are marked. A review marker names the changed columns on the nodes the change reaches.
- A person decides. Adopt the new columns or keep them out. A full run warns you while reviews are open.
Take notebook work to production without rewriting it.
Scripts and models are written in notebooks, then handed to another team to package and run on the cluster. Something is always lost along the way.
How DataFloem handles it
- Write in a Studio. Jupyter Lab or VS Code sessions run on a cluster assigned to the project. Save the files to the Code Flow from there.
- Package it on your servers. A Build Recipe orders the build steps, and an agent on your own server builds the image.
- Run it with the project's data. A Run Recipe runs the image on your cluster with the project's sources, targets and resource limits.
Build a lakehouse your analysts can query and your data office can govern.
Analysts need tables they can find and query. The data office needs to know who can read which table, and to show it.
How DataFloem handles it
- Jobs fill the catalogs. Write to Iceberg REST, JDBC or file catalogs, and keep tables in shape with compaction and snapshot cleanup.
- SQL next to the jobs. Analysts browse the catalog tree and query tables on the cluster they pick.
- Access down to the table. Policies, roles and bindings decide who can see each catalog, namespace and table.
Keep data work inside a regulated network.
Customer data cannot leave the network, and every access has to be explained to auditors. Cloud-only tools are ruled out before the evaluation starts.
How DataFloem handles it
- Self-hosted. Install DataFloem in your data center or your own cloud account. Jobs read and write your systems directly.
- Your directory, your identity provider. Sign in with LDAP or single sign-on, and let permissions follow your groups.
- Every request on record. Audit and change logs, kept from 0 to 730 days, module by module.
Run one data platform for many teams.
Each team runs its own tools and clusters. Nobody can see what the others built, and access requests travel by email.
How DataFloem handles it
- A space for each team. Namespaces keep teams, environments or customers apart; folders and projects organize the work inside.
- Shared infrastructure, assigned. Connections, clusters, Studios and catalogs are assigned to the projects that use them.
- Requests as tasks. Asking for a resource opens a task for the group that approves it.
- No overwritten work. The first edit locks a job or a Code Flow until its owner is done.
Move work from development to production without rebuilding it.
Jobs are rebuilt by hand in each environment, and a value meant for testing ends up in production.
How DataFloem handles it
- Export and import. Export a folder, a project, a job or a single version as a package, and import it into the next environment.
- Parameters for what differs. Keep the values that change between environments in parameters, not in the job.
- A default version. Keep a default version while you work on the next one.
Chosen by data teams in banking, retail, automotive and manufacturing.
Have a case that is not on this page?
Tell us about your team and your data, and we will show you how DataFloem fits.