Why we’re talking about this
Companies using Arivonix are handling more data than they used to, with more transformations to run, more rules to follow, and less time to spend waiting for results. That part isn’t really new, since data has been growing for years and most teams are used to that story by now.
What’s changed lately is how uneven the work has become. Some days a pipeline runs in a few minutes without any trouble at all. Other days, that same customer needs to push through a much bigger job, with more rows, more joins, and more cleanup, and it still needs to finish quickly. It isn’t always obvious in advance which kind of day it’s going to be.
We kept hearing a similar thing from customers, in one form or another: they didn’t want to rebuild how they work, they just wanted a bit more room for when a job got bigger than usual.
What was actually missing
It’s easy to assume the answer is simply more compute. Bigger machines, more nodes and the problem takes care of itself. But looking closer, that wasn’t really the issue.
Most of the time, a small everyday pipeline and a much bigger, heavier one were being pushed through the same engine, tuned the same way. That tends to work fine for a while, but it starts to show its limits as jobs get more varied. On a quiet day, you might be paying for more capacity than you actually need. On a heavier day, that same setup can leave you waiting, simply because it wasn’t really built to absorb something that size.
Looking at it this way, the gap had less to do with raw power and more to do with having options. There was really only one lane for every kind of workload to travel through, with no easy way to send the bigger, heavier jobs down a different path when they needed one.
What we built: two paths instead of one
Rather than replace what already works, we added a second path alongside it, so pipelines have somewhere to go depending on what they need.
- Native engine – your regular pipelines keep running much like they do today, with the same setup and the same logic, and nothing you need to change on your end.
- Databricks – for the bigger, heavier, or harder-to-predict jobs, Arivonix can now send that work over to Databricks, which is built with that kind of scale in mind.
Here’s a simple picture of how that looks in practice:
A pipeline comes in, and depending on what it needs, it can run on the native engine as it always has, or it can be routed over to Databricks for some extra muscle. Either way, the results come back to you the same way you’re used to.
This also means we’re able to offer Databricks’ own data warehouse as an option, instead of routing everything through AWS by default. For customers whose setup or governance rules line up better with that, it’s there to use. For everyone else, nothing about their current setup needs to change.
What this actually looks like day to day
Say you run a hundred pipelines a week. Ninety-five of them are the usual sort of thing, like reports, refreshes, and standard transformations, and those continue running on the native engine much as they always have. You likely won’t notice anything different about them.
The other five tend to be the ones that used to slow everything else down. Maybe it’s a monthly job pulling in a much larger batch of records, or a one-off analysis that’s bigger than your typical load. Those are the kinds of jobs that can now be routed through Databricks instead, so they don’t end up dragging down the rest of your pipelines or forcing you to over-provision your whole system just to get through a few busier days.
None of this has to be decided once and locked in either. It can be turned on for a single pipeline and left off everywhere else, and if a workload changes down the road, the path it runs on can change along with it.
Why this matters more than it might sound like
On its own, adding Databricks support doesn’t sound like a dramatic shift. But the real point is that Arivonix can now scale a bit unevenly, on purpose, because real workloads tend to be uneven too. Ideally, nobody should have to redesign their entire pipeline setup just because one job happened to outgrow it.
This also tends to matter for teams with heavier governance requirements. Some customers have specific rules about where data can live and how it gets processed, and having Databricks available as an option, including its own data warehouse instead of always routing through AWS, gives those teams a path that fits their rules without asking them to step outside the rest of Arivonix.
And because none of this is required, adoption is really up to each customer. Some may never need to touch it at all. Others might route just one heavy pipeline through it and leave it there. Both are perfectly fine outcomes.
Bottom line
It’s still the same Arivonix, with the same pipelines and the same logic you’ve already built. What’s different is what happens once a job outgrows the usual path, since there’s now somewhere for it to go instead of everyone else waiting on it.
Use it where it helps, and skip it where it doesn’t.
