ZML is an open source inference tool from a French startup, available for free. Its selling point is running AI models across chips from multiple vendors without having to write separate code for each platform. The key strength is cutting down on repetitive integration work when deploying models on different chip brands — a good fit for teams tired of dealing with vendor lock-in. Right now it’s still new to the market, worth watching where it goes, but it’s too early to say for sure whether it can really replace existing tooling.
TL;DR: A free cross-chip AI inference tool from a French team — an interesting idea, but you’ll want to wait for wider real-world adoption before deciding it’s worth switching to.
First look: what is ZML
ZML is a French startup that released a free product to help with inference across multiple AI chip brands at once. The selling point is that dev teams no longer need to write separate code for each chip — write it once and run it across platforms.
This sounds like it addresses a problem many teams actually face: each chip maker has its own toolchain, which makes moving workloads across platforms a hassle. Having a middle layer like this is appealing for teams that don’t want to be locked into a single manufacturer.
When your model hits a bottleneck because you picked the wrong chip vendor
Most infra teams run into the same problem: they write inference code tied to CUDA from the start of a project, because at the time NVIDIA was the easiest to get and had the most complete library support.
As the model grows and traffic increases, they want to expand to AMD or TPU to cut cost per request — only to find the code they wrote doesn’t work with other chip vendors at all. Nearly the whole stack needs to be rewritten.
This is where you lose both dev team time and the chance to save on costs, because you’re locked into a single supplier even as the inference chip market keeps offering more and more options.
This is exactly the problem ZML says its new product solves from the ground up.
Where ZML stands in the AI infrastructure battlefield
Today’s AI infra market is split into two main layers: the dev-facing framework layer (PyTorch, XLA) and the layer that talks directly to hardware, like CUDA, which is tied to a single chip vendor.
ZML positions itself as a vendor-neutral compiler/runtime layer sitting in between. Simply put, it’s an interpreter that translates code to run on multiple chip vendors without needing to rewrite it every time you switch suppliers.
The target audience is clear: ML infra teams managing production-grade inference pipelines, and companies that want an exit plan from single-vendor chips — not general developers just casually fine-tuning models for fun.
This positioning is risky, though, because it has to compete with both NVIDIA, which dominates the market, and the framework layer, where major players are already trying to build their own portability. So ZML has to prove that a middle layer like this is genuinely necessary, not just redundant.
From single-chip lock-in to cross-hardware execution in one codebase
Previously, if you trained a model on NVIDIA and wanted to move it to run on a different chip vendor, you’d have to rewrite almost the entire stack, because each vendor has its own compiler. ZML fixes this by letting you write inference code once, then having the middle layer translate it to run on different hardware instead.
What’s interesting is the foundation it’s built on — Zig and MLIR — an approach that prioritizes portability at the compiler level itself, not just an API wrapper slapped on top the way many others do.
| Factor | Before ZML | After ZML |
|---|---|---|
| Locked to which chip vendor | Locked to one vendor | Cross-vendor |
| Separate code per platform | Rewrite every time | Write once, run anywhere |
| Cross-platform deploy speed | Slow, manual stack adjustments | Faster, handled by the middle layer |
| Technology foundation | Tied to a vendor-specific compiler | Zig + MLIR |
What it’s like in practice
Here’s how it maps to situations infra teams commonly run into.
GPU shortage mid-project — Models compiled with ZML aren’t tied to a single chip. When NVIDIA has a long queue, you can switch to running on AMD or TPU without rewriting any code.
Migrating clouds without touching code — Since the middle layer handles talking to hardware for you, teams spreading workloads across multiple cloud providers don’t need to maintain a separate stack for each one.
Picking chips based on price/performance for the job — Some inference workloads suit certain chip models better than others. A vendor-independent system gives you more freedom to choose, instead of being stuck using whatever chip you originally bought forever.
Faster deployment of new models — Because it’s built on Zig + MLIR, the compiler handles the middle layer for you, so dev teams don’t have to keep adjusting the stack every time the target hardware changes.
How it stacks up against competitors in the same arena
ZML isn’t the only player in the vendor-neutral inference engine market — but what sets it apart is its underlying philosophy.
vLLM and TensorRT-LLM are already widely used, but TensorRT-LLM is fairly tightly bound to NVIDIA, while vLLM is more open but still primarily optimized for GPUs.
Modular MAX is the closest to ZML in philosophy — both share the “cross-chip” vision. The difference is that Modular uses its own Mojo language, while ZML chose Zig + MLIR.
| Factor | ZML | vLLM | TensorRT-LLM |
|---|---|---|---|
| Multi-vendor chip support | Yes (cross-platform) | Mainly GPU | NVIDIA only |
| Open source | Yes | Yes | Partial |
| Language/technology foundation | Zig + MLIR | Python/CUDA | C++/CUDA |
Bottom line: if an organization wants to commit fully to a single vendor, TensorRT-LLM is still the strongest option. But if you want to keep your options open, ZML is worth keeping an eye on.
Pros and cons to know before putting it into real use
Before jumping into using ZML on a real project, weigh it carefully — this is new tech that hasn’t been battle-tested for long.
The clear upside is that it’s free and not tied to any single chip vendor, giving teams the option to switch hardware without rewriting all their code — unlike TensorRT-LLM, which is locked to NVIDIA only.
On the flip side, the product is still very new. Documentation and usage examples aren’t as complete as competitors that have been on the market longer, and there aren’t enough real-world production cases yet to speak to its long-term stability.
Pros
- +Free, no license fees, ready to use immediately
- +Vendor-neutral, runs across multiple chip vendors, reducing the risk of being locked into a single hardware platform
Cons
- −Still a new product; documentation and usage examples aren't complete yet
- −Long-term stability in production workloads hasn't been proven yet
”Free” doesn’t mean there’s no cost
ZML being “free” only covers the license fee — the real cost shows up elsewhere.
First is the engineering team’s time spent learning a new system and migrating existing inference code tied to the old stack. The bigger the production system, the longer that adjustment takes.
Second is the risk of depending on a small startup that doesn’t yet have a long track record. If ZML shuts down or changes business direction one day, teams that built their systems around it will have to find their own way out.
So before putting it into real production, you need to evaluate two things separately — the cost of adopting it now, and the long-term risk of maintaining it — not just look at the “free” price tag on the website and call it done.
Who should try it now, who can wait
If you’re on an R&D team with time to experiment with new tools, or a startup already struggling with AI chip costs, ZML is worth trying right away — it’s free, and it isn’t tied to any one chip vendor. The only costs are setup time and the risks mentioned above. For organizations running production systems serving large numbers of real users and requiring strict SLA-level stability, it’s better to wait a while and let ZML build up a track record and a larger community before migrating core systems onto it.
Made for
- R&D teams that want to experiment with new infra ahead of competitors
- Startups looking to cut AI chip costs and willing to accept some risk
- Developers who enjoy trying out new technology
Think twice
- Teams with mid-sized production systems should run a separate pilot before migrating fully
Skip this one
- Organizations with mission-critical systems that require maximum stability — wait until ZML has a longer track record
What to watch next
The really interesting part isn’t just how fast ZML is — it’s that it’s “free” and can run across multiple chip vendors. If this model catches on, it could shake up the lock-in that the big GPU vendors have been trying to build all along.
Picture a team that no longer has to be tied to a single vendor — able to negotiate on price, move workloads based on actual cost. The AI chip market would shift from “who’s the fastest” to “who’s the most flexible” instead.
But all of this is still just a direction, not a conclusion — new tech from a startup still has to prove itself in the field for a good while longer.
If you want to know right away whether it fits your own workload, we’d recommend trying a small pilot first, then sharing your results in the comments — or follow Prism, and we’ll keep you updated on ZML’s progress.