Pilot
NeuroBlock pilot: 3.2× structured output throughput on the same GPUs
Feniria Labs 3 min read
We’re partnering with NeuroBlock to help them get more out of the AI infrastructure they already own. The first numbers are in: on structured output, throughput went from 193 to 614 tokens per second (3.2×) on the same NVIDIA GPUs and the same model, with sub-second time to first token.
3.2×
Structured output throughput
193 → 614 output tokens per second
<1 s
Time to first token
Median ~170 ms at that load
0
GPUs added
Same NVIDIA GPUs, same model
659
Peak tokens per second (3.4×)
Queues start to form above this point
Who NeuroBlock is
NeuroBlock runs private AI models for its clients on its own GPUs, so the data never leaves their infrastructure. That is the reason their clients choose them, and it is also the constraint: every new client means more load on hardware NeuroBlock owns and operates.
Feniria works alongside their team to tune how those models run, so they can keep growing efficiently without adding hardware every time demand goes up.
For a fast-moving AI team like NeuroBlock, this means:
- More capacity from the GPUs already in place.
- Budget that goes into product, not extra infrastructure.
- Faster responses for the customers using their models.
Why structured output
Structured output was NeuroBlock’s priority for this first phase: the model answers in JSON that follows a schema, so another system can read it directly. It is a common pattern in private AI (extraction, classification, data generation), and it puts specific pressure on the inference engine. Outputs are short, many requests share the same prompt prefix, and the schema has to hold on every response.
We benchmarked that shape of traffic: inputs of around 800 tokens with a shared 200-token prefix, and JSON outputs of roughly 90 to 135 tokens.
What changed
The “before” is NeuroBlock’s serving configuration as it ran in production, stressed with benchmark traffic: 193 output tokens per second.
Feniria then searched the inference engine’s configuration for this workload: how many requests run at once, how the KV cache is stored, prefix caching for the shared part of the prompt, and a 4-bit build of the same model weights. Every candidate was measured under load, and output quality was checked for valid JSON and schema compliance.
The “after” is 614 output tokens per second with median time to first token around 170 ms.
| Before | After | |
|---|---|---|
| Output tokens per second | 193 | 614 |
| Gain | 3.2× | |
| Time to first token | Sub-second (median ~170 ms) | |
| GPUs | Same NVIDIA GPUs | Same NVIDIA GPUs |
| Model | Same | Same, with 4-bit weights |
Why we report 614, not 659
If you keep raising the request rate, the tuned setup reaches 659 tokens per second (3.4×). At that point requests start to queue and first-token latency climbs past a second. A number your users would feel as slowness is not a useful capacity number, so we report the highest point where responses stay fast.
That is the general rule we follow: throughput only counts at the latency target.
What’s next
These are benchmark results on traffic shaped like NeuroBlock’s workload. Next we confirm them over longer sustained runs, test them on NeuroBlock’s production cluster, and extend the work to their other models and task types.
There’s more to learn as we grow together, and we’ll share what we find along the way. Thank you to the NeuroBlock team for the trust.
What teams usually want to know
Questions about this post
What is NeuroBlock?
NeuroBlock runs private AI models for its clients on its own GPUs, so client data never leaves their infrastructure.
What did the pilot measure?
Output tokens per second on a structured output workload (JSON responses that follow a schema), before and after tuning, on the same NVIDIA GPUs, the same model and the same prompt set. The result was 193 → 614 tokens per second, a 3.2× gain, with sub-second time to first token.
Why 3.2× and not 3.4×?
The tuned setup reaches 659 tokens per second at its peak, but at that rate requests start to queue and first-token latency rises. 614 tokens per second is the highest point where responses stay fast, so that is the number we report.
Did NeuroBlock have to buy more GPUs?
No. All results are on the GPUs NeuroBlock already had. The point of the pilot is to grow capacity without adding hardware every time demand goes up.
Work with us
Find out if self-hosting pays off for you.
Tell us your model, request volume, traffic pattern and current cost. We'll come back with an initial estimate, and an honest answer if an API is still the better choice.