Introduction to Flux.1 – The new standard for image generation
Flux.1 is a new series of models from Black Forest Labs that has set the new standard in quality and prompt adherence, and it can even render legible text. The Flux.1-Schnell version of the model generates high quality images in just 4 steps, and is released under the permissive and commercially usable Apache 2 license. In this benchmark, we measure speed and cost performance of this new model on SaladCloud.
Benchmark design
We deployed the “Flux.1-Schnell (FP8) – ComfyUI (API)” recipe on Salad, using the default configuration, but setting priority to “batch”, and requesting 10 replicas. We started the benchmark when we had at least 9/10 replicas running.
We used Postman’s collection runner feature to simulate load , first from 10 concurrent users, then ramping up to 18 concurrent users. The test ran for 1 hour. Our virtual users submit requests to generate 1 image like this:
- Prompt: photograph of a futuristic house poised on a cliff overlooking the ocean. The house is made of wood and glass. The ocean churns violently. A storm approaches. A sleek red vehicle is parked behind the house.
- Resolution: 1024×1024
- Steps: 4
- Sampler: Euler
- Scheduler: Simple
We ran this on an RTX 4090 (24GB vram) with 4 vCPU and 30GB ram.
What we measured:
- Cluster Cost: Calculated using the maximum number of replicas that were running during the benchmark. Only instances in the ”running” state are billed, so actual costs may be lower.
- Reliability: % of total requests that succeeded.
- Response Time: Total round-trip time for one request to generate an image and receive a response, as measured on my laptop.
- Throughput: The number of requests succeeding per second for the entire cluster.
- Cost Per Image: A function of throughput and cluster cost.
- Images Per $: Cost per image expressed in a different way
Deployment of Flux.1-Schnell model on SaladCloud
Log in to your portal.salad.com account. Click through the FLUX.1-Schnell recipe, available from the Container Groups interface, and set replica count to 10. Optionally, set a non-default priority, and/or enable authentication. For our benchmark, we used “Batch” priority, and did not enable authentication.
Results from the Flux.1 benchmark
Our cluster of 9 replicas showed very good overall performance, returning images in as little as 4.1s / Image, and at a cost as low as 4265 images / $.
In this test, we can see that as load increases, average round-trip time increases for requests, but throughput also increases. We did not always have the maximum requested replicas running, which is expected. Salad only bills for the running instances, so this really just means we’d want to set our desired replica count to a marginally higher number than what we actually think we need.
While we saw no failed requests during this benchmark, it is not uncommon to see a small number of failed requests that coincide with node reallocations. This is expected, and you should handle this case in your application via retries.
RTX 4090 (24gb vram)
- Maximum running replicas during test: 9, consistent throughout benchmark
- Total Cost of 9 running replicas at “batch” priority: $2.034 / hour ($0.000565 / second)
- Reliability: 100% of requests succeeded
- Performance at 10 Virtual Users:
- Average Response Time: 4.1 seconds
- Average Throughput: 1.93 requests / second
- Cost Per Image: $0.00029274611
- Images Per $: 3415
- Performance at 18 Virtual Users:
- Average Response Time: 6.3 seconds
- Average Throughput: 2.41 requests / second
- Cost Per Image: $0.00023443983
- Images Per $: 4265
- Performance over total duration:
- Average Response Time: 5.531 seconds
- Average Throughput: 2.26 requests / second
- Cost Per Image: $0.00025
- Images Per $: 4000
Conclusion
The Flux.1-Schnell model is a significant advancement in AI image generation, delivering high-quality results while maintaining impressive speed and cost efficiency. Our benchmark on SaladCloud demonstrated its capability to produce images fast, achieving an average response time of just 5.5 seconds and an outstanding cost efficiency of up to 4265 images per dollar. With RTX 4090 (24GB) GPUs available on SaladCloud from just $0.18/hour, AI image generation tools can save significantly on inference cost by running Flux on SaladCloud.
These results not only highlight the model’s performance under varying loads but also underscore the potential for scalability and reliability in real-world applications. As developers and creatives seek robust tools for generating visual content, Flux.1-Schnell stands out as a compelling option.
Shawn Rushefsky is a passionate technologist and systems thinker with deep experience across a number of stacks. As Generative AI Solutions Architect at Salad, Shawn designs resilient and scalable generative ai systems to run on our distributed GPU cloud. He is also the founder of Dreamup.ai, an AI image generation tool that donates 30% of its proceeds to artists.
