Top all-time · #4021
Bento: Run Inference at Scale
Company blog · bentoml.com/blogaiprogramming
Bento: Run Inference at Scale is the company blog at bentoml.com, writing about ai and programming. Since 2019, 19 different Hacker News readers have linked it 54 times, most often “Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC”. No feed was found, so read it on the site itself.
Too few sampled comments (7) to say how often HN recommends it.?
Why HN links it
HN readers link BentoML for its LLM inference benchmarking post and as an option among model-hosting platforms with usage-based pricing. Commenters cite the benchmark post as evidence in discussions of GPU and LLM serving.
Summarised by Claude from a sample of 7 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Inference Platform built for speed and control. Deploy any model anywhere, with tailored inference optimization, efficient scaling, and streamlined operations.
Most-linked posts?
- Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC3 people · 4 mentions · first linked 2024-06-06
- Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TensorRT-LLM, TGI2 people · 2 mentions · first linked 2024-06-08
- BentoML: One Model to Rule Them All2 people · 2 mentions · first linked 2019-04-17
- Building a Production-Ready LangChain Application with BentoML and OpenLLM2 people · 2 mentions · first linked 2023-09-20
- /2 people · 2 mentions · first linked 2022-04-21
- The Complete Guide to DeepSeek Models: From V3 to R1 and Beyond2 people · 2 mentions · first linked 2025-03-07
- Why should you care about compound AI?2 people · 2 mentions · first linked 2024-08-16
- 2024 State of AI Inference Infrastructure Survey Results2 people · 2 mentions · first linked 2025-02-26
What Hacker News says?
Seems there’s still some confusion around what DeepSeek-OCR really does. Learn about the model, Contexts Optical Compression, and its impact on LLMs here: https://www.bentoml.com/blog/deepseek-ocr-contexts-optical-c...
Comment linking /blog/deepseek-ocr-contexts-optical-compression-explained
on “DeepSeek OCR”Submitted LLM Inference Handbook as a story · 366 points
HN discussion> What about alternatives? Custom models? Apart from the Big 3 (in no particular order): - https://together.ai/ - https://replicate.com/ - https://anyscale.com/ - https://baseten.co/ - https://modal.com/ - https://banana.dev/ - https://runpod.io/ - https://bentoml.com/ - https://brev.dev/ - https://octo.ai/ - https://cerebrium.ai/ ...
Comment linking /
on “Fly.io has GPUs now”Submitted Navigating the World of Large Language Models as a story · 48 points
HN discussionThis is really not true my friend. I would love to help you if I had some more time, but let me look for a tutorial. Because the FineWeb Dataset is already super good. You can train 7B or 32B Param models at home The >600B Param model isn't really using all the data effectively, but with a MacStudio Farm you can also train that one at home (if you have enough money to buy at least 100). Here's the easy way: https://…
Comment linking /blog/the-complete-guide-to-deepseek-models-from-v3-to-r1-and-beyond
on “Why DeepSeek is cheap at scale but expensive to run locally”Submitted Is LMDeploy the Ultimate Solution? Why It Outshines VLLM, TRT-LLM, TGI, and MLC as a story · 16 points
HN discussionPretty bad benchmarks to the point of being deliberately misleading. They benchmarked vllm which is less than half the speed of the inference leader lmdeploy: https://bentoml.com/blog/benchmarking-llm-inference-backends They also used Flywheel for AMD while not bothering to turn on Flywheel for Nvidia, which is crazy since Flywheel improves Nvidia performance by 70%. https://mk1.ai/blog/flywheel-launch In this conte…
Comment linking /blog/benchmarking-llm-inference-backends
on “AMD's MI300X Outperforms Nvidia's H100 for LLM Inference”Submitted Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TensorRT-LLM, TGI as a story · 15 points
HN discussionThere's also the option of platforms such as BentoML (I have no affiliation) that offer usage-based pricing so you can at least take the 100% utilization assumption off the table. I'm not sure how the price compares to EKS. https://www.bentoml.com/
Comment linking /
on “Cost of self hosting Llama-3 8B-Instruct”Submitted Benchmarking LLM Inference Back Ends: VLLM, LMDeploy, MLC-LLM, TRT-LLM, and TGI as a story · 12 points
HN discussionHow does this compare to BentoML ( https://www.bentoml.com/ )?
Comment linking /
on “Cog: Containers for Machine Learning”Submitted A List of Top Open-Source Embedding Models as a story · 5 points
HN discussionIs it really that close in production? The benchmarks I've seen show it's almost 3x slower compared to the SOTA. https://bentoml.com/blog/benchmarking-llm-inference-backends
Comment linking /blog/benchmarking-llm-inference-backends
on “AMD MI300x GPUs with GEMM tuning improves throughput and latency by up to 7.2x”Submitted Nvidia Data Center GPUs Explained: From A100 to B200 and Beyond as a story · 4 points
HN discussionThere are already several fully open source LLMs. You can start participating in those projects today. https://www.bentoml.com/blog/navigating-the-world-of-open-so...
Comment linking /blog/navigating-the-world-of-open-source-large-language-models
on “The Future of Everything Is Lies, I Guess: Where Do We Go from Here?”Submitted The Shift to Distributed LLM Inference as a story · 4 points
HN discussionSubmitted Building RAG with Open-Source and Custom AI Models as a story · 4 points
HN discussionSubmitted Three Levels of Running LLMs from Laptop to Cluster-Scale Distributed Inference as a story · 3 points
HN discussionSubmitted Where to Buy or Rent GPUs for LLM Inference: The 2026 GPU Procurement Guide as a story · 3 points
HN discussionSubmitted How to Beat the GPU CAP theorem in AI Inference as a story · 3 points
HN discussionSubmitted From Ollama to OpenLLM: Running LLMs in the Cloud as a story · 3 points
HN discussionSubmitted Stable Diffusion 3: Text Master, Prone Problems? as a story · 3 points
HN discussionSubmitted A Guide to Open-Source Image Generation Models as a story · 3 points
HN discussionSubmitted BentoML: One Model to Rule Them All as a story · 3 points
HN discussionSubmitted Modular Acquires BentoML as a story · 2 points
HN discussionSubmitted The Best Open-Source Small Language Models as a story · 2 points
HN discussionSubmitted What Is InferenceOps as a story · 2 points
HN discussionSubmitted Cold-Starting LLMs on Kubernetes in Under 30 Seconds as a story · 2 points
HN discussionSubmitted Six Infrastructure Pitfalls Slowing Down Your AI Progress as a story · 2 points
HN discussionSubmitted The Complete Guide to DeepSeek Models: From V3 to R1 and Beyond as a story · 2 points
HN discussion
