Top all-time · #2124
Baseten
Company blog · baseten.co/blog · on HN as philipkielyaiprogramming
Baseten is the company blog at baseten.co, writing about ai and programming. Since 2021, 34 different Hacker News readers have linked it 63 times, most often “We built the fastest API for GLM-5.2 (280 TPS)”. No feed was found, so read it on the site itself.
Too few sampled comments (21) to say how often HN recommends it.?
Why HN links it
HN readers link Baseten's blog and guides for explanations of inference engineering, including batching, GPU throughput and prefill versus token generation. Otherwise the site is mentioned as a hosting provider or for hiring.
Summarised by Claude from a sample of 21 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Serve and scale open-source and custom AI models on the fastest, most reliable inference platform.
Most-linked posts?
- We built the fastest API for GLM-5.2 (280 TPS)3 people · 3 mentions · first linked 2026-06-23
- /pricing/3 people · 3 mentions · first linked 2023-12-11
- Inference Engineering by Philip Kiely – Digital Download3 people · 3 mentions · first linked 2026-07-14
- A guide to LLM inference and performance2 people · 2 mentions · first linked 2024-09-27
- Deploying Stable Diffusion in Production Using Truss2 people · 2 mentions · first linked 2022-09-01
- Continuous vs. dynamic batching for AI inference2 people · 2 mentions · first linked 2025-08-06
- BaseTen: The fastest way to build ML-powered applications2 people · 2 mentions · first linked 2021-05-20
- /careers2 people · 2 mentions · first linked 2022-01-03
- Faster Mixtral inference with TensorRT-LLM and quantization2 people · 2 mentions · first linked 2023-12-27
- /2 people · 2 mentions · first linked 2023-05-02
What Hacker News says?
Most models can be inferenced-upon with merely borderline-consumer hardware. Even the fancy models where you need to buy compute (rails) that's about the price of a new car, they have a power draw of ~700W[0] while running inference at 50 tokens/second. But! The constraint with current hardware isn't compute, the models are mostly constrained by RAM bandwidth: back of the envelope estimate says that e.g. if Apple to…
Comment linking /blog/evaluating-nvidia-h200-gpus-for-llm-inference/
on “GPT-5”Submitted Running GPT-OSS-120B at 500 tokens per second on Nvidia GPUs as a story · 247 points
HN discussion> What about alternatives? Custom models? Apart from the Big 3 (in no particular order): - https://together.ai/ - https://replicate.com/ - https://anyscale.com/ - https://baseten.co/ - https://modal.com/ - https://banana.dev/ - https://runpod.io/ - https://bentoml.com/ - https://brev.dev/ - https://octo.ai/ - https://cerebrium.ai/ ...
Comment linking /
on “Fly.io has GPUs now”Submitted The efficient frontier of LLM inference as a story · 155 points
HN discussionBaseten serves models as a service, at scale. There’s quite a lot of interesting engineering both for inference and infrastructure perf. This is a pretty good deep dive into the tricks they employ: https://www.baseten.co/resources/guide/the-baseten-inference...
Comment linking /resources/guide/the-baseten-inference-stack/
on “Ask HN: How can ChatGPT serve 700M users when I can't run one GPT-4 locally?”Submitted Show HN: Baseten – Build ML-powered applications as a story · 112 points
HN discussionBaseten | Frontend, Backend, Infra & Dev Advocate | San Francisco & Remote | https://www.baseten.co/careers Hey! We're baseten, just launched last week, we build a platform to host AI models and create applications over it easily. (see our techrunch article here https://techcrunch.com/2022/04/26/baseten-nabs-20m-to-make-i... ) - We have some nice clients already, such as Patreon and Primer. - We aim to provide 90th…
Comment linking /careers
on “Ask HN: Who is hiring? (May 2022)”Submitted BaseTen: The fastest way to build ML-powered applications as a story · 20 points
HN discussionBaseten | Tech Leads, Software Engineers, and more | SF, NY, Remote | Full-time 2 years and 8 months ago, I joined Baseten after seeing them in a "Who is Hiring" post on HN. Doing so was one of the three best decisions I've ever made in my life. Baseten is now a fast-growing Series B AI infrastructure startup focused on inference. We have PMF and are growing fast in a highly competitive market. We are actively hirin…
Comment linking /blog/ten-reasons-to-join-baseten/
on “Ask HN: Who is hiring? (October 2024)”Submitted Base Labs releases top-ranked decision models mammoth-1 and and elephant-1 as a story · 12 points
HN discussionBaseten | Infrastructure engineer / Site Reliability Engineer (SRE) | SF, Remote | Full time Hey, I'm P-A infrastructure engineer at Baseten. We're searching for an Engineer focused on Site Reliability and Infrastructure, someone that wants to do projects around core infrastructure, be it in the python stack or in the kubernetes parts. Someone that knows how to log into a machine but knows that every time you log in…
Comment linking /careers
on “Ask HN: Who is hiring? (September 2022)”Submitted Show HN: Baseten Chains – Framework and SDK for Multi-Model AI Products as a story · 9 points
HN discussionBaseten | Remote (US, Canada, Europe, and more), SF US | Full-time | https://baseten.co Baseten is an IaaS for data scientist teams that wants to build apps out of their AI models. We've got multiple clients, a successful series A and are carefully expanding our team. We're still under 15, and fly over to SF around once every 3 months. If python, typescript, lots of kubernetes tools, and a really diverse team from a…
Comment linking /careers
on “Ask HN: Who is hiring? (January 2022)”Submitted We built the fastest API for GLM-5.2 (280 TPS) as a story · 6 points
HN discussionBaseten | REMOTE (US, Canada, Europe, and more), SF US | Full-time | https://baseten.co A personal note: I joined Baseten just over a month ago after seeing a post in January's "Who is Hiring" on HN, and I am very happy here. Baseten is an IaaS for data scientist teams that wants to build apps out of their AI models. We have customers like Patreon and Pipe, are well-funded, and are carefully expanding our team. We'r…
Comment linking /careers
on “Ask HN: Who is hiring? (March 2022)”Submitted Serving four million Riffusion requests in two days as a story · 5 points
HN discussionhttps://www.baseten.co/library/mercury-3/
Comment linking /library/mercury-3/
on “Mercury 3: Diffusion at scale”Submitted Our Partnership with OpenAI as a story · 3 points
HN discussionHow much does it cost to train a cutting edge LLM? Those costs need to be factored into the margin from inferencing. They don't, though! I can buy hardware off of the shelf, host open source models on it, and then charge for inference: https://parasail.io , https://www.baseten.co
Comment linking /
on “Tech Titans Amass Multimillion-Dollar War Chests to Fight AI Regulation”Submitted Baseten raised a $1.5B Series F and achieved a $13B valuation as a story · 3 points
HN discussionYep, agreed on the more testing. I don't actually have any allegiance to vLLM. We've just been happy with it. A few things in the linked article that make my eyebrows raise. Notably, they claim to achieve 206 t/s at bs=1 on a single A100 80GB (2 TB/s bandwidth) in fp16. That model is 14.5GB in size, and best case would require aggregate memory bandwidth of almost 3 TB/s to achieve. The BentML you linked to, also ran…
Comment linking /blog/unlocking-the-full-power-of-nvidia-h100-gpus-for-ml-inference-with-tensorrt/
on “AMD MI300x GPUs with GEMM tuning improves throughput and latency by up to 7.2x”Submitted The Math Behind TurboQuant as a story · 3 points
HN discussionWhere can I find technical performance comparison? What models does Groq run at 500tok/sec? In what mode (e.g. batch size)? UPD. found the model, it is Mixtral 8x7B-32k. AFAIK 8xH100 will do 100+tok/sec with batch size 1. But that does not look too impressive for batch sizes higher than 1: https://www.baseten.co/blog/faster-mixtral-inference-with-te...
Comment linking /blog/faster-mixtral-inference-with-tensorrt-llm-and-quantization/
on “Groq Raises $640M to Meet Soaring Demand for Fast AI Inference”Submitted SDXL inference in under 2 seconds as a story · 3 points
HN discussionObviously gb200 and m2 studios are very different price points and power envelopes, per other comments. I agree that Apple does not have a competitive server offering right now. But I think I'd still be spending toward that goal if I were reserving all the 2nm chips in the world next year. Your claim that NVIDIA has better performance per watt really surprised me, so I did some digging. My prior is that opencl is me…
Comment linking /blog/benchmarking-fast-mistral-7b-inference/
on “Secret meeting between Apple and TSMC reported to reserve all 2nm capacity”Submitted Deploying Stable Diffusion in Production Using Truss as a story · 3 points
HN discussionInference (token generation) is memory-bound, KV cache prefill (prompt processing) is compute-bound. The ARM Macintoshes have lots of memory bandwidth but not a lot of compute power, so they're great for outputting text but terrible for tasks like analyzing documents. I've never done fine-tuning but my understanding is that that is a highly-parallelizable compute hog as well. You might like this article, which looks…
Comment linking /blog/llm-transformer-inference-guide/
on “Leak claims RTX 5090 has 600W TGP, RTX 5080 hits 400W”Submitted How to double tokens per second for Llama 3 with Medusa as a story · 2 points
HN discussionOP does not mention that he's employed by BaseTen, the company behind "Blueprint" (which is mentioned repeatedly in the blog post). I noticed the repeated mentions and did some digging to find this out. Not cool, OP, not cool. See: https://www.baseten.co/blog/deploying-stable-diffusion
Comment linking /blog/deploying-stable-diffusion
on “Show HN: I fine-tuned Flan-T5. Can it cook?”Submitted Show HN: Automatically Build Nvidia TRT-LLM Engines as a story · 2 points
HN discussion
