Top all-time · #6063
MLC
Company blog · blog.mlc.aiaiprogramming
MLC is the company blog at blog.mlc.ai, writing about ai and programming. Since 2023, 13 different Hacker News readers have linked it 19 times, most often “Scaling LLama2-70B with Multiple Nvidia/AMD GPU”. No feed was found, so read it on the site itself.
Too few sampled comments (3) to say how often HN recommends it.?
Why HN links it
MLC blog posts describe running and compiling large language models on consumer GPUs, including multi-GPU Llama2-70B and CodeLlama 70B inference. Commenters cite the posts for tokens-per-second figures.
Summarised by Claude from a sample of 3 HN comments that link here. Each comment is marked recommending, critical or neutral (citing without judging); how this works.
Most-linked posts?
- Scaling LLama2-70B with Multiple Nvidia/AMD GPU2 people · 3 mentions · first linked 2023-10-20
- High-Throughput Low-Latency LLM Serving with MLCEngine2 people · 2 mentions · first linked 2024-10-10
- XGrammar-2: 80x Faster Structured Generation for Agent Tool Calling2 people · 2 mentions · first linked 2026-05-04
- LLM Microserving: a new RISC-style approach to design LLM serving API2 people · 2 mentions · first linked 2025-01-07
- Making AMD GPUs competitive for LLM inference (2023)2 people · 2 mentions · first linked 2023-08-09
What Hacker News says?
I run CodeLlama70B in my home lab using two RTX 4090s and the MLC AI framework. It provides Muti-GPU support and is extremely fast on consumer grade hardware. I'm seeing about 30 tokens/sec. https://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infer...
Comment linking /2023/10/19/Scalable-Language-Model-Inference-on-Multiple-NVDIA-AMD-GPUs
on “Ask HN: Code Llama 70B on a dedicated server”Submitted Making AMD GPUs competitive for LLM inference (2023) as a story · 313 points
HN discussionYeah thanks for sharing! This is definitely super valuable data and insights :) Regarding exllama-V2, MLC/TVM does benchmark against it: - Single GPU: https://github.com/mlc-ai/llm-perf-bench#int4-quantized-sing... - Multi GPU: Figure 2 in the blog: http://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infere... > vLLM focuses more on batching performance Exactly. vLLM doesn’t optimize for latency-first scenarios as…
Comment linking /2023/10/19/Scalable-Language-Model-Inference-on-Multiple-NVDIA-AMD-GPUs
on “Scaling LLama2-70B with Multiple Nvidia/AMD GPU”Submitted Universal LLM Deployment Engine with ML Compilation as a story · 17 points
HN discussionMachine Learning Compilation (MLC) now supports compiling LLMs to multiple GPUs. For Llama2-70B, it runs 4-bit quantized Llama2-70B at: - 34.5 tok/sec on two NVIDIA RTX 4090 at $3k - 29.9 tok/sec on two AMD Radeon 7900XTX at $2k - Also it is scales well with 8 A10G/A100 GPUs in our experiment. Details: - Blog post: https://blog.mlc.ai/2023/10/19/Scalable-Language-Model-Infer... - Project: https://github.com/mlc-ai/m…
Comment linking /2023/10/19/Scalable-Language-Model-Inference-on-Multiple-NVDIA-AMD-GPUs
on “Scaling LLama2-70B with Multiple Nvidia/AMD GPU”Submitted In-browser LLM inference engine with WebGPU and OpenAI API as a story · 16 points
HN discussionSubmitted Scaling LLama2-70B with Multiple Nvidia/AMD GPU as a story · 11 points
HN discussionSubmitted TIRx Harness: An Open Compiler Harness for Agentic GPU Programming as a story · 9 points
HN discussionSubmitted High-Throughput Low-Latency LLM Serving with MLCEngine as a story · 8 points
HN discussionSubmitted GPU-Accelerated LLM on a $100 Orange Pi as a story · 5 points
HN discussionSubmitted XGrammar-2: 80x Faster Structured Generation for Agent Tool Calling as a story · 4 points
HN discussionSubmitted LLM Microserving: a new RISC-style approach to design LLM serving API as a story · 4 points
HN discussionSubmitted PithTrain – a compact, agent-native MoE training system as a story · 3 points
HN discussionSubmitted XGrammar-2: Fast, Customizable Structured Generation for Tool Calling and Agents as a story · 2 points
HN discussionSubmitted MLCEngine: Universal LLM Deployment to Both Cloud and Local Devices as a story · 2 points
HN discussionSubmitted KCoral: Lightweight Benchmark Server for Agentic GPU Programming as a story · 1 points
HN discussionSubmitted Microserving LLM Engines as a story · 1 points
HN discussionSubmitted Optimizing and Characterizing High-Throughput Low-Latency LLM Inference as a story · 1 points
HN discussionSubmitted Making AMD GPUs competitive for LLM inference as a story · 1 points
HN discussion
