That's the e-ink monopoly for you. Though you could probably use a slightly smaller b&w eink display with a second hand raspberry 4 to shave a lot from that number.
"It's possible to just connect two computers together with Ethernet" - on the front page of hackernews. Incredible. Ethernet over USB4 interdomain protocol at least had some novelty, but this...
It does seem to be a surprisingly common blindspot of programmers in general. I strongly recommend programmers learn it, it can save a lot of headaches in system architecture.
Especially since most machines will just WORK with auto-assigned magic weird IPs and you'll actually be able to share files directly via the OS if you do this, no need for Linux netcat tricks at all.
Now I still remember when target disk mode over ethernet was being talked about.
Programmers have very little understanding of computers, in general.
They'll think hard about a beautiful unrealistic idealisation of the problem, and/or treat a software abstraction as if it's the underlying reality, e.g. (of the latter) thinking networking starts with HTTP and languages start with JavaScript.
Just read the contract from the other side! If focusing on idealistic software abstractions instead of the problem is bad, the implication is that focusing on the problem is good. If treating a software abstraction as the underlying reality is bad, the implication is that looking below it to the actual underlying reality is good.
Not full precision. I've only benchmarked 27B across Q3-6 quants using lm-eval. I lack the hardware to bench 27B at BF16 but I might be able to do Q8_0. I haven't gotten around to doing 35B. I really should upload my collection of results to Github or somewhere.
Here's a summary of what I have for 27B. I used unsloth's UD-Q{3-6}_K_XL quants across 11 evals. The values are pretty linear between Q3 and Q6.
Apparently some other HMD devices (HMD 105, HMD 110) use "RTOS". Most likely their proprietary feature phone OS, don't expect it will provide an option for user-installable apps.
Have you tried with a single CPU to get rid of the NUMA penalty? I understand this likely means halving the memory but I am interested in how much of a difference it makes
I have (192GB machine with two CPUs), pretty much does the trick. It just runs some small models used for embedding, etc. and has those on one CPU / memory node and all the Docker containers on the other one.c
I have a dual xeon also, same as OP: Ivy Bridge + 128GB DRAM, and was never really able to get decent LLM performance out of it. So I ended up biting the bullet and adding a "budget tier" A4000 20GB GPU. Too bad all my DRAM is wasted now--not sure if there is a way to take advantage of lots of DRAM once you move over to having inference happening on the GPU.
Have you tried putting the KV cache on the GPU and running inference from RAM? From what I gather, prompt processing is particularly painful using RAM alone.
I haven't, since the machine has almost no GPU to speak of. It's an old machine, although in theory it would make sense to pop an R9700 in there and load a bunch of models into RAM and swap them in and out.
reply