I have avoided AI (LLMS) because it didn’t seem nearly useful enough to warrant supporting these tech assholes. Is there anyway to use it locally on a normal device (16 Gb ram, 8 year old Geforce graphics, etc?). Can I just slow it down and get it to work locally?
Man you can absolutely run something on ollama yourself. Right now I have ollama running qwen 3:4b-instruct on 6vCPU and no GPU and for very simple conversations it’s quick enough. TTFT is slow when prompt is uncached but otherwise TTFT is like 2s and I get maybe 5 words/s out.
If I gave it my 1660 I could probably get ~100tps but my card won’t pass through to my VM properly…
Shittiest local inference engine. Use llama.cpp instead.
Old version 3 model from Alibaba. 4 billion parameters total. It will struggle with anything other than natural language generation.
Running on a 6 vcore VM with no GPU pass through.
Time.To.First.Token. How long it takes the model to start responding after you hit enter. Dependent on not only processing speed but also context length.
Cold start = no cache = slow. But faster when mid conversation and the key-value cache is hot.
100 tokens per second. 1 token is roughly 1/4 word.
I’ll just leave this here. https://sleepingrobots.com/dreams/stop-using-ollama/
So… I’m with you/them about the vendor lock in but I think they’re slightly underestimating what ollama brings to the table. For example, ollama bundles CUDA libraries so I don’t need to worry about that as I wrangle with my GPU. It’s also well supported by other ecosystems like home assistant in a way that raw openai compatible API server isn’t.
Overall I don’t know if I find these arguments compelling at this time.