SuspiciousCarrot78@aussie.zonedeleted by creator to Selfhosted@lemmy.worldEnglish · 4 months agoDo you host your own AI?message-squaremessage-square194linkfedilinkarrow-up1165arrow-down139file-text
arrow-up1126arrow-down1message-squareDo you host your own AI?SuspiciousCarrot78@aussie.zonedeleted by creator to Selfhosted@lemmy.worldEnglish · 4 months agomessage-square194linkfedilinkfile-text
minus-squareSuspiciousCarrot78@aussie.zonedeleted by creatorOPlinkfedilinkEnglisharrow-up13·4 months agoLlama.cpp or death!
minus-squaretristynalxander@mander.xyzlinkfedilinkEnglisharrow-up2·4 months agoIt’s not that hard to use llama.cpp directly anyway. Why would I use a wrapper when I can just run a python script?
minus-squarebrucethemoose@lemmy.worldlinkfedilinkEnglisharrow-up1·4 months agoOr exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.
Llama.cpp or death!
It’s not that hard to use
llama.cppdirectly anyway. Why would I use a wrapper when I can just run a python script?Or exllama! Vllm, sglang, Lorax. Koboldcpp, Aphrodite, text-generation-webui, LM Studio, powerinfer, ktransformers, mlc-LLM, really whatever floats your boat. Just not ollama, specifically.