'/%3e%3cpath%20d='M50.3943%2022.237V39.5876H43.5185V22.7481C43.5185%2019.2029%2042.0411%2017.3949%2039.036%2017.3949C35.7325%2017.3949%2034.0778%2019.5338%2034.0778%2023.7585V32.9759H27.2434V23.7585C27.2434%2019.5338%2025.5857%2017.3949%2022.2822%2017.3949C19.2949%2017.3949%2017.8027%2019.2029%2017.8027%2022.7481V39.5876H10.9298V22.237C10.9298%2018.6918%2011.835%2015.8754%2013.6453%2013.7877C15.5128%2011.7049%2017.9623%2010.6355%2021.0028%2010.6355C24.522%2010.6355%2027.1813%2011.9885%2028.9542%2014.6917L30.665%2017.5633L32.3788%2014.6917C34.1517%2011.9885%2036.811%2010.6355%2040.3243%2010.6355C43.3619%2010.6355%2045.8114%2011.7049%2047.6847%2013.7877C49.4931%2015.8734%2050.3963%2018.6899%2050.3943%2022.237Z'%20fill='white'/%3e%3cdefs%3e%3clinearGradient%20id='paint0_linear_89_8'%20x1='30.5'%20y1='0'%20x2='30.5'%20y2='65'%20gradientUnits='userSpaceOnUse'%3e%3cstop%20stop-color='%236364FF'/%3e%3cstop%20offset='1'%20stop-color='%23563ACC'/%3e%3c/linearGradient%3e%3c/defs%3e%3c/svg%3e)
Verwaltet von:
Today, I found a great presentation by @okias about running #OpenCL-accelerated #LLMs on @postmarketOS (see: https://fosdem.org/2024/events/attachments/fosdem-2024-3364-why-not-run-opencl-accelerated-llm-on-your-phone-/slides/22383/Why_not_run_OpenCL-accelerated_LLM_on_your_phon_nK2DudB.pdf). @okias showed a benchmark with a OnePlus 6 device. I wonder whether it would be possible to use even older hardware to run a functional #LLM. Is there maybe even a list of supported devices? 
@LukasBrausch @postmarketOS Thanks. Adreno 630 is a pretty powerful high-end mobile GPU and I was running only GPT2-level LLM. I think older GPUs exposing OpenCL could be still leveraged to accelerate some simpler tasks (e.g. translators etc.), but bigger LLM would be out of scope.
@okias @postmarketOS I see. Thanks a lot for the clarification. What about the other extreme? What's the most potent #LLM that can currently be run on a phone (preferably using only free solutions, such as @postmarketOS)? To me, the idea of having an #OpenSource LLM on a phone instead of a large server farm is just very intriguing.
@LukasBrausch @postmarketOS I think someone mentioned some success regarding some simple small GPT-3. I didn't continue pursuing it, because I want to address a hack I had to do for in Mesa for Tinygrad to work. We also need to get the driver part finished and reviewed. I see the "future" more in smart translators and simple one-language speech recognition on these low-power devices.
@okias @postmarketOS Thanks a lot for your insights and the hard work you're doing here. Much appreciated. 
