{"id":554,"date":"2025-06-27T15:15:14","date_gmt":"2025-06-27T15:15:14","guid":{"rendered":"https:\/\/onursonmez.com\/home\/?p=554"},"modified":"2025-06-27T15:29:52","modified_gmt":"2025-06-27T15:29:52","slug":"realtime-llm-unity-integration","status":"publish","type":"post","link":"http:\/\/onursonmez.com\/home\/realtime-llm-unity-integration\/","title":{"rendered":"Realtime LLM &#8211; Unity Integration"},"content":{"rendered":"","protected":false},"excerpt":{"rendered":"","protected":false},"author":1,"featured_media":556,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[],"class_list":["post-554","post","type-post","status-publish","has-post-thumbnail","hentry","category-work"],"grid":"{\r\n  \"colCount\": 12,\r\n  \"colGutter\": 1,\r\n  \"rowGutters\": [\r\n    1\r\n  ],\r\n  \"frameMargin\": 5,\r\n  \"leftFrameMargin\": 5,\r\n  \"rightFrameMargin\": 5,\r\n  \"topFrameMargin\": 75,\r\n  \"mus\": {\r\n    \"colGutterMu\": \"%\",\r\n    \"rowGutterMu\": \"%\",\r\n    \"topFrameMu\": \"px\",\r\n    \"bottomFrameMu\": \"%\",\r\n    \"frameMu\": \"%\"\r\n  },\r\n  \"bottomFrameMargin\": 5,\r\n  \"rowAttrs\": [\r\n    {\r\n      \"relid\": 1\r\n    },\r\n    {\r\n      \"relid\": 3\r\n    }\r\n  ],\r\n  \"bgColor\": null,\r\n  \"bgImage\": null,\r\n  \"cont\": [\r\n    {\r\n      \"type\": \"img\",\r\n      \"cont\": \"\/wp-content\/uploads\/2025\/06\/llm.jpg\",\r\n      \"align\": \"middle\",\r\n      \"row\": 0,\r\n      \"col\": 0,\r\n      \"colspan\": 5,\r\n      \"offsetx\": 0,\r\n      \"offsety\": 0,\r\n      \"spaceabove\": 0,\r\n      \"spacebelow\": 0,\r\n      \"yvel\": 1,\r\n      \"push\": 0,\r\n      \"relid\": 4,\r\n      \"attid\": 555,\r\n      \"sizes\": {\r\n        \"full\": \"\/wp-content\/uploads\/2025\/06\/llm.jpg\",\r\n        \"_1024\": \"\/wp-content\/uploads\/2025\/06\/llm-1024x853.jpg\",\r\n        \"_768\": \"\/wp-content\/uploads\/2025\/06\/llm-768x640.jpg\",\r\n        \"_512\": \"\/wp-content\/uploads\/2025\/06\/llm-512x427.jpg\",\r\n        \"_265\": \"\/wp-content\/uploads\/2025\/06\/llm-265x221.jpg\"\r\n      },\r\n      \"w\": 1093,\r\n      \"h\": 911,\r\n      \"ar\": 0.8334858188472095,\r\n      \"alt\": \"\",\r\n      \"absolute_position\": false,\r\n      \"frameOverflow\": \"\"\r\n    },\r\n    {\r\n      \"type\": \"text\",\r\n      \"cont\": \"<p><span class=\\\"break-words\\n          tvm-parent-container\\\"><span dir=\\\"ltr\\\">With Marco Pisano<\/span> we've been exploring the possibility of connecting large language models (LLMs) with Unity and controlling real-time systems. <span><br \/><\/span><span><br \/><\/span>We developed a working prototype that runs a local LLM (in this case Mistral) and uses it to real-time control 3D assets in Unity. By communicating Python and C# and fine-tuning the communication pipeline, we achieved smooth interaction between the LLM and our virtual environment.<span><br \/><\/span><span><br \/><\/span>This involved customization of the model file and experimentation with core inference parameters like system prompts, temperature, top_p, and top_k values to find the right balance between coherence and ingenuity. Finding the sweet spot was key to getting responses that are not only context-aware but also stable enough to form a response logic in Unity. <span><br \/><\/span><span><br \/><\/span>The result was a fully local setup running on a laptop where our language model respond intelligently and almost instantly to user inputs, converting them to commands and sending them to control the 3D assets within a Unity Scene. This opens doors for AI-driven, dynamic storytelling all without relying on cloud-based APIs. <span><br \/><\/span><span><br \/><\/span>Open source communities, large language models and the ability to run them locally are the key factors to develop innovative projects with greater privacy, control, freedom and customization.\u00a0<\/span><\/p>\",\r\n      \"align\": \"top\",\r\n      \"row\": 1,\r\n      \"col\": 0,\r\n      \"colspan\": 5,\r\n      \"offsetx\": 0,\r\n      \"offsety\": 0,\r\n      \"spaceabove\": 0,\r\n      \"spacebelow\": 0,\r\n      \"yvel\": 1,\r\n      \"push\": 0,\r\n      \"relid\": 2,\r\n      \"absolute_position\": false\r\n    }\r\n  ]\r\n}","phonegrid":"","_links":{"self":[{"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/posts\/554","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/comments?post=554"}],"version-history":[{"count":0,"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/posts\/554\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/media\/556"}],"wp:attachment":[{"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/media?parent=554"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/categories?post=554"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/onursonmez.com\/home\/wp-json\/wp\/v2\/tags?post=554"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}