adi-barda

[adi-barda]

Unity 3D Short Review

* this text is 100% human. No AI is involved in generating it In the last year I've been using Unity for my studies in game development. I was able to create and publish 3 small games in quite a short time. I was totally new to Unity, never used it before but I knew that it is one of the most popular game engine on the market.

Night Dancer - A game I've created in Unity in about 3 weeks in my spare time. The game includes video rendering, procedural generation and constant shape changes of the dancing stage in run-time, user input with hand gestures using open source machine learning model for the gesture recognition, AWS dynamo DB integration for presenting a Leader Board and the usual stuff - UI, particle system, Humanoid animation, events, triggers etc. Actually I never programed a game before in any other game engine except for my own SceneMax3D engine which is based on JME3 engine and a custom made programming language. So this is my short review for what I liked and what I didn't like about Unity3D. I consider myself as a Unity beginner level programmer so in that perspective you should read this text.

The good things I'm not going to write about each and every good feature in Unity (there's a lot of them) rather I would like to focus some good general components and concepts. For the most part, I think it is a fantastic game engine, very comprehensive with an amazing assets store and eco system. If I will ever aim for developing a serious full scale game, Unity probably will be my choice for game engine to be used.

Performance - this engine has great performance. I used a lot of scene objects, expensive particle systems, video rendering, just-in-time hundreds of objects (prefabs) loading and creation, music and sound effects and the engine handled them exceptionally good performance wise. Most of the time I don't even think about the performance cost of my game objects - that good is the engine's optimizations. Of course from time to time I do hit the performance limits and the FPS is dropping, then I apply some coding best practices and usually the problem is solved.

Slider Glider - a fast arcade game where you need to slide and jump on monstrous sliders while navigating your way in a procedurally generated maze. It took me about 3 weeks to develop this game in my spare time. The sliders models are placed in run-time and the maze is randomly generated in the beginning of each stage. There is also a simple navigation system assisting you to cope with the complex maze navigation reaching the treasure. The big challenge here was using curves for the sliding movement. Unity curves editor was very helpful in this case. Ease of use - Unity IDE is superb. The technique of dragging a prefab (pre defined composite of objects) onto a public field of some other game object and having it ready to be used at run time is very easy and intuitive to use. The usage of Mono Behaviors for injecting specific logic, the animation state machine, events and triggers are all contributing to the ease of using this engine. Rendering - Unity uses the best graphics APIs available for each targeted platform such as Vulkan and DirectX12. This affects the quality, portability and performance of your project.

Assets store - This one is by far my favorite feature of Unity. Absolutely amazing assets store with every library/graphic asset you can think of for your game and with superb reliable integration to your project for installing / uninstalling it. Some of the assets costs money and some of them are free. I was able to get anything I needed for free but for serious projects I will not hesitate to spend some money for getting state of the art libraries / assets. Build system - Unity supports building your projects for all major platforms including mobile, game consoles, PCs and web. This portability in my opinion is a big advantage Coding - For those of us (myself included) who prefers to build the scene graph using coding as much as possible in run time, Unity allows that freedom leaving you only the need to setup your game's resources (pre-fabs).

Dark Chaos - my first ever Unity game. Made in 3 weeks with absolutely no prior knowledge. Tons of particle systems, no best practices used, 3D models from Mixamo, Sketchfab and some I assembled by myself using Asset Forge. I think that demonstrates the ease of creating your first game with Unity Generative AI - Unity is probably the most popular game engine in the market using one the most popular general purpose programming language available (C#) so it makes it really easy for Vibe programing - letting an LLM to create some of your coding and code reviews for you. Harnessing Gen AI for writing some of your game's code is truly a revolution in game creation. Really hard tasks which might take days or even weeks for research and implementation is reduced to minutes. Let me give you an example - in my Night Dancer game I wanted a 10 by 10 dancing floor which changes its shape in real-time from "O" shape to "X" to "Love" to "Square" etc. in random sizes at random speed. That would take me quite a bit of time to code myself and it was perfectly done by the Gen AI in just a few minutes. Absolutely amazing. The bad things

As I mentioned earlier, I consider myself as a beginner programmer in Unity and also don't have much experience in any other game engines so the "bad things" that I'm going to list here might not be that bad if I had a wider perspective. External code editor - I find it really annoying that you can't code inside Unity IDE and need to switch back and forth from Unity IDE to an external code editor such as MS Visual Studio or VS Code. Each time you switch from one IDE to another, a compilation process starts followed by a synchronization process. It take a few seconds (sometimes quite a few!) and you need to wait for those processes to end before switching IDEs. Not fun especially for developers like myself who spend a lot of their time coding. UI Canvas - I find it a little bit confusing to arrange my UI components (2D) in the game's 3D world IDE freezes - It happened to me a few times that the IDE froze for some reason and sometimes the build process was stuck forever. I suspect that it is related to the anti virus installed on my Windows machine. Anyway I needed to hard stop Unity's process and restart the system risking some state loss

Summarize

I was really impressed with what Unity offers you out of the box and as an extension from its assets store and I highly recommend using this game engine for any serious game you have in mind. it is guaranteed that the engine will not be the one stopping you from completing your game. As for myself I'll probably be back working on simple games using my SceneMax3D and JME3 because this what I enjoy most - creating tools and software engineering. Cheers!

[adi-barda]

Back To Business

On Aug-12 2024 I have submitted a project for a global AI coding competition, Then for ~2 month I didn't (couldn't) touch any of my personal projects. I guess that's what happens when you "overflow" putting too much effort in 1 area and neglecting other areas. At work, I have accepted the challenge of becoming cloud security CIEM team leader. I'm excited for this new role but from past experience as a team leader in previous work place, I know that it comes with a lot of pressure. In addition, these days I'm starting my master degree in Digital Games Design. I scheduled my courses for the coming two semesters and (as expected) it consumes almost all of my free time. It also costs a lot of money but I feel that I owe it to myself to keep learning and developing my knowledge and meeting new people and having fun. I will not be able to finish it in 2 years so it is going to be a long 3 years journey... SceneMax3D might be involved in an interesting secret project. I can't tell much and I will not be doing the POC myself since I don't have any spare time but I will probably guide other developers how to use the engine. That should be a lot of fun to do :) What else - I take my time to learn the Irish Tin Whistle, It's such a fun addictive and practical piece of musical instrument. I highly recommend trying it out. It's not that easy to learn tough... Here is me trying to perform "The Foggy Dew" Irish war song:

Cheers!

[adi-barda]

Chapter 5 - Gemini API Developer Competition - Project Submission

The last two week were devoted to finalization of the AI Game Maker project: 1. Added attack strategy to the opponent in a fighting game - this feature allows the Gemini AI to assemble a fight strategy according to some statistics provided by the current game state. This is big topic by itself involving a lot of fine-tuning and cost optimizations (you don't want to use the AI to often while the game is running as it might cost quite a lot) 2. Prepare a new SceneMax3D installation with the new plugins system and upload it to the SceneMax3D website 3. Push the latest SceneMax3D mobile engine to the public repo in GitHub. The mobile engine is an open source project and it's 99% similar to the PC engine. 4. Prepare 2 sound tracks for the demo video clip. I used Suno AI for that and it was a very good experience. I highly recommend it for anyone who seeks to create amazing music for his products. 5. Prepare a short 3 minutes clip for the competition demo. I used Davinci Resolve for this task. I absolutely love this software and it's well worth the ~300$ I paid for it. The free version is also very good 6. Prepare Zip file with all the source code of this AI Game Maker project + binaries + instructions. This is a mandatory delivery for the competition.

7. Submission - on Aug-02-2024 I submitted my innovation. The last day for submission is on Aug-12 and then starts the fun part of reviewing other projects, voting etc. In about two months I start my master's degree in computer game design and the thesis is planned to be on the subject of AI generated games so a big part of that project is already made while participating in the Gemini competition - a win-win :)

My vision is to embed AI in major parts of SceneMax3D language and IDE. It opens a big opportunity for innovation.

Cheers!

[adi-barda]

Chapter 4 - Gemini API Developer Competition - Fighting game & Android Export

As planned, I spent the last days on adding fighting game capability to the engine and Android exporting feature. The fighting game has much more details in the puzzle for the AI agent to cope with. There are complex animations for the player and for the opponent, they need to constantly look at each other, you need to be able to demo their kick, punch, block animations, the player needs to be able to move in 3D space etc. Overall I'm very pleased with the results so far. The user can speak freely enough with the AI, get instant results and funny reactions. What's more, I've been able to add Android exporting of the game and automatically open it in Android studio. It was challenging because the Java code worked different on PC and on the mobile device specifically handling of Zip files and all kind of Gradle dependency hell. ChatGPT was on my side all the way, assisting me to resolve configuration issues and coding problems such as selecting the best Zip 3rd party library.

This video clip, demonstrates the current status of the project. It shows a complete story from the user perspective - you have a conversation with the AI, a game is created and finally you export it to Android studio for deployment in Google play store or any other market place.

What's next

Better and shorter presentation

Prepare the installation of all the components as well as SceneMax3D dev studio

Get feedback from the community

Prepare documentation for the architectural strategies, entities diagram etc.

So far I'm getting very good vibes from the game dev. community, and friends on various WhatsApp groups.

[adi-barda]

Chapter 3 - Gemini API Developer Competition - The importance of the unimportant input

The system is advancing as planned. I invest every minute of my spare time before and after work to push this project to the finish line.

So far I have a kind of complete story to show - The user starts the session with the computer, during that time he can freely speak to the Gemini agent and a game is developed in front of his eyes according to his wishes.

As mentioned in previous chapters, the role of Gemini AI is not to produce game code but rather to understand what the user wants up to a certain level of resolution so if we take a typical user sentence for example: "Hi Gemini, How are you? I would like to create a car racing game with a fast car and a professional racetrack. Can you help me please?" The Gemini prompt instruct the AI to dig just the "important" things, in this example - the user wants: 1. To create a car racing game 2. A professional race track 3. A fast car

The "unimportant" parts are: 1. Hi Gemini 2. How are you 3. Can you help me 4. Please I decided to response also to the "unimportant" parts so that the agent will react as fast as possible to user input such as "Thank you", "Hi", "Great" etc. The response can be an Emoji drawn to the game's screen along with a text and even some speech. This greatly helps the user to feel like he is having a session with a human being rather than a machine. My wife, which tested the system, liked this feature even more than the game creation itself!

What's next

1. Add support for fighting game creation 2. Allow exporting to Android native application

[adi-barda]

Chapter 2 - Gemini API Developer Competition

The last few days were intense. I needed to have a complete flow running with consistent results. Last time I created the plug-ins system for the SceneMax3D platform and an in-process webserver plugin for accepting commands from the outside world into the running game and modify it on-the-fly. This time, I was focused on creating a set of plugins for the SceneMax3D IDE as follows:

Speech To Text - I decided to use AssemblyAI as my speech to text provider. They provide an extremely easy to use speech to text Java SDK and registration process along with some nice features and a reasonable pricing model. For the speech to text feature, I decided to use the real-time streaming model. It was the coolest and easiest to implement.

Prompt Manager - the text provided by the speech-to-text plugin was handed to a prompt manager component which produced a well constructed and formatted prompt for Gemini AI including system instruction and user prompt.

Gemini AI API - the after mentioned prompt was delivered to a Gemini API component which is responsible for sending it to the Gemini AI service and return its result as plain text.

Configuration to code converter - the Gemini AI JSON text result is passed to a configuration to code converter which reproduces a valid SceneMax3D code. The code converter component has a built-in specific code converters for all kind of game genres such as racing games, fighting games, empty unknown games etc.

Gemini SceneMax3D Integration - this component is responsible for managing the entire workflow, orchestrating the various components described above from the user triggering the speech recognition and up to the generated SceneMax3D code which is sent to the run-time engine in real-time.

What's next

Now, that we have a complete working flow from the user interaction to the run-time game engine, we need to add the following features:

Enhance to configuration to code converter - support as much as possible user requests scenarios from updating a running scene up to switching to a new game concept while supporting as much as possible different game types.

Enhance the "Wow Factor" - make it look as impressive as possible emphasizing the Gemini AI contribution for the success of this process.

Export to Android devices - this is a requirement for this competition to have a Gemini AI product running on Android devices

Create a 3 minutes video clip demonstrating the product's capabilities.

[adi-barda]

Gemini API Developer Competition

So, I decided to compete in Google's AI developer competition. It's about writing an innovative application using the Gemini AI to perform some cool logic, submit a video clip explaining your innovation along with the source code and there's a chance of winning cool prizes. It's also exciting to test your innovation & coding skills. I decided to write a kind of real-time, AI based game creator - the user will talk to the AI agent (Gemini in this case), explain how the scene should look like and behave and according to that chat a game will be created and evolve in real-time. I already have built my own a game engine, SceneMax3D which uses the open source Java project JMonkey Engine for the scene rendering and other open source libraries such as Minie for the physics simulation etc. SceneMax3D has its own dedicated programming language and IDE which makes the AI developer competition project so much simpler to implement since I have control on all the components used for creating a 3D game and all that's left is to integrate Gemini AI API for creating a game…

So what is the initial plan?

Plugins system - Google requires the application's source code and there's no way to submit the entire SceneMax3D project sources so I decided to create a plug-ins mechanism for SceneMax3D and submit the Gemini integration plug-ins source code.

Language grammar - SceneMax3D programming language should support using any of the installed plugins so a new plugins command was added to the language. In this case, I need to activate an in-process web server while the engine is running. This will allow direct communication from the outside world to the engine while the game is running so after Gemini AI will generate its product, it could be streamed through a REST API using this in-proc web server.

Speech to text - the end-user should be able to express his thoughts by talking to the agent. The user's speech will be converted to text, which will be used inside some clever prompt, which will be sent to Gemini AI to do his magic.

IDE plugin - in addition to writing a plugin for the run-time engine, we will need one more plugin for the IDE. Its role is to accept the Gemini product, convert it to some SceneMax3D code and send that code to the engine for execution. In rare cases we will ask Gemini to actually create the final SceneMax3D code. It's possible but the results might need manual corrections to make it fit to the game.

Export to Android - the generated game should be able to run on Android devices so all kind of design aspects should be taken in consideration while generating the game's code.

So this is the high level design of the SceneMax3D-Gemini AI integration system. I believe that this design will allow people with no coding skills at all to create some fun games.

What is developed so far

Plug-ins system for SceneMax3D - the run-time engine as well as the IDE can now load pre-installed plug-ins

SceneMax3D language modifications - the programming language now supports a new "plugins" command which can start and stop any installed plugin. for example, we can open our game to accept commands from the outside world by writing the command: plugins.ws.start 8080 This means that an internal web server will start, listening on port 8080, accepting commands from the outside world via the /run REST API

In-process web server plugin for the run-time engine - allows starting an in-process web server while the game is running. For the web server support I chose using the NanoHttpd open source project. It's very simple, small and runs perfectly on any platform including mobile devices. I have very good experience using it in my Android off-road navigation application, Adv Rally where it is used for rendering an offline maps.

I tested this configuration and it works perfectly. I was able to run a scene, start the in-proc web server using the "plugins.ws.start" command and send commands to the scene from the outside world using postman client. The engine accepted the commands and executed them affecting the game's scene in real-time.

What's next

IDE plugin for converting configuration (which will be generated in the future by Gemini) to SceneMax3D code

IDE plugin for sending prompts to Gemini and accepting the results as a JSON configuration

Speech to text component - can be either a Java Swing form (implemented as an IDE plugin), an Android application or a Web application. IDE form and Web application can run locally and send their commands to the engine's in-proc server. Android app or a remote website will need an additional mechanism to push their commands to the engine. A good candidate can be socket.io client or a public SQS queue.