adi-barda

[adi-barda]

Back To Business

On Aug-12 2024 I have submitted a project for a global AI coding competition, Then for ~2 month I didn't (couldn't) touch any of my personal projects. I guess that's what happens when you "overflow" putting too much effort in 1 area and neglecting other areas. At work, I have accepted the challenge of becoming cloud security CIEM team leader. I'm excited for this new role but from past experience as a team leader in previous work place, I know that it comes with a lot of pressure. In addition, these days I'm starting my master degree in Digital Games Design. I scheduled my courses for the coming two semesters and (as expected) it consumes almost all of my free time. It also costs a lot of money but I feel that I owe it to myself to keep learning and developing my knowledge and meeting new people and having fun. I will not be able to finish it in 2 years so it is going to be a long 3 years journey... SceneMax3D might be involved in an interesting secret project. I can't tell much and I will not be doing the POC myself since I don't have any spare time but I will probably guide other developers how to use the engine. That should be a lot of fun to do :) What else - I take my time to learn the Irish Tin Whistle, It's such a fun addictive and practical piece of musical instrument. I highly recommend trying it out. It's not that easy to learn tough... Here is me trying to perform "The Foggy Dew" Irish war song:

Cheers!

[adi-barda]

Chapter 5 - Gemini API Developer Competition - Project Submission

The last two week were devoted to finalization of the AI Game Maker project: 1. Added attack strategy to the opponent in a fighting game - this feature allows the Gemini AI to assemble a fight strategy according to some statistics provided by the current game state. This is big topic by itself involving a lot of fine-tuning and cost optimizations (you don't want to use the AI to often while the game is running as it might cost quite a lot) 2. Prepare a new SceneMax3D installation with the new plugins system and upload it to the SceneMax3D website 3. Push the latest SceneMax3D mobile engine to the public repo in GitHub. The mobile engine is an open source project and it's 99% similar to the PC engine. 4. Prepare 2 sound tracks for the demo video clip. I used Suno AI for that and it was a very good experience. I highly recommend it for anyone who seeks to create amazing music for his products. 5. Prepare a short 3 minutes clip for the competition demo. I used Davinci Resolve for this task. I absolutely love this software and it's well worth the ~300$ I paid for it. The free version is also very good 6. Prepare Zip file with all the source code of this AI Game Maker project + binaries + instructions. This is a mandatory delivery for the competition.

7. Submission - on Aug-02-2024 I submitted my innovation. The last day for submission is on Aug-12 and then starts the fun part of reviewing other projects, voting etc. In about two months I start my master's degree in computer game design and the thesis is planned to be on the subject of AI generated games so a big part of that project is already made while participating in the Gemini competition - a win-win :)

My vision is to embed AI in major parts of SceneMax3D language and IDE. It opens a big opportunity for innovation.

Cheers!

[adi-barda]

Chapter 4 - Gemini API Developer Competition - Fighting game & Android Export

As planned, I spent the last days on adding fighting game capability to the engine and Android exporting feature. The fighting game has much more details in the puzzle for the AI agent to cope with. There are complex animations for the player and for the opponent, they need to constantly look at each other, you need to be able to demo their kick, punch, block animations, the player needs to be able to move in 3D space etc. Overall I'm very pleased with the results so far. The user can speak freely enough with the AI, get instant results and funny reactions. What's more, I've been able to add Android exporting of the game and automatically open it in Android studio. It was challenging because the Java code worked different on PC and on the mobile device specifically handling of Zip files and all kind of Gradle dependency hell. ChatGPT was on my side all the way, assisting me to resolve configuration issues and coding problems such as selecting the best Zip 3rd party library.

This video clip, demonstrates the current status of the project. It shows a complete story from the user perspective - you have a conversation with the AI, a game is created and finally you export it to Android studio for deployment in Google play store or any other market place.

What's next

Better and shorter presentation

Prepare the installation of all the components as well as SceneMax3D dev studio

Get feedback from the community

Prepare documentation for the architectural strategies, entities diagram etc.

So far I'm getting very good vibes from the game dev. community, and friends on various WhatsApp groups.

[adi-barda]

Chapter 3 - Gemini API Developer Competition - The importance of the unimportant input

The system is advancing as planned. I invest every minute of my spare time before and after work to push this project to the finish line.

So far I have a kind of complete story to show - The user starts the session with the computer, during that time he can freely speak to the Gemini agent and a game is developed in front of his eyes according to his wishes.

As mentioned in previous chapters, the role of Gemini AI is not to produce game code but rather to understand what the user wants up to a certain level of resolution so if we take a typical user sentence for example: "Hi Gemini, How are you? I would like to create a car racing game with a fast car and a professional racetrack. Can you help me please?" The Gemini prompt instruct the AI to dig just the "important" things, in this example - the user wants: 1. To create a car racing game 2. A professional race track 3. A fast car

The "unimportant" parts are: 1. Hi Gemini 2. How are you 3. Can you help me 4. Please I decided to response also to the "unimportant" parts so that the agent will react as fast as possible to user input such as "Thank you", "Hi", "Great" etc. The response can be an Emoji drawn to the game's screen along with a text and even some speech. This greatly helps the user to feel like he is having a session with a human being rather than a machine. My wife, which tested the system, liked this feature even more than the game creation itself!

What's next

1. Add support for fighting game creation 2. Allow exporting to Android native application

[adi-barda]

Chapter 2 - Gemini API Developer Competition

The last few days were intense. I needed to have a complete flow running with consistent results. Last time I created the plug-ins system for the SceneMax3D platform and an in-process webserver plugin for accepting commands from the outside world into the running game and modify it on-the-fly. This time, I was focused on creating a set of plugins for the SceneMax3D IDE as follows:

Speech To Text - I decided to use AssemblyAI as my speech to text provider. They provide an extremely easy to use speech to text Java SDK and registration process along with some nice features and a reasonable pricing model. For the speech to text feature, I decided to use the real-time streaming model. It was the coolest and easiest to implement.

Prompt Manager - the text provided by the speech-to-text plugin was handed to a prompt manager component which produced a well constructed and formatted prompt for Gemini AI including system instruction and user prompt.

Gemini AI API - the after mentioned prompt was delivered to a Gemini API component which is responsible for sending it to the Gemini AI service and return its result as plain text.

Configuration to code converter - the Gemini AI JSON text result is passed to a configuration to code converter which reproduces a valid SceneMax3D code. The code converter component has a built-in specific code converters for all kind of game genres such as racing games, fighting games, empty unknown games etc.

Gemini SceneMax3D Integration - this component is responsible for managing the entire workflow, orchestrating the various components described above from the user triggering the speech recognition and up to the generated SceneMax3D code which is sent to the run-time engine in real-time.

What's next

Now, that we have a complete working flow from the user interaction to the run-time game engine, we need to add the following features:

Enhance to configuration to code converter - support as much as possible user requests scenarios from updating a running scene up to switching to a new game concept while supporting as much as possible different game types.

Enhance the "Wow Factor" - make it look as impressive as possible emphasizing the Gemini AI contribution for the success of this process.

Export to Android devices - this is a requirement for this competition to have a Gemini AI product running on Android devices

Create a 3 minutes video clip demonstrating the product's capabilities.

[adi-barda]

Gemini API Developer Competition

So, I decided to compete in Google's AI developer competition. It's about writing an innovative application using the Gemini AI to perform some cool logic, submit a video clip explaining your innovation along with the source code and there's a chance of winning cool prizes. It's also exciting to test your innovation & coding skills. I decided to write a kind of real-time, AI based game creator - the user will talk to the AI agent (Gemini in this case), explain how the scene should look like and behave and according to that chat a game will be created and evolve in real-time. I already have built my own a game engine, SceneMax3D which uses the open source Java project JMonkey Engine for the scene rendering and other open source libraries such as Minie for the physics simulation etc. SceneMax3D has its own dedicated programming language and IDE which makes the AI developer competition project so much simpler to implement since I have control on all the components used for creating a 3D game and all that's left is to integrate Gemini AI API for creating a game…

So what is the initial plan?

Plugins system - Google requires the application's source code and there's no way to submit the entire SceneMax3D project sources so I decided to create a plug-ins mechanism for SceneMax3D and submit the Gemini integration plug-ins source code.

Language grammar - SceneMax3D programming language should support using any of the installed plugins so a new plugins command was added to the language. In this case, I need to activate an in-process web server while the engine is running. This will allow direct communication from the outside world to the engine while the game is running so after Gemini AI will generate its product, it could be streamed through a REST API using this in-proc web server.

Speech to text - the end-user should be able to express his thoughts by talking to the agent. The user's speech will be converted to text, which will be used inside some clever prompt, which will be sent to Gemini AI to do his magic.

IDE plugin - in addition to writing a plugin for the run-time engine, we will need one more plugin for the IDE. Its role is to accept the Gemini product, convert it to some SceneMax3D code and send that code to the engine for execution. In rare cases we will ask Gemini to actually create the final SceneMax3D code. It's possible but the results might need manual corrections to make it fit to the game.

Export to Android - the generated game should be able to run on Android devices so all kind of design aspects should be taken in consideration while generating the game's code.

So this is the high level design of the SceneMax3D-Gemini AI integration system. I believe that this design will allow people with no coding skills at all to create some fun games.

What is developed so far

Plug-ins system for SceneMax3D - the run-time engine as well as the IDE can now load pre-installed plug-ins

SceneMax3D language modifications - the programming language now supports a new "plugins" command which can start and stop any installed plugin. for example, we can open our game to accept commands from the outside world by writing the command: plugins.ws.start 8080 This means that an internal web server will start, listening on port 8080, accepting commands from the outside world via the /run REST API

In-process web server plugin for the run-time engine - allows starting an in-process web server while the game is running. For the web server support I chose using the NanoHttpd open source project. It's very simple, small and runs perfectly on any platform including mobile devices. I have very good experience using it in my Android off-road navigation application, Adv Rally where it is used for rendering an offline maps.

I tested this configuration and it works perfectly. I was able to run a scene, start the in-proc web server using the "plugins.ws.start" command and send commands to the scene from the outside world using postman client. The engine accepted the commands and executed them affecting the game's scene in real-time.

What's next

IDE plugin for converting configuration (which will be generated in the future by Gemini) to SceneMax3D code

IDE plugin for sending prompts to Gemini and accepting the results as a JSON configuration

Speech to text component - can be either a Java Swing form (implemented as an IDE plugin), an Android application or a Web application. IDE form and Web application can run locally and send their commands to the engine's in-proc server. Android app or a remote website will need an additional mechanism to push their commands to the engine. A good candidate can be socket.io client or a public SQS queue.