#debugging

[charyou-tree]

When you're debugging code, there are a number of possible scenarios, in order of increasing scariness:

The code works, but with some known issues.

The code doesn't work, but I know why.

The code doesn't work, and I don't know why.

The code used to work, but I know what changed.

The code used to work, and I don't know what changed.

The code used to work, failed, and I fixed it so it works again, but now that I fixed it I'm not sure how it EVER worked before.

The code works, even though it clearly shouldn't, and I don't know why.

Just fixed a #6 issue at work so I'm a little shook.

[techav]

On Celebrating Errors

Isn't it beautiful? The lovely formatted tables of register and stack contents, the trace of function addresses and parameters, the error message ... it's the most beautiful kernel panic I have ever seen.

Why on earth would I be so excited to see a computer crash? What could possibly be beautiful about a kernel panic?

This kernel panic is well-earned. I fought hard to get it.

This kernel panic came from a current NetBSD kernel, freshly compiled and running on Wrap030, my 68030 homebrew computer. It is the result of hours upon hours of work reading through existing code, scattered documentation and notes, writing and rewriting, and endless compiling.

And it's just the start.

As I've said before, a goal of this project has always been to build something capable of running some kind of Unix-like operating system. Now that I finally have all the necessary pieces of hardware, plus a good bootloader in ROM, it's time to give it a shot. I'm not that great with this type of programming, but I have been getting better. I might just be able to brute force my way through hacking together something functional.

It is hard.

There is some documentation available. The man(9) pages are useful, and NetBSD has a great guide to setting up the build environment for cross-compiling the kernel. There are some published papers on what some people went through to port NetBSD to this system or that. But there's nothing that really explains what all these source code files are, and which parts really need to be modified to run on a different system.

I had a few false starts, but ultimately found an existing 68k architecture, cesfic, which was a bare minimum configuration that could serve well as a foundation for my purposes. I copied the cesfic source directory, changed all instances of the name to wrap030, made sure it still compiled, then set about removing everything that I didn't need. It still compiled, so now it's was time to add in what I did need.

... how ... do I ... ?

This is where things get overwhelming very quickly. There is documentation on the core functions required for a new driver, there's documentation on the autoconf system that attaches drivers to devices in the tree, and there's plenty of drivers already to reference. But where to start?

I started by trying to add the com driver for the 16550 UARTs I'm using. It doesn't compile because I'm missing dependencies. The missing functions are missing because of a breaking change to bus.h at some point; the com driver expects the new format but the cesfic port still uses the old. So I needed to pull in the missing functions from another m68k arch. Which then required more missing functions and headers to be pulled in. Eventually it compiled without error again, but that doesn't mean it will actually run. I still needed to add support for my new programmable timer, customize the startup process, update hardware addresses, make sure it was targeting 68030 instead of 68040 ...

So many parts and pieces that need to be updated. Each one requiring searching for the original function or variable declaration to confirm expected types or implementation, then searching for existing usages to figure out what it needs ... which then requires searching for more functions and variable types.

But I got something that at least appeared to have all the right parts and compiled without error. It was time to throw it on a disk, load it up, and see what happened.

Nothing happened, of course. It crashed immediately.

I have no debugging workflow I can rely on here, and at this stage there isn't even a kernel console yet. All I could do was add little print macros to the locore startup code and see where it failed. Guess, test, and revise.

I spent a week debugging the MMU initialization. If the MMU isn't properly configured, everything comes to an abrupt halt. Ultimately, I replaced the cesfic machine-specific initialization code and pmap bootstrapping code with functions from yet another m68k arch. And spent another day debugging before realizing I had missed a section that had comments suggesting it wasn't for the 68030 CPU, but turned out to be critical for operation of kernel memory allocation.

Until this point, I was able to rely on the low-level exception handling built into my bootloader if my code caused a CPU exception. But with the MMU working, that code was no longer mapped.

So then came another few hours learning how to create a minimal early console driver. An early console is used by the kernel prior to the real console getting initialized. In this case, I'm using the MC6850 on my mainboard for the early console, since that's what my bootloader uses. And finally the kernel was able to speak for itself.

It printed its own panic.

The first thing the kernel does is initialize the console. Which requires that com driver and all the machine-specific code I had to write. The kernel is failing at its step #1.

But at least it can tell me that now. And given all the work necessary to get to this point, that kernel panic data printing to the terminal is absolutely beautiful.

[mrbreaknfix]

Other people follow rules. I follow results

[pressplay-if]

I’ve noticed with the siblings background MC when you give them different names they still show up at Linda and Elliot is that a bug or intentional <:D

OH DAMMIT DAMMIT DAMMIT

...needless to say from that reaction, it's not intentional

[el-ffej]

Text below:

Angels: so what's up with the platypus

God: yeah, about that

Angels: seriously

God: idk it doesn't do anything but it I take it out the world won't compile

[python-programming-language]

Different ways of debugging my source code ...

Post #88: www.pythonforbiologists.com, My code isn't working, 2023.

[lost-coder]

[code-tryperse]

Bugs in Sky: CotL: Broken Gates and Lonely Areas

Disclaimer: I am not affiliated in any way to TGC. Barring confirmation from the developers, everything below is a hypothesis based on observed game behavior.

From January 16, 04:02 GMT (the release of update 0.28.0) up until January 20, 04:03 GMT, Sky players experienced a bug where players rarely encountered other players, resulting in a rather lonely experience playing the game. At the same time, gates and other objects which required lighting an interactable element to activate didn't activate even after lighting said element. Most players attributed this bug to server merging (i.e.: a function where two servers have their players and state combined into one) not working as intended.

This wasn't the case, however, as it was still entirely possible to encounter other players at times. It was just happening a lot rarer than it used to. Instead, a related function — server splitting — was not working as intended which, in turn, caused server merging to happen extremely rarely.

Potential explanation of what happened below the cut.

TLDR: The 0.28.0 update to the game likely accidentally disabled server splitting in all areas of the game, causing server merging to rarely happen and for some gates to stop working.

Keeping Players Close: Server Splits and Server Merges

Have you ever had an experience where you were with your friends when all of a sudden, they all disappear in front of you? And when you teleport back to them, they apparently were still in the same place as they were? Congratulations, you just encountered a server split. A server split is where a player or group of players get disconnected from a server they're currently in and pushed into their own server. Sometimes, the server will separate everyone out of it in what will be called here a "full server split". In that case, everyone gets pushed into a server where they're the only player. The scenario mentioned above is a very noticeable example of a full server split happening. Most of the time, however, server splits don't happen to every player in the server. Instead, it only happens to one player or group of players in that server. This kind of separation is what this blog will refer to when mentioning the term "server split".

The exact criteria needed for a server to split a player or group of players isn't known. However, one criterion that can be easily observed is that the server will split a player or group of players when there are no other players within close proximity to them. "Close proximity" is not defined in strict terms here as it is currently not known how far this distance needs to be. However, to observe this behavior, you can do the following steps:

Stand on one of the floating islands in Sanctuary Island.

Have a friend fly towards the main island.

Keep honking in order to see your friend's name.

As they fly down to the main island, eventually, your friend's name will stop appearing when you honk. A server split just happened. Because you are far from your friend, the server thinks that it is alright to split the two of you in order for a server merge to bring either of you to a server where players are close to you or your friend.

This is necessary behavior because there's a limit of 8 players per server. In order for a server merge to be successful, there must be at most 8 players in both servers combined. By splitting players from the server they're in, it increases the chances that the game will find a server that they can be merged in by reducing the number of players in their server. This is why waiting inside the 8-player puzzle room when there's already 7 players there usually takes a long time. By having 7 players in the server, only a server with one player can be merged into it. However, the two-player door before the room means that most servers with players in that room will have at least two players. Those servers cannot be merged into your 7-player server.

Determining the Absence of Server Splits

To say that server splitting isn't working as intended, we can establish the existence of players that should have been split away from us within the server. Normally, it would be hard to determine such a case, as that entails having to search for the presence of players in areas we cannot see. Additionally, if we do spot a player, we cannot know if that's a player who was already there, or one who was just merged into the server. Thankfully, there is a way to determine the presence of players without seeing them: the chat history.

When you send a chat message in Sky, it is sent to all players in the server you are in, including yourself. This prompts the game to display that chat message on top of the player who said it as well as in the chat history. However, if you don't have chat unlocked with the player, or if either of you aren't in a chat bench (doesn't have to be the same one), the game will replace the text of the message with a bunch of asterisks when displaying it to you. Additionally, if you have a player muted, the game will not display the messages they are sending, but it will still store them. This can be verified by muting a player, having them say something, then unmuting them. The game will then show the messages they sent while you have them muted, which means that it stores their messages even when they're muted.

Because player messages are sent to the entire server, you can see the messages of players in other locations, provided that they are in the same server as you. You can verify this by joining a friend (chat can be locked or unlocked), going into a level transition, then having them chat something. Their messages will still appear in the chat history until the server splits you off from them.

We know that this happened during the period of the bug because there are numerous bug reports in the Sky Discord mentioning the appearance of messages from players who are nowhere near the reporter. These messages can only appear if a player had sent them, which means that there's a player somewhere in that server sending these messages. This player is far from the reporter, which means that they should have been split off from them. However, that is not the case, which shows that server splitting is not working as intended.

Gates and Their Dependencies on Server Merging

Another part of the bug is that none of the gates and other places which require lighting some sort of pedestal or key to open work. While the reason for this cannot be definitively determined, it is possible that these required a check that involved a server merge for it to start working properly. Given that a server merge usually happens often and sometimes unnoticeable, it is likely that a server merge was made as a requirement to ensure that some gates only work when there's multiple players in a given area. The presence of this requirement doesn't make much sense, as two-player and four-player doors already require multiple players on their own, and requiring multiple players for one-player doors is not intuitive and sometimes even harmful for user experience. However, this is the most likely explanation as to why this bug is related to the server splits bug.

Reworking Maps and Inadvertent Changes

To give a possible explanation as to how this bug happened, we need to look at a handful of places and scenarios, namely:

Homespace and Aviary Village

Eye of Eden and Orbit

Valley of Triumph Races and Red Shard Memory Collections

Homespace and Aviary Village serve as central hubs for players to go through. As such, it is important that players be capable of gathering in these areas. Server splits would require players to not explore the central hub while waiting for others, which can make for a boring experience for some. Additionally, full server splits may also happen, which will force the players to repeat the process of gathering, which can be annoying to deal with. As such, server splitting is disabled in these areas.

Past the first area of Eye of Eden, server merging is disabled. While the game's tip for Eye of Eden is to stick together, keeping server splitting enabled is too punishing, as it prevents a player straying too far ahead or too far behind from receiving help from other players. Additionally for Orbit, keeping server splitting enabled can prevent players from hanging around with the people they shared the experience of the end with, which can reduce player satisfaction. As such, server splitting is disabled in these areas.

Races in Valley of Triumph as well as memory collection in Red Shard events sum up all the light/memories that all players in a server receive. If a player within a race or within the memory collection activity were to get split off from the server, it could make the light calculation inconsistent. To prevent these, server splitting is disabled while a race or any other activity is active.

What this all shows is that server splitting is something that can be enabled or disabled when needed. In preparation for the new season, nearly all areas of the map have been updated for the new dye light mechanic. It is possible that, while updating the maps, server splitting was accidentally disabled in these areas, causing server splitting to be disabled in all areas of the game.

A Quick Hotfix

An update was released on January 20, 04:03 GMT which fixed this issue completely. Server splitting now works as intended again. This didn't require a manual update of the game, which meant that this was a fix that involved only editing the assets of the game and not the actual code. Either way, players can now continue playing the game, finding others who are also playing near their character.

[adafruit]

A little Espressif helper for programming

While developing boards, we often want to program ESP chips without going through the onboard USB port; this adapter will help us (and others) do that! It has a CP2102N USB-serial chip

https://www.digikey.com/short/bm7n3p5z

with RX/TX signal LEDs and two transistors wired up to the DTR/RTS line for the 'esptool standard' reset procedure technique. The output IO, plus a 3.3V 500mA regulated output, is available on a socket header, so you can plug wires in for quick programming and debugging. You can use this for everything from an ESP8266 to an ESP32-P4!

[thehattyguy]

Been deveveloping the world system for my game and ended up separating physics and logic from the rendering aspect of the game

That made it a bit hard to debug collisions and things related to the "logic" world because its all hidden within their own World3D so that I can have multiple worlds that don't interact with each other

So I made this small modular debug menu and added the first debug option which creates a duplicate camera inside the main logic world (maybe I'll make it move to the current focused world when I make ways of travelling between worlds) and overlays the rendered image on top of the regular game

This already helped me fix some bugs and there are still some others that showed up since I split logic and rendering that I'm still working on

Also! I had made the equipment section of the inventory binder and didn't end up making a post about it but it came out pretty nice aswell, I'll maybe post some more details about it when its more finished since for now you can only see and edit your deck

Here's how it's looking right now:

[techav]

On Recovering From Disaster

Just before disaster struck with my 68030 homebrew, I had a plan to fork the project and take it in two directions in parallel.

The first would be my minimal Multibasic setup which was running great and was something I would be proud to exhibit at VCFSW '25.

Since that was working well, I didn't want to do anything that might make it not ready for the show. So I had assembled a second main board that I could use with other peripheral cards to continue development. My plan was to rework the memory map to move ROM and I/O addresses to the upper half of the 32-bit address space, which would allow me to use a DRAM card for up to 16MB of contiguous memory starting at address 0 — a perfect setup for running a proper OS

And then I burned up my good mainboard.

I had a spare CPLD on-hand, and I was able to get another 40MHz 68EC040 for cheap. But the 4Mbit SRAM chips are expensive, and I thought that if I was going to be paying for more SRAM, I should get the 12ns parts that would allow me to run RAM with no wait states at 50MHz. Which would require adapter boards, and more pin headers, and would have just driven the cost up even more.

Paralyzed by indecision and the rising cost, I decided to switch gears. I already have the DRAM board I built years ago, I just needed to get it working. And if I'm doing that, I may as well do the work on reorganizing my memory map.

So that's what I did. I made a new branch on my newly cleaned up repository, and set to work. I updated the mainboard bus controller logic to remove the no-longer-needed SRAM control, adjust the ROM address to start at 0x8000,0000, and move the 6850 UART address to 0x8008,0000. This freed up the entire 2GB space between 0x0000,0000 and 0x7FFF,FFFF for the DRAM card to use for main memory (my bus connector layout only allows for up to 16MB main memory, but that's plenty for my purpose).

Next, I needed to build the control logic for my DRAM board. I've done this once before on my Wrap030-ATX project a couple years ago, so I used that logic as a starting point. It … kind-of worked. Sometimes. Maybe.

I was getting random errors all over the place. Sometimes it was able to copy all of ROM into RAM and verify the copy without error, but it would fail when running code from RAM. Sometimes it wouldn't even pass the first two simple tests of writing a longword to RAM and reading it back.

Back to the data sheets to plan out the logic. I drew out a new timing diagram down to 1ns resolution, accounting for the specified propagation time for my CPLDs, and the measured signal delays for my CPU. This gave me the information I needed to build out a new state machine for the DRAM controller. In the process I also caught a few other bugs in my original logic, such as not ensuring the DRAM Write Enable strobe did not get asserted during refresh cycles.

The new state machine worked much better, but I got my timing off a bit. It worked reliably at 16MHz, but not at the intended 25MHz. At least it was working though; I can move forward with 16MHz.

The next thing to do was get my 8-port serial card working with this new setup. Every time the computer tried to access one of the 8 serial ports, it would get a bus error. This was a tough one that had me stuck for a while.

It looked like the serial card was getting addressed and asserting DSACK0# as expected, but I couldn't confirm the signal was making it back to the CPU. Every time I tried to measure the DSACK signals, the probes would increase the line capacitance (I think) too much and cause the whole computer to fail before it even got to the point of trying to address the serial ports.

I have the DSACK signals in a wired-or configuration. The logic on each card connects to the proper DSACK signal via an open-collector buffer, and is able to pull the signal low to signal that card acknowledging the bus cycle. The signal is then pulled back up to +5V by a 1kΩ resistor. This works well enough for small, slow systems, but with long bus wires or lots of chips on the bus, the extra capacitance means it can take a long time for that pull-up resistor to get the signal back up to +5V.

I made a change to my DRAM controller to actively drive the DSACK signals to +5V briefly at the end of a bus cycle before setting the pin to high-impedance. This helps ensure a fast rise time at the end of the cycle so it doesn't disrupt the next bus cycle.

It didn't fix the problem with the serial card, but it did allow me to actual probe the signals live to see what was happening — the Bus Error signal was getting asserted immediately after the CPU addressed the serial card. This didn't make much sense to me. My main board logic will assert Bus Error if no peripherals acknowledge a bus transaction within 64 clock cycles. Nothing should be asserting Bus Error so quickly.

Except … the main board bus controller. My original memory map used the lower half of the address space for the main board, and peripheral cards used the upper half. So my original main board logic was built to assert Bus Error immediately for an unused address in the lower half of the address space. I hadn't properly removed that bit of logic, so it was now asserting Bus Error for any accesses to the upper half of the address space that weren't used by the main board.

The challenges of working on a project after six years. I had forgotten that bit of logic was even there.

With that erroneous Bus Error assertation removed, the machine was finally able to properly load my Multibasic kernel and run BASIC for 8 users again.

At the moment it is running stable at 24MHz on my 25MHz-rated full 68030. It may not be the 50MHz high I was at, but after the frustration and disappointment of the past few weeks, I'm absolutely taking that as a win. 24MHz is incredible.

[charyou-tree]

The perpetual question while I'm debugging code on the cluster:

Is it not running because I fucked something up, or just because the job queue is full and its waiting to start?

[pressplay-if]

My saves were gone so I've been replaying that whole story. Just got into the new bit and think I've found a bug.

If you block Lincoln it says;

"With a few taps on my screen, he’s out of sight and out of mind. It's as easy as that.

“I simply didn’t feel like it.”

I understand. I just hope I’ll see you again soon."

It seems to then carry on a conversation that I think is with him, it doesn't really say it's just back and forth and a bit jarring.

Thanks for reporting! I fixed it, you should be rid of the slug (Lincoln) now

[peachgull]

Debug scraps I found at :

[thoughtportal]