drawpile-dev-diary

[drawpile-dev-diary]

Thoughts on Drawpile 2.2 server architecture

Drawpile 2.2 (beta) is a major reworking of the client application's internals, but the server has remained untouched. To better support the "thick server" operating mode (introduced in 2.1,) it's time to take another look at the architecture. (Thick server is what I call a server that has its own copy of the paint engine so that it can generate a snapshot of the canvas state for new users on its own, thus eliminating the need for session resets and speeding up logins. A thick server is not, however, strictly better than a thin server as it requires more resources on the server side and is tied to a specific protocol version.)

Before we can start planning the next generation architecture, we need to take a look at the current state. Here's a rough architectural diagram of the 2.1 server:

At the heart is the Session class. This represents a running Drawpile session. It contains Session History, which is the session content, past and present. The session also holds the Clients, which are the users presently logged in.

The Session class is abstract and has two concrete implementations: ThinSession and ThickSession.

The SessionHistory class is also abstract and has two implementations: one that holds the session content in memory and one that keeps it in a file. The latter is useful for very long running sessions and for running on a server with limited memory.

Now, here is the kink that needs ironing out: the ThickSession implementation does not use the SessionHistory! (Or it does but only partially in a weird way.) Instead, most of the session content is kept in the paint engine. So what bits are kept where? I for one don't remember! A neater design would be to put the paint engine into a SessionHistory implementation.

Next, the Client abstract class has a concrete implementation for Thin and Thick server modes. There is barely any code in either implementation, so they probably don't need to be two separate classes.

A better architecture might look something like this:

Less classes, no inheritance, easier to understand. The above diagram was made with the C++ implementation in mind, but the Rust version should look just about the same. One difference between Rust and C++ is that Rust doesn't have class inheritance. It does, however, have traits which are similar to interfaces in other languages. In the above diagram, SessionHistory has been changed to an interface, which should make the design compatible with the Rust way of doing things.

Another complication in the present architecture is that a Session instance keeps a list of Clients, but each Client also keeps a pointer to the Session it belongs to. In Rust, this kind of two-way reference is difficult to represent and, in any language, is more difficult to reason about. The client has a connection object through which it receives messages before passing those messages to the session it belongs to, or the login handler if not yet part of a session.

Luckily, we don't have to try replicate this architecture 1:1 in the Rust version. Just a small modification is needed:

When a client connects, an asynchronous task is spawned to handle incoming messages. Received messages are first passed to the login handler. When the login phase completes, the client is added to a session. Further messages are then passed directly to the session's message handler, removing the need for the client to have a reference to the owning session. The client instance will still contain an object representing the network connection, but this will only be used for sending, not receiving.

The new server probably won't be ready for 2.2.0 stable release, but I plan on finishing it not long after.

[drawpile-dev-diary]

A half-baked idea for better undoing

The fairly short undo depth limit is a common irritation for many users. There is a technical reason for it, however. Undo in Drawpile works by rolling back the canvas to a point before the sequence to be undone, then replaying the history with the undone parts omitted. If a user who hasn't drawn anything in a while decides to hit Ctrl-Z, they would effectively cause a denial of service attack as the clients would have to roll back and replay a lot of history, which is a very heavy operation.

This got me thinking, the current implementation is a very naive approach: roll back and replay everything. But often we might not need to. If the undo section is not causally dependent on the rest of the canvas (which is often the case, as users rarely draw over each other,) then we only need to revert that part. We can do that because the layers are tiled: we just keep track of which tiles are affected by the undo sequence and swap those out, then replay just that sequence.

Say we have a canvas like this:

And the (simplified) session history looks something like this:

We could partition that history into concurrent partitions:

Then, when we hit Ctrl-Z, the application finds the last partition for that user, reverts it and replays just that partition. A global undo stack depth limit would still be needed, but that depth limit could be per-partition.

Just something to think more deeply about in the future.

[drawpile-dev-diary]

Drawpile 2.1 architecture overview

Recently, I dusted off the old "Rustpile" branch (my attempt to integrate my experimental Rust based reimplementation of the paint engine to Drawpile itself) and, to my pleasant surprise, discovered it was in a much better shape than I remembered. So, I've been reading through Drawpile's code and drawing a diagram of its overall architecture to get a better idea what it would take to complete the integration.

Here are my notes on the current state of Drawpile's architecture. If it seems unnecessarily complex, that's because it is. There are vestigial structures that no longer make sense and are streamlined away in the Rustpile version. But more on that in the future, here is the present:

The CanvasModel class that contains the state of the canvas, including the layer stack itself, as well as associated models needed by the GUI.

The StateTracker handles drawing commands and applies them to the layer stack. A big change in version 2.1 was that brush state is now entirely local, what is sent over the wire is a list of precomputed dabs. This made the protocol much less stateful, meaning the only thing the state tracker needs to keep track of anymore is the undo history. The Rustpile equivalent is called "CanvasState".

LayerList is a Qt list model. It is used by the layer list GUI widget. Its content is kept in sync with the actual layers in the layerstack.

LayerStack contains the actual layers, as well as other things related to the actual canvas content: the background tile and the annotations.

Layer contains tiles which contain the actual pixel data. Layers are sparse: fully transparent tiles can be presented by null pointers. This is an important optimization, as when multiple layers are used, most layers tend to be mostly transparent. Layers and tiles utilize copy-on-write, which makes copying layers very efficient. This is essential for the undo functionality, that relies on snapshots of the canvas state.

AnnotationModel is a Qt model that contains all the annotations. Annotations work much like text layers, but are not true layers: their stacking order is undefined, there are no guarantees that they render the same on all clients and they are always drawn on top of the canvas. This is a Qt model so that it can be easily accessed via QML (which is not yet implemented, so there is presently no need for this to be a Qt model.)

LayerStack Savepoint is a snapshot of the LayerStack's content. A savepoint can be created from a LayerStack and can then be used to revert the LayerStack to that point. This is unnecessarily complex. In the Rustpile implementation, whole LayerStack instances can be copied cheaply and lack interior mutability thus have no need for savepoints.

A LayerStackObserver is registered with a LayerStack to be notified of changes. An EditableLayer is created to wrap a Layer and add editing functions to it. It notifies the observers of the owning LayerStack when changes are made. In the Rustpile implementation, there is no EditableLayer wrapper, as layers cannot be mutated in place. Instead, all editing operations return an area of effect object that describes the affected area. These can be merged together and dispatched to observers when the LayerStack is updated.

A layerstack can have multiple observers, but in practice just one is enough. A specialized observer class instance that caches the flattened canvas as a QPixmap is shared by all GUI widgets (the canvas view and the navigator.)

The AclFilter stores the state of the access control list used to filter incoming messages. When a message is received, it's first passed to the AclFilter, which either accepts or rejects it. Certain messages affect the ACL itself (e.g. those setting layer lock bits.)

When a Recorder instance exists, it writes a copy of each received message into a recording file that can be later played back. When playing back a recording, ACL filtering is not necessary, since rejected messages were not saved. (For debugging purposes, rejected messages can be stored but marked as such, so they are ignored during playback.)

The Lasers, UserCursors and UserList models store the state of laser pointer lines, the positions of each users cursors and the list of logged in users, respectively. They are Qt models used by the GUI widgets. (Lasers and UserCursors are Qt models only for use in QML, which isn't presently done.)

Additionally, not visible in the diagram, is that the state tracker and the layer stack are referenced in various places:

In the layer list dock, the layerstack's censored bit is checked

The layer stack's view mode is set by an action in the main window

The layerstack is used by the flipbook window

The built-in thick server uses the layer stack, state tracker and ACL filter

The reset dialog needs access to the state trackers savepoints

The canvas view item and navigator reference the cached pixmap layer stack observer

The canvas scene references the annotation, laser and user cursor models

The canvas view widget needs to know the size of the canvas

The annotation editor references the annotation model

The canvas saver runnable needs a copy of the layer stack

The document class references the state tracker and the layerstack

The playback controller uses the state tracker

The annotation tool uses the annotation model

The bezier tool creates preview sublayers

The floodfill tool needs read-only access to the layer stack

Freehand tool needs read-only access to sample colors

The selection tool copies pixel data and creates temporary eraser sublayers

One big problem with the present architecture is that when the paint engine is heavily loaded (for example, when logging into a session and downloading the session history,) it blocks the main thread which leads to the GUI locking up and even disconnects as network traffic isn't being processed.

The solution to this would be to run the paint engine in a separate thread. However, this has proven challenge. From the list above, one can see that the layer stack is referenced in many places. The current workaround is to periodically relinquish control back to the eventloop when paint command execution is taking too long. However, the Rustpile work is an opportunity to fix the architecture to be more multithread compatible. Since in Rustpile, LayerStacks are easily copied, one can be kept around in the main thread for read-only access while a new version is being processed in the paint engine thread. More on this later...

[drawpile-dev-diary]

Improving the brush preset palette

The brush preset palette broke pretty badly in version 2.1.12 and is still a little broken in 2.1.13. This will change in the upcoming 2.1.14 version. 

In the new version, you will be able to sort your brushes into folders. (An import/export feature is planned as well.) This paves the way for adding support for more advanced brush engines in the future.

[drawpile-dev-diary]

Better selection tool

Currently, when you make a selection, you see something like this:

You’ve got the border of the selection and little handles to indicate where you can drag to resize it. Did you know that you can also rotate and skew the selection by holding down alt and shift-alt? (But only when dragging the center of the selection, not the handles!) If not, I don’t blame you. The only way to discover this feature is by accident or reading the “keyboard shortcuts” help page.

In the upcoming version, selections will look like this:

The effect of the drag handles is now more obvious. But this is not all. Clicking on the selection will toggle the transformation mode:

Rotation and shearing is now done using the same handles in the second mode, keyboard no longer needed!

[drawpile-dev-diary]

Detachable chat

The ability to detach the chat box into a separate window has been requested often, and it’s finally making its debut in the soon-to-be-released version 2.1.11.

When right clicking on the chat box, there will be a new option: Detach. Clicking it will move the chat box into a window of its own. When this window is closed, the chat box returns back to the main window where it came from.

[drawpile-dev-diary]

Thin & thick servers and session resets

First a TL;DR: when using the built-in server in version 2.1.9, there are no more autoresets. The dedicated server version of this feature is currently at an experimental stage.

Let’s talk about session resetting. First of all, why is it even necessary?

The Drawpile server is what I call a thin server: it does not know anything about the content of the session and merely passes the data along to each connected user. The session history is all the changes to the canvas, including things like “set canvas size,” “create layer,” and “draw some brush dabs here.”

Because the server doesn’t know what any of that means, it needs to keep the full history in memory to send to newly logged in users. Of course, the amount of data that has to be sent grows the more people paint. This is where autoresetting comes in. When the session history reaches a certain size, the server asks one of the users to reset: replace the session with a new one that’s hopefully smaller than the previous history.

Autoresetting has one purpose and one purpose only: to compact the history smaller so new users can log in faster. If you don’t expect new users to join, you can turn autoresetting off.

But what if you have a very long running session with users coming and going all the time? You want to keep the login time as short as possible, but that means resetting often and disrupting the users already there. This is where the thick server comes in.

A thick server is a server that understands what the session history means. It has its own drawing engine, so it can keep its own live copy of the canvas. When a user logs in, it does not need to ask anyone to perform a reset, it can instead perform an internal soft reset. This means it effectively does an autoreset, but without disturbing any of the users already logged in. So, the end result is that there are no autoresets and every user who logs in gets a fresh reset image, meaning logins are always fast.

However, this comes at a cost. Since the server now needs to perform all the same drawing operations as the client, it means more CPU power and more memory is needed on the server side, meaning less simultaneous sessions per server. Also, since the server must now understand the exact same protocol as the client, it can’t support multiple version (e.g. 2.0.x and 2.1.x) at the same time. (Although there is a way around this. More on this later...)

The dedicated version of the thick server is still experimental, but in version 2.1.9, the application’s built-in server is now a thick server. For the built-in server, the thick server model is actually a better fit than the thin one. Since the session is already tied to a specific Drawpile version, version agnosticism doesn’t matter. There is also no extra overhead, since the built-in server can use the canvas from the main application. This means, the thick server should actually be more lightweight than the thin server, when used as the built-in server, since there is no need to store the full session history!

[drawpile-dev-diary]

Recording indexes

When you play back a Drawpile recording, there is a button labeled “Build Index”. When you click it, a progress bar fills up and snapshots appear on the filmstrip. You can now use the << button to skip backwards and double click on the filmstrip to jump to a specific position in the recording.

So, what exactly does the index do?

Normally, you can only move forward in the recording. The recording is a log of all the drawing actions, the very same ones that are sent over the network to synchronize each Drawpile instance in the session. These actions are not reversible; there is no way to unpaint a brush stroke to restore what was under it. If we want to jump back to some point, we have to go back all the way to the beginning and fast-forward to the desired point.

This is where the index helps. The index contains snapshots of the canvas taken at regular intervals. When you jump to a specific point, the snapshot closest to that point (but before it) is loaded, and the remaining commands from the recording executed. This lets you quickly jump backwards in the recording and also jump all the way to the end without having to replay the whole thing.

The upcoming 2.1.8 version introduces a new optimized index file format. A problem with the old format is that the files tend to be very large. Each snapshot is independent and, for a large canvas, this adds up quickly.

However, when you compare two consecutive snapshots, you may find that they are mostly the same. Typically, there are only few areas that have changed, especially if multiple layers are used. In the new format snapshots reuse unchanged content from previous ones, leading to massive savings.

For example, using the old version, a 20 megabyte recording produced a 90 megabyte index and took about 50 seconds to build. (Note: index sizes vary greatly depending on the canvas content.) With the new format, the index was now only 20 megabytes and took just 30 seconds to build!

[drawpile-dev-diary]

Minimum zoom

Here’s a small but neat new feature. The zoom slider seen in the bottom center of the above screenshot used to have a fixed “50%” minimum value. In the upcoming 2.1.8 version, the minimum value now adjusts automatically so that it is the zoom level needed to fit the whole canvas on screen. (Exceptions: the true minimum is 1% and if the canvas is smaller than the window, the minimum will be 100%.)

[drawpile-dev-diary]

Night mode, coming finally to the Windows version! (Linux users have had this from the beginning and it will be available for Mac later.)

[drawpile-dev-diary]

Reset to image

A new feature in the upcoming 2.1.6 release. When resetting a session, you can now load an external image to reset to. If you’re saving periodically, this allows you to go back further than the built-in memory allows.

[drawpile-dev-diary]

Canvas Inspector Tool

A new feature coming soon in version 2.1.3 is the canvas inspector.

Click on the canvas to see who last modified the given area. All areas last touched by that user will be highlighted as long as you're holding down the mouse button. Currently, the last-edited-by tag is applied per tile (a 64x64 pixel chunk of canvas.) It would technically be possible to tag individual pixels, so you could tell who was responsible for each brush stroke, but this requires much more memory and processor power, so I'm not sure the trade-off is worth it. It's something that can be added in the future, though, if it's considered useful. Also worth noting is that the inspector only shows you who last touched the area on the canvas, not who drew the original work. Adding even a single pixel is enough to change the tag, so this feature is not meant as an abuse proof way of determining attribution. For that, the best way is to look at a session recording.

[drawpile-dev-diary]

More navigator enhancements

Version 2.1.3 will include a few more enhancements to the navigator.

Behind the scenes, the way the canvas is drawn has been changed. The main canvas is drawn using a Qt feature called QGraphicsScene. A QGraphicsScene can be observed by multiple QGraphicsViews simultaneously, a feature I took advantage of to render the navigator view. However, this has a couple of downsides.

First, the exact same view is rendered, including stuff like the user cursors which, on a busy canvas, end up blocking most of the view in the small navigator.

Another problem is that both views are updated in realtime. This destroys the “lazy refresh” performance optimization that defers the recomposition of canvas areas not currently visible.

A behind-the-scenes change to the code made it possible to easily draw the canvas without using QGraphicsView. Now, just the pixel content are drawn, annotations, selections, user markers and other extra decorations are omitted so you get a clear view of the content. Additionally, the navigator refresh rate is capped to 2 FPS to ease the load on the CPU.

Finally, I also added a small new feature. When the canvas is rotated, a short line segment is drawn to indicate which side is the righthand side of the viewport.

[drawpile-dev-diary]

Background layers

Version 2.1 will soon be out of beta, but there’s still time to put in one more feature.

In a previous post I wrote about the new canvas background feature. When adding this feature, I removed the concept of a background layer. Previously, the bottom-most layer in the stack was considered special: it would be always be drawn, even in solo layer view mode or in onionskin mode.

The canvas background feature removed the need for a background layer in the common use case, however there are still situations where a real background layer is needed because, well, you need a real background and not just a solid fill.

So, to support that use case, I added a new layer attribute: fixed. When a layer is marked as fixed, it is always drawn, like the old background layer. But now, you can have more than one background layer. A fixed layer can be at any position, so they can act as foreground layers as well.

Fixed layers are also automatically skipped in the flipbook and when exporting an animation, so you don’t need to manually exclude them by setting a frame range.

[drawpile-dev-diary]

Servers nearby

For a long time, Drawpile has had support for announcing and finding servers on the local network using Bonjour/Zeroconf. However, this feature has not been enabled on Windows, except in version 1.0.0.

The problem was that Bonjour is not reliably available on Windows and bundling the dnssd.dll file with the application requires a special license. (Last time I checked, by physical mail (!) to Apple.) To work around this, Drawpile now loads the library on demand, if it is available.

So, starting with version 2.1.0, this feature can be used in the Windows version again, assuming you have Bonjour installed.

[drawpile-dev-diary]

Zoom tool

Continuing the work I wrote about in the previous post, I went ahead and removed the zoom slider from the status bar. To make up for it, I implemented a proper zoom tool, like in Photoshop and Krita. With it, you can click to zoom in, right click to zoom out, and make a selection to zoom in on a region.

When the zoom tool is selected, the tool dock shows two buttons: Normal Size and Fit To Window. This provides a quick way to get an overview of the whole canvas and zoom in on a region of interest.

Even though the slider is now gone from the status bar, there are still quite a few ways to manage the zoom level:

Using the keyboard shortcuts

With a pinch gesture on a touch screen or a mac touchpad

Ctrl + mouse wheel

Ctrl + middle mouse (or stylus rocker button) drag up&down

Using the controls in the navigator

Mouse wheel on navigator view

The combo box in the status bar

[drawpile-dev-diary]

Navigator dock improvements

A small UI improvement. The navigator dock now has controls for zooming, rotating and flipping the canvas. At the same time, I removed the angle slider from the status bar, since I felt it added too much clutter and was not all that useful. The slider is now available in the navigator, and the keyboard shortcuts and ctrl+space dragging of course still work.

The flip toggle buttons I kept in the status bar, since it might not be immediately obvious when the canvas has been flipped if the navigator is not open.

The zoom slider also stayed, but I removed the zoom in & out buttons. I might add a proper zoom tool later. The zoom percentage label is now a combobox that you can edit directly or click on the arrow to quickly set the scale. A similar feature exists in the current version: right clicking on the slider brings up a popup menu with zoom options. I had no recollection that this feature existed, even though I had added it myself. That says something about how (un)discoverable it was..

I might still remove the slider too, because the new zoom combobox seems to be working pretty well on its own, and the slider is redundant with the one in the navigator dock.