Interactive whiteboard apps seem to be a popular use-case vector for the HTML5 canvas. I’m a fan of the Konva JavaScript 2d canvas library so I wondered what it would take to synchronise canvases.
Design thoughts
My end goal is a meeting or classroom-style whiteboard where there could be multiple users sharing and co-operating on a Konva diagram. It could have a presentation mode where only one user could draw and all others were restricted to viewing, or a co-drawing mode where 2 or more users could draw at the same time.
The architecture is for each user to have a browser with a web page hosting a Konva instance. There is no natural sharing mechanism built-in to Konva, which is fine with me because I want Konva to do the one job of being a canvas lib, which it does very well. So I would need to think about the communications process.
But what would be communicated? Starting out, I knew the most simple design revolved around listening to the Konva objects API stream somehow. Then, like an MI5 wire tap, I could observe all the API calls going into a Konva object and broadcast them out from the client where the API calls happened to the listening clients where I would replay those API calls at each client.
Looking in more detail at how Konva gets things done:
- Declaring a Stage and linking that to a page element that will host the canvas. Konva uses a Stage & shape metaphor – you define shapes and place then on the Stage.
- Defining layers and adding to the stage. Layers refine the shape-stage metaphor by providing layers over the stage onto which shapes are placed, giving versatility and performance gains.
- Making shapes – rectangles, paths, lines, ellipses, custom shapes, etc, and adding to a layer.
Once you have shapes on a stage they can be:
- Scaled
- Positioned,
- Rotated
- Moved ‘up’ and ‘down’ the visual shape stack (z-index)
- Moved between parent containers (groups or layers).
- Transformed individually or as a collection using a visual Transformer
- Have listeners for events
- Yada, yada, and much more yada!
There are hundreds of Konva API calls to have to package and share between clients. I wonder how we’ll capture them.
Maybe the JavaScript proxy has an answer?
The JS proxy is a thing of beauty. It sits in front of an object and lets you catch calls into that object just before they get to the object itself. A proxy object can do something as simple as add input validation, or as complex as adding thousands of lines of code to run when a single property of the target object is changed.
We used to have to do a lot of hacky stuff to make proxies work in the old days, but now JavaScript has the proxy pattern backed right in, so we can proxy library code – like Konva!
Putting a bit more detail down, this example, from the MDN web site page explaining proxies, shows a generic approach to capturing all ‘get’ requests. At line 1 we see a simple JS object with two message properties being defined, and at line 6 the proxy object is defined. It has one method – get. Once attached in front of the target object, this get method will receive all get requests for all properties sent to the target. We can see at line 8 some logic testing for a get call for a specific property of target and when a match is found, the proxy returns a different value. Line 15 shows how the proxy is assigned to the target, and at the end we see the results – instead of ‘Hello’ & ‘everyone’ we see ‘Hello’ and ‘world’ – magic!
const target = {
message1: "hello",
message2: "everyone"
};
const handler3 = {
get: function (target, prop, receiver) {
if (prop === "message2") {
return "world";
}
return Reflect.get(...arguments);
},
};
const proxy3 = new Proxy(target, handler3);
console.log(proxy3.message1); // hello
console.log(proxy3.message2); // world
A quick note – in the example above, MDN chose to create a new object ‘Proxy3’ to combine the original target object and the new handler. It’s important to know that we can also re-assign the combined set as the original target. Why this is important will become clear, and it looks like this.
target = new Proxy(target, handler3);
console.log(target.message1); // hello
console.log(target.message2); // world
So you got the Proxy – how to use it ?
Good question. In a perfect world there would be one Konva phone number for the proxy wire-tap. We could listen to that and capture all the API calls direct at our Konva object on any specific client, and interpret them into some simple message structure that we beam out to all the listening clients.
However, looking inside the Konva source code (it’s all there on GitHub) it is apparent that for efficiency, Konva uses the factory pattern approach to handle much of what it does. Without forking the Konva source and creating our own Frankenstein’s monster, it’s going to be hard to achieve a single wire tap situation. So it seems there’s going to be some trade-off for our solution where we’ll have to wrap some of the API calls into Konva with our own.
However, there is light in the tunnel in the form of the Konva.Shape attrs. This property array is a list of many of the properties assigned to a shape. Internally it’s an array of simple two-property JS objects each holding the name & value of a Shapes property. In code we don’t access it directly, we use Shape.getAttr(), getAttrs(), setAttr() and setAttrs() which pretty much explain themselves. We can also store arbitrary attrs here, which might be of some use later.
Back on the proxy conversation, if only we could make a proxy on the Shape.attrs list, it might go a long way to being that single wire-tap we want to keep things elegant. Good news – we can. Because of the object-like nature of JavaScript Arrays, we can set a proxy on an array. Game on !
Summary so far
This is shaping up to be a long post so I’ll do an interim summary. I’m trying to share the creation and changes of a Konva canvas across browsers to make a shared whiteboard app. I’ve recognised that there’s no one-stop means to tap into the API calls going into the Konva object (by the way I did try proxying the Konva object but the internals of Konva meant it didn’t yield a viable solution), so I’ve settled on having to somehow handle what I can’t proxy in my solution (like stage create, adding shape to a parent container, and whatever else comes up later – I’m hoping it’s a manageable list!).
And I’m hoping that proxying the Shape.attrs array of properties will be viable and so reduce my work hugely. Again, I have to accept that maybe not all properties of a Shape are fully visible or handled in its attrs – I’ll have to ride that out as the code develops.
I haven’t mentioned Konva events yet – like when a user clicks to select an object. In this phase of my R&D I’m thinking that events are going to have to be handled outside of any proxy solution, but also that events are far more solution-specific and I’ll aim to produce a standard event handling process that works with my property-transporting mechanism.
The Code
I’m going to have a good chunk of code to make all this work so I’m going to wrap it all up in a single JS object. I’ll do something like hand the stage to that object, and provide wrapper methods in that object for things like layer and shape creation, changing shape parent, and all the communications. My aim is to leave the JS that creates the Konva diagram as untouched as possible and only require wrappers where I need them. This object will also be constructed so that any instance can be both a sender and receiver of diagram data. We’ll likely have most of the methods be about capturing Konva API calls, and a couple of send and receive message methods.
At this stage, for simplicity, I’m going to use two canvases on a single browser page. Later, I’ll modify the send & receive code to use a peer-to-peer communication process via peerjs. The gif below shows what the code produces at the moment – the top canvas is sending and the bottom is receiving.
The code for this blog is all here at CodePen – its all commented but I’m going to explain the juicy parts so click the ‘Edit on CodePen’ link and follow along.
I’ve named my object syncHandler. It takes as initialising parameters a Konva.Stage object, and a prefix for the objects trace output. There’s a simple trace function that throws the messages out onto a visible div below the second canvas and this prefix is useful to know the source of the traced messages.
The process of creating and assigning the syncHandler is as shown below. The line that assigns stage1 is a standard Konva.Stage create command – it is passed the id of the html element that will host the stage, and the width and height of the canvas (in this case derived from the html element size). I chose not to wrap this in a syncHandler method since every client will have its own stage to construct and I decide that there might be a presenter HTML page and a different viewer html page – this way I don’t lock those pages in to having to have identical element id’s or construction.
// Making the stage is standard Konva API code.
let stage1 = new Konva.Stage({container: "container1", width: $('#container1').width(), height: $('#container1').height()});
// And now we create the handler object.
let handler1 = new syncHandler(stage1, 'sh1');
Back in the syncHandler, we have a followers array, and a shapes object. Followers will be a list of listening syncHandlers. Every connected whiteboard app user will be running a syncHandler and they will be listed here. When we switch to peer-to-peer coms later, we’ll need to know their connection identifier and possibly other information so we’ll stash that in this list as needed.
I’ll cover the shapes object in a moment.
As we communicate changes to any Konva.Shape between clients, we’ll need to indicate exactly which shape is being changed. For that purpose we will be rigorous about applying an id property to each shape that we create. Given that we could have multiple clients collaborating on the whiteboard, we need a way to ensure that the id’s generated are unique. To that end, the getGuid() function is provide in the syncHandler.
Aside re GUID’s: A proper GUID is a Universally unique id – meaning unique in The Universe. There’s no way to generate real GUIDs in JavaScript, because they depend on properties of the local computer that browsers do not expose. There are various solutions to this, I’m using this one.
Anyway, the point is that a unique id will be assigned to each Konva.Shape, regardless of which client creates it. As we create each Konva.Shape and communicate the creation across the listening clients, we will send that id. Then later, when we need to send a message about, say, a change to an objects fill color, we can give the object ID and the listeners will find the shape in their list via it’s id.
And that’s what the shapes object is about. When we add a shape we will use the shape id as the property name – this makes it quick to find and check if it is known. Effectively it gives us a named list.
Making the proxy
The makeProxy function does the work of making the proxy on the passed-in array. We use this to observe changes to each Shape.attrs array, and use a generic set handler to invoke the changeAttr() function which packages up the change and sends it out to the listening clients.
Making Konva shapes
The makeShape() function is a wrapper for the call to the Konva.Shape constructor. The standard syntax for creating a Konva Rect is shown below, as is the process for adding to a layer. Also shown below are the same processes but using the syncHandler equivalents. The wrappers add a small amount of change but this is a manageable overhead.
// Make a rect - normal process
let rect1 = new Konva.Rect({x: 20, y: 10, width: 100, height: 80});
// Add rect1 to layer1 - normal process
layer1.add(rect1);
// Make a rect - via the syncHandler wrapper.
let rect1 = handler1.makeShape("Rect", {x: 20, y: 10, width: 100, height: 80});
// Add rect1 to layer1 via syncHandler
handler1.changeParent(rect1, layer1);
What the syncHandler’s makeShape() function does is create the shape using the standard Konva commands, give the shape a unique id, make the call to add the proxy to the shape’s attrs array, stash the shape in the shapes list and send the message about the shape’s creation to the listening clients.
The changeParent() function follows a similar approach, executing the standard Konva commands on the local Konva object and notifying the listening clients of the change.
Changing shape properties
The changeAttr() function is the last one related to changes in the Konva diagram. This one is invoked via any change to the Shape.attrs proxy. it’s main job is to communicate the attr change to the listening clients, but it includes a small amount of code to limit sending unnecessary messages, since message transport is a costly process and any message we can avoid sending is a performance boost. Specifically, the code seeks to limit changes to numeric attribute values where the change is small, for example when rotation changes from 45 to 45.00001 degrees. Some tuning may need to be done here – we shall have to see how it pans out.
Once a shape’s attr change is recognised as needing to be communicated, it is sent out to the listening clients.
Sending the message
The sendMessage() function handles sending whatever message it is given. Each of the ‘makeShape’, ‘changeParent’, and ‘changeAttr’ functions sends a different message but they have in common that the message is a plain JS object and it contains a type attribute to indicate the meaning of the message.
// Message format for makeShape message
let msg = {type: 'makeShape', name: typeName, attrs: attrs };
// Message format for changeParent message
let msg = {type: 'changeParent', id: shapeObj.id(), parentId: newParentObj.id(), parentType: newParentObj.getType()};
// Message format for changeAttr message
let msg = {type: 'changeAttr', id: target["id"], name: prop, value: value };
These objects are passed into the sendMessage() function where they are converted to a JSON string and sent on their way. In this demo we are using two Konva stages, each with their own attendant syncHandler, in one page. In a later article I will add cross browser communication. For the time being the listening client (found in the followers[] array) is a simple pointer to the second stage’s syncHandler.
Receiving the message
Finally, the syncHandler function processMessage() is used to process the messages sent via sendMessage(). After parsing the JSON back into a JS object, we use a switch statement to determine which type of change we are processing, with a different block of code for each case.
For the makeShape message, we have all that we need in the message to invoke the syncHandler.makeShape() function – yes the same one we used in the setup of the main Konva Stage! This can be done because the receiving syncHandler instance is linked to it’s own stage (stage2 in the demo code) meaning that is where it’s shapes appear, and the listening syncHandler has no followers of its own. There is a potential issue for us to solve later around how we can continue to re-use the syncHandler.makeShape() function in a true bi-directional situation.
The changeParent message needs to know if the new parent is the stage or something else (layer or group). For the case of adding the shape to the stage we fork one way, and for the second case we have all the info we need in the message to find the parent and child shapes (lookups via the shapes object) before applying the change on the Komva stage.
Finally for the changeAttr message, we get the shape id, attr name and value, simply applying them after looking up the shape in the shapes object. Of the hundreds of attrs a shape can have, we use this one piece of code to apply any and all of them.
Remainder of the code
The remainder of the demo code is all about setting up a stage with new shapes, then applying some arbitrary changes to them to confirm the attrs-sharing message works.
Interactive change test
There’s some code in place on the top canvas to enable a transformer – click & drag to make a selection rect and any shape that the selection rect touches will be added to a transformer.
Using the transformer you can scale and rotate the three objects in the top stage and watch the object(s) follow along in the canvas below, and see the message list, courtesy of the two syncHandlers.
Summary
We’ve created the fundamentals of a whiteboard solution – ok there are no neat menu’s or UI for creating and deleting objects etc, but we’ve established what the code for those activities will invoke and we’ve created a way to synchronise those changes to another canvas.
Next steps will be to modify the syncHandler code to cover bidirectional messaging, and to modify the messaging process to use a peer-to-peer comms library, Peerjs.
Thanks for reading.
VW July 2021
Coding for the long view 