Sketches In The Window

An installation of user submitted p5.js sketches displayed in my office window.

Origins

The original concept for Sketches in the Window began to form in autumn of 2021. At the time, I had gotten back into algorithmic art, and had ported some of my old pieces over to p5.js. I wanted a place to show off some of the stuff I had made, and originally considered using some kind of digital photo frame in my office. However, I realized that only the people stopping by my office would get to enjoy it, and only if I was actually there.

I then considered putting a monitor in one of the larger windows above my door. After some searching on Amazon, I managed to find a monitor that was the exact width of the narrow window on my door. I purchased it, as well as a Raspberry Pi, and spent about 100 hours over the course of two weeks writing the first version of the display system. I also spent a ton of time fighting with operating systems and configuration for the Raspberry Pi. Eventually, though, I got something up and running.

Somewhere in the middle of all that time, I realized I could also allow others to submit sketches to be displayed. Our New Media Interactive Development students learn p5 early in their time at RIT, and I figured they could use it as a cool creative outlet. I then set out to program this website and a submission approval system to ensure only appropriate sketches would end up in the queue.

Over the last several years, I've put quite a lot of work into the installation. This includes writing, maintaining, and upgrading the Node.js server that runs the actual display, the website itself, and the discord bot I use to interact with all of it. It has also included a major hardware upgrade where I replaced the Raspberry Pi with a Windows PC, the addition of a webcam for motion tracking, and the integration of several quality of life features. The project itself has also been an awesome example in my IGME 430 class, where we cover backend systems development using many of the tools that the installation does. Below is a description of how those various systems work together.

Technologies

Node Server

At the core of the entire installation is a Node.js server that takes on several roles. Originally the server was only responsible for hosting the submission page so that people could easily submit sketches without having to email me. However, after realizing that it would be nice to have a web-viewable version of the installation, the server expanded to also hosting and maintaining the web viewer, as well as the webpage that the main display uses. This had the added benefit of allowing form submissions to go straight into the database that the display pulls from.

Discord Bot

Speaking of submissions, I quickly realized I would need a system for vetting and approving of submissions. I like to believe in the good of humanity, but I also anticipated inappropriate submissions or ones that deviated too much from the requirements. My original plan was to have submissions get emailed to me for review, and then manually add them to the system. The developer in me figured there must be a better way, though. I had recently been working with the Discord.js npm package, and figured I might be able to integrate a discord bot to cut out the need for setting up an email server.

After a bunch of iteration and updates, the discord bot is now central to the installation's management. When submissions come in, I get pinged by the bot with all of the information, as well as a link to the original sketch. I can then view the sketch on my computer or phone and review its output. Provided it warrants addition to the installation, I can send back a command to approve the sketch (as well as set its background color, and its duration). Once I do, the Node server downloads a copy of the sketch from the p5 editor, unzips it, combines all the JavaScript files, and uses SFTP to store the code and all resources (such as images) on a file server that the installation pulls from.

After getting the discord bot up and running for submissions, I realized I could use it as a convenient interface for a lot of other functions that I had envisioned. When people began submitting sketches, I had to go through a very inconvenient process of manually editing booleans in MongoDB (the database the installation uses) to get the new sketch to show up on the installation itself. I then implemented a queue system by which I could enqueue sketches through the bot. This was a big step forward as it eliminated the need for me to directly interface with the database. However, it still had some issues. For example, even after enqueuing a sketch I had to wait for the last one to time out which could take up to 11 minutes. There had to be a better way.

Socket.IO

I realized that what I needed was a way to push updates to the display, as well as the web viewer. I had recently been covering Socket.IO in my IGME 430 class, and realized it could be used to solve the exact problems I was running into. Without too much difficulty, I was able to integrate Socket into the Node server and the client pages to allow for realtime communication between them. This allowed me to add a "forceEnqueue" command to the Discord bot which would not only enqueue a sketch to be displayed next, but it could also notify all viewers to jump ahead to the next sketch instantly.

After adding the forceEnqueue command, I began brainstorming other things I could do with the websocket connection I had set up. In the past when I would step out of my office, I would usually put a sticky note on the door describing where I had gone and when I would be back. I realized I could save the paper and instead use the giant display I had on my door to do this same thing. I added "notice" commands to the Discord bot which would let me post a message with a duration and level of importance to the display. Unlike the forceEnqueue command, this notice command only affects the display itself, not the web viewer. Since adding it, I have also come to appreciate the benefit of being able to post things to my door from my phone. This is especially convenient in situations where someone is waiting for me and I am running late.

I have also been slowly working on an idea for interactive games and sketches using Socket.IO for realtime communication. I currently have a proof of concept made where someone could scan a QR code to go to a "controller" website, and then use their smartphone as a controller for a game that would come up on the display. I think this would get some good use, as I often come out of my office to find students looking at whatever is being displayed.

Sketches in the Window is run by Austin Willoughby (Senior Lecturer, IGM). Projects submitted by you!