Collaborative editing on wiki notes

Jan 17, 2023 • Yousef Amar • 5 min read

I added something pretty cool to this website today and here are the details, with a bonus surprise at the end!

Problem

Currently, I edit the vast majority of my textual content using Obsidian by modifying markdown files directly. These are converted into what you see using 11ty.

However, I have a class of notes that could benefit from wiki-style collaboration. These are things like collections of tools (think awesome lists) or other living documents that are maintained together. I won't get into my taxonomy of documents, as I'm still developing it, but the "wiki" has always been on there. Crucially:

Non-technical folks need to be able to collaborate seamlessly
Simple to maintain; I don't want to host anything complicated or have a lot of moving parts
It needs to blend in well with my existing website
No databases; we operate on markdown files
History needs to be stored somewhere
It needs to be simple

So this precludes complicated git workflows, as well as self-hosted wikis with opinionated stacks and UI (MediaWiki, Wiki.js, etc). I decided the path of least resistance was to build my own very simple mechanism for editing wiki notes.

Authentication is currently out of scope. This is not a problem I want to solve at the moment, as it hasn't been a problem practically yet. It is currently enough for me to make a note fully public, and have the ability to roll back any vandalism. If I need more control/granularity, I do have some ideas (which might involve IndieAuth Token Auth), but for now I've deliberately left this out of my bullet points.

Updating markdown

I started off by trying to solve the most immediate problem, which is to update the source markdown files. As mentioned, when I do this, I use Obsidian, and sync with my server using Syncthing. Other people need a different way.

I thought I could kill two birds with one stone here. Currently, I run 11ty locally when I want to compile the markdown to HTML, which then syncs to the server, but I've always wanted some kind of Continuous Deployment setup on the server itself, as sometime I update notes from the Obsidian mobile app.

So I decided to use 11ty's programmatic API to make a small node.js server (managed by pm2 which I already had running on that server) that watches the source files and rebuilds the HTML files when they change. This same server also has Express listening for any PATCH requests on the same URL that the actual notes are served from. To get this to work, I tweaked my nginx configuration like this:

location /memo {
	if ($request_uri ~ ^/(.*)\.html$) {
		return 302 /$1;
	}

	if ($request_method = PATCH) {
		proxy_pass http://localhost:8102;
	}

	try_files $uri $uri.html $uri/ =404;
}

I added the if block in the middle that checks if the request method is PATCH, and in that case routes it to the server running on port 8102, otherwise it just serves the static files as usual (with .html removed).

Then the server takes these PATCH requests and overwrites the files with their content. There are a few checks though to prevent funny business:

Only files under my /memo/ path can be updated (I haven't checked what happens if you put a bunch of /../../../s in it, but it doesn't really matter because of the remaining points)
Wikis must exist in an index file under a directory (I'll explain why later) so we check that the path corresponds to a directory, and an index file exists
We check the front matter of both the old file and the new file. They must match (meaning only I can actually edit this). This is so people don't modify the access control list, the public status, the listed status, and other metadata. I may relax this in the future.
If the old file is marked private, meaning someone made a request to a guessed path, or a path that was historically public, I assume funny business, so I pretend the file doesn't exist and respond in kind. This is to prevent leaking information about the existence of private files.
Finally, I replace < and > with their HTML entities to prevent people from writing arbitrary HTML. I may relax this in the future, but for now it's pure markdown to prevent XSS attacks and other funny business.

And thus, if one of my notes is editable, you can PATCH it at the same URL!

Storing history

The most obvious answer here is git. In fact, you might be asking yourself why I don't already use git to keep track of my own edit history on top of my entire knowledge base. I won't get too deep into it, but I actually believe having an edit history is an anti-pattern for note-keeping -- a discussion for another day. Similarly, it seems kind of overkill to create a new GitHub repo for every publicly-editable, single-file note.

My first idea was to use GitHub Gist as some kind of GitHub-account-powered backend for these notes, but there were a lot of problems with that idea. Then I thought about creating the infrastructure for a mini-gist that makes things a bit more accessible to non-technical folks, as I certainly don't want to be telling them to install a git client and do this and that.

But then I snapped out of it and realised that git is overkill. I only need a linear history of past files, and this is where the directory and index file structure comes into play. Whenever a new revision of a note is PATCH'd over, I simply rename the old file to rev-[time].md where [time] is a Unix timestamp of last modification time of the old file, and save the new file to index.md. This works perfectly, and I can use file modification times to keep track of history.

I also unlist these copies so they don't clog up the tree home page visualisations.

UI

All public wikis now have an Edit and History link on the bottom:

I decided not to explicitly mark a note as wiki: true or something like that for now. Instead, I added an access control list to the front matter called acl. At the moment, we only check the existence of this to decide if a note should be editable. These only contain one value at the moment, a wildcard indicating that the note is publicly editable by anyone.

If a note is a wiki, I also pass along the source markdown of the page encoded as base64 (to avoid any unescaped markup issues and save a bit of space) in a script tag. When the user clickes Edit, this source markdown is loaded into a text area that they can edit. When they're done, they can click save on the top right and their changes will be uploaded.

But wait, what's that on the top there?

Real-time collaboration

Thanks to the excellent Yjs, and y-textarea, I was able to add real-time collaboration to these textareas, over WebRTC (Yjs has multiple providers; I used y-webrtc). This means that not only can you edit these notes wiki-style, but can also collaborate with someone on them at the same time, like a Google Doc or Etherpad. This is incredibly useful for brainstorming in a call with people, or keeping meeting minutes, etc.

When you first hit edit, you're assigned a random animal name (I generated the list with ChatGPT by asked it for "nice" animals so it doesn't call someone a pig) and then you can change it while editing. That last part (editing name) wasn't part of the lib, so I contributed that functionality to the repo.

I also found a small vulnerability, where people could name themselves arbitrary HTML, as the lib didn't escape that, meaning they could cover your screen with a div and run an XSS payload on mouse move or similar. I raised an issue about this, and the maintainer is looking at it (the fix is quite small in theory), but it's also kind of fun, so I'm wondering if I should leave it in!

Demo

If you'd like to try it, I put up a test wiki note here. As of writing this, I'm still hacking away at it, so please bear that in mind! For example, I haven't sorted out displaying history yet, and I made the save button wait for the upload to finish, but not for the 11ty recompile, so you end up having to refresh the page once or twice to actually see your changes. All things that are soon to be fixed however!