Roadmap for Doom Sprint #2 (2022-10)

Development
#1

:pushpin: Overview:

  • Started: 2022-09-30T22:00:00Z
  • Ends: 2022-11-21T23:00:00Z
  • Current status: In progress (-4d behind schedule)

Introduction

My second Doom sprint began yesterday afternoon (2022-09-30T22:00:00Z). I’ve spent the day planning out this roadmap and setting up a proxmox box.

A Doom sprint means 4-8 weeks of fulltime Doom development (minus weekends). It gives me the breathing room to deal with larger, systemic issues that I couldn’t fit into the little free time I can normally afford to my OSS capers. This has only recently become possible thanks to my lovely sponsors.

The first sprint was in July, where I wrote a transactional package manager, a profile system, new CLI (which will power our future CI/CD), and laid the ground work for moving our modules out into separate projects. Sadly, there wasn’t enough time to actually merge most of it into master, so that’s what I spent the next two months doing. Slowly. With this sprint, I hope to run the final stretch and release 3.0.

There’s much more to say about it, but this is not the place to say it. I’ll write another post here later to summarize the state of this project, but for now I’ll focus on the sprint:

Goals

I list my targets below. It’s only the highlights, and I realize some points are not well explained, but I’ll add footnotes and update it as I progress through it. Feel free to ask me about them on Discord or in the comments below.

  • Week 1

    • Publish docs/{users,developers,contributors}.org.
    • Update and generalize doom make docs.
    • Update https://docs.doomemacs.org.
    • “State of Doom” Discourse post.
    • Flesh out 3.0, 3.1, and 3.2 goals in the Development Roadmap.
    • Some CLI changes bottlenecking the rest:
      • Introduce “option grouping” in defcli!, replacing &multiple and &flags.
      • Add data migration step to doom upgrade (for detecting and resolving cache/persisted data inconsistencies between Doom versions). One-way only, for now. Will be two-way later, for rollbacks.
      • Implement .doomproject files (an alternative to .doomrc).

  • Week 2

    • Use optional .doommodule files to declare module metadata and roots (and rely less on parsing paths to detect them).
    • Replace doctor.el with .doommodule checks.
    • Add new sanity checks for doom! (in case of misspelled, misplaced, or malformed entries in your doom! blocks).
    • Add syntax to doom! for pulling remote module libraries and modules.
    • Add syntax to doom! for toggling flags globally (and for negating them, per-module, e.g. -lsp).
    • Move modules out of core into separate repos:
      • doomemacs/modules (official modules)
      • doomemacs/contrib-modules (community-maintained modules)

  • Week 3

    • Add spinoff profiles API, to allow for on-the-fly profile generation (needed for the sandbox, test framework, and transactional package manager)
    • Fix M-x doom/sandbox.

  • Week 4-6

    • Replace straight.el with elpaca and adapt+backport rest of transactional package manager
    • Adopt XDG more completely (abandoning $EMACSDIR/.local).
    • Add docs/news.org (finally, a changelog!)
    • Release 3.0.0

Stretch Goals

Some goals slated for 3.1, but could be squeezed into 3.0 if time permits:

  • Improve doom sync:
    • Make doom sync see changed or repinned recipes + dependencies (no more doom sync -u except to forcibly update unpinned packages).
    • Make doom sync rebuild packages if Emacs version has changed (no need to know the magical doom build && doom sync incantation when up/downgrading Emacs).
    • Move doom build to doom sync --rebuild (and doom sync will replace doom build -r).
    • Add --aot flag to doom sync and prompt to doom install (ahead-of-time native compilation).
  • First-class literate config support (moves :config literate into Doom core; allows folks to tangle more of their Doom config and modules, along with the ability to declare what version/commit of Org (or alternative profile) to tangle it with).
  • Finish our hermetic, distributed test framework (uses your choice of ERT or Buttercup, can distribute tests over SSH and/or multiple processes) so we can have CI again.
  • Remove use-package dependency (fully move it into :config use-package, though it won’t become opt-in until 3.1 – giving some folks time to see the deprecation notices and adjust).
  • Add safe-mode profile for user-friendlier startup errors.
  • Add syntax to doom! for toggling flags globally (e.g. +lsp, +tree-sitter, and +pretty) plus negated flags (-lsp).
  • Add a “build docs” step to doom sync, so we can treat docs/*.org as source code, rather than documentation. Then doom-docs-mode and internal links can be greatly simplified.
  • Allow doom upgrade in dirty worktree (only failing if merge fails), forks, or non master branches (will rebase from remote).
  • A :app tutorial module with per-module tutorials (where appropriate).
  • A logo.

Challenges

  • Elpaca:

    • I’ve tested Elpaca a good deal and am very impressed with it. Switching to it gives us a more configurable package manager and an opportunity to cut ~2.4k LoC of advice formerly requires to shoehorn straight.el into the role. That said, I’m sure it won’t be smooth sailing. Unexpected incompatibilities and breaking, upstream changes are sure to be a significant source of delays.
    • Windows support: Elpaca requires symlinks; a burden I don’t want to impose on users, so I’ll likely implement an emulation layer myself. Doom will be a staging ground for an eventual PR (assuming upstream wants it).

  • Breaking changes (and a migration path) must be clearly communicated on Discourse before pushing them. For example:

    • Modules will no longer be included with Doom. doom! blocks will need updating to use the new syntax for pulling remote module libraries,
    • The move to elpaca+XDG will require a complete reinstall/rebuild of your packages, and in a new location.
    • If you’re using the CLI framework, changes to defcli! may warrant changes.

  • Doom v3 will be quite different from the “starter kit” the internet knows it as. It’ll be closer to a “config generator” and “elisp development tool”, and by 3.1 could be an alternative to eldev, chemacs, cask/pallet, makefiles, and alphapapa/with-emacs.sh (while its module libraries would be its “starter kit”).

    I wrote these features for my own tinkering. Generalizing it for wider user and making them discoverable will require time to feel out.

This post is a WIP. If I’ve forgotten something, feel free to poke me on Discord or comment below!

15 Likes
#2

Some good and bad news:

Bad news: the sprint is paused for a couple days while I recover from a flu (not COVID, fortunately).

Good news: a contract I had for mid November was pushed back two weeks, so that’s two more weeks for the sprint. I’ve added start and end dates to the top of the original post.

1 Like
#3

And I’m back! I’m still a bit weak, but with the fever gone and (most of) my focus is back, I’m well enough to resume my sprint. Huzzah!

12 Likes