XKB Incompatability

[CodedBinary]

EDIT: Totally bad diagnosis. Error is that RedirectKey doesn't work.

The new XKB compatability patches have been really nice. However, the functionality does not match up completely.

I have a somewhat peculiar setup. Basically, I've rebound the following, and get the following instead:

Key combo	Xorg Outcome	sway Outcome
Caps + a	-	nothing
Caps + s	+	nothing
Caps + d	backspace	-
Caps + f	enter	+
Caps + g	/	enter
Caps + hjkl	left down up right	left down up right
Caps + qwe..	123...	123...
Caps + Shift + qwe...	!@#...	!@#...
Caps + x	`	nothing
Caps + c	comma	*
Caps + v	period	/
Caps + Alt + q	f1	nothing
Caps + Super + h	home	nothing

Heres my setup:
my .xkeymap: https://pastebin.com/wph6WY1k

First, CAPS (the key, not the sym) is the key i use for accessing the extra layers. This is then bound to NumLock to avoid weirdness when programs look for the caps state, and to stop the light on my keyboard blinking. The rest of the config looks at the state of NumLock, not Lock, for whether they should enter my extra layers (stored in xkb_types, of course). When a key is pressed, it is meant to redirect the keystroke to a different key (not a sym) and clear the relevant modifiers. This part is done in xkb_symbols, next to each key.

One reason this seems so convoluted is I need things like caps+alt+j (caps+j is down) to look like alt+down, with no other modifiers. Similarly, I need shift+caps+x to give me ~ (caps+x gives `) with no modifiers.

Here is a quick rundown of my types:

• CAPS_Movement is for my hjkl, which has super+caps+h for home, super+caps+l for end, super+caps+j for pgdown, super+caps+k for pgup. Note the 5th level hasn't been used yet.
• CAPS_NumSuper is for instances where I want precisely Num and Super to change the level I'm on. I don't think I need the super functionality at the moment though. This is what a lot of my simple caps+thing binds use.
• CAPS_NumSuperShift is for when I want to use Super and Shift to change the level, too - for instance, i dont need to shift backspace, so here I can use shift to make it delete. There are a few media keys that I will try to fix soon that also use this. Similarly, I don't need super functionality at the moment, but I plan to use it for the media keys.
• CAPS_Num should really be called CAPS_NumAlt. Only alt and numlock change the layer I'm on. This is useful for getting function keys ontop of the qwerty row with caps+alt, and number keys on qwerty row with caps.

In vim, it doesn't seem to scroll down more than once when I hold down j anymore, which is weird. Similarly, if I hold caps+j (rebound to down arrow), it doesn't work. But if I hold down arrow, it does work. In X, my keybindings are indistinguishable from the buttons they simulate. In xev, their outputs are (were?) identical.

[tarmack]

Inspired by your issue I dabbled in xkb key maps and ran into limitations involving RedirectKey actions. I found this old xkbcommon issue stating that it is not supported. I think that most (all?) of the differences you report are actually solvable by putting the correct symbols with your key definitions (your redirects and symbols don't match up). You still won't be able to clear the modifier though.

It would be really nice if we could get the RedirectKey functionality, or at least a way to clear modifiers since that seems to be the most regularly used part of that action, however I think it is unlikely to be deemed important enough to ever be implemented (which I understand).

If you really need the RedirectKey functionality to make your keymap work, you may want to look into programmable keyboards using QMK/TMK opensource firmwares.

[CodedBinary]

Oh, I hadn't noticed I left those symbols in. My bad. I guess that gets me a fragment of the functionality. I do plan on buying a programmable keyboard, but covid put it out of stock. That xkbcommon issue makes some sense. Am I right in leaving this open as a sway issue?