Alright, I took a nap and cleared my head. I noticed that on my one machine (which does not run systemd) the permissions on the joystick device were 0644, but on my other machine which does run systemd the permissions were 0600 and when I would change the permissions to 0644 sdl would correctly detect my joystick.
I spent some time on the sytemd irc channel, it turns out startx may startx on a different tty than where you logged in for that session, causing the systemd session to break. I opened a bug against the xinit package. #743015 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=743015
