Stupid work around:
userMarker.addListener('drag', function() {
console.log("hi");
userMarker.setPosition(myLatlng);
});
This way it's selectable, so it doesn't crash, but they can't move the marker. Far from perfect, but it works.