How about having each pager program use update-alternatives to provide a possible variant of /usr/bin/pager, and then having man configured by default to use /usr/bin/pager ?
This would mean that man couldn't tell that less was being used and give it all those funky arguments with the name of the manpage, &c, but it would work better than the current scheme. Ian.