The return value specified in "8.1.2 associated Legendre polynomials" of ISO/IEC JTC 1/SC 22/WG 21 N3060 (which is identical to the expression in the doxygen comment of the patched function) is well- defined for m>l: it is always zero because $ P_l(x) $ is a polynomial of degree l.
The standard does not enforce an exception in this case because none of the requirements in 8.1 (5) on page 11 of ISO/IEC JTC 1/SC 22/WG 21 N3060 are met. Note: the implementation of st::assoc_legendre in Visual Studio 2017 (tested with Visual Studio 15.9.7) silently returns zero.
Index: libstdc++-v3/include/tr1/legendre_function.tcc =================================================================== --- libstdc++-v3/include/tr1/legendre_function.tcc (revision 269352) +++ libstdc++-v3/include/tr1/legendre_function.tcc (working copy) @@ -67,13 +67,13 @@ /** * @brief Return the Legendre polynomial by recursion on degree * @f$ l @f$. - * + * * The Legendre function of @f$ l @f$ and @f$ x @f$, * @f$ P_l(x) @f$, is defined by: * @f[ * P_l(x) = \frac{1}{2^l l!}\frac{d^l}{dx^l}(x^2 - 1)^{l} * @f] - * + * * @param l The degree of the Legendre polynomial. @f$l >= 0@f$. * @param x The argument of the Legendre polynomial. @f$|x| <= 1@f$. */ @@ -120,17 +120,17 @@ /** * @brief Return the associated Legendre function by recursion * on @f$ l @f$. - * + * * The associated Legendre function is derived from the Legendre function * @f$ P_l(x) @f$ by the Rodrigues formula: * @f[ * P_l^m(x) = (1 - x^2)^{m/2}\frac{d^m}{dx^m}P_l(x) * @f] - * + * * @param l The degree of the associated Legendre function. * @f$ l >= 0 @f$. * @param m The order of the associated Legendre function. - * @f$ m <= l @f$. + * @f$ m >= 0 @f$. * @param x The argument of the associated Legendre function. * @f$ |x| <= 1 @f$. * @param phase The phase of the associated Legendre function. @@ -146,8 +146,7 @@ std::__throw_domain_error(__N("Argument out of range" " in __assoc_legendre_p.")); else if (__m > __l) - std::__throw_domain_error(__N("Degree out of range" - " in __assoc_legendre_p.")); + return _Tp(0); else if (__isnan(__x)) return std::numeric_limits<_Tp>::quiet_NaN(); else if (__m == 0) @@ -192,7 +191,7 @@ /** * @brief Return the spherical associated Legendre function. - * + * * The spherical associated Legendre function of @f$ l @f$, @f$ m @f$, * and @f$ \theta @f$ is defined as @f$ Y_l^m(\theta,0) @f$ where * @f[ @@ -202,7 +201,7 @@ * @f] * is the spherical harmonic function and @f$ P_l^m(x) @f$ is the * associated Legendre function. - * + * * This function differs from the associated Legendre function by * argument (@f$x = \cos(\theta)@f$) and by a normalization factor * but this factor is rather large for large @f$ l @f$ and @f$ m @f$ @@ -210,7 +209,7 @@ * and @f$ m @f$. * @note Unlike the case for __assoc_legendre_p the Condon-Shortley * phase factor @f$ (-1)^m @f$ is present here. - * + * * @param l The degree of the spherical associated Legendre function. * @f$ l >= 0 @f$. * @param m The order of the spherical associated Legendre function.