https://sources.debian.org/src/autopkgtest/5.49/doc/README.package-tests.rst/#L569 pandas has network-using tests and they're passing.
Hence, the Forbidden in this particular test is probably coming from the data source, not Debian. Hence, substituting a different online data source (if a suitable one can be found) should work.