Hello again,

just talking to myself here :) . After making sure that there was no issue with the backend resolvers, I noticed a lot of TCP resets from the testing subnet towards the DoH endpoint, so I went ahead and increased setMaxTCPClientThreads from 128 to 4096, I also increased file descriptors to 2^18 and finally upgraded to dnsdist 1.6.0 . Everything is OK now, I can easily reach 70k QpS (probably more)

cheers,
Yannis

On 7/25/21 7:41 PM, Dez C via Pdns-users wrote:
Hi,

I'm also controlling the backend servers and no such rate limit exists. As I mentioned, each backend server can easily handle 60k QpS and the numbers of queries that should be reaching the backend servers from the performance testing, should not be more than 100-200 QpS per backend server.

The reason I sent my dnsdist configuration is so that people -more experienced than me- could spot obvious omissions or errors.

cheers

On 7/24/21 11:41 AM, Winfried Angele via Pdns-users wrote:
Maybe there is some kind of ratelimt in the backend. Keep in mind that from the backend point of view all queries come from the same source

Winfried

Am 24. Juli 2021 10:28:49 MESZ schrieb Winfried Angele via Pdns-users <pdns-users@mailman.powerdns.com>:

    Right. Therefore I would recommend rather to troubleshoot the backend

    Winfried


    Am 23. Juli 2021 20:56:55 MESZ schrieb Dez C via Pdns-users
    <pdns-users@mailman.powerdns.com>:

        Hi,

        On 7/23/21 4:14 PM, Winfried Angele via Pdns-users wrote:
        Does this only happen with DoH frontends? Did you try with
        UDP frontends as well? Sounds like a bottleneck on your
        backends imo.

        I'm only using dnsdist for DoH so I haven't tried with
        UDP/53. I doubt it'll make a difference because the problem
        manifests itself when dnsdist is trying to query the backends
        (eventhough most replies already exist in dnsdist's cache)

        cheers


        Winfried


        Am 23. Juli 2021 13:32:39 MESZ schrieb Yannis via Pdns-users
        <pdns-users@mailman.powerdns.com>:

            hello,

            We're using dnsdist (1.5.1 on Ubuntu 20.04, 16 cores, 32GB RAM) as 
a DoH
            proxy/LB with normal DNS/53 resolvers as backend. This is a test
            installation and we're trying to figure out the performance. It can
            barely handle 1.5k QpS, which I consider pretty low (each backend
            resolver can easily handle >60k QpS). It seems that each time the
            queries rate is higher than ~1.5k, all backend servers are marked 
"DOWN"
            until the rate goes below 1k. I understand that dnsdist marks the
            servers down because it's not receiving a response on its 
healthcheck
            query and I wonder why.

            Should I increase "checkTimeout" and "checkInterval"? Should I use a
            large number for "sockets"? Am I missing other tuning options or 
maybe
            something more important?

            Here's the relevant config (addresses, etc changed)

            setLocal('0.0.0.0:5300')
            addLocal('[::1]:5300')
            controlSocket('local_public_address:xxxx')
            setKey("XXX")
            setConsoleACL('x.x.x.x/24')
            NotRule(MaxQPSRule(50000))
            setMaxUDPOutstanding(65535)
            setMaxTCPClientThreads(128)
            setMaxTCPQueuedConnections(10000)
            setMaxTCPConnectionDuration(600)
            PrimaryCache = newPacketCache(30000000, { keepStaleData=true,
            maxTTL=86400, minTTL=0, numberOfShards=8, maxNegativeTTL=600, 
staleTTL=60 })
            getPool(""):setCache(PrimaryCache)
            addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('10.2.3.4', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            addDOHLocal('2001:DB8::443', 'cert.pem', 'key.key', "/dns-query", {
            reusePort=true, minTLSVersion='tls1.2' })
            newServer({address="2001:DB8::62", qps=10000})
            newServer({address="2001:DB8::61", qps=10000})
            newServer({address="2001:DB8::60", qps=10000})
            newServer({address="2001:DB8::59", qps=10000})
            newServer({address="2001:DB8::58", qps=10000})
            newServer({address="2001:DB8::57", qps=10000})
            newServer({address="2001:DB8::56", qps=10000})
            newServer({address="2001:DB8::55", qps=10000})
            newServer({address="2001:DB8::48", qps=10000})
            newServer({address="2001:DB8::47", qps=10000})
            newServer({address="10.10.10.62", qps=10000})
            newServer({address="10.10.10.61", qps=10000})
            newServer({address="10.10.10.60", qps=10000})
            newServer({address="10.10.10.59", qps=10000})
            newServer({address="10.10.10.58", qps=10000})
            newServer({address="10.10.10.57", qps=10000})
            newServer({address="10.10.10.56", qps=10000})
            newServer({address="10.10.10.55", qps=10000})
            newServer({address="10.10.10.48", qps=10000})
            newServer({address="10.10.10.47", qps=10000})
            setServerPolicy(roundrobin)

            thanks in advance, I'd appreciate any input :)
            
------------------------------------------------------------------------
            Pdns-users mailing list
            Pdns-users@mailman.powerdns.com
            https://mailman.powerdns.com/mailman/listinfo/pdns-users  
<https://mailman.powerdns.com/mailman/listinfo/pdns-users>


        _______________________________________________
        Pdns-users mailing list
        Pdns-users@mailman.powerdns.com
        https://mailman.powerdns.com/mailman/listinfo/pdns-users


--
Diese Nachricht wurde von meinem Android-Gerät mit K-9 Mail gesendet.

_______________________________________________
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users


_______________________________________________
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

_______________________________________________
Pdns-users mailing list
Pdns-users@mailman.powerdns.com
https://mailman.powerdns.com/mailman/listinfo/pdns-users

Reply via email to