Hi, I wouldn't be able to sleep if I had to make sure to keep 5M zones consistent with AXFR/IXFR.
Having a database with a tranaction-ids and where I can monitor "slave seconds behind master" on the slaves makes my job doable. I think I have not yet understood your concern about scaling horizontally. Lets say your database size is 40GB. You buy 64GB RAM Servers with 16 cores, install Linux, MariaDB, PowerDNS. PowerDNS connects locally over unix domain socket. Set MariaDB Page Cache to 40GB. Upon startup of MariaDB a script is run to select everything, so that no query from pdns triggers a physical IO. This setup will easily do 40k req/s. If you have 12 locations with 4 servers each, you have potentially ~2M qps. Cheers Thomas On 11/28/20 11:01 AM, Jianan Wang via Pdns-users wrote:
Hi Chris, Thanks for your detailed explanation on the scaling plan for power dns deployment! It is very helpful from my current standing point, where we use mostly bind based solutions and replication is more through axfr/ixfr. Since it is pretty inspiring, I would like to follow up with some question if you would not mind: (1) Let's say you have master location A and B serving as writer, so it means location C, D's local regional zone will also be written into location A & B's dns instance? Sounds like the trade-off is between simplicity of management compared to write availability for non-master locations. Is the location you mentioned more like within a country or world-wide? (2) In your deployment here, it seems like your authoritative servers are the ones being replicated, and you will have local resolvers pointing to them behind the dnslist pairs you mentioned, right? (3) I bet managing 5 million zones will be a pretty complicated question, so just curious are you guys more a UI based management or git based management in this case? (4) Is there any good practice to manage the mysql instance ip? Are you using lvs, vip style or ip anycast solution to point to master so it would be free of changes? Normally we use domain names so that changing ip of mysql instance will not impact all the services. However, since powerdns is relying on this database anyways, meaning we could only use ip or dns served by other instances. Thanks for your help! Jianan. On Tue, Nov 17, 2020 at 10:11 PM Chris <m...@gbe0.com <mailto:m...@gbe0.com>> wrote: Hi, The way I scale it out (this is scaled out to 12 locations currently): We picked 2 of our large locations to act as master locations. These 2 different locations have an active/standby mariadb server cluster using corosync/drbd (so only 1 server in each location is master at a time). The two locations are setup as master/master (mysql replication). These are the servers for which all write changes are applied to. I only put them in two locations as this is much easier to manage and cuts down on replication problems. Each of the 12 locations we are in then has at least a pair of servers for PowerDNS. Each of these servers has their own individual copy of the MySQL DB; their copy is read only and they replicate it from the master servers. We only use MySQL to replicate the databases, we don't handle that at the DNS level (eg. axfr/ixfr). This way if the masters are taken offline (since they are only in two locations) at least DNS continues to work from the read only copy without changes. It would have to be a very bad day for all 4 master servers to do be dead anyway. Each of the 12 locations also has at least a pair of dnsdist frontends to accept queries from clients and direct it to the various PowerDNS backends. There is a bit more to it (delayed slaves for DR purposes, different local/remote pools for dnsdist etc.) of course, but thats the basics. We have over 5 million zones and over 40 million records (plus additional things like DNSSEC keys, metadata entries and so on) and so far this has scaled nicely for us. ------------------------------------------------------------------------ *From:* Pdns-users <pdns-users-boun...@mailman.powerdns.com <mailto:pdns-users-boun...@mailman.powerdns.com>> on behalf of Jianan Wang via Pdns-users <pdns-users@mailman.powerdns.com <mailto:pdns-users@mailman.powerdns.com>> *Sent:* Wednesday, 18 November 2020 00:05 *To:* pdns-users@mailman.powerdns.com <mailto:pdns-users@mailman.powerdns.com> <pdns-users@mailman.powerdns.com <mailto:pdns-users@mailman.powerdns.com>> *Subject:* Re: [Pdns-users] Question about Powerdns remote database backend best practices Hi Thomas, Thanks for your response on this! IIUC, what you recommend is still having MySQL and PowerDNS created on one instance and scale the instance setup horizontally? If that’s the case, will the MySQL on the single node be bounded by its resources locally to scale when we have a lot of records? Thanks. Jianan.-- Jianan Wang -- Jianan Wang _______________________________________________ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
_______________________________________________ Pdns-users mailing list Pdns-users@mailman.powerdns.com https://mailman.powerdns.com/mailman/listinfo/pdns-users
