Hello Kenneth Having to "restart" the master on a regular basis! That's quite odd.
Is this a good scenario ? >>>> If your MySQL DB >>>> is not dedicated to PDNS, then you will have to balance the needs >>>> of all of the applications using the DB. It sounds like you may >>>> be running some pretty poorly written ones. Good luck. PDNS has it's own dedicated Mysql server and for the future i would appreciate if we stick to the subject and not talk nonsense about how our running applications are written. Cheers, On Jan 29, 2008 10:27 PM, Kenneth Marshall <[EMAIL PROTECTED]> wrote: > Stephen, > > You are completely correct. It is a bandaid (needed) until the > patched version is available. I am in the same situation currently > with 2.9.20 and the TCP connection handle problem and have to > restart my master on a regular basis. Monitoring is definitely > the way to go for issues such as memory leaks. > > Ken > > On Tue, Jan 29, 2008 at 12:18:54PM -0800, Stephen Manchester wrote: > > Increasing the wait_timeout on the mysql server will help, but it does > not > > actually solve the memory leak, as dead connections do happen for a > number > > of reasons (server failure, network failure, stateful firewalls, etc.) > > > > Until a patch can be applied though, increasing the wait_timeout value > on > > the MySQL server to something very high (say 1 day or more) is the only > > thing you can do. From what I've read, it looks like the default for > MySQL > > 4.x was 30s. The default for 5.x is 28800 seconds, but if there was > > already a my.cnf file laying around (which there often is a 4.x my.cnffile > > by default on many distros) then you might inherit this old value which > > will cause the problem to happen very often. Again, this is only a work > > around, not a true fix. You should still monitor pdns virtual memory > usage > > from time to time to make sure it isn't getting crazy. > > > > ----- Original Message ----- From: "Kenneth Marshall" <[EMAIL PROTECTED]> > > To: "Stephen Manchester" <[EMAIL PROTECTED]> > > Cc: <Pdns-users@mailman.powerdns.com> > > Sent: Tuesday, January 29, 2008 11:56 AM > > Subject: Re: [Pdns-users] Backend error: Failed to > > executemysql_query,perhaps connection died? Err=1: MySQL server has > > goneaway > > > > > >> Again, this problem, for which a patch already exists, is triggered > >> by the DB connection being terminated abnormally by the backend > >> server, in this case via a connection time_limit. I do not use > >> MySQL as a PDNS backend, but the PostgreSQL backend does correctly > >> reconnect to the database should the connection die. I assume that > >> the PDNS will reconnect to the MySQL server when the connection > >> dies. It seems that increasing the timeout value in MySQL is that > >> best measure to ensure more robust service from PDNS. > >> > >> Cheers, > >> Ken > >> > >> > >> On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote: > >>> There is currently a memory leak in 2.9.21 from gmysql backend threads > >>> dying in this fashion. The threads themselves are not being cleaned > up. > >>> this is only noticeable on very light loads where the backends do not > get > >>> enough traffic to keep the mysql connection open. You will notice it > by > >>> abnormally high virtual memory usage after pdns has been running for a > >>> time. Each backend that dies leaves 10MB of virtual memory allocated > >>> (the > >>> thread stack?) and this will not get cleaned up until pdns is > restarted. > >>> > >>> Hopefully they will implement mysql connection reconnects in the > future, > >>> which solves both the memory leak and the backends failing when the > >>> connection is closed. The only solution I found was to apply a patch > and > >>> compile from source. > >>> > >>> ----- Original Message ----- From: "Kenneth Marshall" <[EMAIL PROTECTED]> > >>> To: "Catalin Constantin" <[EMAIL PROTECTED]> > >>> Cc: <Pdns-users@mailman.powerdns.com> > >>> Sent: Tuesday, January 29, 2008 5:25 AM > >>> Subject: Re: [Pdns-users] Backend error: Failed to execute > >>> mysql_query,perhaps connection died? Err=1: MySQL server has gone away > >>> > >>> > >>>> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote: > >>>>> Hello, > >>>>> > >>>>> This is a funny question. > >>>>> Mysql is FINE and it's up and running on localhost. > >>>>> > >>>>> Here is the startup log of PDNS. > >>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006 > >>>>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc > >>>>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up > >>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO > >>>>> WARRANTY. > >>>>> This is free software, and you are welcome to redistribute it > according > >>>>> to > >>>>> the terms of the GPL version 2. > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109 > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113 > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching > >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > >>>>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads > for > >>>>> UDP > >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > >>>>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master > >>>>> serial > >>>>> 1201611548, our serial 1201610827 > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master > >>>>> serial > >>>>> 1201611548, our serial 1201610827 > >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master > >>>>> serial > >>>>> 1201611548, our serial 1201610827 > >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > >>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', > >>>>> transaction > >>>>> started > >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful > >>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone > >>>>> committed > >>>>> > >>>>> > >>>>> The problem is that mysql closes the connection if there is no > >>>>> activity. > >>>>> PDNS does not make sure that MySQL is still "alive" and from time to > >>>>> time > >>>>> it > >>>>> shows in the log errors like: > >>>>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute > >>>>> mysql_query, perhaps connection died? Err=1: MySQL server has gone > away > >>>>> > >>>>> Does PDNS have a timeout for backend connection "recheck" ? > >>>>> Can this be configured ? > >>>>> > >>>>> Is there anything we can do to make sure the connection does not go > >>>>> away > >>>>> ? > >>>>> > >>>>> Note: the NS server activity for now is not much (we're still > testing > >>>>> and > >>>>> just have a couple of domains on it). > >>>>> > >>>>> Thanks, > >>>>> > >>>> Dear Ms. Constantin, > >>>> > >>>> If a properly functioning application opens a connection to a > database, > >>>> the connection should not drop until closed. The typical fix is to > >>>> increase your timeout on your MySQL server appropriately. This same > >>>> thread comes up periodically. In order for PDNS to notice that a > >>>> connection has died, it tries to use it with a query. This causes > >>>> the errors in your logs. The amount of errors will be proportional > >>>> to how long your timeout is set. I certainly would not like to see > >>>> a patch that added overhead to every query to check the status of > >>>> a connection and re-establish if needed. PDNS will re-connect if > >>>> the connection drops, but you will get an error. If your MySQL DB > >>>> is not dedicated to PDNS, then you will have to balance the needs > >>>> of all of the applications using the DB. It sounds like you may > >>>> be running some pretty poorly written ones. Good luck. > >>>> > >>>> Ken > >>>> > >>>>> On 1/29/08, Nico van Royen <[EMAIL PROTECTED]> wrote: > >>>>> > Hello, > >>>>> > > >>>>> > Do you infact have a running mysql server ? Also check the > database > >>>>> > > > >>>>> that > >>>>> > you have specified in your pdns.conf matches your actual database > >>>>> > information (db name, user/password etc). > >>>>> > > >>>>> > Regards, > >>>>> > Nico > >>>>> > WARP3.COM > >>>>> > ________________________________ > >>>>> > From: Catalin Constantin [mailto:[EMAIL PROTECTED] > >>>>> > To: pdns-users@mailman.powerdns.com > >>>>> > Sent: Tue, 29 Jan 2008 10:48:09 +0100 > >>>>> > Subject: [Pdns-users] Backend error: Failed to execute > mysql_query, > >>>>> perhaps > >>>>> > connection died? Err=1: MySQL server has gone away > >>>>> > > >>>>> > > >>>>> > Hello, > >>>>> > > >>>>> > We're getting quite many errors: > >>>>> > Backend error: Failed to execute mysql_query, perhaps connection > > >>>>> died? > >>>>> > Err=1: MySQL server has gone away latelly. > >>>>> > > >>>>> > I checked the docs and the mailing lists for this error and i > found > >>>>> > little informations about it. > >>>>> > > >>>>> > What is actually the solution for this error not to happen ? > >>>>> > Nice would be to have a real example. > >>>>> > > >>>>> > Note: > >>>>> > - we use latest pdns package from Debian Etch (stable). > >>>>> > > >>>>> > Thanks, > >>>>> > > >>>>> > -- > >>>>> > Catalin Constantin > >>>>> > Dazoot Software > >>>>> > http://www.dazoot.eu/ > >>>>> > _______________________________________________ > >>>>> > Pdns-users mailing list > >>>>> > Pdns-users@mailman.powerdns.com > >>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users > >>>>> > > >>>>> > _______________________________________________ > >>>>> > Pdns-users mailing list > >>>>> > Pdns-users@mailman.powerdns.com > >>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users > >>>>> > > >>>>> > > >>>>> > >>>>> > >>>>> -- > >>>>> Catalin Constantin > >>>>> Dazoot Software > >>>>> http://www.dazoot.eu/ > >>>> > >>>>> _______________________________________________ > >>>>> Pdns-users mailing list > >>>>> Pdns-users@mailman.powerdns.com > >>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users > >>>> > >>>> _______________________________________________ > >>>> Pdns-users mailing list > >>>> Pdns-users@mailman.powerdns.com > >>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users > >>> > >>> _______________________________________________ > >>> Pdns-users mailing list > >>> Pdns-users@mailman.powerdns.com > >>> http://mailman.powerdns.com/mailman/listinfo/pdns-users > >>> > > > > _______________________________________________ > > Pdns-users mailing list > > Pdns-users@mailman.powerdns.com > > http://mailman.powerdns.com/mailman/listinfo/pdns-users > > > _______________________________________________ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > http://mailman.powerdns.com/mailman/listinfo/pdns-users > -- Catalin Constantin Dazoot Software http://www.dazoot.eu/
_______________________________________________ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
