Stephen, You are completely correct. It is a bandaid (needed) until the patched version is available. I am in the same situation currently with 2.9.20 and the TCP connection handle problem and have to restart my master on a regular basis. Monitoring is definitely the way to go for issues such as memory leaks.
Ken On Tue, Jan 29, 2008 at 12:18:54PM -0800, Stephen Manchester wrote: > Increasing the wait_timeout on the mysql server will help, but it does not > actually solve the memory leak, as dead connections do happen for a number > of reasons (server failure, network failure, stateful firewalls, etc.) > > Until a patch can be applied though, increasing the wait_timeout value on > the MySQL server to something very high (say 1 day or more) is the only > thing you can do. From what I've read, it looks like the default for MySQL > 4.x was 30s. The default for 5.x is 28800 seconds, but if there was > already a my.cnf file laying around (which there often is a 4.x my.cnf file > by default on many distros) then you might inherit this old value which > will cause the problem to happen very often. Again, this is only a work > around, not a true fix. You should still monitor pdns virtual memory usage > from time to time to make sure it isn't getting crazy. > > ----- Original Message ----- From: "Kenneth Marshall" <[EMAIL PROTECTED]> > To: "Stephen Manchester" <[EMAIL PROTECTED]> > Cc: <Pdns-users@mailman.powerdns.com> > Sent: Tuesday, January 29, 2008 11:56 AM > Subject: Re: [Pdns-users] Backend error: Failed to > executemysql_query,perhaps connection died? Err=1: MySQL server has > goneaway > > >> Again, this problem, for which a patch already exists, is triggered >> by the DB connection being terminated abnormally by the backend >> server, in this case via a connection time_limit. I do not use >> MySQL as a PDNS backend, but the PostgreSQL backend does correctly >> reconnect to the database should the connection die. I assume that >> the PDNS will reconnect to the MySQL server when the connection >> dies. It seems that increasing the timeout value in MySQL is that >> best measure to ensure more robust service from PDNS. >> >> Cheers, >> Ken >> >> >> On Tue, Jan 29, 2008 at 11:43:43AM -0800, Stephen Manchester wrote: >>> There is currently a memory leak in 2.9.21 from gmysql backend threads >>> dying in this fashion. The threads themselves are not being cleaned up. >>> this is only noticeable on very light loads where the backends do not get >>> enough traffic to keep the mysql connection open. You will notice it by >>> abnormally high virtual memory usage after pdns has been running for a >>> time. Each backend that dies leaves 10MB of virtual memory allocated >>> (the >>> thread stack?) and this will not get cleaned up until pdns is restarted. >>> >>> Hopefully they will implement mysql connection reconnects in the future, >>> which solves both the memory leak and the backends failing when the >>> connection is closed. The only solution I found was to apply a patch and >>> compile from source. >>> >>> ----- Original Message ----- From: "Kenneth Marshall" <[EMAIL PROTECTED]> >>> To: "Catalin Constantin" <[EMAIL PROTECTED]> >>> Cc: <Pdns-users@mailman.powerdns.com> >>> Sent: Tuesday, January 29, 2008 5:25 AM >>> Subject: Re: [Pdns-users] Backend error: Failed to execute >>> mysql_query,perhaps connection died? Err=1: MySQL server has gone away >>> >>> >>>> On Tue, Jan 29, 2008 at 03:04:46PM +0200, Catalin Constantin wrote: >>>>> Hello, >>>>> >>>>> This is a funny question. >>>>> Mysql is FINE and it's up and running on localhost. >>>>> >>>>> Here is the startup log of PDNS. >>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS 2.9.20 (C) 2001-2006 >>>>> PowerDNS.COMBV (Mar 10 2007, 00:36:58, gcc >>>>> 4.1.2 20061115 (prerelease) (Debian 4.1.1-21)) starting up >>>>> Jan 29 14:59:05 k2 pdns[30294]: PowerDNS comes with ABSOLUTELY NO >>>>> WARRANTY. >>>>> This is free software, and you are welcome to redistribute it according >>>>> to >>>>> the terms of the GPL version 2. >>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective group id to 109 >>>>> Jan 29 14:59:05 k2 pdns[30294]: Set effective user id to 113 >>>>> Jan 29 14:59:05 k2 pdns[30294]: Creating backend connection for TCP >>>>> Jan 29 14:59:05 k2 pdns[30294]: Master/slave communicator launching >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>>>> Jan 29 14:59:05 k2 pdns[30294]: About to create 10 backend threads for >>>>> UDP >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>>>> Jan 29 14:59:05 k2 pdns[30294]: 14 slave domains need checking >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test1.com is stale, master >>>>> serial >>>>> 1201611548, our serial 1201610827 >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test2.com is stale, master >>>>> serial >>>>> 1201611548, our serial 1201610827 >>>>> Jan 29 14:59:05 k2 pdns[30294]: Domain test3.com is stale, master >>>>> serial >>>>> 1201611548, our serial 1201610827 >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR started for 'test1.com', >>>>> transaction >>>>> started >>>>> Jan 29 14:59:05 k2 pdns[30294]: gmysql Connection succesful >>>>> Jan 29 14:59:05 k2 pdns[30294]: AXFR done for 'test1.com', zone >>>>> committed >>>>> >>>>> >>>>> The problem is that mysql closes the connection if there is no >>>>> activity. >>>>> PDNS does not make sure that MySQL is still "alive" and from time to >>>>> time >>>>> it >>>>> shows in the log errors like: >>>>> Jan 29 11:33:57 k2 pdns[28941]: Backend error: Failed to execute >>>>> mysql_query, perhaps connection died? Err=1: MySQL server has gone away >>>>> >>>>> Does PDNS have a timeout for backend connection "recheck" ? >>>>> Can this be configured ? >>>>> >>>>> Is there anything we can do to make sure the connection does not go >>>>> away >>>>> ? >>>>> >>>>> Note: the NS server activity for now is not much (we're still testing >>>>> and >>>>> just have a couple of domains on it). >>>>> >>>>> Thanks, >>>>> >>>> Dear Ms. Constantin, >>>> >>>> If a properly functioning application opens a connection to a database, >>>> the connection should not drop until closed. The typical fix is to >>>> increase your timeout on your MySQL server appropriately. This same >>>> thread comes up periodically. In order for PDNS to notice that a >>>> connection has died, it tries to use it with a query. This causes >>>> the errors in your logs. The amount of errors will be proportional >>>> to how long your timeout is set. I certainly would not like to see >>>> a patch that added overhead to every query to check the status of >>>> a connection and re-establish if needed. PDNS will re-connect if >>>> the connection drops, but you will get an error. If your MySQL DB >>>> is not dedicated to PDNS, then you will have to balance the needs >>>> of all of the applications using the DB. It sounds like you may >>>> be running some pretty poorly written ones. Good luck. >>>> >>>> Ken >>>> >>>>> On 1/29/08, Nico van Royen <[EMAIL PROTECTED]> wrote: >>>>> > Hello, >>>>> > >>>>> > Do you infact have a running mysql server ? Also check the database >>>>> > > >>>>> that >>>>> > you have specified in your pdns.conf matches your actual database >>>>> > information (db name, user/password etc). >>>>> > >>>>> > Regards, >>>>> > Nico >>>>> > WARP3.COM >>>>> > ________________________________ >>>>> > From: Catalin Constantin [mailto:[EMAIL PROTECTED] >>>>> > To: pdns-users@mailman.powerdns.com >>>>> > Sent: Tue, 29 Jan 2008 10:48:09 +0100 >>>>> > Subject: [Pdns-users] Backend error: Failed to execute mysql_query, >>>>> perhaps >>>>> > connection died? Err=1: MySQL server has gone away >>>>> > >>>>> > >>>>> > Hello, >>>>> > >>>>> > We're getting quite many errors: >>>>> > Backend error: Failed to execute mysql_query, perhaps connection > >>>>> died? >>>>> > Err=1: MySQL server has gone away latelly. >>>>> > >>>>> > I checked the docs and the mailing lists for this error and i found >>>>> > little informations about it. >>>>> > >>>>> > What is actually the solution for this error not to happen ? >>>>> > Nice would be to have a real example. >>>>> > >>>>> > Note: >>>>> > - we use latest pdns package from Debian Etch (stable). >>>>> > >>>>> > Thanks, >>>>> > >>>>> > -- >>>>> > Catalin Constantin >>>>> > Dazoot Software >>>>> > http://www.dazoot.eu/ >>>>> > _______________________________________________ >>>>> > Pdns-users mailing list >>>>> > Pdns-users@mailman.powerdns.com >>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users >>>>> > >>>>> > _______________________________________________ >>>>> > Pdns-users mailing list >>>>> > Pdns-users@mailman.powerdns.com >>>>> > http://mailman.powerdns.com/mailman/listinfo/pdns-users >>>>> > >>>>> > >>>>> >>>>> >>>>> -- >>>>> Catalin Constantin >>>>> Dazoot Software >>>>> http://www.dazoot.eu/ >>>> >>>>> _______________________________________________ >>>>> Pdns-users mailing list >>>>> Pdns-users@mailman.powerdns.com >>>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users >>>> >>>> _______________________________________________ >>>> Pdns-users mailing list >>>> Pdns-users@mailman.powerdns.com >>>> http://mailman.powerdns.com/mailman/listinfo/pdns-users >>> >>> _______________________________________________ >>> Pdns-users mailing list >>> Pdns-users@mailman.powerdns.com >>> http://mailman.powerdns.com/mailman/listinfo/pdns-users >>> > > _______________________________________________ > Pdns-users mailing list > Pdns-users@mailman.powerdns.com > http://mailman.powerdns.com/mailman/listinfo/pdns-users > _______________________________________________ Pdns-users mailing list Pdns-users@mailman.powerdns.com http://mailman.powerdns.com/mailman/listinfo/pdns-users
