This issue has been reported in Github. I think recent versions of Transmission all have memory leak.
For version 3.0, I think the first mention is here: https://github.com/transmission/transmission/issues/3077 The latest bug fix in 4.0.0 branch is here: https://github.com/transmission/transmission/pull/5748
