Repository: spark Updated Branches: refs/heads/branch-1.2 94bafd8b0 -> 5d5ee40cf
[SPARK-5282][mllib]: RowMatrix easily gets int overflow in the memory size warning JIRA: https://issues.apache.org/jira/browse/SPARK-5282 fix the possible int overflow in the memory computation warning Author: Yuhao Yang <[email protected]> Closes #4069 from hhbyyh/addscStop and squashes the following commits: e54e5c8 [Yuhao Yang] change to MB based number 7afac23 [Yuhao Yang] 5282: fix int overflow in the warning (cherry picked from commit 4432568aac1d4a44fa1a7c3469f095eb7a6ce945) Signed-off-by: Xiangrui Meng <[email protected]> Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/5d5ee40c Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/5d5ee40c Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/5d5ee40c Branch: refs/heads/branch-1.2 Commit: 5d5ee40cfac5dea4b9f268aa8e2ac3da67c102d8 Parents: 94bafd8 Author: Yuhao Yang <[email protected]> Authored: Mon Jan 19 10:10:15 2015 -0800 Committer: Xiangrui Meng <[email protected]> Committed: Mon Jan 19 10:10:22 2015 -0800 ---------------------------------------------------------------------- .../org/apache/spark/mllib/linalg/distributed/RowMatrix.scala | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/5d5ee40c/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala ---------------------------------------------------------------------- diff --git a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala index 10a515a..0e6665b 100644 --- a/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala +++ b/mllib/src/main/scala/org/apache/spark/mllib/linalg/distributed/RowMatrix.scala @@ -131,8 +131,8 @@ class RowMatrix( throw new IllegalArgumentException(s"Argument with more than 65535 cols: $cols") } if (cols > 10000) { - val mem = cols * cols * 8 - logWarning(s"$cols columns will require at least $mem bytes of memory!") + val memMB = (cols.toLong * cols) / 125000 + logWarning(s"$cols columns will require at least $memMB megabytes of memory!") } } --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
