Repository: spark Updated Branches: refs/heads/branch-0.9 706e38fd7 -> 8e9f479ed
SPARK-1990: added compatibility for python 2.6 for ssh_read command https://issues.apache.org/jira/browse/SPARK-1990 There were some posts on the lists that spark-ec2 does not work with Python 2.6. In addition, we should check the Python version at the top of the script and exit if it's too old Author: Anant <[email protected]> Closes #941 from anantasty/SPARK-1990 and squashes the following commits: 4ca441d [Anant] Implmented check_optput withinthe module to work with python 2.6 c6ed85c [Anant] added compatibility for python 2.6 for ssh_read command Conflicts: ec2/spark_ec2.py Project: http://git-wip-us.apache.org/repos/asf/spark/repo Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/8e9f479e Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/8e9f479e Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/8e9f479e Branch: refs/heads/branch-0.9 Commit: 8e9f479edd5cbacbd6622d61e614b36a2ce2b86b Parents: 706e38f Author: Anant <[email protected]> Authored: Mon Jun 16 23:42:27 2014 -0700 Committer: Patrick Wendell <[email protected]> Committed: Mon Jun 16 23:46:10 2014 -0700 ---------------------------------------------------------------------- ec2/spark_ec2.py | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/spark/blob/8e9f479e/ec2/spark_ec2.py ---------------------------------------------------------------------- diff --git a/ec2/spark_ec2.py b/ec2/spark_ec2.py index cfc61b2..8f6a69c 100755 --- a/ec2/spark_ec2.py +++ b/ec2/spark_ec2.py @@ -629,9 +629,23 @@ def ssh(host, opts, command): time.sleep(30) tries = tries + 1 +# Backported from Python 2.7 for compatiblity with 2.6 (See SPARK-1990) +def _check_output(*popenargs, **kwargs): + if 'stdout' in kwargs: + raise ValueError('stdout argument not allowed, it will be overridden.') + process = subprocess.Popen(stdout=PIPE, *popenargs, **kwargs) + output, unused_err = process.communicate() + retcode = process.poll() + if retcode: + cmd = kwargs.get("args") + if cmd is None: + cmd = popenargs[0] + raise subprocess.CalledProcessError(retcode, cmd, output=output) + return output + def ssh_read(host, opts, command): - return subprocess.check_output( + return _check_output( ssh_command(opts) + ['%s@%s' % (opts.user, host), stringify_command(command)])
