Remote Spark Compute Context using PuTTY on Windows
Published Mar 23 2019 10:40 PM 1,211 Views
Microsoft
First published on MSDN on Feb 12, 2017
If you are running Microsoft R Server / Microsoft R Client from a Windows computer equipped with PuTTY , you can create a compute context that will run RevoScaleR functions from your local client in a distributed fashion on your Hadoop cluster. You use RxSpark to create the compute context, but use additional arguments to specify your user name, the file-sharing directory where you have read and write access, the publicly-facing host name or IP address of your Hadoop cluster’s name node or an edge node that will run the master processes, and any additional switches to pass to the ssh command (such as the -i flag if you are using a ppk file for authentication).

RxSpark assumes that the directory containing the plink.exe command (PuTTY) is in your path. If not, you can specify the location of these files using the sshClientDir argument.

In some cases, you may find that environment variables needed by Hadoop are not set in the remote sessions run on the sshHostname computer. This may be because a different profile or startup script is being read on ssh login. You can specify the appropriate profile script by specifying the sshProfileScript argument to RxSpark (this should be an absolute path).

Here is an example of using remote Spark Compute Context:



OUTPUT:
























Troubleshooting




Using RClient 3.3.2/MRS 9.0.1, you might get the following error messages while running the above program:
Error in rxFindWindowsSsh("plink.exe", "ssh.exe", sshClientDir = "") : Could not find neither plink.exe nor ssh.exe.

plink: unknown option \"-p\"


We have released Hotfix to address the above error messages. You can download the fix and instructions here .


Few other possible error messages and troubleshooting steps:
Error in checkSshReturnMessage(printOut) : Access denied. Please validate the host connection credential file.

Error in checkSshReturnMessage(printOut) : Host key verification failed. Please validate host connection using the specified ssh tool directly and try again.

Error: Fail to fetch remote MRS version; text output is: Unable to open connection:, Host does not exist


If you get the above error messages, ensure the following:

  1. The public key is set properly in /home/sshuser/.ssh/authorized_keys file. Check for extra spaces, newline, extra characters etc and remove them.

  2. Key comment property of the public key does not have a space since OpenSSH's SSH-2 key format contains no space for a comment

  3. sshuser has read access to /home/sshuser/.ssh/authorized_keys

  4. azureuser has read access C:\Users\azureuser\Documents\privateKey.ppk

  5. Connection to the edge node is successful using PuTTY.


Here is a detailed document on how to Use SSH with HDInsight (Hadoop) from PuTTY on Windows .
Please use the comments section in this article to report any other SSH issues when using RxSpark.
Version history
Last update:
‎Mar 23 2019 03:40 PM
Updated by: