Hadoop on Azure labs

[gina_cooper]

Hello-

I'm working on installing Hadoop on the Azure lab environment. I have followed these steps:https://kontext.tech/column/hadoop/377/latest-hadoop-321-installation-on-windows-10-step-by-step-gui... and am using this template: 

Windows 10 Pro, Version 1909

Medium|4 cores|7GB RAM

I keep getting an error when I try to format the namenode and I can't figure it out. I'm wondering if anyone has had success installing hadoop on Azure lab. This is the error I'm getting

2020-10-13 18:07:46,044 INFO util.GSet: VM type = 64-bit
2020-10-13 18:07:46,046 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB
2020-10-13 18:07:46,046 INFO util.GSet: capacity = 2^15 = 32768 entries
2020-10-13 18:07:46,220 INFO namenode.FSImage: Allocated new BlockPoolId: BP-2025207759-10.140.18.4-1602612466198
2020-10-13 18:07:46,375 WARN namenode.NameNode: Encountered exception during format:
ExitCodeException exitCode=-1073741515:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1307)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1289)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1350)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:1341)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:1294)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:456)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:591)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:613)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:188)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1206)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1649)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)
2020-10-13 18:07:46,386 ERROR namenode.NameNode: Failed to start namenode.
ExitCodeException exitCode=-1073741515:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)
at org.apache.hadoop.util.Shell.run(Shell.java:901)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1307)
at org.apache.hadoop.util.Shell.execCommand(Shell.java:1289)
at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1350)
at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:1341)
at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:1294)
at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:456)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:591)
at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:613)
at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:188)
at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1206)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1649)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)
2020-10-13 18:07:46,406 INFO util.ExitUtil: Exiting with status 1: ExitCodeException exitCode=-1073741515:
2020-10-13 18:07:46,418 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ML-RefVm-145321/10.140.18.4
************************************************************/

[#2]

@gina_cooper - we'll follow up with you directly on this to better understand your scenario.

Thanks,

Nicole

[#3]

@nicolehaugen I got it to work! If any schools need to install Hadoop on the Azure lab environment and are having trouble, I wrote down my steps and would be glad to help. It's a pain but it is working now!

[#4]

Hi @gina_cooper, any advise would be appreciated on how you resolved this issue, thanks. 

[#5]

Hi-

I haven't had a chance to validate these steps myself, but here are steps that were previously shared with me by @gina_cooper that may be helpful here:

-anaconda
-chrome
-java
-7zip
-visual c++ redistributable 2008, 2010
-mongodb
-mysql
-hadoop following: https://www.datasciencecentral.com/profiles/blogs/how-to-install-and-run-hadoop-on-windows-for-begin... and https://towardsdatascience.com/installing-hadoop-3-2-1-single-node-cluster-on-windows-10-ac258dd48ae...
-set JAVA_HOME and HADOOP_HOME in environment variables. JAVA_HOME should go to the jdk and HADOOP_HOME to Hadoop directory (not bin)
-set path environment variable to %JAVA_HOME%\bin and %HADOOP_HOME%\bin
-Create data/namenode and data/datanode and then change the 4 xml files per the second link using the full directory path to namenode and datanode.
-run powershell command as administrator: hdfs namenode –format
-if you get an error replace the .jar file: Download hadoop-hdfs-3.2.1.jar file from the following link: https://github.com/FahaoTang/big-data/blob/master/hadoop-hdfs-3.2.1.jar
-In powershell then type in %HADOOP_HOME%\sbin\start-dfs.cmd and it works!

Please let us know if this helps.

Thanks,
Nicole