Hadoop on Azure labs

Question

Hello-I'm working on installing Hadoop on the Azure lab environment. I have followed these steps:https://kontext.tech/column/hadoop/377/latest-hadoop-321-installation-on-windows-10-step-by-step-guide&nbsp;and am using this template:&nbsp;Windows 10 Pro, Version 1909Medium|4 cores|7GB RAMI keep getting an error when I try to format the namenode and I can't figure it out. I'm wondering if anyone has had success installing hadoop on Azure lab. This is the error I'm getting&nbsp;2020-10-13 18:07:46,044 INFO util.GSet: VM type = 64-bit2020-10-13 18:07:46,046 INFO util.GSet: 0.029999999329447746% max memory 889 MB = 273.1 KB2020-10-13 18:07:46,046 INFO util.GSet: capacity = 2^15 = 32768 entries2020-10-13 18:07:46,220 INFO namenode.FSImage: Allocated new BlockPoolId: BP-2025207759-10.140.18.4-16026124661982020-10-13 18:07:46,375 WARN namenode.NameNode: Encountered exception during format:ExitCodeException exitCode=-1073741515:at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)at org.apache.hadoop.util.Shell.run(Shell.java:901)at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)at org.apache.hadoop.util.Shell.execCommand(Shell.java:1307)at org.apache.hadoop.util.Shell.execCommand(Shell.java:1289)at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1350)at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:1341)at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:1294)at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:456)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:591)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:613)at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:188)at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1206)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1649)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)2020-10-13 18:07:46,386 ERROR namenode.NameNode: Failed to start namenode.ExitCodeException exitCode=-1073741515:at org.apache.hadoop.util.Shell.runCommand(Shell.java:1008)at org.apache.hadoop.util.Shell.run(Shell.java:901)at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:1213)at org.apache.hadoop.util.Shell.execCommand(Shell.java:1307)at org.apache.hadoop.util.Shell.execCommand(Shell.java:1289)at org.apache.hadoop.fs.FileUtil.execCommand(FileUtil.java:1350)at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:1341)at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:1294)at org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.clearDirectory(Storage.java:456)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:591)at org.apache.hadoop.hdfs.server.namenode.NNStorage.format(NNStorage.java:613)at org.apache.hadoop.hdfs.server.namenode.FSImage.format(FSImage.java:188)at org.apache.hadoop.hdfs.server.namenode.NameNode.format(NameNode.java:1206)at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1649)at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1759)2020-10-13 18:07:46,406 INFO util.ExitUtil: Exiting with status 1: ExitCodeException exitCode=-1073741515:2020-10-13 18:07:46,418 INFO namenode.NameNode: SHUTDOWN_MSG:/************************************************************SHUTDOWN_MSG: Shutting down NameNode at ML-RefVm-145321/10.140.18.4************************************************************/&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;

nicolehaugen · Answer

gina_cooper&nbsp;- we'll follow up with you directly on this to better understand your scenario.
&nbsp;
Thanks,
Nicole

gina_cooper · Answer

nicolehaugen&nbsp;I got it to work! If any schools need to install Hadoop on the Azure lab environment and are having trouble, I wrote down my steps and would be glad to help. It's a pain but it is working now!

rudiminty · Answer

Hi gina_cooper,&nbsp;any advise would be appreciated on how you resolved this issue, thanks.&nbsp;

nicolehaugen · Answer

Hi-I haven't had a chance to validate these steps myself, but here are steps that were previously shared with me by gina_cooper that may be helpful here:-anaconda-chrome-java-7zip-visual c++ redistributable 2008, 2010-mongodb-mysql-hadoop following: https://www.datasciencecentral.com/profiles/blogs/how-to-install-and-run-hadoop-on-windows-for-beginners and https://towardsdatascience.com/installing-hadoop-3-2-1-single-node-cluster-on-windows-10-ac258dd48aef -set JAVA_HOME and HADOOP_HOME in environment variables. JAVA_HOME should go to the jdk and HADOOP_HOME to Hadoop directory (not bin)-set path environment variable to %JAVA_HOME%\bin and %HADOOP_HOME%\bin-Create data/namenode and data/datanode and then change the 4 xml files per the second link using the full directory path to namenode and datanode.-run powershell command as administrator: hdfs namenode –format-if you get an error replace the .jar file: Download hadoop-hdfs-3.2.1.jar file from the following link: https://github.com/FahaoTang/big-data/blob/master/hadoop-hdfs-3.2.1.jar-In powershell then type in %HADOOP_HOME%\sbin\start-dfs.cmd and it works!Please let us know if this helps.Thanks,Nicole

Forum Discussion

Hadoop on Azure labs

Share

Resources