1 Azure vs AWS

The main differences in this article is that I’m using Azure to instantiate a single node HDP sandbox (for training) and I’m using AWS to instantiate a reale multi-node cluster (for real use cases). But you may also instantiate a real cluster using Azure.

2 Amazon AWS

Overview : https://fr.hortonworks.com/blog/quickly-launch-hortonworks-data-platform-amazon-web-services/

Procedure : http://hortonworks.github.io/hdp-aws/

2.1 Subscribe to HDP services and Controller Services

2.1.1 Hortonworks Data Cloud - Controller Service

Controller Service will let you instantiate AMI (Amazon Machine Image) Subscribe to Controller Services

2.1.2 Hortonworks Data Cloud - HDP Services

The HDP services subscription will make the HDP AMI available Subscribe to HDP Services

In the following steps, be sure to select the same region on the top right corner.

First of all, you need to generate a key pair : Generate my key pair

Once you have subscribed to both these services and generated the key pair you may:

1. Get back to your “Controller service” Get back to Controller Service and click on “Launch with CloudFormation Console”
2. Fill in the email field and keep the default options. Concerning the “remote access” you may input 0.0.0.0/0 to permit access to the cluster from any IP address.
3. Passer la page option Puis cliquer sur “Create” => La création du “Controller Service” peut prendre jusqu’à 15 min
   • Une fois crée, on clique sur output en bas de page et on récupère le lien pour accéder à l’interface du controller service.

Ensuite on clique sur “Create Cluster”, on nomme notre cluster : plus d’infos sur les conf ici http://hortonworks.github.io/hdp-aws/create/index.html

J’ai sauvegardé un template qu’il est possibde de dupliquer désormais.

2.2 Utilisation du clsuter

accéder à https://aws.amazon.com/fr/ Mon compte > AWS Mangaement Console Se loguer Cliquer sur Service en haut à gauche puis cliquer sur EC2 piur arriver sur la liste de mes instances EC2 Cliquer sur CloudForamtion pour arriver sur la liste de mes clusters et de mes instance “Controller service” => En cochant sur la ligne Controller Service puis sur “Sorties” en bas de page je retrouve l’URL d’accès à mon “controller service” pour instancier de nouveaux cluster ou accéder aux clusters existants =>En cliquant sur le bloc de mon cluster j’obtiens la lisre des urls de mes noeuds et aussi les commandes SSH pour s’y connecter en ssh => En cochant sur le cluster puis sur “Sorties” j’obtiens l’URL d’ambari

Pour créer un nouveau cluter Cliquer sur “Create cluster” à patir de ma page “Controller service” Sélectionner le template Sélectionner un mot de passe admin Mettre un email pour recevoir la confirmation de création

2.3 Billing

Check your detailed reports Check your detailed bill

3 AZURE

3.1 Creating your virtual machine

Prerequisities : You needd to have a Micrsoft account (Outlook/Hotmail)

Official procedure available here

Go to to HortonWorks product in Azure’s Marketplace

Create a new service “HDP Sandbox 2.5” using the button on the bottom of the page and fill in the form:

• Name your service (ex: MySandbox)
• Select SSD disk type
• You may choose 2 types of authentication system (public/private key vs login/password). You should enter your local system username
• Name your resource group (ex: MySandbox)
• Validate the form

Select your machine type : Standard DS11 v2

Validate the Network step without any modification

Once your cluster is created, it will be displayed on your home page

In order to connect to your machine, you need to create a ssh funnel. You need to edit your own ./.ssh/config file and enter the public IP of your sandbox in the 4th line et your login in the 3rd line.

Host azureSandbox
  Port 22
  User jajab
  HostName 13.81.244.170
  LocalForward 8080 127.0.0.1:8080
  LocalForward 8888 127.0.0.1:8888
  LocalForward 9995 127.0.0.1:9995
  LocalForward 9996 127.0.0.1:9996
  LocalForward 8886 127.0.0.1:8886
  LocalForward 10500 127.0.0.1:10500
  LocalForward 4200 127.0.0.1:4200
  LocalForward 2222 127.0.0.1:2222

You may then connect to the your Sandbox

ssh azureSandbox

Once conected you may access the home page of your sandbox using http://localhost:8888

3.2 Creating an image of your virtual machine

Source : Step-by-Step: Templating VMs in the Cloud with Windows Azure

You should ssh to your machine and run the following commands :

   sudo su – 
   waagent –force –deprovision 
   export HISTSIZE=0 
   shutdown –h now

Once your machine is stopped on windows Azure interface, you may capture your machine image.