Create slurm cluster #291
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,34 @@ | ||
| # Deploy a slurm cluster | ||
|
|
||
| Built by: [YidingZhou](https://github.com/YidingZhou) | ||
|
|
||
| ##Deploy | ||
|
|
||
| <a href="https://portal.azure.com/#create/Microsoft.Template/uri/https%3A%2F%2Fraw.githubusercontent.com%2FYidingZhou%2Fazure-quickstart-templates%2Fmaster%2Fslurm%2Fazuredeploy.json" target="_blank"> | ||
| <img alt="Deploy to Azure" src="http://azuredeploy.net/deploybutton.png"/> | ||
| </a> | ||
|
|
||
| 1. Fill in the 3 mandatory parameters - public DNS name, a storage account to hold VM image, and admin user password. | ||
|
|
||
| 2. Fill in other info and click "OK". | ||
|
|
||
| ## Using the cluster | ||
|
|
||
| Simply SSH to the master node and do a srun! The DNS name is _**dnsName**_._**location**_.cloudapp.azure.com, for example, yidingslurm.westus.cloudapp.azure.com. | ||
|
|
||
| ## Parameters | ||
|
|
||
| | Name | Description | | ||
| |:--- |:---| | ||
| | dnsName | Unique public dns name where the master node will be exposed | | ||
| | newStorageAccountName | Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed | | ||
| | adminPassword | password of the admin user | | ||
| | vmSize | Instance size of the VM worker node | | ||
| | scaleNumber | Number of work nodes in the SLURM cluster | | ||
| | location | This is the location where the availability set will be deployed | | ||
| | masterVMName | The machine name of the master node - not be confused with dnsName which is public visible name. | | ||
| | workerVMName | The machine name of the worker node. | | ||
| | adminUserName | User name for the Virtual Machine. | | ||
|
|
||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| { | ||
|
Contributor
There was a problem hiding this comment. This file should match the parameters in the template |
||
| "dnsName": { | ||
| "value": "deishbai32" | ||
| }, | ||
| "newStorageAccountName": { | ||
| "value": "slurmcluster" | ||
| }, | ||
| "adminUserName": { | ||
| "value": "azureuser" | ||
| }, | ||
| "adminPassword": { | ||
| "value": "" | ||
| }, | ||
| "vmSize": { | ||
| "value": "Standard_D1" | ||
| }, | ||
| "scaleNumber": { | ||
| "value": 2 | ||
| }, | ||
| "location": { | ||
| "value": "West US" | ||
| } | ||
| } | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,303 @@ | ||
| { | ||
| "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#", | ||
| "contentVersion": "1.0.0.0", | ||
| "parameters": { | ||
| "dnsName": { | ||
| "type": "string", | ||
| "metadata": { | ||
| "description": "Unique public dns name where the master node will be exposed" | ||
| } | ||
| }, | ||
| "newStorageAccountName": { | ||
| "type": "string", | ||
| "metadata": { | ||
| "description": "Unique DNS Name for the Storage Account where the Virtual Machine's disks will be placed." | ||
| } | ||
| }, | ||
| "adminUserName": { | ||
| "type": "string", | ||
| "defaultValue": "azureuser", | ||
| "metadata": { | ||
| "description": "User name for the Virtual Machine." | ||
| } | ||
| }, | ||
| "adminPassword": { | ||
|
Contributor
There was a problem hiding this comment. Put this parameter next to adminUser |
||
| "type": "securestring", | ||
| "metadata": { | ||
| "description": "Admin password" | ||
| } | ||
| }, | ||
| "vmSize": { | ||
|
Contributor
There was a problem hiding this comment. Provide allowedValues for this. Use D-sizes instead of A-sizes |
||
| "type": "string", | ||
| "defaultValue": "Standard_D1", | ||
| "allowedValues": [ | ||
| "Standard_D1", | ||
| "Standard_D2", | ||
| "Standard_D3", | ||
| "Standard_D4" | ||
| ], | ||
| "metadata": { | ||
| "description": "Size of the nodes." | ||
| } | ||
| }, | ||
| "scaleNumber": { | ||
| "type": "int", | ||
| "defaultValue": 2, | ||
| "metadata": { | ||
| "description": "This template create N worker node. Use scaleNumber to specify that N." | ||
| } | ||
| }, | ||
| "location": { | ||
| "type": "string", | ||
| "defaultValue": "West US", | ||
| "metadata": { | ||
| "description": "This is the location where the availability set will be deployed" | ||
| } | ||
| } | ||
| }, | ||
| "variables": { | ||
| "imagePublisher": "Canonical", | ||
| "imageOffer": "UbuntuServer", | ||
| "ubuntuOSVersion": "15.04", | ||
| "vmStorageAccountContainerName": "vhd", | ||
| "OSDiskName": "osdisk", | ||
| "publicIPAddressType": "Dynamic", | ||
| "publicIPAddressName": "publicips", | ||
| "masterVMName": "master", | ||
| "workerVMName": "worker", | ||
| "nicName": "nic", | ||
| "networkSettings": { | ||
| "virtualNetworkName": "virtualnetwork", | ||
| "addressPrefix": "10.0.0.0/16", | ||
| "subnet": { | ||
| "dse": { | ||
| "name": "dse", | ||
| "prefix": "10.0.0.0/24", | ||
| "vnet": "virtualnetwork" | ||
| } | ||
| }, | ||
| "statics": { | ||
| "workerRange": { | ||
| "base": "10.0.0.", | ||
| "start": 5 | ||
| }, | ||
| "master": "10.0.0.254" | ||
| } | ||
| }, | ||
| "vnetID": "[resourceId('Microsoft.Network/virtualNetworks',variables('networkSettings').virtualNetworkName)]", | ||
| "subnetRef": "[concat(variables('vnetID'),'/subnets/',variables('networkSettings').subnet.dse.name)]", | ||
| "sshKeyPath": "[concat('/home/',parameters('adminUsername'),'/.ssh/authorized_keys')]", | ||
| "templateBaseUrl": "https://raw.githubusercontent.com/YidingZhou/azure-quickstart-templates/master/slurm/", | ||
|
Contributor
There was a problem hiding this comment. This url needs to be updated for this repo as well |
||
| "installationCLI": "[concat('sh azuredeploy.sh ', variables('masterVMName'), ' ', variables('networkSettings').statics.master, ' ', variables('workerVMName'), ' ', variables('networkSettings').statics.workerRange.base, ' ', variables('networkSettings').statics.workerRange.start, ' ', parameters('scaleNumber'), ' ', parameters('adminUsername'), ' ', parameters('adminPassword'), ' ', variables('templateBaseUrl'))]", | ||
| "storageAccountType": "Standard_LRS" | ||
| }, | ||
| "resources": [ | ||
| { | ||
| "type": "Microsoft.Storage/storageAccounts", | ||
| "name": "[parameters('newStorageAccountName')]", | ||
| "apiVersion": "2015-05-01-preview", | ||
| "location": "[parameters('location')]", | ||
| "properties": { | ||
| "accountType": "[variables('storageAccountType')]" | ||
| } | ||
| }, | ||
| { | ||
| "apiVersion": "2015-05-01-preview", | ||
| "type": "Microsoft.Network/virtualNetworks", | ||
| "name": "[variables('networkSettings').virtualNetworkName]", | ||
| "location": "[parameters('location')]", | ||
| "properties": { | ||
| "addressSpace": { | ||
| "addressPrefixes": [ | ||
| "[variables('networkSettings').addressPrefix]" | ||
| ] | ||
| }, | ||
| "subnets": [ | ||
| { | ||
| "name": "[variables('networkSettings').subnet.dse.name]", | ||
| "properties": { | ||
| "addressPrefix": "[variables('networkSettings').subnet.dse.prefix]" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "type": "Microsoft.Network/publicIPAddresses", | ||
| "apiVersion": "2015-05-01-preview", | ||
| "name": "[variables('publicIPAddressName')]", | ||
| "location": "[parameters('location')]", | ||
| "properties": { | ||
| "publicIPAllocationMethod": "[variables('publicIPAddressType')]", | ||
| "dnsSettings": { | ||
| "domainNameLabel": "[parameters('dnsName')]" | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "apiVersion": "2015-05-01-preview", | ||
| "type": "Microsoft.Network/networkInterfaces", | ||
| "name": "[variables('nicName')]", | ||
| "location": "[parameters('location')]", | ||
| "dependsOn": [ | ||
| "[concat('Microsoft.Network/publicIPAddresses/', variables('publicIPAddressName'))]", | ||
| "[concat('Microsoft.Network/virtualNetworks/', variables('networkSettings').virtualNetworkName)]" | ||
| ], | ||
| "properties": { | ||
| "ipConfigurations": [ | ||
| { | ||
| "name": "ipconfig1", | ||
| "properties": { | ||
| "privateIPAllocationMethod": "Static", | ||
| "privateIPAddress": "[variables('networkSettings').statics.master]", | ||
| "publicIPAddress": { | ||
| "id": "[resourceId('Microsoft.Network/publicIPAddresses',variables('publicIPAddressName'))]" | ||
| }, | ||
| "subnet": { | ||
| "id": "[variables('subnetRef')]" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "apiVersion": "2015-05-01-preview", | ||
| "type": "Microsoft.Compute/virtualMachines", | ||
| "name": "[variables('masterVMName')]", | ||
| "location": "[parameters('location')]", | ||
| "dependsOn": [ | ||
| "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]", | ||
| "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'))]" | ||
| ], | ||
| "properties": { | ||
| "hardwareProfile": { | ||
| "vmSize": "[parameters('vmSize')]" | ||
| }, | ||
| "osProfile": { | ||
| "computername": "[variables('masterVMName')]", | ||
| "adminUsername": "[parameters('adminUsername')]", | ||
| "adminPassword": "[parameters('adminPassword')]" | ||
| }, | ||
| "storageProfile": { | ||
| "imageReference": { | ||
| "publisher": "[variables('imagePublisher')]", | ||
| "offer": "[variables('imageOffer')]", | ||
| "sku" : "[variables('ubuntuOSVersion')]", | ||
| "version":"latest" | ||
| }, | ||
| "osDisk" : { | ||
| "name": "osdisk", | ||
| "vhd": { | ||
| "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),variables('masterVMName'),'.vhd')]" | ||
| }, | ||
| "caching": "ReadWrite", | ||
| "createOption": "FromImage" | ||
| } | ||
| }, | ||
| "networkProfile": { | ||
| "networkInterfaces": [ | ||
| { | ||
| "id": "[resourceId('Microsoft.Network/networkInterfaces',variables('nicName'))]" | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "apiVersion": "2015-05-01-preview", | ||
| "type": "Microsoft.Compute/virtualMachines/extensions", | ||
| "name": "[concat(variables('masterVMName'), '/Installation')]", | ||
| "location": "[parameters('location')]", | ||
| "dependsOn": [ | ||
| "[concat('Microsoft.Compute/virtualMachines/', variables('masterVMName'))]" | ||
| ], | ||
| "properties": { | ||
| "publisher": "Microsoft.OSTCExtensions", | ||
| "type": "CustomScriptForLinux", | ||
| "typeHandlerVersion": "1.1", | ||
| "settings": { | ||
| "fileUris": [ | ||
| "[concat(variables('templateBaseUrl'), 'azuredeploy.sh')]" | ||
| ], | ||
| "commandToExecute": "[variables('installationCLI')]" | ||
| } | ||
| } | ||
| }, | ||
| { | ||
| "apiVersion": "2015-05-01-preview", | ||
| "type": "Microsoft.Network/networkInterfaces", | ||
| "name": "[concat(variables('nicName'), 'worker', copyindex())]", | ||
| "location": "[parameters('location')]", | ||
| "dependsOn": [ | ||
| "[concat('Microsoft.Network/virtualNetworks/', variables('networkSettings').virtualNetworkName)]" | ||
| ], | ||
| "copy": { | ||
| "name": "foo", | ||
| "count": "[parameters('scaleNumber')]" | ||
| }, | ||
| "properties": { | ||
| "ipConfigurations": [ | ||
| { | ||
| "name": "ipconfig1", | ||
| "properties": { | ||
| "privateIPAllocationMethod": "Static", | ||
| "privateIPAddress": "[concat(variables('networkSettings').statics.workerRange.base, copyindex(variables('networkSettings').statics.workerRange.start))]", | ||
| "subnet": { | ||
| "id": "[variables('subnetRef')]" | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| }, | ||
| { | ||
| "apiVersion": "2015-05-01-preview", | ||
| "type": "Microsoft.Compute/virtualMachines", | ||
| "name": "[concat(variables('workerVMName'), copyindex())]", | ||
| "location": "[parameters('location')]", | ||
| "dependsOn": [ | ||
| "[concat('Microsoft.Storage/storageAccounts/', parameters('newStorageAccountName'))]", | ||
| "[concat('Microsoft.Network/networkInterfaces/', variables('nicName'), 'worker', copyindex())]" | ||
| ], | ||
| "copy": { | ||
| "name": "foo", | ||
| "count": "[parameters('scaleNumber')]" | ||
| }, | ||
| "properties": { | ||
| "hardwareProfile": { | ||
| "vmSize": "[parameters('vmSize')]" | ||
| }, | ||
| "osProfile": { | ||
| "computername": "[concat(variables('workerVMName'), copyindex())]", | ||
| "adminUsername": "[parameters('adminUsername')]", | ||
| "adminPassword": "[parameters('adminPassword')]" | ||
| }, | ||
| "storageProfile": { | ||
| "imageReference": { | ||
| "publisher": "[variables('imagePublisher')]", | ||
| "offer": "[variables('imageOffer')]", | ||
| "sku" : "[variables('ubuntuOSVersion')]", | ||
| "version":"latest" | ||
| }, | ||
| "osDisk" : { | ||
| "name": "osdisk", | ||
| "vhd": { | ||
| "uri": "[concat('http://',parameters('newStorageAccountName'),'.blob.core.windows.net/',variables('vmStorageAccountContainerName'),'/',variables('OSDiskName'),variables('workerVMName'),copyindex(),'.vhd')]" | ||
| }, | ||
| "caching": "ReadWrite", | ||
| "createOption": "FromImage" | ||
| } | ||
| }, | ||
| "networkProfile": { | ||
| "networkInterfaces": [ | ||
| { | ||
| "id": "[resourceId('Microsoft.Network/networkInterfaces',concat(variables('nicName'), 'worker', copyindex()))]" | ||
| } | ||
| ] | ||
| } | ||
| } | ||
| } | ||
| ] | ||
| } | ||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be updated to point to this repo