Wednesday, 11 January 2017

Deploying Azure Virtual Networks with multiple CIDR addresses using ARM templates

I recently had to craft an ARM template in Azure for a Virtual Network that had an address space that wasn't sized as a "to the power of two".

i.e. this would be simple if you're creating a Vnet for an address space of say 256 addresses, such as
10.1.0.0 - 10.1.0.255 = 10.1.0.0/24
You can use a parameter of type "string" in the template, so assuming you arrange your parameters and template into 2 files, the parameters would include:
"vnetAddressPrefix": {
    "value": "10.1.0.0/24",
}
And the template itself would include the parameters and resource definitions for the Vnet as follows:
"vnetAddressPrefix": {
    "type": "string"
}
...
"resources": [ {
    "type": "Microsoft.Network/virtualNetworks",
    "name": "my256Vnet",
    "apiVersion": "2016-03-30",
    "location": "[resourceGroup().location]",
    "properties": {
        "addressSpace": {
            "addressPrefixes": [
                "[parameters('vnetAddressPrefix')]"
            ]
        },
        "subnets": [ {
            "name": "default",
            "properties": {
                "addressPrefix": "10.1.0.0/28"
            }
        }
        ]
    }
}
]

But, lets say you need a Vnet with 384 addresses, such as
10.1.0.0 - 10.1.1.127
Because you can't specify this range in a single CIDR address, you would instead need to create an address space in CIDR notation of
10.1.0.0/24
10.1.1.0/25
The simple "string" parameter used in the first example will not work here, instead you should use a parameter of type "array", modifying the parameter file to include the CIDR ranges as follows:
"vnetAddressPrefixes": {
    "value": [
            "10.1.0.0/24",
            "10.1.1.0/25"
    ]
}
And then changing your template file to include parameters and resource definitions as follows:
"vnetAddressPrefixes": {
    "type": "array"
}
...
"resources": [ {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/virtualNetworks",
    "name": "my384Vnet",
    "location": "[resourceGroup().location]",
    "properties": {
        "addressSpace": {
            "addressPrefixes": 
                "[parameters('vnetAddressPrefixes')]"
        },
        "subnets": [ {
            "name": "default",
            "properties": {
                "addressPrefix": "10.1.0.63/24"
            }
        }
        ]
    }
}
]

That's it! Although it's unlikely that a deploying Vnets in a greenfield environment would require this solution, it still may happen, and is more likely to happen in an environment where address space is already constrained by other Vnets and addressing conventions.

Wednesday, 4 January 2017

AWS Glue: The future of ETL is nigh

AWS will soon be launching a fully managed ETL as-a-service which promises to automate "data discovery, conversion, mapping, and job scheduling tasks" wherever your data lives, this is not just for your data stored in AWS.

AWS Glue, will provide the ETL layer that links AWS data services (S3RDS and Redshift etc) to their and analytics services, such as Quicksight.

I for one am fascinated to see what it can do and discover real-world use-cases that can take advantage of AWS Glue, building it into the architecture of Big Data Cloud solutions.

Wednesday, 28 December 2016

Globally Unique Identifiers - Creating GUIDs in Powershell

Powershell Pea...

New-Guid

I've been working more and more with Powershell, specifically with Azure Automation and scripting and wanted to share a useful Cmdlet for creating GUIDs. This link provides more detail, but ultimately this command will provide you with a unique ID that can be assigned to arbitrary objects that require a unique ID.

Bear in mind that this doesn't protect you from duplicates, but the probability of them occurring is small. So if you need GUIDs for relatively small record/object sets and your application can handle the prospect of duplicates, then this function is a quick way to generate a "unique" ID.

See this link for more details on GUIDs/UUIDs.

Wednesday, 21 December 2016

Azure Premium Storage Blob Snapshot Error - 409 Conflict

I've recently been working in more detail with Azure blob snapshots and (by trial and error) discovered that Premium storage imposes limitations on the number and frequency of snapshots that can be performed on a single blob.

If you create a snapshot of a Premium storage blob and then another in quick succession (within 60 seconds of each other), then you might get a 409/Conflict error message.

This article suggests that there are two possibilities for this error, either

  • SnapshotCountExceeded - You've exceeded the limit of 100 snapshots per blob, or
  • SnaphotOperationRateExceeded - You've exceeded the limit of snapshots within a time window (stated as 10 minutes, but I observed this as closer to 1 minute)
Whilst it's unlikely that these limits would cause a problem in practice, it's something to keep in mind when developing/testing solutions that make use of the blob snapshot facility.

Wednesday, 14 December 2016

Azure Automation Troubles

A recent venture into Azure Automation threw up some unexpected problems.

A simple script, run locally, that called Get-AzureRmVM was working to my subscription, but the same script did not work within an Azure Runbook.

I'd created the AzureRunAsConnection by default with the Azure Automation account and followed the same code as given in the AzureAutomationTutorialScript powershell runbook to login to the subscription and execute Cmdlets.

The problem came when executing the Get-AzureRmVM Cmdlet in the runbook, receiving the error:
Get-AzureRmVM : Run Login-AzureRmAccount to login.
+     $VM = Get-AzureRmVM | ? {$_.Name -eq $VMName}
+           ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Get-AzureRmVM], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.Azure.Commands.Compute.GetAzureVMCommand

After several googles I found this post...
https://feedback.azure.com/forums/246290-automation/suggestions/16590229-get-azurermvm-failing-in-azure-automation

Therefore, if anyone else sees errors when running basic Cmdlets in a runbook (that work fine from PowerShell), try updating all the modules in the Automation account using a script like this one...
https://www.powershellgallery.com/packages/Update-ModulesInAutomationToLatestVersion/1.03/DisplayScript
(Note that the deploy to Azure button didn't work for me, so I had to import the script and run manually.)

Note also that (depending on when you created your Automation Account) the above may, or may not work, and you may have to create a new Automation Account and update the modules before starting to use the account. Microsoft suggest setting the script to run on a schedule in the automation account to keep the modules up to date.

Sunday, 27 November 2016

Azure Pea of the day

For those of you who care about the aesthetics of their Azure cloud experience, I accidentally discovered how to quickly change your portal theme...

Simply double-click on the dashboard background and it will cycle through the available themes.

Wednesday, 23 November 2016

Run your own Gitlab server on Azure

For those of you out there wanting to host and control your own git server in the cloud, the Azure marketplace has a Gitlab Community Edition VM ready to deploy, see here. Of course, gitlab.com takes away the need to host and removes the cost involved, but if you want control over where your repository is located and the underlying VM, then this looks like a good option.

I've spun this up using the newly available A1_V2 VM size, which handles light load amicably. So for less then £20 you can have control over all those features that the community edition offers, including SSH and HTTPS access.

The gitlab CE documentation is a little outdated, but still good enough to help you configure the instance and get going with your own git repo.

Wednesday, 21 September 2016

Using environment variables to create a generic WebJob script in Azure

I recently had to write a script for a WebJob in an Azure App Service Web App which was reused across multiple copies of the same Web App.

The script was simple, it simply needed to make a request to a scripted action (in my case PHP) within the website hosted by the Web App. The request was via a URL, which was dependent upon the name of the web app, e.g.if the web app name is "mycoolwebapp" then the URL of the scripted action would be
https://mycoolwebapp.azurewebsites.net/bin/mytask.php
Instead of writing different scripts each time, with the web app name hardcoded, it's possible to use environment variables from within the WebJob script to refer to the Web App environment variables. In this case, "mycoolwebapp.azurewebsites.net" can be referred to via the WEBSITE_HOSTNAME environment variable in a CMD script as follows
curl https://%WEBSITE_HOSTNAME%/bin/mytask.php
This single script can then be uploaded to multiple Web Jobs for your websites, without needing a different version of the script for every website.

Monday, 12 September 2016

How to enable PHP extensions by adding modules to an Azure Web App

Using Azure, you can easily deploy a Web App (a website) that uses PHP.

An additional step that you may be required to perform is to enable additional PHP extensions that contain modules your PHP code uses.

For example, if you need to add the php_sockets.dll to the Web App, you would follow these steps:

  1. Obtain the extension dll, one way to do this is to download the correct PHP installation for windows from here
  2. Extract the PHP zip file and open the ext directory, locate the required dll file(s)
  3. Add FTP credentials to your Web App and log into your Web App using an FTP client
  4. Add a bin directory to the root directory here: D:\home\site\wwwroot
  5. Copy the dll files to the bin directory
  6. In Azure click on AppServices->"MyPHPWebApp"->Settings->Application Settings
  7. Under App Settings, create a PHP_EXTENSIONS key and add a value which is a relative path to the extension dll in the root directory, e.g. bin\php_sockets.dll


Once you save the application settings, you can run phpinfo() to check that the new extension has been loaded.

See this post for more details and an alternative method.

Monday, 5 September 2016

Buying an SSL certificate for a single custom sub-domain in Azure

The process of buying an SSL certificate in Azure is relatively simple and well-documented via the App Service Certificates option in Azure.

However, I recently had to buy an SSL certificate for a specific custom sub-domain and found the process and advice to be somewhat confusing prior to committing to the purchase. This article states:
Make sure to enter correct host name (custom domain) that you want to protect with this certificate. DO NOT append the Host name with WWW. For example, if your custom domain name is www.contoso.com then just enter contoso.com in the Host Name field, the certificate in question will protect both www and root domains.
This information is specifically aimed at those who are creating a website certificate for the www sub-domain. In terms of a custom sub-domain you shouldn't follow the same advice.

Therefore, this post will help you if you're in a similar situation as follows:

  • You've deployed an Azure App Service Web App to Azure, e.g. mywebapp.azurewebsites.net
  • You own a custom domain, e.g. necloud.uk
  • You've associated a sub-domain such as mywebapp.necloud.uk to mywebapp.azurewebsites.net
Then purchase the SSL Certificate as follows:
  • Log into Azure->App Service Certificates->Add
  • Here's the catch... under the "Naked Domain Host Name" field, enter the full sub-domain, e.g. mywebapp.necloud.uk, i,.e. DO NOT enter just necloud.uk unless you specifically want a wild card certificate.
  • Once purchased, you can add the certificate to your sub-domain via Azure->App Service->mywebapp->SSL certificates->Import App Service Certificates