Use tags to drill into Azure Enterprise Billing with Power BI

Whilst Power BI's Azure Enterprise Content Pack gives you pre-built dashboards to expose your Azure Enterprise costs, it may not provide the level of detail required to really drill-down into the finer detail of resource costs for your Azure subscriptions.

What if you wanted to tag resources to identify those that belong to your enterprises products or services and report on costs for each. For example, perhaps you'd like to be able to find out how much was spent on Azure Compute resources (Virtual Machines) and storage during the first month of an online service that you've just launched. Your VMs might be spread across Resource Groups, with their storage being held in storage accounts in multiple regions. In order to easily differentiate those resources that are dedicated to the new service from existing resources you could tag those resources as follows:

"service" : "ournewservice"

Using Microsoft's Power BI you can then connect to your Azure Enterprise billing data via the Azure Enterprise (Beta) Data Connection, see here.

The Power BI approach then enables reports and dashboards to be built that display Azure Resource costs per service, by grouping and filtering on resource tags. If you have multiple tag names, then Power BI gives you the ability to split these out into dedicated columns from which tag-specific reports can be built.

PowerShell Pea - Check Disk Usage on Remote Windows Servers using PowerShell

Found this useful script today that (given a list of Windows server names) will check the disk usage on each Windows Server (whose name is supplied in a file) and can also report results back via email.

It demonstrates an alternative to PowerShell remoting, if you want to build reports across a number of servers using a script and need to avoid labour-intensive checks of each server via RDP, for example.

Deploying Azure Virtual Networks with multiple CIDR addresses using ARM templates

I recently had to craft an ARM template in Azure for a Virtual Network that had an address space that wasn't sized as a "to the power of two".

i.e. this would be simple if you're creating a Vnet for an address space of say 256 addresses, such as

10.1.0.0 - 10.1.0.255 = 10.1.0.0/24

You can use a parameter of type "string" in the template, so assuming you arrange your parameters and template into 2 files, the parameters would include:
"vnetAddressPrefix": {
    "value": "10.1.0.0/24",
}
And the template itself would include the parameters and resource definitions for the Vnet as follows:
"vnetAddressPrefix": {
    "type": "string"
}
...
"resources": [ {
    "type": "Microsoft.Network/virtualNetworks",
    "name": "my256Vnet",
    "apiVersion": "2016-03-30",
    "location": "[resourceGroup().location]",
    "properties": {
        "addressSpace": {
            "addressPrefixes": [
                "[parameters('vnetAddressPrefix')]"
            ]
        },
        "subnets": [ {
            "name": "default",
            "properties": {
                "addressPrefix": "10.1.0.0/28"
            }
        }
        ]
    }
}
]

But, lets say you need a Vnet with 384 addresses, such as

10.1.0.0 - 10.1.1.127

Because you can't specify this range in a single CIDR address, you would instead need to create an address space in CIDR notation of

10.1.0.0/24

10.1.1.0/25

The simple "string" parameter used in the first example will not work here, instead you should use a parameter of type "array", modifying the parameter file to include the CIDR ranges as follows:

"vnetAddressPrefixes": {
    "value": [
            "10.1.0.0/24",
            "10.1.1.0/25"
    ]
}
And then changing your template file to include parameters and resource definitions as follows:
"vnetAddressPrefixes": {
    "type": "array"
}
...
"resources": [ {
    "apiVersion": "2016-03-30",
    "type": "Microsoft.Network/virtualNetworks",
    "name": "my384Vnet",
    "location": "[resourceGroup().location]",
    "properties": {
        "addressSpace": {
            "addressPrefixes": 
                "[parameters('vnetAddressPrefixes')]"
        },
        "subnets": [ {
            "name": "default",
            "properties": {
                "addressPrefix": "10.1.0.63/24"
            }
        }
        ]
    }
}
]

That's it! Although it's unlikely that a deploying Vnets in a greenfield environment would require this solution, it still may happen, and is more likely to happen in an environment where address space is already constrained by other Vnets and addressing conventions.

AWS Glue: The future of ETL is nigh

AWS will soon be launching a fully managed ETL as-a-service which promises to automate "data discovery, conversion, mapping, and job scheduling tasks" wherever your data lives, this is not just for your data stored in AWS.

AWS Glue, will provide the ETL layer that links AWS data services (S3, RDS and Redshift etc) to their and analytics services, such as Quicksight.

I for one am fascinated to see what it can do and discover real-world use-cases that can take advantage of AWS Glue, building it into the architecture of Big Data Cloud solutions.

Globally Unique Identifiers - Creating GUIDs in Powershell

Powershell Pea...

New-Guid

I've been working more and more with Powershell, specifically with Azure Automation and scripting and wanted to share a useful Cmdlet for creating GUIDs. This link provides more detail, but ultimately this command will provide you with a unique ID that can be assigned to arbitrary objects that require a unique ID.

Bear in mind that this doesn't protect you from duplicates, but the probability of them occurring is small. So if you need GUIDs for relatively small record/object sets and your application can handle the prospect of duplicates, then this function is a quick way to generate a "unique" ID.

See this link for more details on GUIDs/UUIDs.

Azure Premium Storage Blob Snapshot Error - 409 Conflict

I've recently been working in more detail with Azure blob snapshots and (by trial and error) discovered that Premium storage imposes limitations on the number and frequency of snapshots that can be performed on a single blob.

If you create a snapshot of a Premium storage blob and then another in quick succession (within 60 seconds of each other), then you might get a 409/Conflict error message.

This article suggests that there are two possibilities for this error, either

• SnapshotCountExceeded - You've exceeded the limit of 100 snapshots per blob, or
• SnaphotOperationRateExceeded - You've exceeded the limit of snapshots within a time window (stated as 10 minutes, but I observed this as closer to 1 minute)

Whilst it's unlikely that these limits would cause a problem in practice, it's something to keep in mind when developing/testing solutions that make use of the blob snapshot facility.

Azure Automation Troubles

A recent venture into Azure Automation threw up some unexpected problems.

A simple script, run locally, that called Get-AzureRmVM was working to my subscription, but the same script did not work within an Azure Runbook.

I'd created the AzureRunAsConnection by default with the Azure Automation account and followed the same code as given in the AzureAutomationTutorialScript powershell runbook to login to the subscription and execute Cmdlets.

The problem came when executing the Get-AzureRmVM Cmdlet in the runbook, receiving the error:

Get-AzureRmVM : Run Login-AzureRmAccount to login.
+     $VM = Get-AzureRmVM | ? {$_.Name -eq $VMName}
+           ~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Get-AzureRmVM], PSInvalidOperationException
    + FullyQualifiedErrorId : InvalidOperation,Microsoft.Azure.Commands.Compute.GetAzureVMCommand

After several googles I found this post...
https://feedback.azure.com/forums/246290-automation/suggestions/16590229-get-azurermvm-failing-in-azure-automation

Therefore, if anyone else sees errors when running basic Cmdlets in a runbook (that work fine from PowerShell), try updating all the modules in the Automation account using a script like this one...
https://www.powershellgallery.com/packages/Update-ModulesInAutomationToLatestVersion/1.03/DisplayScript
(Note that the deploy to Azure button didn't work for me, so I had to import the script and run manually.)

Note also that (depending on when you created your Automation Account) the above may, or may not work, and you may have to create a new Automation Account and update the modules before starting to use the account. Microsoft suggest setting the script to run on a schedule in the automation account to keep the modules up to date.

Azure Pea of the day

For those of you who care about the aesthetics of their Azure cloud experience, I accidentally discovered how to quickly change your portal theme...

Simply double-click on the dashboard background and it will cycle through the available themes.

Run your own Gitlab server on Azure

For those of you out there wanting to host and control your own git server in the cloud, the Azure marketplace has a Gitlab Community Edition VM ready to deploy, see here. Of course, gitlab.com takes away the need to host and removes the cost involved, but if you want control over where your repository is located and the underlying VM, then this looks like a good option.

I've spun this up using the newly available A1_V2 VM size, which handles light load amicably. So for less then £20 you can have control over all those features that the community edition offers, including SSH and HTTPS access.

The gitlab CE documentation is a little outdated, but still good enough to help you configure the instance and get going with your own git repo.

Using environment variables to create a generic WebJob script in Azure

I recently had to write a script for a WebJob in an Azure App Service Web App which was reused across multiple copies of the same Web App.

The script was simple, it simply needed to make a request to a scripted action (in my case PHP) within the website hosted by the Web App. The request was via a URL, which was dependent upon the name of the web app, e.g.if the web app name is "mycoolwebapp" then the URL of the scripted action would be

https://mycoolwebapp.azurewebsites.net/bin/mytask.php

Instead of writing different scripts each time, with the web app name hardcoded, it's possible to use environment variables from within the WebJob script to refer to the Web App environment variables. In this case, "mycoolwebapp.azurewebsites.net" can be referred to via the WEBSITE_HOSTNAME environment variable in a CMD script as follows

curl https://%WEBSITE_HOSTNAME%/bin/mytask.php

This single script can then be uploaded to multiple Web Jobs for your websites, without needing a different version of the script for every website.