heimdall808-blog

[heimdall808-blog]

Setting up Key Vault secured Pipelines with Azure DevOps

One of the first things you need to do when setting up a new Azure DevOps pipeline is to pull secrets from Azure Key Vault into a Variable Group. Variable Groups can be used by your project to setup how things like Terraform in your pipelines will authenticate themselves with Azure, AWS, or GCP.

By default, when setting up a Variable Group, you can manually add them. The problem with this is…

[heimdall808-blog]

You will find that CosmosDB Emulator will fail, randomly, for no apparent reason doing simple things like get an instance of a container or create the database. In the world of cloud, it’s important to handle Transient Faults, or errors that are not repeatable or consistent in when the appear.

They might look like this:

Business.Tests.BuildingTests.CreateNewBuilding [3s 650ms]

Error Message:

Microsoft.Azure.Cosmos.CosmosException : Response status code does not indicate success: 500 Substatus: 0 Reason: (Microsoft.Azure.Documents.DocumentClientException: Unknown server error occurred when processing this request.ActivityId: b55227fb-15d1-4506-a685-c6b7751271c3, Microsoft.Azure.Documents.Common/2.9.2, {"RequestStartTimeUtc":"2020-03-22T04:41:43.6841102Z","RequestEndTimeUtc":"2020-03-22T04:41:44.0570433Z","RequestLatency":"00:00:00.3729331","IsCpuOverloaded":false,"NumberRegionsAttempted":1,"ResponseStatisticsList":[],"AddressResolutionStatistics":[{"StartTime":"2020-03-22T04:41:43.6842253Z","EndTime":"2020-03-22T04:41:44.0570433Z","TargetEndpoint":"https://192.168.231.161:8081/dbs/mydb/colls"}],"SupplementalResponseStatistics":[],"FailedReplicas":[],"RegionsContacted":[],"ContactedReplicas":[]}, Windows/10.0.14393 cosmos-netstandard-sdk/3.4.2 at Microsoft.Azure.Cosmos.GatewayStoreClient.ParseResponseAsync(HttpResponseMessage responseMessage, JsonSerializerSettings serializerSettings, DocumentServiceRequest request) at Microsoft.Azure.Cosmos.GatewayStoreClient.InvokeAsync(DocumentServiceRequest request, ResourceType resourceType, Uri physicalAddress, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.GatewayStoreModel.ProcessMessageAsync(DocumentServiceRequest request, CancellationToken cancellationToken) at Microsoft.Azure.Cosmos.Handlers.TransportHandler.SendAsync(RequestMessage request, CancellationToken cancellationToken)).

Stack Trace:

at Microsoft.Azure.Cosmos.ResponseMessage.EnsureSuccessStatusCode()

at Microsoft.Azure.Cosmos.CosmosResponseFactory.ProcessMessageAsync[T](Task`1 cosmosResponseTask, Func`2 createResponse)

at Microsoft.Azure.Cosmos.DatabaseCore.CreateContainerIfNotExistsAsync(ContainerProperties containerProperties, Nullable`1 throughput, RequestOptions requestOptions, CancellationToken cancellationToken)

at Common.DataAccess.BaseDataAccess.GetContainerAsync() in D:a1sCommonDataAccessBaseDataAccess.cs:line 27

at DataAccess.BaseCrudDataAccess`1.CreateAsync(T entity) in D:a1sCommonDataAccessBaseCrudDataAccess.cs:line 24

at Business.BuildingRepository.CreateAsync(BuildingDetail entity) in D:a1sLocation.BusinessBuildingRepository.cs:line 88

at Business.Tests.BuildingTests.CreateNewBuilding() in D:a1sLocation.Business.TestsBuildingTests.cs:line 26

I’ve done that using some simple retry logic that I found from a great post on Stack Overflow that I modified to include async operations (DoAsync and DoAsync<T> where T is the return type of the async method).

Example usage, in he cosmos DB setting is this:

Just adding some simple re-try logic around my Cosmos DB data access code drastically improved the reliability of automated tests.

Improving CosmosDB Test Automation Reliability with Retry Logic You will find that CosmosDB Emulator will fail, randomly, for no apparent reason doing simple things like get an instance of a container or create the database.

[heimdall808-blog]

Improving CosmosDB Test Automation Reliability using Simple Retry Logic

You will find that CosmosDB Emulator will fail, randomly, for no apparent reason doing simple things like get an instance of a container or create the database. In the world of cloud, it’s important to handle Transient Faults, or errors that are not repeatable or consistent in when the appear.

They might look like this:

Business.Tests.BuildingTests.CreateNewBuilding [3s 650ms]

Error Message:

Micr…

[heimdall808-blog]

The cosmos DB Emulator is a custom action available on the Azure DevOps portal, however, that doesn’t exactly make it turnkey to use. The custom task will spin up a container running Cosmos DB, however it does so with a specific local DNS / port that you need to pipe into your test running.

There is documentation but it’s a bit dated.

It will output a pipeline variable called "$(CosmosDbEmulator.Endpoint)". This isn’t exactly clear from the output variables GUI within the task editor. The trick is getting this into a variable that you future tasks can use. I prefer using Environment Variables over Test Runner configuration files as the dated documentation from Microsoft advises.

Those of you familiar with Azure DevOps will know that it’s a bit tricky working with Environment Variables. Some tasks support them, some don’t. Unfortunately the DotNetCoreCLI task type, you know the one that executes my .NET Core Xunit tests does not support environment variables. Therefore I have to ensure the environment variable is made available before that task is executed.

I do that in my "Setup Cosmos DB Endpoint" task. Which is simply there to grab the pipeline variable and set it up as an Environment Variable so that future steps can use it.

The data flow is like this:

$(CosmosDbEmulator.Endpoint) –> EMULATOR_ENDPOINT –> COSMOS_DB_ENDPOINT

I’m jamming in an Environment Variable using an Environment Variable? Weird, I know right? But that’s because how the Environment Variables work within Tasks. If I pipe in the pipeline variable $(CosmosDbEmulator.Endpoint) as an Environment Variable "EMULATOR_ENDPOINT" then it is only available within the scope of that task!

I have to use some special syntax to ensure I setup an Environment Variable that will be available across all tasks (going forward). That’s where this comes into play:

echo Task.setvariable

echo ##vso[task.setvariable variable=COSMOS_DB_ENDPOINT]%EMULATOR_ENDPOINT%

When we analyze the Azure DevOps pipeline execution we see what’s happening:

image005

image006

Notice that during the Setup of the CosmosDB Endpoint the values look like this:

$(CosmosDbEmulator.Endpoint) https://6f99b2565caf:8081/ EMULATOR_ENDPOINT https://6f99b2565caf:8081/ COSMOS_DB_ENDPOINT NULL

Now we move onto the confirmation step…

Notice in the next step things have changed:

$(CosmosDbEmulator.Endpoint) https://6f99b2565caf:8081/ EMULATOR_ENDPOINT NULL COSMOS_DB_ENDPOINT https://6f99b2565caf:8081/

Notice that the change I requested to COSMOS_DB_ENDPOINT does NOT take into effect in the CURRENT task where the value is set. This can be very perplexing but it is available in all subsequent steps.

You might be asking, why not just use the pipeline variable. The answer is simple. This whole approach, is designed to make the Unit Tests and Unit Test configuration SIMPLER. Therefore, the unit tests should not have to worry about where or how the CosmosDB Endpoint URL gets into the "COSMOS_DB_ENDPOINT" Environment Variable it will work the same way. Developers can set the variable themselves on their environment if they want to use a different endpoint for testing, or you can do NULL checks in your Unit Test and use the local emulator settings when running the tests locally.

Now your test running will execute and connect to a REAL Cosmos DB server without issue. Isn’t that wonderful?! Yes and no.

Unfortunately, CosmosDB takes a hella long time to load.

Like 10 bloody minutes long.

The conspiracy theorist in me might suspect that this was an obnoxious attempt by Microsoft to increase the build minutes incurred on Azure DevOps to drive an increase in build minutes per month or parallel builds unit which of course are the main monetization strategy. But it’s a new Microsoft, right? Maybe they will fix the Cosmos DB emulator so it isn’t so ridiculously long to boot….a dev can dream, no?

DISCLAIMER ========== This e-mail may contain privileged and confidential information which is the property of Persistent Systems Ltd. It is intended only for the use of the individual or entity to which it is addressed. If you are not the intended recipient, you are not authorized to read, retain, copy, print, distribute or use this message. If you have received this communication in error, please notify the sender and delete all copies of this message. Persistent Systems Ltd. does not accept any liability for virus infected mails.

Cosmos DB Emulator on Azure DevOps The cosmos DB Emulator is a custom action available on the Azure DevOps portal, however, that doesn't exactly make it turnkey to use.

[heimdall808-blog]

CosmosDB Emulator on Azure DevOps

The cosmos DB Emulator is a custom action available on the Azure DevOps portal, however, that doesn’t exactly make it turnkey to use. The custom task will spin up a container running Cosmos DB, however it does so with a specific local DNS / port that you need to pipe into your test running.

There is documentation but it’s a bit dated.

It will output a pipeline variable called…

[heimdall808-blog]

Cosmos DB Emulator on Azure DevOps

The cosmos DB Emulator is a custom action available on the Azure DevOps portal, however, that doesn’t exactly make it turnkey to use. The custom task will spin up a container running Cosmos DB, however it does so with a specific local DNS / port that you need to pipe into your test running.

There is documentation but, in true Microsoft fashion, it’s a bit dated. :o)

It will output a pipeline…

[heimdall808-blog]

Azure Functions + EventGrid Subscriptions With Terraform

Azure Functions + EventGrid Subscriptions With Terraform

I think I figured it out. I have been working with Serverless Microservices Architecture on Azure for quite some time. I’ve been trying to leverage Terraform to effectively manage the extensive configuration of PaaS services on Azure. Due to immaturity of the Terraform AzureRM provider I’ve encountered well-documented challenges. This is the last major hurdle that I’ve overcome.

Thanks to this…

[heimdall808-blog]

Using Terraform to Provision Microservices with Azure API Management Backed by Azure Functions

Using Terraform to Provision Microservices with Azure API Management Backed by Azure Functions

So I’ve talked about the fact that the Azure Function resource in the AzureRM provider does not support the exporting of the Key necessary to integrate Azure Functions with Azure API Management. However, I have recently employed a work around, whereby you are able to export the Azure Function Key using the Resource Group Template Terraform Resource (i.e. azurerm_template_deployment).

To make it…

[heimdall808-blog]

Azure API Management Backend Management with PowerShell

Azure API Management Backend Management with PowerShell

If you are using API Management you Might have noticed that there is no Azure Portal Access to view the Backends that are created automatically when you import you API from wherever. You might have also noticed it takes a ridiculously long time to provision and API Management resource. Therefore, destroying it completely and re-creating is less desirable than just cleaning up after yourself.

I…

[heimdall808-blog]

Terraform + Azure DevOps Environment Variable Piping Tips

Terraform + Azure DevOps Environment Variable Piping Tips

When establishing good security processes around your software release pipeline it’s important to ensure that secrets are handled with care. As a best practice, I always ensure secrets that need to be used in a CI / CD pipeline are stored in Key Vault and imported into an Azure DevOps Variable Group. This allows the secrets to be reused across multiple pipelines and ensures they are stored and…

[heimdall808-blog]

Azure Serverless Architecture Terraform Explained

Azure Serverless Architecture Terraform Explained

Here is a higher level view of the architecture and all the things that Terraform is provisioning. Based on my previous post, we know that certain things still require manual intervention but Terraform can provision a lot of stuff:

Resource Group

Event Grid Topics (but no Topics Subscriptions due to Function Access Key limitation)

CosmosDB Account (but not databases or collections due to…

[heimdall808-blog]

We’ve been exercising the AzureRM and AzureAD Terraform providers with a healthcare client who wants to go serverless with a new product they are building. We’ve been able to automate a significant portion of the infrastructure with relative ease, however, there are limitations to both the Terraform providers and the operational models of Azure Functions that are causing some pain.

The objectives of our architecture are:

Store all secrets in KeyVault and apply least privledge access control around secrets

Codify as much of the environment configuration in Terraform to minimize additional post-deployment manual effort

Employ a Microservices Architecture utilizing Azure Serverless & PaaS Offerings (Azure Functions, EventGrid and CosmosDB) to minimize development time, run cost and operational overhead.

Terraform ‘AzureRM’ Provider Issues

The AzureRM provider for Terraform boasts a large number of resources, unfortunately, we’ve found that many of these are incomplete or lack basic documentation required to quickly get up and running that it’s older and more actively developed, peer, the AWS provider, benefits from.

Event Subscriptions for EventGrid Triggered Azure Functions must be created manually

Creating a simple EventGrid trigger will produce some very simplistic code. The missing piece of this puzzle is establishing an EventGrid subscription for this particular Azure Function. This is done by using a WebHook.

You’ll need to create an Event Grid Subscription. From the portal we can gain some insight into this process. The Azure Functions portal interface automatically detects if your Function is using EventGridTrigger and will create this hand shortcut to kick off the process.

Notice this ‘Endpoint Details’ section down at the bottom.

Taking a closer look at the webhook this is generating we can see that the WebHook URL is embedding a code that EventGrid will use to authenticate itself with the Azure Function.

When you look at the ‘azurerm_eventgrid_event_subscription’ resource and see the ‘webhook_endpoint’ property you might feel a rush of excitement! However, alas, the problem is not with the Event Subscription resource but with the Azure Function resource ‘azurerm_function_app’. The ‘code’ parameter in the above WebHook Endpoint URL is the same for all functions in this Function App. However, the Function App resource does not output the master key. There is also an open issue / feature request (#699) but it doesn’t appear to be on the ‘short term roadmap’. There is a shim / work around using a ‘azurerm_template_deployment’ but building JSON text in a Terraform file feels very hacky.

CosmosDB Database and Collections must be created manually.

This is because the ‘azurerm_cosmosdb_sql_database’ resource does not support setting ‘shared throughput’. Normally, when you go and create a Cosmos DB SQL Database you are able to specify whether the Throughput Units are shared across the collections in the database or not.

#gallery-0-5 { margin: auto; } #gallery-0-5 .gallery-item { float: left; margin-top: 10px; text-align: center; width: 33%; } #gallery-0-5 img { border: 2px solid #cfcfcf; } #gallery-0-5 .gallery-caption { margin-left: 0; } /* see gallery_shortcode() in wp-includes/media.php */

0cd9fa7a136b4cb78c4e01bbb6eb8668

72c51a1e39fd4ed6bc067ca1dd964d85

If you don’t select this option then every collection you create will have its own dedicated Throughput Units (RUs) which will cost you a staggering amount of money! In a Dev/Test environment, this is clearly not needed and should be avoided.

There is an open issue in GitHub (#3623) that has been open since June 2019. There appears to be a pull request (#4616) that promises a fix so hopefully our pain is temporary.

Serverless Infrastructure Automation on Azure using Terraform We’ve been exercising the AzureRM and AzureAD Terraform providers with a healthcare client who wants to go serverless with a new product they are building.

[heimdall808-blog]

Azure Speech API is game changer

Azure Speech API is game changer

For advanced speech based use cases never before has so much power been put in the hands of developers. Whether you are transcribing customer calls to improve your customer service or using chat bots to create new speech based interfaces with your employees the Azure Speech API has a sophisticated feature set that opens up a flurry of new opportunities in enabling the now burgeoning user input…

[heimdall808-blog]

Configuring AzureAD provider in Terraform to manage App Registrations

Configuring AzureAD provider in Terraform to manage App Registrations

Automating the provisioning of Application Registrations(i.e. App Registrations) can be a very useful way of organizing your application’s Infrastructure as Code. Keeping track of all these disparate app registrations, who they belong to, what they are for, can be very tedious and challenging. It can get messy very quickly as well and a wary Azure AD administrator might be tempted to delete…

[heimdall808-blog]

Configuring AzureAD provider in Terraform to manage App Registrations

Configuring AzureAD provider in Terraform to manage App Registrations

Automating the provisioning of Application Registrations(i.e. App Registrations) can be a very useful way of organizing your application’s Infrastructure as Code. Keeping track of all these disparate app registrations, who they belong to, what they are for, can be very tedious and challenging. It can get messy very quickly as well and a wary Azure AD administrator might be tempted to delete…

[heimdall808-blog]

I guess my expectations are a bit high that when I install Visual Studio 2019 to include cross-platform mobile development (Xamarin.Forms) it would properly configure itself. It turns out that when you try and get UI Test Automation working with Xamarin.Forms that is just not the case.

Let’s peel this onion.

Missing ANDROID_HOME environment variable.

For me the environment variable needed to be set to this:

ANDROID_HOME C:Program Files (x86)Androidandroid-sdk

#gallery-0-5 { margin: auto; } #gallery-0-5 .gallery-item { float: left; margin-top: 10px; text-align: center; width: 33%; } #gallery-0-5 img { border: 2px solid #cfcfcf; } #gallery-0-5 .gallery-caption { margin-left: 0; } /* see gallery_shortcode() in wp-includes/media.php */

1286add3ba8242b085d2c52be6db1a65

9c337a17625341a5ab0945bc64ed4e84

f6ef084017274a53babd477b9690e800

SetUp : System.Exception : Android SDK not found. Please install it and if it is still not located, please set the ANDROID_HOME environment variable to point to the directory.

Searched locations:

Windows Registry

(No path) – Not set. [ Source: ANDROID_HOME ]

Missing JAVA_HOME environment variable.

For me the environment variable had to be set to this:

JAVA_HOME C:Program FilesAndroidjdkmicrosoft_dist_openjdk_1.8.0.25bin

Manually specify the APK location.

This is probably more of a weakness in the Xamarin UI Test Framework rather than Xamarin SDK Setup. You just need to explicitly specify where the APK is for your Android app. Kinda lame but workable.

Too many Android Devices.

This is pretty reasonable. The test automation framework needs to know where to run your tests. In my case, I have a physical phone attached via USB and a simulator. The trick is figuring out what the device serial is. Luckily the device serial numbers are output in the error message. What a pleasant surprise! An actual useful error message!

SetUp : System.Exception : Found 2 connected Android devices. Either only have 1 connected or select one using DeviceSerial during configuration. Devices found: <s1>, <s2>

Xamarin UI Test Automation Android Setup I guess my expectations are a bit high that when I install Visual Studio 2019 to include cross-platform mobile development (Xamarin.Forms) it would properly configure itself.

[heimdall808-blog]

Plex Server Setup Part II: Making a Filesystem

Plex Server Setup Part II: Making a Filesystem

After the clean install of ubuntu, I only had a 4GB OS partition setup for me. This partition lived on a single physical volume (PV), volume group (VG) and logical volume (LV). These are important concepts to understand Linux filesystem structures.

The first step in expanding my storage is to create a separate logical volume within my volume group. Create the logical volume and verify its…