Steven Wright Blog

We have moved our infrastructure to the Azure cloud, as mentioned in previous posts .  With this move, we also started the practice of infrastructure as code.  Our preferred method of deploying our infrastructure is using Terraform.  Recently I updated our Terraform code to use the Terraform Databrick provider to automate the creation of the clusters and not just the Azure Databrick Workspace.  This post covers some of my lessons learned while deploying our Databricks solution with Terraform. Configuring the Databricks provider When running the apply action of our terraform code, the first problem was trying to configure the Databricks provider and start the deployment of the cluster resource before Azure Databrick Workspace was fully operational.  Even with the implied and explicitly dependence on the azurerm_databricks_workspace resource. My method to resolve this early start off configuring Databricks provider objects was to use the data.azurerm_databricks_workspace object.  Unfortu

If you are using Octopus Deploy to automate the deployment of solution(s) and part of your solution is using Databricks, then this post might interest you. We are using Databrick within my current project, and due to tight security controls, we cannot link our workspace to our git repository. We were manual importing our Databricks workbooks as our method of deployment. Until we automate the process using Octopus Deploy and the newly created community step template: Import Databricks Workbooks. Currently, the community step template only imports workbooks with the ‘.ipynb’or ‘.scala’ file extension and only communicates to the DataBricks instance using HTTPS. The task uses the DataBricks rest API v 2.0 to update the DataBricks instance, so there is no requirement to install any extra supporting software. The template only requires the parameters: DataBricks Workbook Package: The package of the Databricks workbooks for deployment. Databricks Instance Uri: The hostname of the Databrick

This post covers one of the challenges we face when moving our infrastructure from on-prem to Microsoft Azure: Moving SQL Server data from on-premise to Microsoft SQL Server Paas. Moving our data was a challenge for the following reasons: Our company has tight controls and rules about what software/connectivity is allowed on our network The database that we need to migrate after compression was over 100GB in size. No information about how long other migration processes would take. As needed to ensure our migration was complete before the next Monday on the weekend of our migration. For most people migrating their data, the best option is to use Azure Database Migration Service , especially when transferring a database size that we were. We couldn't use the Azure Database Migration Service for the following reasons: Azure Database Migration Service requires software installed on a server within our network Connections to both databases

As mention in a previous post, we are using Azure SQL Server IaaS to help our transition to the cloud. The main reason for this is how some of our SSIS projects, which we inherited, were created. Some were using SSIS to manage the transactions, which uses MSDTC not supported in Azure. Use extra .Net libraries to read JSON and office files; external programs to do file operations like decompression. Migration timelines and priorities we didn't have time to refactor these projects. While ago we started to deploy our SSIS projects to our Azure SQL Server IaaS virtual machine, a Windows 2019 server with SQL Server 2019, to get the following error: SSIS Deployment Error 6522 "A required privilege is not held by the client". We able to resolve our problem by using the steps we found in the following blog posts: System.ComponentModel.Win32Exception: A required privilege is not held by the client while Deploying SSIS Project | Microsoft Docs . Fixing SQL 2012 SSIS Deployment Err