Steven Wright Blog

We have moved our infrastructure to the Azure cloud, as mentioned in previous posts .  With this move, we also started the practice of infrastructure as code.  Our preferred method of deploying our infrastructure is using Terraform.  Recently I updated our Terraform code to use the Terraform Databrick provider to automate the creation of the clusters and not just the Azure Databrick Workspace.  This post covers some of my lessons learned while deploying our Databricks solution with Terraform. Configuring the Databricks provider When running the apply action of our terraform code, the first problem was trying to configure the Databricks provider and start the deployment of the cluster resource before Azure Databrick Workspace was fully operational.  Even with the implied and explicitly dependence on the azurerm_databricks_workspace resource. My method to resolve this early start off configuring Databricks provider objects was to use the data.azurerm_databricks_wo...

If you are using Octopus Deploy to automate the deployment of solution(s) and part of your solution is using Databricks, then this post might interest you. We are using Databrick within my current project, and due to tight security controls, we cannot link our workspace to our git repository. We were manual importing our Databricks workbooks as our method of deployment. Until we automate the process using Octopus Deploy and the newly created community step template: Import Databricks Workbooks. Currently, the community step template only imports workbooks with the ‘.ipynb’or ‘.scala’ file extension and only communicates to the DataBricks instance using HTTPS. The task uses the DataBricks rest API v 2.0 to update the DataBricks instance, so there is no requirement to install any extra supporting software. The template only requires the parameters: DataBricks Workbook Package: The package of the Databricks workbooks for deployment. Databricks Instance Uri: The hostname of the Databrick...