Import a directory into Databricks using the Workspace API in Python

Another fairly easy thing that I couldn't find in the docs. I wanted to be able to upload a directory into my Databricks Workspace from my CI server so I could test the current branch.

Luckily enough, the databricks-cli library was written in Python, so we can just use that. But first you'll need to generate a token for yourself to use in the API. Of course, you need to follow the instructions to be able to use the API in the first place, but from there it's pretty straightforward.

from databricks_cli.workspace.api import WorkspaceApi  
from databricks_cli.sdk.api_client import ApiClient


client = ApiClient(  
    host='https://your.databricks-url.net',
    token=api_key
)
workspace_api = WorkspaceApi(client)  
workspace_api.import_workspace_dir(  
    source_path=base_path,
    target_path="/Users/user@example.com/MyFolder",
    overwrite=True,
    exclude_hidden_files=True
)