Table: databricks_ml_model - Query Databricks ML Models using SQL
Databricks ML Models is a feature within Databricks that allows users to manage the full lifecycle of machine learning models. It offers capabilities to track, compare, and visualize machine learning experiments. It also enables the deployment of machine learning models in a consistent and reproducible manner.
Table Usage Guide
The databricks_ml_model
table provides insights into ML models within Databricks. As a data scientist or machine learning engineer, you can explore model-specific details through this table, including the model name, version, creation timestamp, and user ID of the creator. Utilize it to track and manage the lifecycle of your machine learning models, compare different models, and ensure reproducibility of your experiments.
Examples
Basic info
Explore the creation and update details of machine learning models on Databricks. This can help you assess the activity and usage of models over time, providing insights into your data science operations.
select name, creation_timestamp, description, last_updated_timestamp, user_id, account_idfrom databricks_ml_model;
select name, creation_timestamp, description, last_updated_timestamp, user_id, account_idfrom databricks_ml_model;
List models modified in the last 7 days
Discover the modifications made to machine learning models in the past week. This is useful for keeping track of recent changes and understanding how your models are evolving over time.
select name, creation_timestamp, description, last_updated_timestamp, user_id, account_idfrom databricks_ml_modelwhere last_updated_timestamp > now() - interval '7' day;
select name, creation_timestamp, description, last_updated_timestamp, user_id, account_idfrom databricks_ml_modelwhere last_updated_timestamp > datetime('now', '-7 day');
Get users permission level for each model
Explore which users have certain permission levels for each machine learning model in your Databricks environment. This can help ensure appropriate access rights are maintained across your team.
select name, user_id, permission_level, account_idfrom databricks_ml_model;
select name, user_id, permission_level, account_idfrom databricks_ml_model;
List all models with a specific permission level
Explore which machine learning models within your Databricks account have a specific permission level. This could be beneficial in managing access control and understanding who has the ability to manage certain models.
select name, user_id, permission_level, account_idfrom databricks_ml_modelwhere permission_level = 'CAN_MANAGE';
select name, user_id, permission_level, account_idfrom databricks_ml_modelwhere permission_level = 'CAN_MANAGE';
List details of version for each model
Determine the version details for each machine learning model in your Databricks environment, including its creation time, status, and source. This can be useful for tracking model evolution and understanding the current stage of each model version.
select name, user_id, permission_level, account_id, v ->> 'Version' as version, v ->> 'CreationTimestamp' as creation_time, v ->> 'Name' as version_name, v ->> 'RunId' as run_id, v ->> 'Status' as version_status, v ->> 'UserId' as user_id, v ->> 'Source' as version_source, v ->> 'CurrentStage' as current_version_stagefrom databricks_ml_model, jsonb_array_elements(latest_versions) as v;
select name, user_id, permission_level, account_id, json_extract(v.value, '$.Version') as version, json_extract(v.value, '$.CreationTimestamp') as creation_time, json_extract(v.value, '$.Name') as version_name, json_extract(v.value, '$.RunId') as run_id, json_extract(v.value, '$.Status') as version_status, json_extract(v.value, '$.UserId') as user_id, json_extract(v.value, '$.Source') as version_source, json_extract(v.value, '$.CurrentStage') as current_version_stagefrom databricks_ml_model, json_each(latest_versions) as v;
Schema for databricks_ml_model
Name | Type | Operators | Description |
---|---|---|---|
_ctx | jsonb | Steampipe context in JSON form. | |
account_id | text | =, !=, ~~, ~~*, !~~, !~~* | The Databricks Account ID in which the resource is located. |
creation_timestamp | timestamp with time zone | Timestamp recorded when this model was created. | |
description | text | Description of the model. | |
last_updated_timestamp | timestamp with time zone | Timestamp recorded when this model was last updated. | |
latest_versions | jsonb | Collection of latest model versions for each stage. | |
name | text | = | Unique name for the model. |
permission_level | text | Permission level of the requesting user on the object. | |
sp_connection_name | text | =, !=, ~~, ~~*, !~~, !~~* | Steampipe connection name. |
sp_ctx | jsonb | Steampipe context in JSON form. | |
tags | jsonb | Additional metadata key-value pairs for this model. | |
title | text | The title of the resource. | |
user_id | text | User ID of the user who created this model. |
Export
This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.
You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh
script:
/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- databricks
You can pass the configuration to the command with the --config
argument:
steampipe_export_databricks --config '<your_config>' databricks_ml_model