turbot/databricks
steampipe plugin install databricks

Table: databricks_ml_model - Query Databricks ML Models using SQL

Databricks ML Models is a feature within Databricks that allows users to manage the full lifecycle of machine learning models. It offers capabilities to track, compare, and visualize machine learning experiments. It also enables the deployment of machine learning models in a consistent and reproducible manner.

Table Usage Guide

The databricks_ml_model table provides insights into ML models within Databricks. As a data scientist or machine learning engineer, you can explore model-specific details through this table, including the model name, version, creation timestamp, and user ID of the creator. Utilize it to track and manage the lifecycle of your machine learning models, compare different models, and ensure reproducibility of your experiments.

Examples

Basic info

Explore the creation and update details of machine learning models on Databricks. This can help you assess the activity and usage of models over time, providing insights into your data science operations.

select
name,
creation_timestamp,
description,
last_updated_timestamp,
user_id,
account_id
from
databricks_ml_model;
select
name,
creation_timestamp,
description,
last_updated_timestamp,
user_id,
account_id
from
databricks_ml_model;

List models modified in the last 7 days

Discover the modifications made to machine learning models in the past week. This is useful for keeping track of recent changes and understanding how your models are evolving over time.

select
name,
creation_timestamp,
description,
last_updated_timestamp,
user_id,
account_id
from
databricks_ml_model
where
last_updated_timestamp > now() - interval '7' day;
select
name,
creation_timestamp,
description,
last_updated_timestamp,
user_id,
account_id
from
databricks_ml_model
where
last_updated_timestamp > datetime('now', '-7 day');

Get users permission level for each model

Explore which users have certain permission levels for each machine learning model in your Databricks environment. This can help ensure appropriate access rights are maintained across your team.

select
name,
user_id,
permission_level,
account_id
from
databricks_ml_model;
select
name,
user_id,
permission_level,
account_id
from
databricks_ml_model;

List all models with a specific permission level

Explore which machine learning models within your Databricks account have a specific permission level. This could be beneficial in managing access control and understanding who has the ability to manage certain models.

select
name,
user_id,
permission_level,
account_id
from
databricks_ml_model
where
permission_level = 'CAN_MANAGE';
select
name,
user_id,
permission_level,
account_id
from
databricks_ml_model
where
permission_level = 'CAN_MANAGE';

List details of version for each model

Determine the version details for each machine learning model in your Databricks environment, including its creation time, status, and source. This can be useful for tracking model evolution and understanding the current stage of each model version.

select
name,
user_id,
permission_level,
account_id,
v ->> 'Version' as version,
v ->> 'CreationTimestamp' as creation_time,
v ->> 'Name' as version_name,
v ->> 'RunId' as run_id,
v ->> 'Status' as version_status,
v ->> 'UserId' as user_id,
v ->> 'Source' as version_source,
v ->> 'CurrentStage' as current_version_stage
from
databricks_ml_model,
jsonb_array_elements(latest_versions) as v;
select
name,
user_id,
permission_level,
account_id,
json_extract(v.value, '$.Version') as version,
json_extract(v.value, '$.CreationTimestamp') as creation_time,
json_extract(v.value, '$.Name') as version_name,
json_extract(v.value, '$.RunId') as run_id,
json_extract(v.value, '$.Status') as version_status,
json_extract(v.value, '$.UserId') as user_id,
json_extract(v.value, '$.Source') as version_source,
json_extract(v.value, '$.CurrentStage') as current_version_stage
from
databricks_ml_model,
json_each(latest_versions) as v;

Schema for databricks_ml_model

NameTypeOperatorsDescription
_ctxjsonbSteampipe context in JSON form.
account_idtext=, !=, ~~, ~~*, !~~, !~~*The Databricks Account ID in which the resource is located.
creation_timestamptimestamp with time zoneTimestamp recorded when this model was created.
descriptiontextDescription of the model.
last_updated_timestamptimestamp with time zoneTimestamp recorded when this model was last updated.
latest_versionsjsonbCollection of latest model versions for each stage.
nametext=Unique name for the model.
permission_leveltextPermission level of the requesting user on the object.
sp_connection_nametext=, !=, ~~, ~~*, !~~, !~~*Steampipe connection name.
sp_ctxjsonbSteampipe context in JSON form.
tagsjsonbAdditional metadata key-value pairs for this model.
titletextThe title of the resource.
user_idtextUser ID of the user who created this model.

Export

This table is available as a standalone Exporter CLI. Steampipe exporters are stand-alone binaries that allow you to extract data using Steampipe plugins without a database.

You can download the tarball for your platform from the Releases page, but it is simplest to install them with the steampipe_export_installer.sh script:

/bin/sh -c "$(curl -fsSL https://steampipe.io/install/export.sh)" -- databricks

You can pass the configuration to the command with the --config argument:

steampipe_export_databricks --config '<your_config>' databricks_ml_model