- 23405 Views
- 6 replies
- 6 kudos
Resolved! Access the environment variable from the custom container base cluster
Hi Databricks Community, I want to set environment variables for all clusters in my workspace. The goal is to the have environment variable, available in all notebooks executed on the cluster.The environment variable is generated in global init scrip...
- 23405 Views
- 6 replies
- 6 kudos
- 6 kudos
Thanks @Lukasz Lu​ - that worked for me as well. When I used the following script:#!/bin/bash echo MY_TEST_VAR=value1 | tee -a /etc/environment >> /databricks/spark/conf/spark-env.shfor non-docker clusters, MY_TEST_VAR shows up twice in ` /databrick...
- 6 kudos
- 2378 Views
- 2 replies
- 1 kudos
Resolved! Using Datbricks Connect with serverless compute and MLflow
Hi all,I have been using databricks-connect with serverless compute to develop and debug my databricks related code. It worked great so far. Now I started integrating ML-Flow in my workflow, and I am encountering an issue. When I run the following co...
- 2378 Views
- 2 replies
- 1 kudos
- 1 kudos
The error you are encountering, pyspark.errors.exceptions.connect.AnalysisException: [CONFIG_NOT_AVAILABLE] Configuration spark.mlflow.modelRegistryUri is not available. SQLSTATE: 42K0I, is a known issue when using MLflow with serverless clusters in ...
- 1 kudos
- 34 Views
- 0 replies
- 0 kudos
Custom Multi-agent deployment error
Hi. I am deploying a custom multi-agent system comprising of a genie agent and a RAG solution. While deploying, I am getting the following error:I am using 16.1 ML (Node: Standard_D4ads_v5 16GB,4 core) cluster and I am using the following code for lo...
- 34 Views
- 0 replies
- 0 kudos
- 253 Views
- 3 replies
- 0 kudos
Error when uploading MLFlow artifacts to DBFS
Hi everyone,I'm attempting to use MLFlow experiment tracking from a local machine, but I'm encountering difficulties in uploading artifacts.I've tried a sample code as simple as the following.import mlflow import os os.environ["DATABRICKS_HOST"] = "...
- 253 Views
- 3 replies
- 0 kudos
- 0 kudos
It is considered best practice not to store any production data or assets in DBFS (Databricks File System). The primary reason is that DBFS does not provide robust security controls-anyone with workspace access can potentially access items stored the...
- 0 kudos
- 102 Views
- 0 replies
- 0 kudos
workflow not pickingup correct host value (While working with MLflow model registry URI)
Exception: mlflow.exceptions.MlflowException: An API request to https://canada.cloud.databricks.com/api/2.0/mlflow/model-versions/list-artifacts failed due to a timeout. The error message was: HTTPSConnectionPool(host='canada.cloud.databricks.com', p...
- 102 Views
- 0 replies
- 0 kudos
- 2228 Views
- 4 replies
- 2 kudos
Problem serving a langchain model on Databricks
Hi, I've encountered a problem of serving a langchain model I just created successfully on Databricks.I was using the following code to set up a model in unity catalog:from mlflow.models import infer_signatureimport mlflowimport langchainmlflow.set_r...
- 2228 Views
- 4 replies
- 2 kudos
- 2 kudos
Hi,The warnings/errors in the logs of the langchain model log process can give you a good hint, although it may be not that evident at first sight.It happened something similar to me - same error message, and the cause was having used an OpenAI model...
- 2 kudos
- 1873 Views
- 2 replies
- 0 kudos
Accessing Databricks Volumes from a Serving Endpoint Using a Custom Model Class in Unity Catalog
Hi everyone,I’m looking for accessing Unity Catalog (UC) Volumes from a Databricks Serving Endpoint. Here’s my current setup:I have a custom AI model class for inference, which I logged into Unity Catalog using mlflow.pyfunc.log_model.I’ve created a ...
- 1873 Views
- 2 replies
- 0 kudos
- 0 kudos
Hey VELU1122,did you find a solution for it. We are struggling with the same problem currently. Thanks
- 0 kudos
- 428 Views
- 1 replies
- 0 kudos
Problem with ipywidgets and plotly on Databricks
Hi everyone, I am encountering a problem when using ipywidgets with plotly on Databricks. I am trying to pass interactive arguments to a function and then plot with plotly. When I do the followingdef f(m, b) : plt.figure(2) x = np.linspace(-10,...
- 428 Views
- 1 replies
- 0 kudos
- 0 kudos
It would help a lot if you attach a notebook or copy the full code that demonstrates the problem. This would make it easy for someone to copy/paste, run, and troubleshoot it and you are more likely to get effective help that way Also, since your Data...
- 0 kudos
- 1027 Views
- 4 replies
- 0 kudos
Resolved! FeatureEngineeringClient workspace id error
Hi, I am working from local notebook using vscode databricks extension.I am trying to use FeatureEngineeringClient, when I create data set training_set = fe.create_training_set( df=filtered_data_train, feature_lookups=payments_feat...
- 1027 Views
- 4 replies
- 0 kudos
- 0 kudos
I’ve done some additional research and found that the FeatureStoreClient is not officially supported when accessing a managed Databricks environment from an external IDE, even when using Databricks Connect. The client library is designed to operate w...
- 0 kudos
- 2170 Views
- 2 replies
- 2 kudos
"error_code":"INVALID_PARAMETER_VALUE","message":"INVALID_PARAMETER_VALUE: Failed to generate access
Hello everyone,I have an Azure Databricks subscription with my company, and I want to use external LLMs in databricks, like claude-3 or gemini. I managed to create a serving endpoint for Anthropic and I am able to use claude 3.But I want to use a Gem...
- 2170 Views
- 2 replies
- 2 kudos
- 1504 Views
- 1 replies
- 0 kudos
Resolved! Enabled the AI Builder Preview but unable to see the feature on the menu even after 3-4 hours
I am an account admin and enabled the beta feature. Does any additional permissions need to be added before I can see the feature on the workspace.
- 1504 Views
- 1 replies
- 0 kudos
- 0 kudos
Realized that our workspace is hosted in a different region. AI Builder is available for only couple of regions at the moment. I was able to spin up a new workspace and it works. Can close this thread
- 0 kudos
- 256 Views
- 0 replies
- 0 kudos
When does everyone utilize the model register?
Hi, I'm Yuki,I'm considering when I should use register_model.In my case, I'm running the training batch once a week and if the model is good, I want to update the champion.I have created the code to register the model if the score is the best.# star...
- 256 Views
- 0 replies
- 0 kudos
- 1336 Views
- 2 replies
- 0 kudos
spark_session invocation from executor side error, when using sparkXGBregressor and fe client
Hi I have created a model and pipeline using xgboost.spark's sparkXGBregressor and pyspark.ml's Pipeline instance. However, i run into a "RuntimeError: _get_spark_session should not be invoked from executor side." when i try to save the predictions i...
- 1336 Views
- 2 replies
- 0 kudos
- 0 kudos
Did you ever find a resolution to this? I've been running into the same error with a Spark XGBoost classification model, and haven't had any success in finding a solution. Setting it to a pyfunc model in logging resulted in an error, and clearly you ...
- 0 kudos
- 417 Views
- 1 replies
- 0 kudos
How to paralellize using R in Databricks notebook?
Hi!I'm using an R library, but it is only using one node, is there a way to paralellize it?Thanks in advance!
- 417 Views
- 1 replies
- 0 kudos
- 0 kudos
To parallelize computations in R while using a Databricks environment, you can utilize two main approaches: SparkR or sparklyr. Both allow you to run R code in a distributed manner across multiple nodes in a cluster. Hope this helps. Louis.
- 0 kudos
- 420 Views
- 1 replies
- 0 kudos
Not able to run end to end ML project on Databricks Trial
I started using Databricks trial version from today. I want to explore full end to end ML lifecycle on the databricks. I observed for the compute only 'serverless' option is available. I was trying to execute the notebook posted on https://docs.datab...
- 420 Views
- 1 replies
- 0 kudos
- 0 kudos
I can take up to 15 minutes for the serving endpoint to be created. Once you initiate the "create endpoint" chunk of code go and grab a cup of coffee and wait 15 minutes. Then, before you use it verify it is running (bottom left menu "Serving") by g...
- 0 kudos
