Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Credentials in azure-identity should be pickle-able #37451

Open
needuv opened this issue Sep 18, 2024 · 0 comments
Open

Credentials in azure-identity should be pickle-able #37451

needuv opened this issue Sep 18, 2024 · 0 comments
Assignees

Comments

@needuv
Copy link
Member

needuv commented Sep 18, 2024

Is your feature request related to a problem? Please describe.

The azure-ai-evaluation SDK exposes a credential field in its evaluators (example: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/evaluators/_content_safety/_violence.py#L14) to be used when calling the Azure AI RAI service. Right now, this credential is optional, and at execution time we choose DefaultAzureCredential if the user did not pass anything in. However, we need to make this credential parameter required in the evaluators since choosing a default credential is not safe SDK behavior.

If the evaluator is being run on a single input OR when it is used in the batch evaluate API with no credential being passed in, the customer will not see any errors. However, if the user passes a credential to the evaluator and uses it in the evaluate function, it will not work since evaluate uses multiprocessing to do batch runs of multiple evaluators. Multiprocessing requires the evaluator to be pickle-able, but the credentials in azure-identity are not pickle-able, which breaks this scenario.

Describe the solution you'd like
The credentials in azure-identity should be pickle-able so that they can be used in evaluators when doing batch evaluation using evaluate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Status: Untriaged
Development

No branches or pull requests

3 participants