You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If the evaluator is being run on a single input OR when it is used in the batch evaluate API with no credential being passed in, the customer will not see any errors. However, if the user passes a credential to the evaluator and uses it in the evaluate function, it will not work since evaluate uses multiprocessing to do batch runs of multiple evaluators. Multiprocessing requires the evaluator to be pickle-able, but the credentials in azure-identity are not pickle-able, which breaks this scenario.
Describe the solution you'd like
The credentials in azure-identity should be pickle-able so that they can be used in evaluators when doing batch evaluation using evaluate.
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem? Please describe.
The
azure-ai-evaluation
SDK exposes acredential
field in its evaluators (example: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/evaluation/azure-ai-evaluation/azure/ai/evaluation/evaluators/_content_safety/_violence.py#L14) to be used when calling the Azure AI RAI service. Right now, this credential is optional, and at execution time we chooseDefaultAzureCredential
if the user did not pass anything in. However, we need to make thiscredential
parameter required in the evaluators since choosing a default credential is not safe SDK behavior.If the evaluator is being run on a single input OR when it is used in the batch
evaluate
API with no credential being passed in, the customer will not see any errors. However, if the user passes a credential to the evaluator and uses it in theevaluate
function, it will not work sinceevaluate
uses multiprocessing to do batch runs of multiple evaluators. Multiprocessing requires the evaluator to be pickle-able, but the credentials inazure-identity
are not pickle-able, which breaks this scenario.Describe the solution you'd like
The credentials in
azure-identity
should be pickle-able so that they can be used in evaluators when doing batch evaluation usingevaluate
.The text was updated successfully, but these errors were encountered: