LiteLLM

​

LiteLLM

From LiteLLM’s docs:

Call 100+ LLMs using the same Input/Output Format

• Translate inputs to provider’s completion, embedding, and image_generation endpoints
• Consistent output. Text responses will always be available at ['choices'][0]['message']['content']
• Retry/fallback logic across multiple deployments (e.g. Azure/OpenAI)
• Track spend & set budgets per project

LiteLLM also supports many providers

​

Using AgentOps with LiteLLM

​

Requires litellm>=1.3.1

AgentOps requires you to make a minor adjustment to how you call LiteLLM.

# Do not use LiteLLM like this
# from litellm import completion
# ...
# response = completion(model="claude-3", messages=messages)

# Use LiteLLM like this
import litellm
...
response = litellm.completion(model="claude-3", messages=messages)
# or
response = await litellm.acompletion(model="claude-3", messages=messages)