Skip to main content

LLM Gateway

Portkey is an enterprise-grade LLM gateway running on-prem:

Portkey Architecture

Broadly it facilitates:

  • Route requests via configurations that can allow for fallbacks, load balance, etc
  • Observability via the control pane that displays your usage statistics and logs. You can retrieve your logs via an API.
  • Prompt management via the prompt playground. You can compare the response for a prompt across multiple LLMs, collaborate with your team and export your prompts.
  • Guardrails usage by allowing you to define them or integrate with third party guardrails.
  • Agentic workflows by integrating with various agentic frameworks like langchain, llamaindex, etc.
  • Security & Governance by allowing you to set budget and rate limits on the API keys created for your workspace

Proceed to the next section to learn more about the on-boarding process.