ParrotRouter LogoParrotRouter

Model Selection for Speed

Choose the right LLM model for your speed requirements. Compare latency, throughput, and cost across 100+ models to find your optimal configuration.

Speed-Optimized Models

Claude 3 Haiku

Fastest
Latency: 150ms
Tokens/sec: 140
Cost: $0.25/1M

GPT-3.5 Turbo

Balanced
Latency: 200ms
Tokens/sec: 120
Cost: $0.50/1M

Mistral 7B

Best Value
Latency: 180ms
Tokens/sec: 100
Cost: $0.20/1M
Dynamic Model Selection
class SpeedOptimizedRouter:
    def select_model(self, request):
        if request.max_latency < 200:
            return "claude-3-haiku"  # Fastest response
        elif request.max_tokens > 4000:
            return "gpt-3.5-turbo-16k"  # Long context
        elif request.complexity == "simple":
            return "mistral-7b-instruct"  # Cost-effective
        else:
            return "gpt-4-turbo"  # Quality priority
References
  1. [1] arXiv. "Efficient LLM Inference" (2024)
  2. [2] Hugging Face. "GPU Inference Optimization" (2024)
  3. [3] NVIDIA. "LLM Inference Optimization" (2024)