syftr: Optimize Agent Workflows with Pareto Front Search

syftr is an optimization framework designed to refine agent workflows under specific budget constraints. By providing a dataset and defining a search space of models and components, users can leverage syftr to sample that space efficiently. The tool employs advanced multi-objective Bayesian optimization and a specialized "Pareto pruner" to map the Pareto front—the optimal trade-off curve between accuracy and competing metrics such as cost, latency, and throughput.

Pareto-optimal search – syftr balances accuracy against secondary targets within a set budget, returning the most efficient workflow combinations.

Optional evaluation – Users can rigorously test generated workflows to validate real-world performance.

Versatile use cases – The tool supports foundation model tuning, synthetic exam generation, corpus processing, and various other agentic tasks.

syftr integrates several prominent open-source projects to power its functionality:

• Ray: Handles scaling across large CPU and GPU clusters to accelerate computation.
• Optuna: Provides a flexible "define-by-run" interface (similar to PyTorch’s eager execution) to support multi-objective algorithms.
• LlamaIndex: Facilitates the construction of complex RAG workflows, both agentic and non-agentic, for natural language processing.
• Hugging Face Datasets: Enables fast, collaborative loading and processing of unified datasets.
• Trace: Optimizes internal workflow components, such as prompts, to enhance text-handling performance.

syftr installation and configuration

Installation steps

1. Clone the syftr repository.
2. Execute the following commands to set up the environment:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv --python 3.12.7
source .venv/bin/activate
uv sync --extra dev

Required credentials

To use the full feature set, you will need:

• An Azure OpenAI API key
• An Azure OpenAI endpoint URL (api_url)
• A PostgreSQL server DSN (the system defaults to a local SQLite instance if this is not provided)

Copy the config.yaml.sample file to config.yaml and update the relevant sections with your credentials.

Configuration options

1. Main configuration file – Use config.yaml to define file paths, logging preferences, database connections, and Ray parameters. Refer to config.yaml.sample for specific examples before populating it with your infrastructure details.
2. Environment variables – Configurations can also be set via environment variables, such as: export SYFTR_PATHS__ROOT_DIR=/foo/bar.
3. Database setup – By default, syftr uses SQLite for Optuna data. For distributed workloads, update the database.dsn field to point to a PostgreSQL instance or any other Optuna-supported relational database.

After configuration is complete, you can verify the setup by running the examples/1-welcome.ipynb notebook.

syftr quick start

1. Run make check to validate your configuration and credentials. Note that connection tests will fail if LLM credentials have not been configured.
2. Explore the Jupyter notebooks located in the examples/ directory.
3. Alternatively, initiate a syftr study directly through the user API:

from syftr import api

s = api.Study.from_file("studies/example-dr-docs.yaml")
s.run()

Retrieve the results once the study is complete:

s.wait_for_completion()
print(s.pareto_flows)
# [{'metrics': {'accuracy': 0.7, 'llm_cost_mean': 0.000258675},
#   'params': {'response_synthesizer_llm': 'gpt-4o-mini',
#    'rag_mode': 'no_rag',
#    'template_name': 'default',
#    'enforce_full_evaluation': True}},
#  ...
# ]

syftr custom extensions

Custom LLMs

In addition to built-in models, you can integrate any OpenAI-API-compatible endpoints via config.yaml.

Example: Generative model configuration

local_models:
  default_api_key: "YOUR_API_KEY_HERE"
  generative:
    - model_name: "microsoft/Phi-4-multimodal-instruct"
      api_base: "http://phi-4-host.com/openai/v1"
      max_tokens: 2000
      context_window: 129072
      is_function_calling_model: true
      additional_kwargs:
        frequency_penalty: 1.0
        temperature: 0.1
    - model_name: "deepseek-ai/DeepSeek-R1-Distill-Llama-70B"
      api_base: "http://big-vllm-host:8000/v1"
      max_tokens: 2000
      context_window: 129072
      is_function_calling_model: true
      additional_kwargs:
        temperature: 0.6

Example: Embedding model configuration

local_models:
...
  embedding:
    - model_name: "BAAI/bge-small-en-v1.5"
      api_base: "http://vllmhost:8001/v1"
      api_key: "non-default-value"
      additional_kwargs:
        extra_body:
          truncate_prompt_tokens: 512
    - model_name: "thenlper/gte-large"
      api_base: "http://vllmhost:8001/v1"
      additional_kwargs:
        extra_body:
          truncate_prompt_tokens: 512

Models defined in the configuration file are automatically included in the default search space, or they can be manually assigned to specific workflow components.