ailinone/collective-intelligence

What it claims

The headline benchmark "97% vs 68–82%" is 37 of 38 tasks answered correctly by the collective. The baselines are GPT-5.5-pro, Claude Opus 4.8, Gemini 3.1 Pro and Grok 4.3, with their scores pooled. The repo makes two further claims:

• Across every run, the verifier never selected an objectively wrong answer.
• A pool of sub-frontier open-weight models, coordinated, out-answered every flagship on the same tasks.

The repo says its custom "ailin" coordinator models are part of the design but are not yet shipping. What ships today is the audit substrate.

How it works

The ailin-auto cascade routes each request down one of three paths. The verifier only runs when the request declares a machine-checkable answer.

flowchart LR
    A[Request] --> B{ailin-auto cascade}
    B -->|simple| C[Single cheapest viable model]
    B -->|answer_check declared| D[Consensus + deterministic verifier]
    B -->|strategy named| E[1 of 32 strategies]
    C --> F[Fallback chains + budget governor]
    D --> F
    E --> F
    F --> G[Quality gate]
    G --> H[Response + ailin_metadata]

• Path selection. The cascade is described as conservative. It defaults to the cheap single-model path and escalates only when quality gating requires it. To get consensus plus verification, the request must set ailin_constraints.answer_check.
• Strategy status. Each of the 32 strategies is labeled auto-selectable, explicit-only or roadmap. The docs use status badges to separate production features from maturing ones.
• Model discovery. Models are discovered and indexed automatically. A route is advertised only after a health check proves it live.
• Beyond chat. The gateway also routes image, audio and video generation by capability. It can render DOCX, XLSX, PDF, PPTX and ZIP files from any model that supports structured output.

How to use it

Requirements: Docker with Compose v2, about 8 GB of free RAM, free ports 3000, 5432 and 6379, and python3. On Windows, use Git Bash or WSL.

1. Write the config. The app refuses weak or default JWT secrets. Replace sk-... with a real key from any of the ~90 providers.

git clone https://github.com/ailinone/collective-intelligence.git
cd collective-intelligence/docker
cat > .env <<EOF
JWT_SECRET=$(openssl rand -base64 48)
AILIN_SHARED_JWT_SECRET=$(openssl rand -base64 48)
SECRETS_PROVIDER_PRIMARY=env
OPENAI_API_KEY=sk-...
EOF

2. Start the stack. First boot runs database migrations and a model-discovery scan, which takes about 1–5 minutes.

docker compose up -d api postgres redis
docker compose logs -f api
curl http://localhost:3000/health

3. Register a user to get a token. The token is used as the API key.

export TOKEN=$(curl -s -X POST http://localhost:3000/v1/auth/register \
  -H 'Content-Type: application/json' \
  -d '{"email":"you@example.com","password":"pick-a-strong-one","name":"You"}' \
  | python3 -c "import sys,json; print(json.load(sys.stdin)['tokens']['accessToken'])")

4. Call it with the OpenAI SDK. Available model aliases are ailin-auto, ailin-best, ailin-fast, ailin-economy and ailin-consensus.

client = OpenAI(base_url="http://localhost:3000/v1", api_key=os.environ["TOKEN"])
r = client.chat.completions.create(model="ailin-auto", messages=[...])
print(r.model_extra["ailin_metadata"])  # strategy_used, models_used, cost_actual, dissent

Running without an API key: set OLLAMA_URL=http://host.docker.internal:11434 in docker/.env. On native Linux, also add extra_hosts: ["host.docker.internal:host-gateway"] to the api service.

Reproducing the benchmark: regenerating every published table from the committed raw CSVs takes about two minutes and uses only Python's standard library. Start from REPRODUCING_THE_BENCHMARK.md.

Limits

• Small benchmark. The headline result rests on 38 tasks and covers only verifiable tasks. The page states its open fronts exist but does not list them in the fetched text. Check the full report's §3 leaderboard for per-entry n and caveats.
• Workloads the project calls a poor fit:
• high-volume, low-stakes traffic
• tight latency SLAs
• documentation-style prose
• Unfinished parts. Production coordinator weights are still in development, and some strategies are roadmap-only.
• AGPL network clause. Under AGPL §13, if you run a modified version as a network service you must offer its source to users. The engine helps by serving /source and /license endpoints and sending X-License and X-Source-Code headers. Point AGPL_SOURCE_URL at your modified source. Commercial licensing is available via licensing@ailin.one, and the trademarks are not covered by the AGPL.
• Secrets setup. Without SECRETS_PROVIDER_PRIMARY=env, the app tries GCP Secret Manager. A log full of Secret retrieval failed means it booted in degraded mode.