Zoho_Tickets

Ticket Router

Python 3.12+ FastAPI OpenRouter Chroma

A dynamic, learning ticket classifier: a ticket comes in, it’s matched against your taxonomy, and if the intent is genuinely ambiguous the system asks a follow-up question before committing to a route. Every human correction is fed back into a vector memory so future, similar tickets are classified more accurately — no retraining or redeploy required.

How it works

ticket text
     │
     ▼
retrieve K most-similar known examples  (Chroma vector store: taxonomy seed
     │                                    examples + past corrected tickets)
     ▼
OpenRouter structured-output call        (taxonomy + retrieved examples in prompt)
     │
     ▼
confident & unambiguous? ──No──► ask ONE clarifying question ──► append answer,
     │                                                             re-run loop
    Yes                                                            (capped by
     │                                                     MAX_CLARIFICATION_TURNS)
     ▼
route ticket + store category/confidence/reasoning
     │
     ▼
ticket transferred in Zoho? ──► webhook sees the new subcategory ──► embed
                                   (text → correct category) into vector memory
                                   immediately = next similar ticket gets it right

This is “dynamic” in the practical sense that matters for a classifier running on an API model you don’t fine-tune: in-context learning from a growing, retrieved example bank, not weight updates. It gets better as corrections accumulate, with no deploy step.

Project layout

app/
  main.py          FastAPI routes — Zoho webhook, Log and Taxonomy pages
  classifier.py    Builds the dynamic prompt, calls OpenRouter, decides finalize vs clarify
  memory.py        Chroma vector store — the "context memory" (seed + corrections)
  taxonomy.py      Loads taxonomy.json, exposes it to the prompt + JSON schema
  taxonomy.json    <-- your real taxonomy (14 categories / 68 routed subcategories)
  db.py            SQLite (local) / Turso (prod) — ticket/session state, conversation turns, correction log
  models.py        Pydantic request/response + structured-output schema
  static/index.html  The Log and Taxonomy pages (one shell, shared header)

Setup

cd ticket-classifier
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env        # optional overrides only, see below
uvicorn app.main:app --reload --host 0.0.0.0 --port 8000

--host 0.0.0.0 matters on Replit specifically - its webview/deployment can’t reach a server bound to 127.0.0.1. See REPLIT_DEPLOY.md for deploying this as an always-on Reserved VM instead of running it locally.

Set OPENROUTER_API_KEY (classification) and OPENAI_API_KEY (embeddings only, see below) in .env before starting the server - there’s no in-browser key input, both are server-side only. Open http://localhost:8000, log in, then type a ticket and watch it get classified (or asked a clarifying question), and use “Correct it” to simulate an agent fixing a bad call.

The taxonomy

app/taxonomy.json is wired to the real taxonomy (14 top-level categories, 67 routed subcategories — QA/instructor evaluation, curriculum, staffing/HR, scheduling, etc., plus a catch-all “Other / Unclear” category for tickets with no discernible intent). Classification targets the subcategory level, since that’s what actually determines routing (assigned_team, poc_primary) — the classifier picks a category_id like G01-S01, and the API/UI resolve that back to its parent category, team, and point of contact.

The live taxonomy is stored in the database; taxonomy.json is only the starting copy. To ship a taxonomy change through git, set its version higher than the live one shown on the Taxonomy page.

Nothing in the code is hardcoded to this taxonomy’s content — the classifier, the JSON schema sent to OpenRouter, and the seed examples are all generated from this file at startup. To update it, re-export the same 2-level shape:

{
  "version": 2,
  "categories": [
    {
      "id": "G01",
      "name": "QA Report / Instructor Evaluation",
      "subcategories": [
        {
          "id": "G01-S01",
          "name": "Feedback Too Generic or Vague",
          "description": "1-3 sentences the model uses to distinguish this from other subcategories.",
          "assigned_team": "IAS/SET",
          "poc_primary": "someone@company.com",
          "examples": ["a real historical ticket", "another one"]
        }
      ]
    }
  ]
}

Uploading ticket data (Taxonomy page)

Once the Taxonomy page is unlocked (same TAXONOMY_EDIT_PASSWORD), Upload tickets CSV takes a Zoho “Instructors Ticketing System” export and upserts it by Ticket ID (POST /api/tickets/import-csv). It previews the counts first and writes only after you confirm.

Pages

The app has three pages: Log (/log, the default), Classify (/classify) and Taxonomy (/taxonomy). The old Daily report, Pulse, Stats and Drill down pages were removed, along with the POC Chrome extension, resolution grading and the ticket/correction APIs behind them; old page addresses redirect to the Log. The Drill down page’s saved groups (shared_state table) are dropped on startup.

On the Classify page, a correction is saved as an example of the chosen subcategory; the model can then draft a definition change, preview its effect on the issue and existing examples, and you can apply it to the live taxonomy (stored in the database) or undo it (taxonomy_changes table), as long as nobody edited those definitions since.

Key config (.env)

Var Default Effect
OPENROUTER_API_KEY (required) Key for classify() - get one at openrouter.ai/keys.
OPENROUTER_CLASSIFY_MODEL openai/gpt-4o-mini Model used for classification, in OpenRouter’s provider/model form.
OPENAI_API_KEY (required) Used ONLY for embeddings now (few-shot memory) - OpenRouter has no embeddings endpoint.
CONFIDENCE_THRESHOLD 0.65 Below this, the ticket needs clarification or human review rather than auto-routing.
MAX_CLARIFICATION_TURNS 2 Caps back-and-forth so the bot doesn’t interrogate the user forever; falls back to needs_human_review.
FEWSHOT_K 5 How many retrieved examples get injected as dynamic few-shot context per call.
REPORTER_OVERRIDE_MIN_CONFIDENCE 0.9 How sure the model must be to override the subcategory the instructor picked.
TICKET_RETENTION_DAYS 60 Tickets that arrived more than this many days ago are deleted with everything stored for them (timeline, clarification turns, corrections), at startup and once a day. The taxonomy and its history are never deleted.

Tune CONFIDENCE_THRESHOLD down if you’re getting too many clarifying questions on tickets a human would consider obvious; tune it up if wrong-but-confident routes are getting through.

The instructor’s own category pick

Instructors pick a category (and optionally a subcategory) on the Zoho form before the pre-submit script classifies and overwrites them. The webhook passes that pick to classify() as a ReporterHint, but the model never sees it - it classifies from the issue text alone, so its pick and confidence are its own. The pick only counts afterwards:

The pre-submit Deluge script must send sub_category_of_the_issue for this to see the subcategory (see Claude outputs/zoho_invoke_script_fixed.deluge).

Each ticket is classified once, at pre-submit. That call stores a draft row (no Zoho ticket id yet) with the instructor’s pick, the model’s own pick and confidence, and the decision (agreed, kept or overrode). The On Add call that follows claims the oldest draft with the same text from the last 2 hours - filling in the ticket id instead of creating a new row - or, with no draft, stores the category the form was saved with. It only calls the model again when the pre-submit classification itself failed or the saved subcategory isn’t usable.

Drafts nobody claims (abandoned forms, or text edited after the suggestion) are never deleted, just hidden from every page, export and count, and the background classifier skips them. The same text from the same instructor (ticket_raised_by) within 7 days is marked as a duplicate.

The Log page (/log) shows one row per ticket - instructor’s pick, model’s pick and confidence, decision, final category, right or wrong - and, from settled tickets, how often each decision was right and which REPORTER_OVERRIDE_MIN_CONFIDENCE would have routed the most disagreements correctly. A ticket counts as wrong when it was corrected in the portal or transferred in Zoho.

OpenRouter vs Gemini accuracy comparison

The app’s own classify path goes through OpenRouter, but app/classifier.py also has classify_gemini() - same taxonomy, prompt, and few-shot retrieval, just a Gemini call instead for the final decision. scripts/compare_ classifiers.py runs both against existing tickets that already carry a Zoho category tag (used as a rough reference, not a hand-labeled eval set) and reports agreement rates:

OPENROUTER_API_KEY=sk-or-... OPENAI_API_KEY=sk-... GEMINI_API_KEY=... python scripts/compare_classifiers.py --limit 20

OPENAI_API_KEY is only needed here for embeddings (few-shot retrieval used by both sides), not classification. Needs GEMINI_API_KEY (and GEMINI_CLASSIFY_MODEL, default gemini-2.5-flash) set too - see .env.example. Not wired into the running app in any way; it’s a standalone script for deciding whether switching or dual-running models is worth pursuing further.

Cloudflare AI Gateway (optional)

Set CLOUDFLARE_ACCOUNT_ID and CLOUDFLARE_AI_GATEWAY_ID (see .env.example) to route OpenAI embedding calls (memory.py) through a Cloudflare AI Gateway instead of hitting OpenAI directly - same OPENAI_API_KEY, same model, but Cloudflare caches repeated identical requests and gives a usage dashboard at dash.cloudflare.com -> AI -> AI Gateway. Does not apply to classify() (it goes to OpenRouter, not OpenAI). Unset means embedding calls go straight to OpenAI, unchanged from before this existed.

Separately, set CLOUDFLARE_API_TOKEN (a token with Workers AI permission specifically, not AI Gateway’s) and classify() automatically fall back to Cloudflare Workers AI (CLOUDFLARE_WORKERS_AI_MODEL, default @cf/meta/llama-3.1-8b-instruct) whenever OpenRouter raises a rate-limit error - a completely separate quota from OpenRouter’s, so classification keeps working instead of stalling until OpenRouter’s own limit resets. Same taxonomy/prompt/few-shot pipeline either way; only the final model call moves. Unset means these calls behave exactly as before (raise on rate limit).

API

All routes below (and the UI itself) require an admin session — log in at /login first (POST /api/login {username, password} sets the session cookie; POST /api/logout clears it).

The one route outside the login is POST /api/webhooks/zoho/tickets, which Zoho calls with the X-Webhook-Secret header (see zoho-invoke-url-setup.md).

Production notes / next steps