AG
TR

// work / mizan

Mizan

A hybrid search and text-analysis engine for the Quran, combining BM25-style keyword search over the Arabic with ISRI stemming, three pgvector cross-language embedding paths, Reciprocal Rank Fusion, and letter/Abjad calculations verified against the Tanzil corpus.

Role
Author and maintainer
Date
Jan 2026

Stack

  • Python 3.11
  • FastAPI
  • PostgreSQL
  • pgvector
  • Redis 7
  • Next.js 16
  • TypeScript
  • SQLAlchemy (async)
  • ISRI stemmer
  • intfloat/multilingual-e5-base

The problem

Searching and analyzing the Quran well is harder than it looks. Arabic is morphologically rich, so naive keyword search misses obvious matches; users query in Turkish and English, not only Arabic; and letter counts and Abjad (numerology) totals circulate widely but are often wrong because nobody shows how they were computed.

Constraints

  • Every count and Abjad total must be verifiable letter-by-letter against an authoritative corpus, not merely asserted.
  • Queries arrive in Turkish and English as well as Arabic; the engine has to bridge languages without an external search engine.

Approach

Mizan combines classical information retrieval with modern cross-language embeddings. The retriever runs four paths in parallel: three pgvector cosine-similarity paths (Arabic, English, Turkish) using intfloat/multilingual-e5-base (768-dim), and one PostgreSQL tsvector/GIN keyword path with an ISRI Arabic stemmer. The four ranked lists are fused with Reciprocal Rank Fusion (RRF), with an optional cross-encoder reranker (ms-marco-MiniLM) on top. Letter counting and Abjad numerology are verified against the Tanzil corpus letter-by-letter, with SHA-checked corpus integrity at load time.

Key decisions

  • Four-path hybrid retrieval with RRF instead of pure lexical or pure semantic search

    The pgvector paths (Arabic, English, Turkish) capture semantic and cross-language matches that keyword search would miss. The tsvector/GIN path with the ISRI stemmer handles morphology-aware exact recall on the Arabic source. Neither alone covers both a scholar's exact-phrase lookup and a casual Turkish question. RRF fuses the ranked lists without requiring score calibration between the paths.

  • Methodology metadata on every result

    Each result carries the counting method and model used to produce it, so the numbers are reproducible. The corpus is SHA-checked at load time to detect any modification before computation begins.

Architecture

A query in Arabic, Turkish, or English enters the FastAPI service. The hybrid retriever runs three pgvector cosine paths (Arabic, English, Turkish) and one tsvector/GIN keyword path (with ISRI stemmer) in parallel, fuses the results with RRF, applies the optional cross-encoder reranker, and returns ranked results alongside letter/Abjad counts from the analysis engine. Every response includes the counting method and model used.

flowchart LR
  q["Query\nArabic / Turkish / English"]
  svc["FastAPI service"]
  vec_ar["pgvector cosine\n(Arabic embeddings)"]
  vec_en["pgvector cosine\n(English embeddings)"]
  vec_tr["pgvector cosine\n(Turkish embeddings)"]
  kw["tsvector/GIN\n+ ISRI stemmer"]
  rrf["Reciprocal Rank Fusion"]
  rerank["Cross-encoder reranker\n(optional, ms-marco-MiniLM)"]
  analysis["Analysis engine\nletter + Abjad counts\n(SHA-checked corpus)"]
  out["Result\n(method + model metadata)"]

  q --> svc
  svc --> vec_ar --> rrf
  svc --> vec_en --> rrf
  svc --> vec_tr --> rrf
  svc --> kw --> rrf
  rrf --> rerank --> out
  svc --> analysis --> out
Three pgvector cosine paths plus one tsvector/GIN keyword path, fused with RRF and an optional reranker, alongside SHA-verified letter and Abjad counts.

Outcome

Mizan ships as a FastAPI service with a Next.js front end, with an extensive test suite. Letter counts and Abjad totals are verified against the Tanzil corpus, the corpus is SHA-checked at load time, and every result reports the counting method and model used. The service is live at mizan.rollingcatsoftware.com and mizan-api.rollingcatsoftware.com on a Hetzner server.

By the numbers

  • 4 Retrieval paths
  • 3 Query languages bridged
  • Tanzil corpus Count verification
  • method + model Methodology tag per result

Deep dive

Mizan (“the scale” / “the balance”) combines classical information retrieval with modern cross-language embeddings to make Quranic text searchable and its letter-level statistics verifiable.

Why hybrid retrieval

Arabic is morphologically rich: the same root surfaces in many inflected forms. A plain keyword index misses obvious matches, so Mizan runs a PostgreSQL tsvector/GIN path with an ISRI Arabic stemmer to get morphology-aware lexical recall on the Arabic source. But a scholar’s exact-phrase lookup and a casual Turkish question are different problems. The second needs to bridge languages, which is what the three pgvector cosine paths do: one each for Arabic, English, and Turkish, all using intfloat/multilingual-e5-base (768-dim). Reciprocal Rank Fusion combines the four ranked lists without requiring score calibration between paths. An optional cross-encoder reranker (ms-marco-MiniLM) sits on top when precision matters more than latency.

Verifiable, not just asserted

The part that matters most for trust is the numbers. Letter counts and Abjad (numerological) totals circulate widely and are frequently wrong because nobody shows the work. Mizan computes them and verifies them letter-by-letter against the Tanzil corpus. The corpus is SHA-checked at load time so any file-level modification is caught before computation begins. Every search and analysis result carries the counting method and the model version used to produce it, so the numbers are reproducible.

Where it runs

Mizan is a FastAPI service (Python 3.11, SQLAlchemy async, PostgreSQL + pgvector HNSW, Redis 7) with a Next.js 16 / React 19 / Tailwind front end, deployed on a Hetzner server. The front end is live at mizan.rollingcatsoftware.com and the API at mizan-api.rollingcatsoftware.com. The source is public on GitHub.

All case studies