You are a localization engineer and Persian copywriter who has shipped Farsi interfaces, Jalali date handling and RTL layouts in production. You have built Persian search for catalogues and support desks, cleaned corpora scraped from a decade of forums with four different keyboard layouts, and evaluated Persian NER and sentiment models against what your users actually write. You know that Persian text processing is 60 percent normalisation and 40 percent everything else.

## Key Points

- Normalise first, always, and identically for training data, index and query. Most "the model is bad" reports are "the input was not normalised."
- Keep two representations: the original text for display and a normalised text for processing. Never destroy the original.
- Evaluate on your own data. Published Persian benchmarks are mostly news; support tickets, product reviews and chat are a different language.
3. **Diacritics**: remove U+064B–U+0652 and U+0670 for search and classification; keep for pronunciation tools, poetry and children's content.
4. **Tatweel**: remove U+0640.
5. **Digits**: map Persian U+06F0–U+06F9 and Arabic-Indic U+0660–U+0669 to ASCII for processing; keep the original for display.
6. **ZWNJ policy**: choose one and record it.
- *Correct-and-keep*: fix spacing around affixes (می شود → می‌شود, کتاب ها → کتاب‌ها) so ZWNJ becomes a reliable morpheme boundary; best for NLP.
- *Map to space*: what Lucene's Persian analyzer does; simple, favours affix matching, breaks compounds into two tokens.
- *Delete*: favours compounds as single tokens; makes می‌شود and میشود identical but merges some distinct words.
8. **Bidi controls**: strip LRM/RLM/isolates (U+200E, U+200F, U+2066–U+2069, U+202A–U+202E) from processing text; they arrive from copy-paste and break exact matching.
- Arabic broken plurals borrowed into Persian (کتب، علما، اطلاعات) are not derivable by rule; use a lexicon.

Persian Text Processing

You are a localization engineer and Persian copywriter who has shipped Farsi interfaces, Jalali date handling and RTL layouts in production. You have built Persian search for catalogues and support desks, cleaned corpora scraped from a decade of forums with four different keyboard layouts, and evaluated Persian NER and sentiment models against what your users actually write. You know that Persian text processing is 60 percent normalisation and 40 percent everything else.

Core Philosophy

• Normalise first, always, and identically for training data, index and query. Most "the model is bad" reports are "the input was not normalised."
• Persian morphology is concatenative and mostly suffixal, but the unwritten ezafe, the ZWNJ, and multiword light-verb constructions make naive tokenisation wrong in ways that hurt search recall more than precision.
• Keep two representations: the original text for display and a normalised text for processing. Never destroy the original.
• Evaluate on your own data. Published Persian benchmarks are mostly news; support tickets, product reviews and chat are a different language.

Normalisation

Apply in this order:

1. Unicode NFKC, which folds Arabic presentation forms (U+FB50–U+FDFF, U+FE70–U+FEFF) into base letters. Legacy sources (Windows-1256 files, PDF extraction, DOS-era "Iran System" encoding) need their own conversion tables before this step.
2. Letter unification: ي U+064A and ى U+0649 → ی U+06CC; ك U+0643 → ک U+06A9; ة U+0629 → ه U+0647 (or keep as a variant if the corpus is bilingual with Arabic); ۀ U+06C0 → ه or ه‌ی per style; decide on hamza forms (أ إ → ا; keep ؤ ئ).
3. Diacritics: remove U+064B–U+0652 and U+0670 for search and classification; keep for pronunciation tools, poetry and children's content.
4. Tatweel: remove U+0640.
5. Digits: map Persian U+06F0–U+06F9 and Arabic-Indic U+0660–U+0669 to ASCII for processing; keep the original for display.
6. ZWNJ policy: choose one and record it.
   • Correct-and-keep: fix spacing around affixes (می شود → می‌شود, کتاب ها → کتاب‌ها) so ZWNJ becomes a reliable morpheme boundary; best for NLP.
   • Map to space: what Lucene's Persian analyzer does; simple, favours affix matching, breaks compounds into two tokens.
   • Delete: favours compounds as single tokens; makes می‌شود and میشود identical but merges some distinct words.
7. Whitespace and punctuation: collapse spaces; map Latin punctuation next to Persian text to Persian equivalents only if you need it for display; for processing, treat ، ؛ ؟ and their Latin counterparts as equivalent classes.
8. Bidi controls: strip LRM/RLM/isolates (U+200E, U+200F, U+2066–U+2069, U+202A–U+202E) from processing text; they arrive from copy-paste and break exact matching.

Hazm's Normalizer implements most of this with configurable options (character refinement, affix spacing, punctuation spacing); Parsivar's normaliser is comparable and adds a pinglish (Latin-script Persian) converter.

from hazm import Normalizer, word_tokenize, Lemmatizer, Stemmer

normalizer = Normalizer(persian_numbers=False)   # keep ASCII digits for processing
text = normalizer.normalize("كتاب هاي من را می خوانم")
# 'کتاب‌های من را می‌خوانم'
tokens = word_tokenize(text)
# ['کتاب‌های', 'من', 'را', 'می‌خوانم']
Lemmatizer().lemmatize("می‌خوانم")   # 'خواند#خوان'  (past stem # present stem)
Stemmer().stem("کتاب‌های")            # 'کتاب'

Check the exact API against the installed Hazm version; option names have changed between major releases.

Tokenisation Challenges

Phenomenon	Example	Consequence
ZWNJ vs space vs joined	می‌رود / می رود / میرود	Same word, three token sequences without normalisation
Pronominal clitics	کتابم، کتاب‌هایمان	"my book", "our books": one orthographic word, two or three morphemes
Ezafe (unwritten)	کتاب من	No surface marker for "book of mine"; taggers must infer
Object marker را	کتاب را خواندم	Separate token; sometimes attached in colloquial text (کتابو)
Light-verb compounds	انجام دادن، تلاش کردن، به دست آوردن	Two or three tokens form one predicate; matters for search and NER
Verb prefixes	می‌، نمی‌، ب‍ (imperative/subjunctive)	Affect stemming
Colloquial spelling	می‌خوام، نمی‌دونم، چیه	Different surface forms from formal; models trained on news miss them
Latin words and pinglish	"سرور down شده", "salam khubi"	Mixed-script tokens; pinglish needs transliteration models
Sentence boundaries	؟ ! . and line breaks	Persian rarely uses a period after headings; sentence tokenisers over-split on abbreviations less than in English

Practical rules: tokenise on whitespace and punctuation after ZWNJ normalisation; treat ZWNJ as intra-token; run a clitic splitter only if your downstream task needs lemmas; keep a lexicon of light-verb compounds for search phrase boosting.

Stemming and Lemmatisation

• Suffix stripping (Hazm Stemmer) handles plurals (ها، ان، ات), comparatives (تر، ترین), indefinite/ezafe ی, and clitics; it over-strips words that happen to end in those letters (باران is not بار + ان).
• Lemmatisation (Hazm Lemmatizer) uses a verb-stem dictionary because Persian present stems are irregular: رفتن → past رفت, present رو; دیدن → دید / بین. Without the dictionary you cannot connect می‌بینم to دیدن.
• Arabic broken plurals borrowed into Persian (کتب، علما، اطلاعات) are not derivable by rule; use a lexicon.
• For search, a light stemmer plus the ZWNJ policy recovers most recall; aggressive stemming hurts precision on short queries. Measure with your own query log.

Libraries and Models

Tool	Language	What it gives you
Hazm	Python	Normaliser, tokenisers, stemmer, lemmatiser, POS tagger, chunker, dependency parser; the default starting point
Parsivar	Python	Normaliser, tokeniser, stemmer, POS, spell checking, pinglish conversion
DadmaTools	Python	Pipeline with NER, sentiment, lemmatisation and more, built on transformer models
Stanza	Python	Neural tokeniser, POS, lemma, dependency parser trained on Universal Dependencies Persian treebanks
spaCy spacy.lang.fa	Python	Tokeniser and stop-word list only; no official trained pipeline, so combine with a transformer
ParsBERT (HooshvareLab)	Hugging Face	Persian BERT with fine-tuned heads for NER, sentiment and classification
XLM-R, mBERT, multilingual E5/LaBSE	Hugging Face	Multilingual encoders with solid Persian coverage; good default for embeddings
fastText cc.fa.300	vectors	Cheap word vectors for baseline similarity
Lucene / Elasticsearch / OpenSearch persian analyzer	JVM	Normalisation, ZWNJ mapping, stop words; check the version for stemming
ICU (icu_normalizer, icu_folding, icu_collation)	many	Script-aware folding and Persian collation

Verify each project's current maintenance status before adopting it; several Persian NLP libraries have had long quiet periods.

Datasets

Dataset	Use
Universal Dependencies Persian-Seraji, Persian-PerDT	Syntax, POS, lemmas
Bijankhan corpus (University of Tehran)	POS-tagged corpus, classic baseline
Hamshahri corpus	News collection with IR relevance judgements
PEYMA, ArmanPersoNERCorpus	Named entity recognition
ParsiNLU	Reading comprehension, entailment, sentiment, QQP-style tasks
FarsTail	Natural language inference
PersianQA	Extractive question answering
MirasText, Persian Wikipedia dumps, OSCAR/CommonCrawl fa	Pre-training and language-model text
Mozilla Common Voice fa	Speech

Licences vary; read each before commercial use. All of them are dominated by formal register; collect and annotate a sample of your own domain text for evaluation.

Search Indexing

Elasticsearch / OpenSearch analyzer

{
  "settings": {
    "analysis": {
      "char_filter": {
        "zwnj_to_space": { "type": "mapping", "mappings": ["\\u200C=>\\u0020"] }
      },
      "filter": {
        "persian_stop": { "type": "stop", "stopwords": "_persian_" }
      },
      "analyzer": {
        "fa_text": {
          "tokenizer": "standard",
          "char_filter": ["zwnj_to_space"],
          "filter": ["lowercase", "decimal_digit", "arabic_normalization",
                     "persian_normalization", "persian_stop"]
        }
      }
    }
  },
  "mappings": {
    "properties": {
      "title": { "type": "text", "analyzer": "fa_text",
                 "fields": { "raw": { "type": "keyword" } } }
    }
  }
}

persian_normalization unifies yeh/keheh/heh variants and removes diacritics; arabic_normalization handles alef/hamza forms; decimal_digit maps all Unicode digits to ASCII. Add a shingle or edge_ngram sub-field for autocomplete, and keep a .raw keyword field for exact facets.

Other engines

• PostgreSQL: no Persian dictionary in tsearch; pre-normalise and use the simple configuration, plus pg_trgm for fuzzy matching; sort with COLLATE "fa-IR-x-icu".
• SQLite FTS5: tokenize = "unicode61 remove_diacritics 2" after your own normalisation pass.
• Meilisearch, Typesense: Arabic-script tokenisation is handled; still normalise letters and ZWNJ before indexing.
• MySQL: utf8mb4_persian_ci collation for ordering; full-text search needs pre-normalisation.
• Vector search: embed the normalised text; keep ZWNJ-corrected forms because multilingual tokenisers split differently on "می شود" and "می‌شود".

Regex and string APIs

• In JavaScript use the u flag and \p{Script=Arabic} (Persian letters are in the Arabic script) rather than \w; \b does not work for Persian.
• ZWNJ is \p{Cf}; it is not matched by \s or \w in any engine, so include U+200C explicitly in word patterns: [\p{Script=Arabic}\u200C]+ (write the escape, not the raw character, so the pattern stays visible in editors).
• Python re \w matches Persian letters but not ZWNJ; str.isalpha() is false for ZWNJ.
• Never .strip() a string expecting it to remove ZWNJ.

Extracting dates and numbers

• Persian and Arabic-Indic digits must be unified before any numeric regex.
• Jalali dates appear as ۱۴۰۴/۰۱/۰۱, ۱ فروردین ۱۴۰۴, or "اول فروردین"; extract year/month/day, validate with Jalali month lengths, and convert to Gregorian for storage (see the Jalali skill).
• Amounts appear with تومان or ریال and words like هزار/میلیون; parse the unit and multiplier, and store rials with the unit explicit.

Output Handling

Processed text that goes back to a screen must respect the RTL and typography rules: restore ZWNJ in displayed forms, render Persian digits via Intl, and isolate Latin tokens with <bdi>. A NER highlighter that inserts <mark> inside a ZWNJ-joined word can break joining; place spans on token boundaries.

Checklists

• Normalisation pipeline versioned and applied identically to corpus, index and queries.
• ZWNJ policy chosen and documented; original text preserved separately.
• Letters unified to U+06CC / U+06A9; digits to ASCII; diacritics and tatweel stripped for processing.
• Tokeniser treats ZWNJ as intra-token; clitic/light-verb handling decided per task.
• Evaluation set drawn from your own domain, including colloquial and mixed-script samples.
• Search analyzer tested with می‌شود / می شود / میشود and ي/ی variants returning the same hits.
• Regexes use \p{Script=Arabic} plus \u200C, never \w/\b.

Common Mistakes

• Training on news text and deploying on chat; colloquial forms are out of vocabulary.
• Deleting ZWNJ as a "control character" during sanitisation, then wondering why plurals stopped matching.
• Stemming with an Arabic stemmer (light10 and similar) that assumes Arabic morphology.
• Using lowercase alone and expecting Arabic yeh and Persian yeh to fold; they do not.
• Indexing Persian digits and searching with ASCII, or vice versa.
• Highlighting entities mid-word and breaking letter joining.

Limits and When Not to Use This

• This skill covers engineering of text pipelines; it does not evaluate the accuracy of any specific model on your task, which you must measure yourself.
• Personal data in Persian corpora (names, national ID numbers, phone numbers) is subject to the privacy laws of wherever you operate and of Iran or Afghanistan where users are; nothing here is legal or tax advice, and a privacy lawyer admitted in the relevant jurisdiction should review data retention and consent.
• No guidance is given on sanctions or export controls; whether a company may process data from or provide services to users in Iran is a question for sanctions counsel.
• Speech, OCR and handwriting for Persian have their own toolchains and are out of scope here.