{"id":49,"date":"2024-06-09T19:30:00","date_gmt":"2024-06-09T16:30:00","guid":{"rendered":"https:\/\/fsaeed.blog\/?p=49"},"modified":"2026-10-31T00:18:53","modified_gmt":"2026-10-30T21:18:53","slug":"working-with-arabic-text-in-python","status":"publish","type":"post","link":"https:\/\/fsaeed.blog\/index.php\/2024\/06\/09\/working-with-arabic-text-in-python\/","title":{"rendered":"Working with Arabic Text in Python: Normalising, Searching, Exporting"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">Hello, tech enthusiasts! \ud83d\udc4b<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Imagine you have a customer list with Arabic names. You search for &#8220;\u0627\u062d\u0645\u062f&#8221; and get nothing. But you know the customer is there. The problem? In the file, the name is written &#8220;\u0623\u062d\u0645\u062f&#8221;, with a hamza on the alef.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Arabic can write the same word in many small ways. Some people add tashkeel (the small vowel marks). Some use tatweel (the long line &#8220;\u0640&#8221; that stretches a word). Some write &#8220;\u0629&#8221; and some write &#8220;\u0647&#8221; at the end. For a human, these are the same word. For a computer, they are completely different.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Today we will fix this. We will clean the text, search it, and export it so Excel shows Arabic correctly.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What you need<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Python 3 installed (<a href=\"https:\/\/www.python.org\">https:\/\/www.python.org<\/a>)<\/li>\n\n\n<li>The pandas and openpyxl libraries<\/li>\n\n\n<li>A file with some Arabic text, for example <code>customers.csv<\/code><\/li>\n\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Install the libraries with this command:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>pip install pandas openpyxl<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Understand normalising<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\"><strong>Normalising<\/strong> means making all the different ways of writing a word look the same. Think of it like making everyone in an office use the same date format. After that, comparing is easy.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We will make these changes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>All alef forms (\u0623 \u0625 \u0622 \u0671) become a plain alef (\u0627).<\/li>\n\n\n<li>Alef maqsura (\u0649) becomes yaa (\u064a).<\/li>\n\n\n<li>Taa marbuta (\u0629) becomes haa (\u0647).<\/li>\n\n\n<li>Tashkeel marks are removed.<\/li>\n\n\n<li>Tatweel (\u0640) is removed.<\/li>\n\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">We only use the clean version for searching. You keep the original text for display.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Write the normalise function<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We use <strong>regex<\/strong> (regular expressions). A regex is like a search pattern with wildcards. It can find many characters at once.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import re\n\n# Tashkeel: fathatan to sukun, plus superscript alef and Quranic marks\nTASHKEEL = re.compile(r\"[\u0610-\u061a\u064b-\u065f\u0670]\")\nTATWEEL = \"\u0640\"\n\ndef normalise(text):\n    \"\"\"Return a clean version of Arabic text for searching.\"\"\"\n    if not isinstance(text, str):\n        return \"\"\n    text = TASHKEEL.sub(\"\", text)        # remove vowel marks\n    text = text.replace(TATWEEL, \"\")     # remove the stretch line\n    text = re.sub(\"[\u0623\u0625\u0622\u0671]\", \"\u0627\", text)   # all alef forms -&gt; \u0627\n    text = text.replace(\"\u0649\", \"\u064a\")        # alef maqsura -&gt; yaa\n    text = text.replace(\"\u0629\", \"\u0647\")        # taa marbuta -&gt; haa\n    return text.strip()\n\n# Quick test\nprint(normalise(\"\u0645\u064f\u062d\u064e\u0645\u064e\u0651\u062f\"))     # \u0645\u062d\u0645\u062f\nprint(normalise(\"\u0623\u062d\u0640\u0640\u0640\u0645\u062f\"))     # \u0627\u062d\u0645\u062f\nprint(normalise(\"\u0641\u0627\u0637\u0645\u0629\"))       # \u0641\u0627\u0637\u0645\u0647<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Run it. All three words now come out clean and simple.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Search your data<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now let us load a file and search it. Our example <code>customers.csv<\/code> has the columns <code>name<\/code> and <code>city<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>import pandas as pd\n\ndf = pd.read_csv(\"customers.csv\", encoding=\"utf-8\")\n\n# Add a hidden helper column with the clean text\ndf[\"name_clean\"] = df[\"name\"].apply(normalise)\n\ndef search(query):\n    q = normalise(query)\n    return df[df[\"name_clean\"].str.contains(q, regex=False)]\n\nresults = search(\"\u0627\u062d\u0645\u062f\")\nprint(results[[\"name\", \"city\"]])<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Now &#8220;\u0627\u062d\u0645\u062f&#8221; finds &#8220;\u0623\u062d\u0645\u062f&#8221;, &#8220;\u0625\u062d\u0645\u062f&#8221; and &#8220;\u0623\u062d\u0640\u0645\u062f&#8221;. The trick is simple: you clean <strong>both<\/strong> sides, the data and the search word. Then they always speak the same language.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We use <code>regex=False<\/code> so that special characters in the search word, like a dot, are treated as normal text.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Export to CSV so Excel shows Arabic<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">This part surprises many people. You save a CSV with Arabic text, open it in Excel, and you see strange symbols like &#8220;\u00d8\u00a3\u00d8\u00ad\u00d9\u2026\u00d8\u00af&#8221;.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The reason: Excel does not always know the file is UTF-8. <strong>UTF-8<\/strong> is the standard way to store letters from every language. The fix is a small hidden mark at the start of the file, called a <strong>BOM<\/strong>. It tells Excel &#8220;this file is UTF-8&#8221;. In Python, you add it with the encoding <code>utf-8-sig<\/code>.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Drop the helper column before saving\noutput = results.drop(columns=[\"name_clean\"])\n\n# utf-8-sig adds the BOM, so Excel reads Arabic correctly\noutput.to_csv(\"search_results.csv\", index=False, encoding=\"utf-8-sig\")<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Open <code>search_results.csv<\/code> in Excel. The names now look perfect.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 5: Or export straight to Excel<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">An <code>.xlsx<\/code> file always stores text as Unicode, so you do not need any encoding trick.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>output.to_excel(\"search_results.xlsx\", index=False)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If you want the sheet to read from right to left, you can set it with openpyxl:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from openpyxl import load_workbook\n\nwb = load_workbook(\"search_results.xlsx\")\nwb.active.sheet_view.rightToLeft = True   # show columns right to left\nwb.save(\"search_results.xlsx\")<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Common problems<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Excel still shows strange symbols<\/strong> \u2192 Check that you used <code>utf-8-sig<\/code>, not <code>utf-8<\/code>. Also open the file by double-clicking, not by pasting the text.<\/li>\n\n\n<li><strong><code>UnicodeDecodeError<\/code> when reading a CSV<\/strong> \u2192 The file may be in an old Windows Arabic format. Try <code>encoding=\"cp1256\"<\/code> in <code>read_csv<\/code>.<\/li>\n\n\n<li><strong>Search finds too much<\/strong> \u2192 Changing &#8220;\u0629&#8221; to &#8220;\u0647&#8221; can join a few different words. If that is a problem for your data, remove that line from <code>normalise<\/code>.<\/li>\n\n\n<li><strong>Empty cells break the code<\/strong> \u2192 Our function returns an empty string for empty cells, so this is already handled.<\/li>\n\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Wrap-up<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">You now have a small toolkit for Arabic text: clean it, search it, and export it without broken letters. These few lines can save you hours of manual checking in big customer or invoice lists.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Next idea: use the same <code>normalise<\/code> function to find <strong>duplicate<\/strong> customers. Group by <code>name_clean<\/code> and see which names appear more than once. You may be surprised! \ud83d\ude04<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Learn how to clean Arabic text in Python so searching works, and how to export it so Excel shows it correctly.<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[20,22,5,12,21],"class_list":["post-49","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-arabic","tag-csv","tag-excel","tag-python","tag-text"],"_links":{"self":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts\/49","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/comments?post=49"}],"version-history":[{"count":1,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts\/49\/revisions"}],"predecessor-version":[{"id":84,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/posts\/49\/revisions\/84"}],"wp:attachment":[{"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/media?parent=49"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/categories?post=49"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/fsaeed.blog\/index.php\/wp-json\/wp\/v2\/tags?post=49"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}