Chinese Wikipedia Corpus: Processing 990k Articles for NLP Tasks

The Chinese Wikipedia dump contains over 990,000 entries, making it an invaluable resource for NLP research—provided it is cleaned effectively. Processing the raw XML, however, is a significant challenge. Most developers rely on either Wikipedia Extractor or gensim’s wikicorpus library, but both are often insufficient for precise work. Wikipedia Extractor frequently strips out essential spaces and brackets, while gensim.corpora.wikicorpus.WikiCorpus is even more aggressive, removing all punctuation. This results in the total loss of sentence boundaries, rendering the text unusable for many linguistic models.

The following method offers a more refined approach to handling this data.

OpenCC: Simplified and Traditional Conversion

OpenCC is the standard tool for converting scripts between Simplified and Traditional Chinese. While some online guides suggest the installation is a complex process, it is actually quite simple.

You can install it directly via pip:

pip install opencc-python-reimplemented

Alternatively, you can clone the repository and run python setup.py install. Both methods are equally effective.

The usage is straightforward:

from opencc import OpenCC

openCC = OpenCC('s2t')  # Converts Simplified to Traditional
# Additional conversions can be set later: openCC.set_conversion('s2tw')
to_convert = '开放中文转换'
converted = openCC.convert(to_convert)

Available conversion modes include:

• hk2s: Traditional (Hong Kong) to Simplified
• s2hk: Simplified to Traditional (Hong Kong)
• s2t: Simplified to Traditional
• s2tw: Simplified to Traditional (Taiwan)
• s2twp: Simplified to Traditional (Taiwan, with phrase mapping)
• t2hk: Traditional to Traditional (Hong Kong)
• t2s: Traditional to Simplified
• t2tw: Traditional to Traditional (Taiwan)
• tw2s: Traditional (Taiwan) to Simplified
• tw2sp: Traditional (Taiwan) to Simplified (with phrase mapping)

Parsing the Chinese Wikipedia Dump

First, download the source file, zhwiki-20180301-pages-articles-multistream.xml.bz2, from the official Wikimedia mirrors. By processing it through wiki_parser.py, you can extract clean, structured text blocks like the following:

=== 词源 ===
英语词语Philosophy（philosophia）源于古希腊语中的φιλοσοφία，意思为「爱智慧」，有时也译为「智慧的朋友」

=== 主分支 ===
哲学可以分为很多不同的分支，主要包括形而上学、知识论、伦理学、逻辑学和美学。
* 形而上学/宇宙论
* 知识论

This parser maintains the document structure, allowing you to segment specific sections while preserving internal links.

Keyword Search: A Work in Progress

Keyword retrieval within this corpus faces the same hurdles as standard Chinese word segmentation. Consider the query: "民用无人机到自主驾驶汽车" (Civilian drones to autonomous cars). Ideally, the system should identify "无人机" (drone) and "自主驾驶汽车" (autonomous vehicle). A naive tokenizer, however, often fragments the input into "民用; 无人机; 自主; 驾驶; 汽车; 无人...", causing the conceptual hierarchy to collapse. For most applications, "驾驶汽车" is the required unit, as "汽车" alone is too broad.

The current prototype processes queries as follows:

search_txt = '民用无人机到自主驾驶汽车'
search(search_txt)

It returns a dictionary containing relevant Wikipedia excerpts:

{
"无人机": "各种类型的无人机。\r\n无人机（Uncrewed vehicle、Unmanned vehicle、Drone）或称无人载具是一种无搭载人员的载具。通常使用遥控、导引或自动驾驶来控制。可在科学研究、军事、休闲娱乐用途上使用。\r\n在日常用语中，“无人机”被特指为“无人飞行载具”。\r\n",
"汽车": "Benz Patent-Motorwagen Nummer 1，第一辆“现代汽车”。\r\n1927年的汽车，福特T型车。\r\n1942年的汽车，纳许大使。\r\n1980年的汽车，大众帕萨特\r\n1999年的汽车，西雅特托莱多\r\n2008年的超级跑车，科尼赛克CCX。\r\n日产Maxima SR\r\n怪兽卡车\r\n汽车或称机动车（英式英语：car；美式英语：automobile；美国口语：auto），即本身具有动力得以驱动，不须依轨道或电缆，得以动力行驶之车辆。广义来说，具有四轮或以上行驶的车辆，普遍多称为汽车。虽然，长久以来学术各界对「谁是第一位汽车发明者」皆有不同的看法及论述，未有完全一致性的看法，但是，绝大部份学者皆将德国工程师卡尔·本茨视为第一位发明者。美国人亨利·福特首先大量生产平价汽车，是使汽车得以普及化的人。\r\n",
"驾驶": "驾驶，指的是人类在操纵交通工具或一些机械设备时的行为，可分为机动车驾驶、船舶驾驶、列车驾驶、航空器驾驶、其它驾驶，这些一般都属于真实驾驶，可采用手动驾驶或自动驾驶的方式进行驾驶。对于通过电子系统以游戏等方式进行模拟真实驾驶情况的行为，则被称为虚拟驾驶。对于交通工具或一些机械设备的驾驶者，被称为驾驶员。对于驾驶交通工具或一些机械设备时应随身携带的证件，则被称为驾驶证。\r\n"
}

The module retrieves full paragraphs, which, while not yet perfect, provides far more context than a simple list of fragmented tokens. The next objective is to implement more sophisticated phrase detection so that terms like "自主驾驶汽车" remain intact. For now, the corpus and parser provide a clean, reliable foundation for further development.