jieba — Chinese Text Segmentation in Python

Introduction

jieba (meaning "to stutter" in Chinese) is the most widely used Chinese word segmentation library for Python. Since Chinese text has no spaces between words, tokenization is the first step in any Chinese NLP pipeline. jieba uses a prefix dictionary combined with dynamic programming and a Hidden Markov Model to segment text quickly and accurately.

What jieba Does

• Segments Chinese text into words using dictionary-based and statistical methods
• Detects out-of-vocabulary words using HMM-based new word discovery
• Supports three segmentation modes: precise, full, and search-engine optimized
• Extracts keywords using TF-IDF and TextRank algorithms
• Allows custom dictionaries and user-defined word frequencies

Architecture Overview

jieba builds a directed acyclic graph (DAG) from a prefix dictionary trie, then applies dynamic programming to find the most probable segmentation path based on word frequencies. For words not in the dictionary, an HMM with Viterbi decoding identifies likely word boundaries. The library loads its dictionary lazily on first use and supports parallel tokenization across multiple CPU cores.

Setup & Configuration

• Install via pip install jieba (pure Python, no compiled dependencies)
• Load custom dictionaries with jieba.load_userdict('custom_dict.txt')
• Add individual words at runtime using jieba.add_word('深度学习', freq=10000)
• Enable parallel mode with jieba.enable_parallel(4) for multi-core processing
• Use jieba.analyse for keyword extraction with TF-IDF or TextRank

Key Features

• Three segmentation modes for different accuracy and recall trade-offs
• HMM-based discovery of words not in the dictionary
• Built-in keyword extraction via TF-IDF and TextRank
• Custom dictionary support for domain-specific vocabularies
• Parallel processing for batch segmentation workloads

Comparison with Similar Tools

• pkuseg — higher accuracy on specific domains, jieba is faster and more flexible
• HanLP — full NLP pipeline with multi-task models, jieba focuses on segmentation
• LAC (Baidu) — neural-based with POS tagging built in, jieba is lighter weight
• spaCy (zh) — transformer-based Chinese support, jieba is simpler to deploy

FAQ

Q: How accurate is jieba for specialized text like medical or legal Chinese? A: Accuracy improves when you load a domain-specific dictionary via load_userdict(). Without one, uncommon terms may be over-segmented.

Q: Does jieba work with traditional Chinese characters? A: The default dictionary covers simplified Chinese. For traditional Chinese, load a traditional character dictionary.

Q: Can I use jieba in production web services? A: Yes. Pre-initialize the dictionary at startup with jieba.initialize() to avoid first-call latency, and use parallel mode for throughput.

Q: Does jieba support Python 3? A: Yes. jieba supports Python 2.7 and Python 3.x.

Sources

• https://github.com/fxsjy/jieba
• https://pypi.org/project/jieba/