# jieba — Chinese Text Segmentation in Python

> Fast and accurate Chinese word segmentation library with HMM-based new word detection, keyword extraction, and custom dictionary support.

## Install

Save in your project root:

# jieba — Chinese Text Segmentation in Python

## Quick Use
```bash
pip install jieba
```
```python
import jieba
words = jieba.lcut("自然语言处理是人工智能的核心技术")
print(words)
# ['自然语言', '处理', '是', '人工智能', '的', '核心技术']
```

## Introduction
jieba (meaning "to stutter" in Chinese) is the most widely used Chinese word segmentation library for Python. Since Chinese text has no spaces between words, tokenization is the first step in any Chinese NLP pipeline. jieba uses a prefix dictionary combined with dynamic programming and a Hidden Markov Model to segment text quickly and accurately.

## What jieba Does
- Segments Chinese text into words using dictionary-based and statistical methods
- Detects out-of-vocabulary words using HMM-based new word discovery
- Supports three segmentation modes: precise, full, and search-engine optimized
- Extracts keywords using TF-IDF and TextRank algorithms
- Allows custom dictionaries and user-defined word frequencies

## Architecture Overview
jieba builds a directed acyclic graph (DAG) from a prefix dictionary trie, then applies dynamic programming to find the most probable segmentation path based on word frequencies. For words not in the dictionary, an HMM with Viterbi decoding identifies likely word boundaries. The library loads its dictionary lazily on first use and supports parallel tokenization across multiple CPU cores.

## Setup & Configuration
- Install via `pip install jieba` (pure Python, no compiled dependencies)
- Load custom dictionaries with `jieba.load_userdict('custom_dict.txt')`
- Add individual words at runtime using `jieba.add_word('深度学习', freq=10000)`
- Enable parallel mode with `jieba.enable_parallel(4)` for multi-core processing
- Use `jieba.analyse` for keyword extraction with TF-IDF or TextRank

## Key Features
- Three segmentation modes for different accuracy and recall trade-offs
- HMM-based discovery of words not in the dictionary
- Built-in keyword extraction via TF-IDF and TextRank
- Custom dictionary support for domain-specific vocabularies
- Parallel processing for batch segmentation workloads

## Comparison with Similar Tools
- **pkuseg** — higher accuracy on specific domains, jieba is faster and more flexible
- **HanLP** — full NLP pipeline with multi-task models, jieba focuses on segmentation
- **LAC (Baidu)** — neural-based with POS tagging built in, jieba is lighter weight
- **spaCy (zh)** — transformer-based Chinese support, jieba is simpler to deploy

## FAQ
**Q: How accurate is jieba for specialized text like medical or legal Chinese?**
A: Accuracy improves when you load a domain-specific dictionary via `load_userdict()`. Without one, uncommon terms may be over-segmented.

**Q: Does jieba work with traditional Chinese characters?**
A: The default dictionary covers simplified Chinese. For traditional Chinese, load a traditional character dictionary.

**Q: Can I use jieba in production web services?**
A: Yes. Pre-initialize the dictionary at startup with `jieba.initialize()` to avoid first-call latency, and use parallel mode for throughput.

**Q: Does jieba support Python 3?**
A: Yes. jieba supports Python 2.7 and Python 3.x.

## Sources
- https://github.com/fxsjy/jieba
- https://pypi.org/project/jieba/

---
Source: https://tokrepo.com/en/workflows/asset-7969ddc0
Author: AI Open Source