# HanLP — Multi-Language NLP Library for Production

> Joint multi-task NLP framework supporting tokenization, POS tagging, NER, parsing, and more across 100+ languages.

## Install

Save in your project root:

# HanLP — Multi-Language NLP Library for Production

## Quick Use
```bash
pip install hanlp
```
```python
import hanlp
HanLP = hanlp.load(hanlp.pretrained.mtl.UD_ONTONOTES_TOK_POS_LEM_FEA_NER_SRL_DEP_SDP_CON_XLMR_BASE)
print(HanLP(['In 2024,�nowledge graphs are transforming AI.', '自然语言处理越来越强大。']))
```

## Introduction
HanLP is a production-ready NLP library that provides joint multi-task learning models for 100+ languages. Originally built for Chinese NLP, it has grown into a multilingual framework with state-of-the-art transformer-based models for tokenization, part-of-speech tagging, named entity recognition, dependency parsing, and semantic role labeling.

## What HanLP Does
- Tokenizes and segments text across Chinese, English, Japanese, and 100+ other languages
- Performs part-of-speech tagging, lemmatization, and morphological feature extraction
- Extracts named entities (person, location, organization) and custom entity types
- Builds dependency parse trees and semantic dependency graphs
- Runs semantic role labeling and constituency parsing in a single forward pass

## Architecture Overview
HanLP v2 uses a multi-task learning architecture where a shared transformer encoder (XLM-R or ELECTRA) feeds into task-specific decoders. This design allows all NLP tasks to share representations, reducing memory and improving accuracy through transfer learning. Models are loaded lazily and cached locally for offline use.

## Setup & Configuration
- Install via `pip install hanlp` for the full package or `hanlp[tf]` for TensorFlow backend
- Download pretrained models on first use; they cache under `~/.hanlp`
- Use `hanlp.pretrained` to browse available model keys by task and language
- Configure GPU usage via standard PyTorch or TensorFlow device placement
- Deploy as a REST service with `hanlp.server` or integrate via the Java API for JVM projects

## Key Features
- Joint multi-task inference runs all NLP tasks in one forward pass
- Pretrained models for 100+ languages based on Universal Dependencies
- Supports both PyTorch and TensorFlow backends
- RESTful server mode for production microservice deployments
- Java API available for JVM-based applications

## Comparison with Similar Tools
- **spaCy** — production-focused single-task pipelines, HanLP offers joint multi-task models
- **Stanza** — Stanford research-oriented, HanLP provides broader language coverage
- **NLTK** — educational and rule-based, HanLP is transformer-based and faster
- **jieba** — Chinese segmentation only, HanLP covers full NLP pipeline

## FAQ
**Q: Does HanLP support languages other than Chinese?**
A: Yes. HanLP v2 supports 100+ languages through multilingual transformer models trained on Universal Dependencies treebanks.

**Q: Can I fine-tune HanLP models on my own data?**
A: Yes. HanLP provides training scripts and configuration files for fine-tuning on custom datasets in CoNLL format.

**Q: What is the difference between HanLP v1 and v2?**
A: v1 uses traditional algorithms and a Java API. v2 is transformer-based, Python-first, and supports multi-task learning.

**Q: How large are the pretrained models?**
A: Base models are around 500 MB. Larger models with more tasks can reach 1-2 GB.

## Sources
- https://github.com/hankcs/HanLP
- https://hanlp.hankcs.com

---
Source: https://tokrepo.com/en/workflows/asset-4dd02b7f
Author: AI Open Source