HanLP — Multi-Language NLP Library for Production

Introduction

HanLP is a production-ready NLP library that provides joint multi-task learning models for 100+ languages. Originally built for Chinese NLP, it has grown into a multilingual framework with state-of-the-art transformer-based models for tokenization, part-of-speech tagging, named entity recognition, dependency parsing, and semantic role labeling.

What HanLP Does

• Tokenizes and segments text across Chinese, English, Japanese, and 100+ other languages
• Performs part-of-speech tagging, lemmatization, and morphological feature extraction
• Extracts named entities (person, location, organization) and custom entity types
• Builds dependency parse trees and semantic dependency graphs
• Runs semantic role labeling and constituency parsing in a single forward pass

Architecture Overview

HanLP v2 uses a multi-task learning architecture where a shared transformer encoder (XLM-R or ELECTRA) feeds into task-specific decoders. This design allows all NLP tasks to share representations, reducing memory and improving accuracy through transfer learning. Models are loaded lazily and cached locally for offline use.

Setup & Configuration

• Install via pip install hanlp for the full package or hanlp[tf] for TensorFlow backend
• Download pretrained models on first use; they cache under ~/.hanlp
• Use hanlp.pretrained to browse available model keys by task and language
• Configure GPU usage via standard PyTorch or TensorFlow device placement
• Deploy as a REST service with hanlp.server or integrate via the Java API for JVM projects

Key Features

• Joint multi-task inference runs all NLP tasks in one forward pass
• Pretrained models for 100+ languages based on Universal Dependencies
• Supports both PyTorch and TensorFlow backends
• RESTful server mode for production microservice deployments
• Java API available for JVM-based applications

Comparison with Similar Tools

• spaCy — production-focused single-task pipelines, HanLP offers joint multi-task models
• Stanza — Stanford research-oriented, HanLP provides broader language coverage
• NLTK — educational and rule-based, HanLP is transformer-based and faster
• jieba — Chinese segmentation only, HanLP covers full NLP pipeline

FAQ

Q: Does HanLP support languages other than Chinese? A: Yes. HanLP v2 supports 100+ languages through multilingual transformer models trained on Universal Dependencies treebanks.

Q: Can I fine-tune HanLP models on my own data? A: Yes. HanLP provides training scripts and configuration files for fine-tuning on custom datasets in CoNLL format.

Q: What is the difference between HanLP v1 and v2? A: v1 uses traditional algorithms and a Java API. v2 is transformer-based, Python-first, and supports multi-task learning.

Q: How large are the pretrained models? A: Base models are around 500 MB. Larger models with more tasks can reach 1-2 GB.

Sources

• https://github.com/hankcs/HanLP
• https://hanlp.hankcs.com