ARC Task Gen — Synthetic ARC-AGI Task Generator

Introduction

ARC Task Gen is a Python tool by Pathway that generates original ARC-AGI-1-style visual reasoning tasks. Each generated task is distribution-matched to the public evaluation set, giving researchers a scalable way to create training and benchmarking data for abstract reasoning systems without manually designing puzzles.

What ARC Task Gen Does

• Generates novel grid-based visual reasoning tasks matching the ARC-AGI-1 format
• Produces input-output example pairs with verifiable transformation rules
• Ensures statistical alignment with the official ARC evaluation distribution
• Outputs tasks in the standard ARC JSON schema for direct use with evaluation harnesses
• Supports configurable difficulty and transformation-type filters

Architecture Overview

The generator uses a pipeline of composable transformation primitives — rotations, reflections, color mappings, object manipulations — that are sampled and chained according to a learned distribution model. A validation layer checks each generated task against solvability constraints and distribution metrics before emitting it. The system runs entirely on CPU and produces hundreds of tasks per minute.

Self-Hosting & Configuration

• Install from PyPI or clone the repository and install with pip
• Configure generation parameters via CLI flags or a YAML config file
• Set a random seed for reproducible task batches
• Output format supports JSON and the ARC-AGI evaluation harness directory layout
• No external API keys or GPU required

Key Features

• Distribution-matched output ensures generated tasks reflect real ARC difficulty curves
• Deterministic seeding for reproducible research experiments
• Extensible primitive library for adding custom transformation types
• Built-in deduplication against the official ARC public training and evaluation sets
• Lightweight with zero heavy dependencies

Comparison with Similar Tools

• Manual ARC authoring — slow and labor-intensive; ARC Task Gen automates the process
• LLM-based task generation — can hallucinate invalid puzzles; ARC Task Gen validates every output
• RE-ARC — another procedural generator; ARC Task Gen focuses on tighter distribution matching
• ARC-AGI official set — fixed 800 tasks; ARC Task Gen provides unlimited new ones

FAQ

Q: Are the generated tasks guaranteed to be solvable? A: Yes. Each task passes a validation step that confirms at least one consistent transformation rule.

Q: Can I use generated tasks for model training? A: Yes. The output follows the standard ARC JSON format and is ready for training pipelines.

Q: How closely do generated tasks match the official distribution? A: The generator is calibrated against the public eval set and reports distribution metrics per batch.

Q: Does it require a GPU? A: No. Task generation is CPU-only and runs on any Python 3.10+ environment.

Sources

• https://github.com/pathwaycom/arc-task-gen
• https://arcprize.org