SpiderFoot — Automated OSINT Collection and Attack Surface Mapping

Introduction

SpiderFoot automates open-source intelligence gathering by correlating data from over 200 public sources including DNS records, WHOIS, search engines, breach databases, and social media platforms. Security teams use it to map attack surfaces, discover shadow IT, and perform due diligence assessments from a single interface.

What SpiderFoot Does

• Queries 200+ data sources automatically given a target domain, IP, email, or name
• Correlates discovered entities (subdomains, IPs, emails, credentials) into a graph
• Identifies potential vulnerabilities like exposed services, leaked credentials, and misconfigurations
• Provides both a web-based UI for interactive exploration and a CLI for automated pipelines
• Exports results to CSV, JSON, or GEXF graph format for external analysis

Architecture Overview

SpiderFoot is a Python application with a modular architecture. Each data source is a self-contained module (200+ modules) that receives scan events and produces new findings. The correlation engine links discoveries across modules, building a directed graph of relationships. The web UI runs on CherryPy and stores scan data in a local SQLite database, while the CLI mode supports headless batch scanning.

Self-Hosting & Configuration

• Install via pip or run the official Docker image for isolated deployments
• Configure API keys for premium data sources (Shodan, VirusTotal, etc.) via the web UI settings
• Set scan scope limits to prevent accidental enumeration beyond authorized targets
• Schedule recurring scans via cron or integrate with CI/CD for continuous attack surface monitoring
• Store scan results externally by configuring database paths or exporting via the REST API

Key Features

• 200+ OSINT modules covering DNS, WHOIS, geo-IP, breach data, social media, and dark web
• Correlation engine automatically links entities across disparate data sources
• Scan types range from passive (no target interaction) to active (direct probing)
• Built-in REST API for integration with SIEM, ticketing, and orchestration platforms
• Visualization of entity relationships as interactive network graphs

Comparison with Similar Tools

• Maltego — powerful GUI-based OSINT with commercial transforms; SpiderFoot is fully open-source with 200+ built-in modules
• Recon-ng — modular CLI framework requiring manual module selection; SpiderFoot auto-chains modules based on discovered data types
• theHarvester — focused on email and subdomain enumeration; SpiderFoot provides broader coverage and entity correlation
• Amass — specialized in subdomain enumeration with DNS brute-forcing; SpiderFoot covers more OSINT categories beyond DNS

FAQ

Q: Is SpiderFoot legal to use? A: SpiderFoot queries publicly available data sources. Always ensure you have authorization to scan targets and comply with local laws regarding reconnaissance activities.

Q: Can I run scans without any API keys? A: Yes. Many modules work without API keys using public endpoints, though premium sources like Shodan or VirusTotal require free or paid keys for full results.

Q: How long does a full scan take? A: Depends on target size and enabled modules. A typical domain scan completes in 5-30 minutes; large organizations with thousands of subdomains may take hours.

Q: Does SpiderFoot actively probe targets? A: It supports both passive (no direct contact) and active scan modes. You can restrict scans to passive-only for non-intrusive reconnaissance.

Sources

• https://github.com/smicallef/spiderfoot
• https://www.spiderfoot.net/