SpiderFoot is an open-source OSINT (Open Source Intelligence) automation tool that aggregates data from over 200 sources to build a comprehensive intelligence picture of a target. Whether you’re conducting authorized penetration test reconnaissance, threat intelligence research, or digital footprint analysis, SpiderFoot dramatically accelerates the enumeration process by running dozens of modules in parallel.
Installation
Python (Direct Install)
# Clone the repository
git clone https://github.com/smicallef/spiderfoot.git
cd spiderfoot
# Install dependencies
pip3 install -r requirements.txt --break-system-packages
# Launch the web interface
python3 sf.py -l 127.0.0.1:5001
Access at http://127.0.0.1:5001
Docker
docker pull smicallef/spiderfoot
docker run -p 5001:5001 smicallef/spiderfoot
Kali Linux
SpiderFoot is pre-installed on Kali:
spiderfoot -l 127.0.0.1:5001
Web Interface vs. CLI
SpiderFoot offers both a web UI and a command-line interface.
Web UI (recommended for beginners): Provides scan management, module selection, interactive results graphs, and export options.
CLI (for automation):
# Run a scan on a domain, all modules
python3 sf.py -s target.com -t INTERNET_NAME -m all -o csv -f results.csv
# Specific scan type
python3 sf.py -s target.com -t INTERNET_NAME -u all
Understanding Scan Types
SpiderFoot targets different entity types:
| Target Type | CLI Flag | Example |
|---|
| Domain name | INTERNET_NAME | example.com |
| IP address | IP_ADDRESS | 1.2.3.4 |
| Email address | EMAILADDR | user@example.com |
| Phone number | PHONE_NUMBER | +15551234567 |
| Username | USERNAME | johndoe |
| Company name | HUMAN_NAME | Acme Corporation |
| BGP ASN | BGP_AS_MEMBER | AS12345 |
Running Your First Scan
In the web UI:
- Click New Scan
- Enter the scan target (e.g.,
example.com)
- Select scan type: All (comprehensive), Footprint (passive only), Investigate (with active modules), or Passive (no direct contact with target)
- Click Run Scan Now
Passive mode is safest for authorized recon — it queries third-party services without directly contacting the target.
Key Modules and What They Find
SpiderFoot’s 200+ modules include:
DNS and Domain Intelligence
- sfp_dnsraw: DNS record enumeration (A, MX, TXT, SPF, DKIM)
- sfp_sublist3r: Subdomain enumeration via passive sources
- sfp_dnsdumpster: DNS recon via DNSDumpster API
- sfp_whois: WHOIS data including registrant info
Email Intelligence
- sfp_hunter: Email discovery via Hunter.io
- sfp_emailrep: Email reputation scoring
- sfp_haveIBeenPwned: Check emails against breach databases
Threat Intelligence
- sfp_virustotal: Check domains/IPs against VirusTotal
- sfp_shodan: Shodan integration for exposed services
- sfp_abuseipdb: Check IPs against AbuseIPDB
- sfp_threatcrowd: IOC lookup
- sfp_linkedin: LinkedIn profile discovery
- sfp_github: GitHub user and repo discovery
- sfp_twitter: Twitter/X account discovery
- sfp_pastebin: Pastes mentioning the target
SSL/TLS Certificates
- sfp_ssl: Certificate inspection and CT log mining
- sfp_crt: Certificate Transparency log queries
Configuring API Keys
Many modules require API keys for full functionality. Configure them in Settings > Global Settings:
| Service | Get Key At |
|---|
| Shodan | shodan.io |
| VirusTotal | virustotal.com |
| Hunter.io | hunter.io |
| SecurityTrails | securitytrails.com |
| FullContact | fullcontact.com |
| HaveIBeenPwned | haveibeenpwned.com |
Free API tiers are sufficient for most OSINT work.
Analyzing Results
After a scan completes, SpiderFoot displays results in several views:
- Graph — Visual relationship map of discovered entities
- Browse — Filter results by data type
- Summary — Count of each entity type found
Look for high-value finds:
- Leaked credentials (via breach database modules)
- Exposed services (Shodan integration)
- Email addresses of key personnel (for phishing simulation in authorized tests)
- Technology stack (web server, CMS, frameworks)
- IP ranges associated with the target ASN
- Related domains (acquisitions, subsidiaries)
Exporting Data
Export scan results for reporting or further analysis:
- CSV: Good for spreadsheet analysis
- JSON: Machine-readable for tool integration
- GEXF: Import into Gephi for advanced graph visualization
- PDF: Via the web UI export button
SpiderFoot HX is the commercial SaaS version at spiderfoot.net. It offers:
- Always-on scanning with alerts
- Team collaboration
- More data sources
- Historical scan comparison
The free tier allows limited scans — sufficient for evaluation.
Operational Security
When using SpiderFoot for authorized testing:
- Use Passive or Footprint mode first — does not directly contact the target
- Be aware some modules use your real IP for queries — use a VPN
- Third-party API queries may alert the target (e.g., VirusTotal URL scans are public)
- Document all recon activity as part of your pentest methodology
SpiderFoot is a force multiplier for recon phases. A scan that would take days of manual work runs in hours, giving you a comprehensive starting point for further investigation. Combine it with Maltego for advanced relationship visualization and Shodan for deeper technical enumeration.