Data Sources & Methodology¶

Last updated: 2026-04-01 07:21 UTC

How This Dashboard Works¶

This dashboard is automatically generated by a Python data pipeline that:

1. Fetches data from multiple threat intelligence sources via APIs and RSS feeds
2. Processes and analyses the raw data to extract key metrics and trends
3. Generates MkDocs markdown pages with embedded data and chart configurations
4. Builds the static site using MkDocs with the Material theme
5. Deploys to GitHub Pages via GitHub Actions

The pipeline runs on a scheduled basis (configurable via GitHub Actions cron) to keep data current.

Data Sources¶

Australian Government¶

International Government¶

Community Threat Intelligence¶

Strategic & Geopolitical¶

Optional (API Key Required)¶

API Key Setup Guide¶

All API keys are optional. The dashboard will still build without them, but enabling them unlocks richer data. Keys should be stored as GitHub Secrets (Settings > Secrets and variables > Actions) for the automated pipeline, or as environment variables for local development.

NVD (National Vulnerability Database)¶

The NVD API works without a key, but rate-limits unauthenticated requests to 5 per 30 seconds. With a key, you get 50 per 30 seconds.

1. Go to https://nvd.nist.gov/developers/request-an-api-key
2. Enter your email address and organisation
3. Check your inbox for the API key (arrives within minutes)
4. GitHub Secret name: NVD_API_KEY

Cost: Free, no usage limits beyond rate throttling.

AlienVault OTX (Open Threat Exchange)¶

OTX provides community-sourced threat intelligence pulses, IOCs, and adversary tracking.

1. Go to https://otx.alienvault.com/ and create a free account
2. Once logged in, go to Settings (click your avatar, top-right)
3. Your API key is displayed under OTX Key on the settings page
4. Optionally subscribe to relevant pulses (e.g. search for "Australia", "APT40", "Critical Infrastructure") to get more targeted data
5. GitHub Secret name: OTX_API_KEY

Cost: Free. No usage limits for the public API.

GreyNoise¶

GreyNoise classifies internet scanning traffic as benign or malicious. The community tier provides basic IP lookups.

1. Go to https://viz.greynoise.io/signup and create a free Community account
2. Once logged in, go to Account > API Key
3. Copy the API key
4. GitHub Secret name: GREYNOISE_API_KEY

Cost: Free Community tier (limited queries/day). Paid tiers available for full trend data and bulk lookups.

Shodan¶

Shodan indexes internet-facing devices and services globally. Used here to query Australian IP space exposure.

1. Go to https://account.shodan.io/register and create an account
2. Once logged in, your API key is shown on the Account page
3. The free tier provides basic search. A paid membership (one-time USD $49 for lifetime) unlocks filters like country:AU
4. GitHub Secret name: SHODAN_API_KEY

Cost: Free tier available. Lifetime membership recommended for country-level queries.

abuse.ch (URLhaus, ThreatFox, MalwareBazaar)¶

abuse.ch threat intelligence APIs have required authentication since June 2025. An Auth-Key is needed for URLhaus, ThreatFox, and MalwareBazaar.

1. Go to https://auth.abuse.ch/ and register for a free account
2. Once registered and logged in, navigate to your profile/account settings
3. Copy your Auth-Key (API authentication key)
4. GitHub Secret name: ABUSE_CH_AUTH_KEY

Cost: Free. Community-driven threat intelligence platform.

Adding Secrets to GitHub¶

In your repository:

1. Go to Settings > Secrets and variables > Actions
2. Click New repository secret
3. Add each key with the exact name shown above (e.g. NVD_API_KEY)
4. The GitHub Actions workflow already references these secrets

Architecture¶

graph LR
    A[RSS Feeds] --> D[Python Fetchers]
    B[REST APIs] --> D
    C[Curated Data] --> D
    D --> E[JSON Data Files]
    E --> F[Page Generator]
    F --> G[MkDocs Markdown]
    G --> H[MkDocs Build]
    H --> I[GitHub Pages]

Running Locally¶

# Install dependencies
pip install -r requirements.txt

# Set API keys (optional, enhances data)
export SHODAN_API_KEY="your-key"
export NVD_API_KEY="your-key"
export OTX_API_KEY="your-key"
export GREYNOISE_API_KEY="your-key"
export ABUSE_CH_AUTH_KEY="your-key"

# Fetch data and generate pages
python -m scripts.build_all

# Preview locally
mkdocs serve

# Build static site
mkdocs build