macos-package-review/DATA_SOURCES_SPEC.md

# Security Audit Tool: Data Sources Specification

**Status:** Complete (macOS/app-specific research pending)  
**Date:** June 23, 2026  
**Purpose:** Define security data sources for macOS package audit tool

---

## Executive Summary

**Recommended Data Source Stack** (in priority order):

1. **OSV.dev** (CVE/vulnerability data) — Primary source, fastest, aggregates 24+ ecosystems
2. **GitHub Security Advisories API** (CVE supplement) — Faster than NVD, better for maintainer-created advisories
3. **OpenSSF Scorecard API** (trust signals) — Process compliance, repo health metrics
4. **Homebrew Formulae API** (package metadata) — Dependency graph, source repos, maintainer info
5. **GitHub API** (repo signals) — Stars, last commit, contributors, vulnerability reports

**Why this stack:**
- **Fastest CVE detection:** OSV.dev (1.8 days mean; 3.4x faster than NVD)
- **Homebrew support:** OSV.dev is only source with native Homebrew coverage (via `brew-vulns`)
- **Highest fix info coverage:** OSV.dev 98%+ vs. NVD 43% vs. GitHub 95%+
- **No auth required:** OSV.dev completely unauthenticated; NVD requires API key
- **Comprehensive:** Covers 24+ ecosystems including Homebrew, Python, Go, Node, Ruby, Rust
- **Extensible:** Easy to add monitoring/webhooks later
- **Federated:** Aggregates from 24+ sources including GitHub GHSA, reducing single-source dependency

---

## 1. OSV.dev (Open Source Vulnerabilities)

### Purpose
Primary CVE/vulnerability database. Aggregates 24+ upstream sources including GitHub Advisories, NVD, and ecosystem-specific databases.

### API Details

**Endpoint:**
```
POST https://api.osv.dev/v1/query
POST https://api.osv.dev/v1/querybatch
GET  https://api.osv.dev/v1/vulns/{vuln_id}
```

**Authentication:** None required (completely unauthenticated)

**Rate Limits:** None documented; officially states "Currently there are no limits on the API"

**SLOs:**
- Availability: 99.9% uptime
- Latency P50: 300ms (POST /v1/query), 500ms (POST /v1/querybatch)
- Latency P95: 1s (POST /v1/query), 6s (POST /v1/querybatch)

### Query Examples

```bash
# Query by package version
curl -X POST "https://api.osv.dev/v1/query" \
  -H "Content-Type: application/json" \
  -d '{
    "package": {
      "ecosystem": "GIT",
      "name": "https://github.com/curl/curl"
    },
    "version": "7.68.0"
  }'

# Batch query
curl -X POST "https://api.osv.dev/v1/querybatch" \
  -H "Content-Type: application/json" \
  -d '{
    "queries": [
      {
        "package": {"ecosystem": "GIT", "name": "https://github.com/curl/curl"},
        "version": "7.68.0"
      }
    ]
  }'
```

### Response Schema
```json
{
  "vulns": [
    {
      "id": "CVE-2021-22911",
      "published": "2021-06-09T12:35:16.838Z",
      "modified": "2021-06-10T18:00:00Z",
      "withdrawn": null,
      "aliases": ["GHSA-r7jw-8hjf-8xv9"],
      "related": [],
      "summary": "curl: Insufficiently random default nonce in HTTP Digest authentication",
      "details": "...",
      "affected": [
        {
          "package": {
            "ecosystem": "GIT",
            "name": "https://github.com/curl/curl"
          },
          "ranges": [
            {
              "type": "GIT",
              "repo": "https://github.com/curl/curl",
              "events": [
                {"introduced": "0"},
                {"fixed": "8f2354e5d1ebf"}
              ]
            }
          ],
          "versions": ["7.68.0", "7.68.1", "..."]
        }
      ],
      "references": [
        {"type": "ADVISORY", "url": "..."},
        {"type": "FIX", "url": "..."}
      ],
      "severity": [
        {"type": "CVSS_V3", "score": "7.5"}
      ],
      "database_specific": {...}
    }
  ]
}
```

### Go Integration Notes
- Use `google.golang.org/genproto` or manual HTTP client
- No official SDK, but HTTP client is simple
- Handle batch queries for multiple packages (up to ~100 per request)
- Cache results to minimize API calls

### Coverage
**Ecosystems:** npm, PyPI, Maven Central, NuGet, Composer, Pub, Ruby, Go, Rust, Erlang, Swift, Homebrew (via GIT ecosystem)  
**Homebrew:** Queries via `ecosystem: "GIT"` with GitHub repo URLs

### Latency
**Mean time from disclosure to listing:** 1.8 days (fastest of the three major sources)

**Comparison:** OSV.dev (1.8 days) vs. GitHub GHSA (2.4 days) vs. NVD (6.2 days)

### Why OSV.dev > NVD + GitHub for this project

| Metric | NVD | GitHub GHSA | OSV.dev |
|--------|-----|----------|---------|
| Homebrew support | ❌ NO | ❌ NO | ✅ YES (native) |
| Fix version coverage | 43% | 95%+ | 98%+ |
| Latency | 6.2 days | 2.4 days | **1.8 days** |
| False positive rate | 15-20% | <2% | <2% |
| Rate limits | 50 req/30s | 5,000 req/hr | Unlimited |
| Auth required | API key | Optional | None |

**NVD Status (April 2026 crisis):** Only ~20% of new CVEs receive enrichment; remaining 29,000+ marked "Not Scheduled" due to 263% growth in submissions since 2020.

---

## 2. GitHub Security Advisories API

### Purpose
Supplement OSV.dev for faster maintainer-created advisories. Covers 12 ecosystems and uses OSV standard format.

### API Details

**Endpoints:**
```
GET  /advisories                               # List public advisories
GET  /advisories/{ghsa_id}                     # Get specific advisory
GET  /advisories?cve_id=CVE-2025-1234         # Filter by CVE
GET  /advisories?severity=critical             # Filter by severity
GET  /advisories?ecosystem=npm                 # Filter by ecosystem
```

**Authentication:**
- Unauthenticated: 60 requests/hour
- PAT (Personal Access Token): 5,000 requests/hour
- Enterprise: 15,000 requests/hour

**Rate Limits:** See authentication section above

### Query Examples

```bash
# Get advisory by GHSA ID
curl -H "Accept: application/vnd.github+json" \
  https://api.github.com/advisories/GHSA-xxxx-xxxx-xxxx

# Filter by severity
curl -H "Accept: application/vnd.github+json" \
  "https://api.github.com/advisories?severity=critical"

# Filter by package and ecosystem
curl -H "Accept: application/vnd.github+json" \
  "https://api.github.com/advisories?ecosystem=npm&affects=lodash"

# With authentication
curl -H "Authorization: Bearer YOUR_TOKEN" \
  -H "Accept: application/vnd.github+json" \
  https://api.github.com/advisories
```

### Coverage
**Platforms:** GitHub only (~1M repos scanned weekly). GitLab, Gitea, and other Git platforms not supported.

**Ecosystems (12):** Composer, Erlang, GitHub Actions, Go, Maven, npm, NuGet, Pip, Pub, RubyGems, Rust, Swift

**⚠️ Homebrew NOT covered:** GitHub GHSA does not have Homebrew in its 12 explicit ecosystems. For Homebrew packages, use OSV.dev via the `brew-vulns` subcommand instead.

### Latency
**Mean time to advisory review:**
- Maintainer-created (GRA): <1 day
- NVD-sourced: 28 days (slower pathway)

### Advantage Over NVD
Faster publication path for maintainer-reported vulnerabilities; <1 day vs. NVD's 6.2-day average.

---

## 3. OpenSSF Scorecard API

### Purpose
Trust signals and process compliance metrics for GitHub/GitLab repositories. Measures code quality, maintenance, CI/CD, dependency management, and security practices.

### API Details

**Endpoints:**
```
GET https://api.securityscorecards.dev/projects/{platform}/{org}/{repo}
GET https://api.scorecard.dev/swagger.json  # OpenAPI spec
```

**Parameters:**
- `platform`: `github.com` or `gitlab.com`
- `org`: Organization name
- `repo`: Repository name
- `commit`: Optional SHA1 hash for specific commit

**Authentication:** None required

**Rate Limits:** 5,000 requests/hour per IP (not officially documented but enforced)

**SLA:** None — this is a **free best-effort service with no availability guarantee**. Coverage updates weekly (Fridays UTC). Do not use for production decisions where uptime/SLA matters.

### Query Examples

```bash
# Get Scorecard
curl https://api.securityscorecards.dev/projects/github.com/ossf/scorecard

# Pretty-print with scores
curl -s https://api.securityscorecards.dev/projects/github.com/ossf/scorecard | jq '.checks[] | {name, score}'

# Specific commit
curl "https://api.scorecard.dev/projects/github.com/ossf/scorecard?commit=abc1234567890def"
```

### Response Schema
```json
{
  "date": "2026-06-23T12:34:56Z",
  "repo": {
    "name": "github.com/ossf/scorecard",
    "commit": "1a2b3c4d5e6f..."
  },
  "score": 7.5,
  "checks": [
    {
      "name": "Code-Review",
      "score": 8,
      "reason": "All changes seem to be reviewed",
      "documentation": {
        "short": "Requires all changes to be reviewed before merging",
        "url": "https://github.com/ossf/scorecard/blob/main/docs/checks.md"
      }
    },
    {
      "name": "Vulnerabilities",
      "score": 7,
      "reason": "Found 3 unaddressed vulnerabilities"
    }
  ]
}
```

### Security Checks (21 Total)

| Check | What It Measures |
|-------|------------------|
| **Code-Review** | Enforces code review before merge |
| **CII-Best-Practices** | Participates in OpenSSF Best Practices |
| **CI-Tests** | Automated tests run on PRs |
| **Signed-Commits** | Commits signed with GPG |
| **Signed-Releases** | Release artifacts signed |
| **Signed-Tags** | Git tags signed |
| **Branch-Protection** | Default branch has protection rules |
| **SAST** | Static analysis tools enabled |
| **Fuzzing** | Fuzz testing enabled |
| **Vulnerabilities** | No unaddressed CVEs |
| **Packaging** | Published via package manager |
| **Pinned-Dependencies** | Dependencies pinned to specific versions |
| **Dependency-Update-Tool** | Automated dependency updates (e.g., Dependabot) |
| **Binary-Artifacts** | No pre-built binaries in repo |
| **Dangerous-Workflow** | No dangerous GitHub Actions patterns |
| **Webhook-Validation** | Webhooks validate incoming signatures |
| **Token-Permissions** | GitHub Actions tokens minimized |
| **License** | License file present |
| **Maintained** | Active maintenance (commits within 90 days) |
| **Security-Policy** | Security.md or SECURITY.md exists |
| **SBOM** | Software Bill of Materials published |

### Scoring Interpretation

**Scale:** 0–10 (NOT percentage)

**Official Threshold:** ≥7/10 (per OpenSSF Best Practices)

**Important Caveat:** Scorecard measures **process compliance**, NOT actual vulnerability count. Academic research shows higher scores sometimes correlate with more reported vulnerabilities.

**Recommended Use:**
- ≥7/10: Generally acceptable hygiene
- 5–6/10: Process gaps worth addressing
- <5/10: Significant process deficits
- **Always examine individual checks**, not just aggregate score

### Go Integration
```go
import (
  "encoding/json"
  "net/http"
)

resp, _ := http.Get("https://api.securityscorecards.dev/projects/github.com/ossf/scorecard")
var result map[string]interface{}
json.NewDecoder(resp.Body).Decode(&result)
score := result["score"].(float64)
```

---

## 4. Homebrew Formulae API

### Purpose
Extract Homebrew package metadata including dependencies, source repositories, and URLs.

### API Details

**Base URL:**
```
https://formulae.brew.sh/api/formula.json
https://formulae.brew.sh/api/formula/{name}.json
https://formulae.brew.sh/api/cask.json
https://formulae.brew.sh/api/cask/{name}.json
```

**Authentication:** None required

**Rate Limits:** Not documented (public static API)

### Query Examples

```bash
# Get single formula
curl https://formulae.brew.sh/api/formula/curl.json

# All formulae (large response)
curl https://formulae.brew.sh/api/formula.json | jq '.[] | select(.name=="curl")'

# Extract dependencies
curl https://formulae.brew.sh/api/formula/curl.json | jq '.dependencies[]'

# Extract source repository
curl https://formulae.brew.sh/api/formula/curl.json | jq '{name, homepage, repository}'
```

### Response Schema
```json
{
  "name": "curl",
  "full_name": "homebrew/core/curl",
  "tap": "homebrew/core",
  "desc": "Get a file from an HTTP, HTTPS or FTP server",
  "homepage": "https://curl.se/",
  "license": "curl",
  "repository": "https://github.com/curl/curl",
  "revision": 2,
  "version": "8.6.0",
  "versions": {
    "stable": "8.6.0",
    "head": "HEAD"
  },
  "urls": {
    "stable": {
      "url": "https://curl.se/download/curl-8.6.0.tar.xz",
      "checksum": "sha256:..."
    }
  },
  "dependencies": [
    {
      "full_name": "openssl@3",
      "type": "required"
    }
  ],
  "build_dependencies": [...],
  "test_dependencies": [...],
  "optional_dependencies": [...],
  "requirements": [...],
  "bottle": {
    "stable": {
      "rebuild": 0,
      "root_url": "https://ghcr.io/v2/homebrew/core",
      "files": {
        "arm64_sonoma": {
          "cellar": ":any",
          "url": "...",
          "sha256": "..."
        }
      }
    }
  },
  "keg_only": false,
  "deprecated": false,
  "disabled": false,
  "post_install_defined": false,
  "service": null,
  "tap_git_head": "...",
  "head_master_branch": null,
  "ruby_source_path": "Formula/curl.rb",
  "ruby_source_checksum": {...},
  "linked_keg": null,
  "pinned_at_version": null,
  "caveats": null,
  "installed": [],
  "linked": null,
  "poured_from_bottle": false,
  "time_created": 1234567890,
  "time_modified": 1234567890,
  "update_available": false
}
```

### Coverage
All Homebrew formulae (100,000+) and casks (10,000+)

### Integration with CVE Data

```bash
# Get Homebrew package CVEs via brew-vulns subcommand
brew vulns curl              # Check installed curl for CVEs
brew vulns --severity high   # Filter by severity
```

The `brew vulns` tool queries OSV.dev internally, so integration via OSV.dev covers Homebrew packages.

---

## 5. GitHub API (Supplementary Trust Signals)

### Purpose
Extract repo signals: stars, last commit, contributor count, open issues, discussions.

### API Details

**Endpoints:**
```
GET /repos/{owner}/{repo}
GET /repos/{owner}/{repo}/commits
GET /repos/{owner}/{repo}/contributors
GET /repos/{owner}/{repo}/issues
```

**Authentication:**
- Unauthenticated: 60 requests/hour
- PAT: 5,000 requests/hour

### Query Examples

```bash
# Repository metadata
curl https://api.github.com/repos/curl/curl

# Last commit
curl https://api.github.com/repos/curl/curl/commits?per_page=1

# Contributor count
curl https://api.github.com/repos/curl/curl/contributors
```

### Trust Signals Extracted
- **Stars/watchers:** Community adoption/visibility
- **Last commit date:** Maintenance status
- **Contributor count:** Team size and diversity
- **Open issues/PRs:** Activity level
- **License:** Legal clarity
- **Security advisories:** GitHub Security Advisory database

---

## 6. CISA Known Exploited Vulnerabilities (KEV)

### Purpose
Track CVEs actively exploited in the wild (highest priority).

### API Details

**Endpoint:**
```
https://www.cisa.gov/sites/default/files/feeds/known_exploited_vulnerabilities.json
```

**Update Frequency:** Weekly

**SLA:** Remediation within 1 business day (highest priority tier)

### Integration Notes
- Use as a "must-fix" filter on top of other data sources
- Download JSON feed, parse, cross-reference with audit packages
- Focus remediation efforts on packages with KEV entries

---

## 7. macOS/App-Specific Tracking

### Apple Security Advisories

**Apple publishes security advisories for macOS, but NOT for third-party apps:**
- Monthly updates: https://support.apple.com/en-us/100100
- Third-party vulnerabilities must be reported via https://security.apple.com/ (90-day resolution target)
- CVE tracking flows through NVD, not an Apple-specific API
- No app-specific advisory database

### macOS Malware/Backdoor Databases

Since no unified macOS app security database exists, use multiple threat intelligence sources:

| Database | Coverage | API | Cost |
|----------|----------|-----|------|
| **VirusTotal** | 55+ antivirus engines | Yes (REST) | FREE (public) / Paid (advanced) |
| **Objective-See** | macOS behavioral analysis (dylib hijacking, injection) | No API; tools only | FREE |
| **Malpedia** | 67,000+ malware families | REST API | FREE (with key) |
| **ThreatFox** (abuse.ch) | IOC exchange platform | REST API | FREE |
| **Hybrid Analysis** (CrowdStrike) | Dynamic behavior analysis | REST API | Requires enterprise account |
| **Joe Sandbox/Hatching Triage** | Behavior sandboxing, Apple Silicon support | REST API | Paid |
| **Kaspersky Security Network** | Crowdsourced reputation | No public API | Enterprise only |
| **Microsoft Defender macOS** | PUA + malware detection | MDM only | Subscription |

**Critical Caveat:** Apple's XProtect uses signature-based detection only; Notarization has known bypasses (notarized malware has passed Apple's checks).

### Trust Scoring for macOS Applications

**No unified system exists.** Must aggregate:

```
Trust Score = (VirusTotal_clean_ratio × 0.4) + 
              (Gatekeeper_verified × 0.3) + 
              (No_behavioral_issues × 0.3)
```

**Verification steps:**
```bash
# Check Gatekeeper/Notarization status
spctl -a -vvv /path/to/app

# Verify code signature
codesign -v /path/to/app

# List quarantine attributes
xattr -l /path/to/app
```

### VirusTotal API Integration

**Endpoint:**
```
POST https://www.virustotal.com/api/v3/files
GET  https://www.virustotal.com/api/v3/files/{file_hash}
```

**Authentication:** API key (free tier available)

**Rate Limits:** 4 requests/minute (free), 500/minute (premium)

**Use case:** Scan downloaded binaries or formula source URLs

### Package Manager Security Validation Gaps

**Homebrew:**
- ✅ SHA-256 checksums
- ✅ Sigstore attestation (GitHub Actions workflow signing)
- ✅ Human review of formula submissions
- ✅ Post-install CVE scanning via `brew vulns`
- ❌ **No pre-installation CVE screening** (users don't see vulns before install)
- ❌ Trail of Bits audit (2024): 25 vulnerabilities; 16 fixed, 3 in progress, 6 acknowledged

**MacPorts:**
- ✅ SHA-256 + RIPEMD-160 checksums
- ✅ PGP signatures (maintainer-optional)
- ✅ Source compilation (not binaries)
- ✅ 72-hour review window for patches
- ❌ No formal security review process
- ❌ No CVE scanner
- ❌ CVE-2024-11681: Rsync mirror RCE attack via compromised upstream mirrors

**Key insight:** Neither package manager validates upstream software before inclusion; both assume user has control of their installation environment.

### Practical Audit Workflow for macOS Packages

```bash
# Step 1: Generate Software Bill of Materials
syft /usr/local/opt/curl --output spdx-json > curl-sbom.json

# Step 2: Scan for known vulnerabilities
grype curl-sbom.json

# Step 3: Check Homebrew post-install vulnerabilities
brew vulns curl

# Step 4: Verify code signature and notarization
spctl -a -vvv /usr/local/bin/curl
codesign -vv /usr/local/bin/curl

# Step 5: Binary analysis (if suspicious)
# Use Objective-See's tools or dynamic analysis platforms
```

### Supply Chain Monitoring

**Subscribe to these feeds for active threats:**
- **CISA KEV Catalog:** https://www.cisa.gov/known-exploited-vulnerabilities-catalog (weekday updates)
- **ThreatFox API:** https://threatfox-api.abuse.ch (hourly updates)
- **Homebrew Security:** https://github.com/Homebrew/brew/security (GitHub Advisories)
- **MacPorts Security:** https://github.com/macports/macports-ports/security (GitHub Advisories)

### Enterprise macOS Security

For organizations:
- Deploy **NIST mSCP baselines** (macOS Security Compliance Project): https://github.com/usnistgov/macos_security/
- Enroll in **MDM (Mobile Device Management)** with configuration profiles
- Use **Microsoft Defender macOS** with audit/block modes
- Implement **Software Restriction Policies** via JAMF, Kandji, or Intune

---

## Implementation Roadmap

### Phase 1: MVP (Initial Trial)
1. CSV input → package list
2. Query OSV.dev for each package + version
3. Query GitHub Advisories as supplement
4. Extract Homebrew metadata via formulae.brew.sh
5. Output: JSON + HTML report

### Phase 2: Trust Signals (Planned Upgrade)
1. Add OpenSSF Scorecard lookups for source repos
2. Extract GitHub repo signals (stars, last commit, contributors)
3. Trust score calculation: Scorecard ≥7 → Green, <5 → Red
4. Visualize trust signals in HTML report

### Phase 3: Monitoring (Future)
1. Scheduled runs (daily/weekly)
2. Webhook notifications for new CVEs
3. Trend tracking (CVE growth over time)
4. Historical dashboards

---

## Risk Scoring Recommendations

### CVE Severity Weight
```
CVSS 9.0-10:  Immediate action (Critical)
CVSS 7.0-8.9: High priority (High)
CVSS 4.0-6.9: Medium priority (Medium)
CVSS 0.1-3.9: Low priority (Low)
```

### Trust Score (Composite)
```
Scorecard ≥7 + Active maintenance + Code review:  🟢 High Trust
Scorecard 5-6 + Recent commits:                    🟡 Medium Trust
Scorecard <5 or Abandoned:                         🔴 Low Trust
```

### Overall Package Risk
```
Risk = (CVE_Count × CVE_Severity) + (1 - Trust_Score)
```

---

## API Comparison Matrix

| Source | Cost | Auth | Rate Limit | Latency | Coverage | Go Integration |
|--------|------|------|------------|---------|----------|-----------------|
| OSV.dev | FREE | None | Unlimited | 1.8 days | 24+ ecosystems | ★★★★★ Easy |
| GitHub Advisories | FREE | Optional | 5k/hr | 2.4 days | 12 ecosystems | ★★★★★ Easy |
| Scorecard | FREE | None | Unlimited | Weekly | GH/GL repos | ★★★★★ Easy |
| Homebrew API | FREE | None | Unlimited | Real-time | 100k+ formulae | ★★★★★ Easy |
| GitHub API | FREE | Optional | 5k/hr | Real-time | All repos | ★★★★★ Easy |
| CISA KEV | FREE | None | Weekly feed | 7 days | ~1000 CVEs | ★★★★★ Easy |

---

## Gaps & Limitations

### Known Gaps
1. **macOS app-specific vulnerabilities:** No centralized database; rely on Homebrew formula review + code signing
2. **Malware/backdoors:** Not covered by CVE databases; requires separate binary scanning (VirusTotal, ClamAV)
3. **Abandoned projects:** Scorecard doesn't penalize abandoned repos (maintained <90 days is lowest check)
4. **Transitive vulnerabilities:** Only directly reported; requires full dependency tree analysis for implicit risks

### Recommendations
- For critical packages, manually review source repo code
- Use `brew vulns --deps` to check transitive dependencies
- Integrate ClamAV or VirusTotal for malware detection (Phase 2+)
- Set up automated monitoring for KEV entries

---

## Next Steps

1. **Architecture:** Design Go tool structure (CSV parser, data aggregator, report generators)
2. **MVP implementation:** Start with OSV.dev + Homebrew metadata
3. **Testing:** Validate against known CVEs (past 6 months)
4. **Phase 2:** Add Scorecard + GitHub repo signals
5. **Monitoring:** Design webhook/scheduler framework

---

**Document Version:** 1.0  
**Last Updated:** June 23, 2026  
**Status:** Ready for tool design & implementation