secubox-openwrt/DOCS/MIGRATION-NETIFYD-TO-NDPID.md

Migration Plan: Netifyd to nDPId

Executive Summary

This document provides a comprehensive migration plan to replace Netifyd v5.2.1 with nDPId in the SecuBox OpenWrt project while maintaining full compatibility with existing CrowdSec and Netdata consumers.

Key Finding: Both Netifyd and nDPId are built on top of nDPI (the underlying DPI library). Netifyd is essentially a feature-rich wrapper around nDPI with cloud integration, while nDPId is a minimalist, high-performance daemon with a microservice architecture.

---

Current Architecture Analysis

Netifyd Integration Overview

Component	Location	Purpose
Base Package	secubox-app-netifyd	Netifyd v5.2.1 DPI engine
LuCI App	luci-app-secubox-netifyd	Web UI with real-time monitoring
RPCD Backend	/usr/libexec/rpcd/luci.secubox-netifyd	15 read + 9 write RPC methods
UCI Config	/etc/config/secubox-netifyd	Feature toggles, plugins, sinks
Status File	/var/run/netifyd/status.json	Summary statistics (NOT flows)
Socket	/var/run/netifyd/netifyd.sock	JSON streaming interface
Collector	/usr/bin/netifyd-collector	Periodic stats to /tmp/netifyd-stats.json

Current Data Consumers

1. CrowdSec: NO direct integration exists. Runs independently.
2. Netdata: Separate dashboard. Reads system metrics via /proc, not DPI data.
3. LuCI Dashboard: Primary consumer via RPCD backend.

Netifyd Output Formats

Summary Statistics (/var/run/netifyd/status.json):

{
  "flow_count": 150,
  "flows_active": 42,
  "devices": [...],
  "stats": {
    "br-lan": {
      "ip_bytes": 1234567,
      "wire_bytes": 1345678,
      "tcp": 1200,
      "udp": 300,
      "icmp": 50
    }
  },
  "dns_hint_cache": { "cache_size": 500 },
  "uptime": 86400
}

Flow Data (when sink enabled, not default):

{
  "flow_id": "abc123",
  "src_ip": "192.168.1.100",
  "dst_ip": "8.8.8.8",
  "src_port": 54321,
  "dst_port": 443,
  "protocol": "tcp",
  "application": "google",
  "category": "search_engine",
  "bytes_rx": 1500,
  "bytes_tx": 500,
  "packets_rx": 10,
  "packets_tx": 5
}

---

nDPId Architecture

Core Components

Component	Purpose
nDPId	Traffic capture daemon using libpcap + libnDPI
nDPIsrvd	Broker that distributes events to multiple consumers
libnDPI	Core DPI library (shared with Netifyd)

nDPId Event System

Message Format: [5-digit-length][JSON]\n

01223{"flow_event_id":7,"flow_event_name":"detection-update",...}\n

Event Categories:

Category	Events	Description
Error	17 types	Packet processing failures, memory issues
Daemon	4 types	init, shutdown, reconnect, status
Packet	2 types	packet, packet-flow (base64 encoded)
Flow	9 types	new, end, idle, update, detected, guessed, detection-update, not-detected, analyse

nDPId Flow Event Example

{
  "flow_event_id": 5,
  "flow_event_name": "detected",
  "thread_id": 0,
  "packet_id": 12345,
  "source": "eth0",
  "flow_id": 1001,
  "flow_state": "finished",
  "flow_src_packets_processed": 15,
  "flow_dst_packets_processed": 20,
  "flow_first_seen": 1704067200000,
  "flow_src_last_pkt_time": 1704067260000,
  "flow_dst_last_pkt_time": 1704067258000,
  "flow_idle_time": 2000,
  "flow_src_tot_l4_payload_len": 1500,
  "flow_dst_tot_l4_payload_len": 2000,
  "l3_proto": "ip4",
  "src_ip": "192.168.1.100",
  "dst_ip": "142.250.185.78",
  "l4_proto": "tcp",
  "src_port": 54321,
  "dst_port": 443,
  "ndpi": {
    "proto": "TLS.Google",
    "proto_id": 91,
    "proto_by_ip": 0,
    "encrypted": 1,
    "breed": "Safe",
    "category_id": 5,
    "category": "Web"
  }
}

---

Migration Strategy

Phase 1: Compatibility Layer Development

Create a translation daemon that converts nDPId events to Netifyd-compatible format.

New Component: secubox-ndpid-compat

nDPId → nDPIsrvd → secubox-ndpid-compat → Existing Consumers
                                        ↓
                    /var/run/netifyd/status.json (compatible)
                    /tmp/netifyd-stats.json (compatible)
                    RPCD backend (unchanged)

Phase 2: Package Development

2.1 New Package: secubox-app-ndpid

Makefile:

PKG_NAME:=ndpid
PKG_VERSION:=1.7.0
PKG_RELEASE:=1
PKG_SOURCE_PROTO:=git
PKG_SOURCE_URL:=https://github.com/utoni/nDPId.git

DEPENDS:=+libndpi +libpcap +libjson-c +libpthread

Build Requirements:

• libnDPI ≥5.0.0
• libpcap
• libjson-c
• CMake build system

2.2 New Package: secubox-ndpid-compat

Translation layer script that:

1. Connects to nDPIsrvd socket
2. Aggregates flow events into Netifyd-compatible format
3. Writes to /var/run/netifyd/status.json
4. Provides the same RPCD interface

Phase 3: Output Format Translation

3.1 Status File Translation Map

Netifyd Field	nDPId Source	Translation Logic
flow_count	Count of flow events	Increment on new, decrement on end/idle
flows_active	Active flow tracking	Count flows without end/idle events
stats.{iface}.tcp	l4_proto == "tcp"	Aggregate per interface
stats.{iface}.udp	l4_proto == "udp"	Aggregate per interface
stats.{iface}.ip_bytes	flow_*_tot_l4_payload_len	Sum per interface
uptime	Daemon status event	Direct mapping

3.2 Flow Data Translation Map

Netifyd Field	nDPId Field	Notes
src_ip	src_ip	Direct
dst_ip	dst_ip	Direct
src_port	src_port	Direct
dst_port	dst_port	Direct
protocol	l4_proto	Lowercase
application	ndpi.proto	Parse from "TLS.Google" → "google"
category	ndpi.category	Direct
bytes_rx	flow_dst_tot_l4_payload_len	Note: reversed (dst=rx from flow perspective)
bytes_tx	flow_src_tot_l4_payload_len	Note: reversed

3.3 Application Name Normalization

nDPId uses format like TLS.Google, QUIC.YouTube. Normalize to lowercase base:

TLS.Google → google
QUIC.YouTube → youtube
HTTP.Facebook → facebook
DNS → dns

Phase 4: Consumer Compatibility

4.1 CrowdSec Integration (NEW)

Since there's no existing CrowdSec integration, we can design it properly:

Acquisition Configuration (/etc/crowdsec/acquis.d/ndpid.yaml):

source: file
filenames:
  - /tmp/ndpid-flows.log
labels:
  type: ndpid
---
source: journalctl
journalctl_filter:
  - "_SYSTEMD_UNIT=ndpid.service"
labels:
  type: syslog

Parser (/etc/crowdsec/parsers/s02-enrich/ndpid-flows.yaml):

name: secubox/ndpid-flows
description: "Parse nDPId flow detection events"
filter: "evt.Parsed.program == 'ndpid'"
onsuccess: next_stage
statics:
  - parsed: flow_application
    expression: evt.Parsed.ndpi_proto
nodes:
  - grok:
      pattern: '%{IP:src_ip}:%{INT:src_port} -> %{IP:dst_ip}:%{INT:dst_port} %{WORD:proto} %{DATA:app}'

Scenario (/etc/crowdsec/scenarios/ndpid-suspicious-app.yaml):

type: leaky
name: secubox/ndpid-suspicious-app
description: "Detect suspicious application usage"
filter: evt.Parsed.flow_application in ["bittorrent", "tor", "vpn_udp"]
groupby: evt.Parsed.src_ip
capacity: 5
leakspeed: 10m
blackhole: 1h
labels:
  remediation: true

4.2 Netdata Integration (NEW)

Create custom Netdata collector for nDPId:

Collector (/usr/lib/netdata/plugins.d/ndpid.chart.sh):

#!/bin/bash
# nDPId Netdata collector

NDPID_STATUS="/var/run/netifyd/status.json"

# Chart definitions
cat << EOF
CHART ndpid.flows '' "Network Flows" "flows" ndpid ndpid.flows area
DIMENSION active '' absolute 1 1
DIMENSION total '' absolute 1 1
EOF

while true; do
    if [ -f "$NDPID_STATUS" ]; then
        active=$(jq -r '.flows_active // 0' "$NDPID_STATUS")
        total=$(jq -r '.flow_count // 0' "$NDPID_STATUS")
        echo "BEGIN ndpid.flows"
        echo "SET active = $active"
        echo "SET total = $total"
        echo "END"
    fi
    sleep 1
done

Phase 5: Plugin System Migration

5.1 IPSet Actions

Netifyd plugins → nDPId external processor:

Netifyd Plugin	nDPId Equivalent
libnetify-plugin-ipset.so	External script consuming flow events
libnetify-plugin-nftables.so	External nftables updater

nDPId Flow Action Script (/usr/bin/ndpid-flow-actions):

#!/bin/bash
# Process nDPId events and update ipsets

socat -u UNIX-RECV:/tmp/ndpid-actions.sock - | while read -r line; do
    # Parse 5-digit length prefix
    json="${line:5}"

    event=$(echo "$json" | jq -r '.flow_event_name')
    app=$(echo "$json" | jq -r '.ndpi.proto' | tr '.' '\n' | tail -1 | tr '[:upper:]' '[:lower:]')

    case "$event" in
        detected)
            case "$app" in
                bittorrent)
                    src_ip=$(echo "$json" | jq -r '.src_ip')
                    ipset add secubox-bittorrent "$src_ip" timeout 900 2>/dev/null
                    ;;
            esac
            ;;
    esac
done

---

Implementation Phases

Phase 1: Foundation (Week 1-2)

1. Create secubox-app-ndpid package
2. Build nDPId + nDPIsrvd for OpenWrt
3. Test basic flow detection
4. Create UCI configuration schema

Phase 2: Compatibility Layer (Week 3-4)

1. Develop secubox-ndpid-compat translation daemon
2. Implement status.json generation
3. Implement flow event aggregation
4. Test with existing LuCI dashboard

Phase 3: RPCD Backend Update (Week 5)

1. Update RPCD methods to use nDPId data
2. Ensure all 15 read methods work
3. Ensure all 9 write methods work
4. Test LuCI application compatibility

Phase 4: Consumer Integration (Week 6-7)

1. Create CrowdSec parser/scenario
2. Create Netdata collector
3. Test end-to-end data flow
4. Document new integrations

Phase 5: Migration & Cleanup (Week 8)

1. Create migration script for existing users
2. Update documentation
3. Remove Netifyd package (optional, can coexist)
4. Final testing and release

---

File Structure After Migration

package/secubox/
├── secubox-app-ndpid/              # NEW: nDPId package
│   ├── Makefile
│   ├── files/
│   │   ├── ndpid.config            # UCI config
│   │   ├── ndpid.init              # procd init script
│   │   └── ndpisrvd.init           # nDPIsrvd init
│   └── patches/                    # OpenWrt patches if needed
│
├── secubox-ndpid-compat/           # NEW: Compatibility layer
│   ├── Makefile
│   └── files/
│       ├── ndpid-compat.lua        # Translation daemon
│       ├── ndpid-flow-actions      # IPSet/nftables handler
│       └── ndpid-collector         # Stats aggregator
│
├── luci-app-secubox-netifyd/       # MODIFIED: Works with both
│   └── root/usr/libexec/rpcd/
│       └── luci.secubox-netifyd    # Updated for nDPId compat
│
└── secubox-app-netifyd/            # DEPRECATED: Keep for fallback

---

Configuration Mapping

UCI Config Translation

Netifyd (/etc/config/secubox-netifyd):

config settings 'settings'
    option enabled '1'
    option socket_type 'unix'

config sink 'sink'
    option enabled '1'
    option type 'unix'
    option unix_path '/tmp/netifyd-flows.json'

nDPId (/etc/config/secubox-ndpid):

config ndpid 'main'
    option enabled '1'
    option interfaces 'br-lan br-wan'
    option collector_socket '/tmp/ndpid-collector.sock'

config ndpisrvd 'distributor'
    option enabled '1'
    option listen_socket '/tmp/ndpisrvd.sock'
    option tcp_port '7000'

config compat 'compat'
    option enabled '1'
    option netifyd_status '/var/run/netifyd/status.json'
    option netifyd_socket '/var/run/netifyd/netifyd.sock'

---

Risk Assessment

Risk	Impact	Mitigation
Detection accuracy differences	Medium	Both use libnDPI; similar results expected
Performance regression	Low	nDPId is lighter; should improve performance
Plugin compatibility	High	Must reimplement flow actions externally
Breaking existing dashboards	High	Compatibility layer ensures same output format
Missing Netifyd features	Medium	Document feature gaps; prioritize critical ones

Features Comparison

Feature	Netifyd	nDPId	Migration Impact
Protocol detection	Yes	Yes	None
Application detection	Yes	Yes	None
Flow tracking	Yes	Yes	None
JSON output	Yes	Yes	Format translation needed
Socket streaming	Yes	Yes	Different format
Cloud integration	Yes	No	Feature removed
Plugin architecture	Built-in	External	Reimplement
Memory footprint	~50MB	~15MB	Improvement
Startup time	~5s	~1s	Improvement

---

Testing Plan

Unit Tests

1. Translation Accuracy: Verify nDPId events correctly map to Netifyd format
2. Statistics Aggregation: Verify flow counts, bytes, packets match
3. Application Detection: Compare detection results between engines

Integration Tests

1. LuCI Dashboard: All views render correctly
2. RPCD Methods: All 24 methods return expected data
3. IPSet Actions: BitTorrent/streaming detection triggers ipset updates
4. CrowdSec Parsing: Flow events parsed and scenarios trigger

Performance Tests

1. Throughput: Measure max flows/second
2. Memory: Compare RAM usage under load
3. CPU: Compare CPU usage during traffic bursts

---

Rollback Plan

If migration fails:

1. Stop nDPId services: /etc/init.d/ndpid stop && /etc/init.d/ndpisrvd stop
2. Start Netifyd: /etc/init.d/netifyd start
3. Compatibility layer auto-detects and switches source
4. No data loss; both can coexist

---

References

• nDPId GitHub Repository
• nDPI Library
• Netifyd Documentation
• CrowdSec Acquisition
• Netdata External Plugins

---

Appendix A: nDPId Event Schema Reference

Flow Event Fields

{
  "flow_event_id": "integer (0-8)",
  "flow_event_name": "string (new|end|idle|update|detected|guessed|detection-update|not-detected|analyse)",
  "thread_id": "integer",
  "packet_id": "integer",
  "source": "string (interface name)",
  "flow_id": "integer",
  "flow_state": "string (skipped|finished|info)",
  "l3_proto": "string (ip4|ip6)",
  "src_ip": "string",
  "dst_ip": "string",
  "l4_proto": "string (tcp|udp|icmp|...)",
  "src_port": "integer",
  "dst_port": "integer",
  "flow_src_packets_processed": "integer",
  "flow_dst_packets_processed": "integer",
  "flow_first_seen": "integer (ms timestamp)",
  "flow_src_tot_l4_payload_len": "integer (bytes)",
  "flow_dst_tot_l4_payload_len": "integer (bytes)",
  "ndpi": {
    "proto": "string (e.g., TLS.Google)",
    "proto_id": "integer",
    "encrypted": "integer (0|1)",
    "breed": "string (Safe|Acceptable|Fun|Unsafe|...)",
    "category_id": "integer",
    "category": "string"
  }
}

Daemon Status Event Fields

{
  "daemon_event_id": 3,
  "daemon_event_name": "status",
  "global_ts_usec": "integer",
  "uptime": "integer (seconds)",
  "packets": "integer",
  "packet_bytes": "integer",
  "flows_active": "integer",
  "flows_idle": "integer",
  "flows_detected": "integer",
  "compressions": "integer",
  "decompressions": "integer"
}

---

Appendix B: Sample Compatibility Layer Code

#!/usr/bin/env lua
-- secubox-ndpid-compat: nDPId to Netifyd format translator

local socket = require("socket")
local json = require("cjson")

local NDPISRVD_SOCK = "/tmp/ndpisrvd.sock"
local OUTPUT_STATUS = "/var/run/netifyd/status.json"
local UPDATE_INTERVAL = 1

-- State tracking
local state = {
    flows = {},
    flow_count = 0,
    flows_active = 0,
    stats = {},
    devices = {},
    uptime = 0,
    start_time = os.time()
}

-- Process incoming nDPId event
local function process_event(raw)
    -- Strip 5-digit length prefix
    local json_str = raw:sub(6)
    local ok, event = pcall(json.decode, json_str)
    if not ok then return end

    local event_name = event.flow_event_name or event.daemon_event_name

    if event_name == "new" then
        state.flows[event.flow_id] = event
        state.flow_count = state.flow_count + 1
        state.flows_active = state.flows_active + 1

    elseif event_name == "end" or event_name == "idle" then
        state.flows[event.flow_id] = nil
        state.flows_active = state.flows_active - 1

    elseif event_name == "detected" then
        if state.flows[event.flow_id] then
            state.flows[event.flow_id].detected = event.ndpi
        end
        -- Update interface stats
        local iface = event.source or "unknown"
        if not state.stats[iface] then
            state.stats[iface] = {ip_bytes=0, tcp=0, udp=0, icmp=0}
        end
        local proto = event.l4_proto or ""
        if proto == "tcp" then state.stats[iface].tcp = state.stats[iface].tcp + 1 end
        if proto == "udp" then state.stats[iface].udp = state.stats[iface].udp + 1 end
        if proto == "icmp" then state.stats[iface].icmp = state.stats[iface].icmp + 1 end
        local bytes = (event.flow_src_tot_l4_payload_len or 0) + (event.flow_dst_tot_l4_payload_len or 0)
        state.stats[iface].ip_bytes = state.stats[iface].ip_bytes + bytes

    elseif event_name == "status" then
        state.uptime = event.uptime or (os.time() - state.start_time)
    end
end

-- Generate Netifyd-compatible status.json
local function generate_status()
    return json.encode({
        flow_count = state.flow_count,
        flows_active = state.flows_active,
        stats = state.stats,
        devices = state.devices,
        uptime = state.uptime,
        dns_hint_cache = { cache_size = 0 }
    })
end

-- Main loop
local function main()
    -- Create output directory
    os.execute("mkdir -p /var/run/netifyd")

    local sock = socket.unix()
    local ok, err = sock:connect(NDPISRVD_SOCK)
    if not ok then
        print("Failed to connect to nDPIsrvd: " .. (err or "unknown"))
        os.exit(1)
    end

    sock:settimeout(0.1)

    local last_write = 0
    while true do
        local line, err = sock:receive("*l")
        if line then
            process_event(line)
        end

        -- Write status file periodically
        local now = os.time()
        if now - last_write >= UPDATE_INTERVAL then
            local f = io.open(OUTPUT_STATUS, "w")
            if f then
                f:write(generate_status())
                f:close()
            end
            last_write = now
        end
    end
end

main()

---

Document Version: 1.0 Created: 2026-01-09 Author: Claude Code Assistant