Skip to content

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Deep Packet Inspection (DPI) Engine

A highly performant, production-grade Deep Packet Inspection system built entirely in Java. This engine parses raw .pcap network capture files, decodes network protocols across the OSI model, deeply inspects TLS/HTTP payloads to extract Server Name Indications (SNI) and Host headers, tracks concurrent connection states, and actively applies firewall blocking rules to drop or forward traffic securely.


Table of Contents

  1. What is this Project?
  2. Core Concepts & Background
  3. Project Architecture
  4. File Structure
  5. Internal Working (Step-by-Step)
  6. Deep Technical Explanation
  7. Important Algorithms / Logic
  8. Build & Run Instructions
  9. Sample Output
  10. Features
  11. Tech Stack
  12. Future Improvements
  13. Learning Outcomes
  14. Resume / Interview Value

1. What is this Project?

Purpose

The Java DPI Engine is a network traffic analysis tool that goes beyond standard port-based routing. While standard firewalls rely merely on IP addresses and TCP/UDP ports, this DPI engine intercepts traffic, unpacks the encapsulated layers (Ethernet -> IPv4 -> TCP/UDP), and dives straight into the Application Layer (Layer 7) payload to identify the exact application being accessed (e.g., YouTube, Netflix, Discord) despite TLS encryption.

Real-World Use Cases

  • Enterprise Security: Blocking unauthorized applications (like TikTok or BitTorrent) on a corporate network.
  • ISPs & Telecommunications: Throttling bandwidth for specific streaming services.
  • Intrusion Detection Systems (IDS): Analyzing payload signatures to detect malware communication.
  • Parental Controls: Implementing router-level blocks for explicit domains.

Why this project matters?

With over 90% of web traffic now encrypted via HTTPS/TLS, standard firewalls are blind to the exact website being visited. This project solves this by exploiting the TLS Client Hello handshake, grabbing the plaintext Server Name Indication (SNI) string before the symmetric encryption keys are exchanged, allowing us to enforce granular, application-level security policies.


2. Core Concepts & Background

To understand this system, one must understand how network data traverses the internet.

The OSI Model & Packet Encapsulation

Data on a network is wrapped like a Russian nesting doll. As data moves down the OSI model on the sender's machine, headers are added. Our DPI engine reverses this process.

┌──────────────────────────────────────────────────────────────────┐
│ Ethernet Header (14 bytes)  [MAC Addresses]                      │
│ ┌──────────────────────────────────────────────────────────────┐ │
│ │ IPv4 Header (20 bytes)  [Source/Dest IPs]                    │ │
│ │ ┌──────────────────────────────────────────────────────────┐ │ │
│ │ │ TCP Header (20 bytes)  [Source/Dest Ports]               │ │ │
│ │ │ ┌──────────────────────────────────────────────────────┐ │ │ │
│ │ │ │ Application Payload (TLS Handshake, HTTP)            │ │ │ │
│ │ │ └──────────────────────────────────────────────────────┘ │ │ │
│ │ └──────────────────────────────────────────────────────────┘ │ │
│ └──────────────────────────────────────────────────────────────┘ │
└──────────────────────────────────────────────────────────────────┘

The "Five-Tuple" Flow Identity

A "flow" or "connection" represents a continuous stream of packets between two machines. We identify a unique flow using a deterministic 5-Tuple:

  1. Source IP
  2. Destination IP
  3. Source Port
  4. Destination Port
  5. Protocol (TCP/UDP)

TLS Server Name Indication (SNI)

When you visit https://youtube.com, the payload is encrypted. However, during the initial handshake, your browser sends a Client Hello packet. Because a single server might host millions of websites, the browser must include the desired domain name in plaintext so the server knows which SSL Certificate to return. We extract this exact string.


3. Project Architecture

The architecture mimics enterprise-grade networking appliances. It reads raw byte streams, parses headers, categorizes flows, runs a ruleset engine, and streams the surviving packets to an output pipeline.

       [Input PCAP Stream]
               │
               ▼
      ┌─────────────────┐
      │   PcapReader    │ Reads Global Header & Packets byte-by-byte
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │  PacketParser   │ Slices Ethernet -> IPv4 -> TCP/UDP
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │ DPIEngine (Main)│ ──► [ConnectionTracker] (State Machine)
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │  SNIExtractor   │ Searches payload for Hex 0x0000 (SNI block)
      └────────┬────────┘
               │
               ▼
      ┌─────────────────┐
      │  BlockingRules  │ Evaluates against IP/App/Domain blacklists
      └────────┬────────┘
               │
          Drop or Forward
        (Discard)     (Keep)
                       │
                       ▼
               [Output PCAP Stream]

4. File Structure

src/main/java/com/dpi/
├── DPIEngine.java             # Entry point. The main orchestrator connecting parsing, tracking, and output.
├── PcapReader.java            # Responsible for low-level byte stream I/O. Reads PCAP file structures natively.
├── PacketParser.java          # Protocol dissecting engine. Extracts MACs, IPs, and Ports using bitwise operations.
├── SNIExtractor.java          # Deep packet inspector. Traverses TLS Handshake extensions to find the Hostname.
├── BlockingRules.java         # Security policy enforcement. Maintains HashSets of banned IPs, Domains, and AppTypes.
├── ConnectionTracker.java     # Local flow state manager. Implements LRU cache eviction and connection aging.
├── GlobalConnectionTable.java # Thread-safe global aggregator for reporting active flows and statistics.
├── AppType.java               # Categorical Enum mapping raw strings to classified Apps (e.g., "youtube" -> YOUTUBE).
├── FiveTuple.java             # Data structure overriding equals()/hashCode() for O(1) connection map lookups.
├── ParsedPacket.java          # DTO holding the sliced networking layers for a single packet.
├── Connection.java            # State holder for a flow (Bytes transferred, timestamps, SYN/ACK/FIN flags).
├── ConnectionState.java       # Enum representing flow lifecycle (NEW, ESTABLISHED, CLASSIFIED, BLOCKED, CLOSED).
└── PacketAction.java          # Enum representing routing decisions (FORWARD, DROP, INSPECT, LOG_ONLY).

5. Internal Working (Step-by-Step)

Tracing the lifecycle of a packet as it hits the engine:

Step 1: Ingestion & Reading

The PcapReader opens the .pcap file, bypassing standard Java object overhead, and reads the 24-byte PCAP Global Header, followed by the 16-byte Packet Header (Timestamp, Length), grabbing the raw payload byte array.

Step 2: Protocol Parsing & Validation

PacketParser reads the raw bytes. It verifies the EtherType is 0x0800 (IPv4). It dynamically calculates the IP Header Length (since it can vary via options) using bitwise masking: int ipHeaderLen = (data[14] & 0x0F) * 4;. It extracts IPs and ports via bit-shifting ((data[offset] & 0xFF) << 8) | (data[offset+1] & 0xFF).

Step 3: Connection State Management

A FiveTuple is constructed. The DPIEngine checks the HashMap<FiveTuple, Flow>. If the flow is new, a state machine is instantiated.

FiveTuple tuple = new FiveTuple(srcIp, dstIp, srcPort, dstPort, protocol);
Flow flow = flows.computeIfAbsent(tuple, k -> new Flow(k));
flow.packets++;

Step 4: Deep Packet Inspection (SNI Extraction)

If the connection is targeting Port 443 (HTTPS) and is not yet classified, the engine dives into the payload. The SNIExtractor offsets the TCP header, looks for the TLS Handshake byte 0x16, and verifies it is a Client Hello 0x01. It calculates the offset to the Extensions block, iterating through them until it finds Extension Type 0x0000 (SNI).

// Check TLS record header
if (payload[0] != 0x16) return null; // Not Handshake
if (payload[5] != 0x01) return null; // Not Client Hello
// ... skip cipher suites / compression ...
if (extType == 0x0000) { // SNI Found!
    int sniLen = readUint16BE(payload, offset + 3);
    return new String(payload, offset + 5, sniLen); // "www.youtube.com"
}

Step 5: Rule Enforcement & Forwarding

The domain is passed to BlockingRules.java. If the flow triggers a rule, flow.blocked = true is set. The engine evaluates this: blocked packets are discarded silently, while valid packets are pushed to the FileOutputStream output PCAP.


6. Deep Technical Explanation

Endianness & Network Byte Order

Network protocols strictly adhere to Big-Endian (Most Significant Byte first) ordering. To parse ports and lengths from the raw byte array natively in Java, the project heavily relies on bitwise operations, masking implicit sign-extensions (& 0xFF) before shifting (<< 8).

Connection Hashing (Five-Tuple)

To track hundreds of thousands of concurrent packets, FiveTuple overrides hashCode() and equals(). This guarantees O(1) time complexity when mapping a packet to an active flow state.

Memory & State Management (LRU Eviction)

In a real-world scenario, maintaining state for every connection leads to Out-Of-Memory (OOM) crashes. ConnectionTracker implements an LRU (Least Recently Used) eviction policy and timeout sweeps (cleanupStale()) to purge closed or dead connections, keeping the memory footprint deterministic.

Concurrent Architecture Readiness

The GlobalConnectionTable utilizes synchronized blocks and AtomicLong (in multi-threaded contexts) to safely aggregate local thread-specific ConnectionTracker stats. This prevents cache contention and race conditions when multiple Fast-Path (FP) worker threads are processing packets simultaneously.


7. Important Algorithms / Logic

The SNI Payload Traversal Algorithm

Purpose: Finding the plaintext domain inside a nested, variable-length TLS header.

Flow:

  1. Assert Payload length.
  2. Verify TLS Record Type 0x16.
  3. Verify Handshake Type 0x01.
  4. Skip 43 bytes to reach Session ID.
  5. Read Session ID Length (N), skip N bytes.
  6. Read Cipher Suites Length (M), skip M bytes.
  7. Read Compression Length (O), skip O bytes.
  8. Read Extensions Length, iterate loop.
  9. For each extension, read Type and Length. If Type == 0x0000, parse the string. Else, advance offset by Length.

Why this matters: We cannot rely on fixed offsets. The client's cipher suites vary in length wildly. The algorithm must dynamically parse the protocol exactly as a Web Server does.


8. Build & Run Instructions

The system requires no external dependencies, Maven, or Gradle. It runs purely on the Java Standard Library.

Prerequisites

  • JDK 17 or higher (java -version)
  • Windows CMD or PowerShell (or Linux equivalent shell)

Installation & Build

Option 1: Using the provided Windows script

.\build.bat

Option 2: Using raw terminal commands (Cross-Platform)

mkdir out
javac -d out src/main/java/com/dpi/*.java

(This compiles all classes into the out/ directory).

Execution

Option 1: Using the provided Windows script

.\run.bat [input_file.pcap] [output_file.pcap] [options]

Option 2: Using raw terminal commands (Cross-Platform)

java -cp out com.dpi.DPIEngine [input_file.pcap] [output_file.pcap] [options]

Supported Options:

  • --block-ip <ip>: Drop packets originating from this IPv4 address.
  • --block-app <app>: Drop specific apps mapped in AppType.java (e.g., YouTube).
  • --block-domain <domain>: Drop domains via exact/substring matching.

Example Usage (Copy & Paste to Run)

1. Basic Usage (Analyze packets without blocking):

.\run.bat test_dpi.pcap output.pcap

2. With Blocking (Block specific apps and IPs):

.\run.bat test_dpi.pcap output.pcap --block-app YouTube --block-ip 192.168.1.50

3. Using raw Java command (If not on Windows):

java -cp out com.dpi.DPIEngine test_dpi.pcap output.pcap --block-domain facebook.com

9. Sample Output

When the engine executes, it generates a real-time console log and a final statistical report.

[Rules] Blocked app: YouTube
[Rules] Blocked IP: 192.168.1.50

╔══════════════════════════════════════════════════════════════╗
║                    DPI ENGINE v1.0 (JAVA)                    ║
╚══════════════════════════════════════════════════════════════╝

[DPI] Processing packets...
[BLOCKED] 192.168.1.100 -> 142.250.185.110 (YOUTUBE: www.youtube.com)
[BLOCKED] 192.168.1.50 -> 172.217.0.100 (HTTPS)

╔══════════════════════════════════════════════════════════════╗
║                      PROCESSING REPORT                       ║
╠══════════════════════════════════════════════════════════════╣
║ Total Packets:              77                             ║
║ Forwarded:                  71                             ║
║ Dropped:                     6                             ║
║ Active Flows:               43                             ║
╠══════════════════════════════════════════════════════════════╣
║                    APPLICATION BREAKDOWN                     ║
╠══════════════════════════════════════════════════════════════╣
║ HTTPS                37  48.1% #########             ║
║ UNKNOWN              16  20.8% ####                  ║
║ HTTP                  4   5.2% #                     ║
║ DNS                   4   5.2% #                     ║
║ YOUTUBE               1   1.3%                       ║
╚══════════════════════════════════════════════════════════════╝

[Detected Applications/Domains]
  - www.youtube.com -> YOUTUBE
  - github.com -> GITHUB
  - open.spotify.com -> SPOTIFY

What is happening internally?

  • The system immediately acknowledges the initialization of our security rules.
  • As the .pcap is parsed, we see real-time triggers [BLOCKED] printing the exact IP hitting the exact blocked domain.
  • The Processing Report displays total throughput, confirming 6 packets were dropped and excluded from output.pcap.
  • The breakdown provides an analytical view of network bandwidth usage per application.

10. Features

  • Layer 7 Application Detection: Classifies traffic based on domain strings, not just ports.
  • TLS Handshake Decryption (SNI): Defeats basic HTTPS blindness by analyzing pre-encryption keys.
  • Zero-Dependency Core: Operates purely on Java java.io and java.nio packages.
  • Dynamic Connection State Machine: Tracks bidirectional packets via custom hash-maps.
  • High-Performance Parsing: Uses bitwise mathematics to avoid heavy string allocations during packet inspection.
  • Dynamic Rule Engine: Supports real-time firewall rule processing.

11. Tech Stack

  • Core Language: Java (JDK 17+)
  • Networking Concepts: OSI Model, Ethernet, IPv4, TCP/UDP, TLS 1.2/1.3, HTTP
  • Architecture: Object-Oriented Design, Protocol Parsing, State Machines
  • Tools: Wireshark / tcpdump (for generating PCAPs)
  • Data Structures: HashMap, ArrayList, HashSet, Bitwise Operations

12. Future Improvements

To elevate this to a distributed, commercial-grade product:

  1. Multi-Threading / Zero-Copy Buffers: Implement java.nio.channels.FileChannel and ExecutorService thread pools to split packet processing across CPU cores.
  2. eBPF / AF_PACKET Integration: Hook the Java engine directly into the Linux Kernel networking stack for live interface sniffing, bypassing PCAP files entirely.
  3. Kafka Streaming: Stream classification metrics to Apache Kafka for live Grafana dashboarding.
  4. Encrypted Client Hello (ECH) Circumvention: Introduce machine learning / heuristic analysis (packet length/timing analysis) to identify apps even when SNI is fully encrypted (TLS 1.3 ECH).

13. Learning Outcomes

Building this project solidifies advanced engineering concepts:

  • Low-Level Systems Programming in Java: Moving away from web frameworks and interacting directly with raw memory bytes and bitwise operators.
  • Deep Networking Expertise: Truly understanding how the internet works under the hood beyond REST APIs.
  • State Management: Managing complex lifecycle states of distributed, asynchronous events (network packets arriving out of order).
  • Performance Optimization: Understanding memory limits, garbage collection pressure, and cache eviction logic.

14. Resume / Interview Value

This project provides massive leverage for technical interviews.

Skills Demonstrated:

  • Advanced Java (Bitwise math, IO, Collections).
  • Deep knowledge of the Network Stack (TCP/IP, OSI Model).
  • Software Architecture & System Design.
  • Security and Firewall Engineering.

Roles this applies to:

  • Backend Software Engineer
  • Infrastructure / Core Systems Engineer
  • Cybersecurity Engineer / Network Analyst

What interviewers will ask:

  • "How did you handle the memory if the PCAP file contained 50 million packets?" (Answer: Streaming I/O and LRU cache eviction on the flow map).
  • "How do you parse data without standard JSON libraries?" (Answer: Reading RFCs and implementing custom bit-shifting logic to slice byte arrays).
  • "How would you scale this to 10 Gbps traffic?" (Answer: By implementing RSS-based Load Balancers mapping 5-tuples to independent thread queues to prevent lock contention).

About

No description, website, or topics provided.

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages