Networking#

HPC Network Traffic

High performance computing centers generate vast amounts of network data. This data is useful to optimize the performance of the center itself but also to look for unexpected behavior due to a malfunction, misconfiguration or malicious use. In this project you’ll use data from a real data center to understand how network data looks and see what is going on. There are a lot of resources to understand the data online so google is a great place to start researching here.

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()
import numpy as np
import pandas as pd
import os.path
import subprocess

def wget_data(url):
    local_path='./tmp_data'
    subprocess.run(["wget", "-nc", "-P", local_path, url])

def locate_data(name, check_exists=True):
    local_path='./tmp_data'
    path = os.path.join(local_path, name)
    if check_exists and not os.path.exists(path):
        raise RuxntimeError('No such data file: {}'.format(path))
    return path

wget_data('https://courses.physics.illinois.edu/phys398dap/fa2025/data/netflow.csv')

(This is a 2GB file and so it takes a few minutes to upload to colab and it will probably take some time to analyze. You might consider making a smaller version of this to develop your code on and then only run the full version once.)

A description of the data is here:

  • first: When the flow (or this chunk of it) started, as Unix epoch seconds with millisecond precision.

  • last: When the last packet of the flow or chunk was seen, in the same format.

  • duration: last − first, in seconds. It’s 0 for single-packet flows like DNS, NTP and scan probes.

  • proto: IP protocol: TCP or UDP. In the v5 record these are the numbers 6 and 17.

  • src, sport: Source IP address and port.

  • dst, dport: Destination IP address and port.

  • tcp_flags: Every TCP flag seen in the flow, combined. It’s shown as six positions UAPRSF (URG, ACK, PSH, RST, SYN, FIN), with a dot for flags not seen. It’s blank for UDP.

  • tos: The IP Type of Service byte in decimal. It’s almost always 0. The value 104 means DSCP 26, which marks RoCE traffic.

  • packets: Number of packets in this record.

  • bytes: Number of bytes in this record, counted at layer 3 (IP header plus payload, no Ethernet framing), as v5 does.

Part 1

How much time does this data cover? How do you know?

Part 2

Make a plot of the data moved as a function of the sources. Which sources are the biggest data movers?
What might you conclude from that?

Part 3

Do you see any unusual repetitive activity around port 22? What makes it anomalous?

What is port 22 typically used for? Does what you see make sense?

Part 4

Visualize the connections in this network. Describe what you see and speculate as to why the traffic is moving in this manner.

Part 5

How many hosts does “10.40.0.1” talk to throughout this data capture? What might this host be doing?