Impact of traffic load on Ping times over Cable Modem links

Impact of traffic load on Ping times over Cable Modem links

Grenville Armitage
May 6th, 2001

In 2001 I had cable internet to my house in Redwood City, California. I was also playing online Quake III Arena and Half Life over this link. Out of curiosity I decided to check how the latency between my house and my favorite game server changed as a function of IP traffic load through the cable modem link. For the benefit of anyone curious enough to come this far, I'm hereby documenting what I did and what I found.

Situation:

In a nutshell here's the situation:

I run a game server in Palo Alto at 192.147.236.5 (FreeBSD 4.2) on a reasonably well-connected edge site (1.5Mbps T1 line).

My home network connection is a cable modem (AT&T@Home) to NetGear RT314 four-port NAPT/router (100baseT switched on the LAN side, 10baseT on the cable modem side).

I have two machines on the home LAN - a Dell Dimension XPST450 running FreeBSD 4.2, and a Dell Precision 8100 running WindowsME.

I use 'scp' to establish bulk file transfers from home to the game server (congesting the cable modem's uplink).

I run 'ping' before and during the scp transfers to measure the impact on RTT between home and the game server at 192.147.236.5
Traffic load causes the cable modem access link to add between 500 and 1000ms to the experienced RTT between my home LAN and all sites beyond my cable modem link.

Pinging from xpst450 while file transfer from xpst450 to game server

Began pinging while the link was otherwise idle:

gja@xpst450 :ping 192.147.236.5
PING 192.147.236.5 (192.147.236.5): 56 data bytes
64 bytes from 192.147.236.5: icmp_seq=0 ttl=246 time=19.105 ms
64 bytes from 192.147.236.5: icmp_seq=1 ttl=246 time=19.101 ms
64 bytes from 192.147.236.5: icmp_seq=2 ttl=246 time=16.447 ms
64 bytes from 192.147.236.5: icmp_seq=3 ttl=246 time=16.729 ms
64 bytes from 192.147.236.5: icmp_seq=4 ttl=246 time=18.666 ms
64 bytes from 192.147.236.5: icmp_seq=5 ttl=246 time=18.000 ms

Now started a 160MB 'scp' file transfer from XPST450 towards 192.147.236.5 (the same box) and the ping times shifted dramatically:

64 bytes from 192.147.236.5: icmp_seq=6 ttl=246 time=1180.611 ms
64 bytes from 192.147.236.5: icmp_seq=7 ttl=246 time=1222.496 ms
64 bytes from 192.147.236.5: icmp_seq=8 ttl=246 time=976.896 ms
64 bytes from 192.147.236.5: icmp_seq=9 ttl=246 time=1110.958 ms
64 bytes from 192.147.236.5: icmp_seq=10 ttl=246 time=1060.045 ms
64 bytes from 192.147.236.5: icmp_seq=11 ttl=246 time=1102.726 ms
64 bytes from 192.147.236.5: icmp_seq=12 ttl=246 time=1087.677 ms
64 bytes from 192.147.236.5: icmp_seq=13 ttl=246 time=1049.636 ms

In other words, the RTT jumps from roughly 18ms (with uplink idle) to over 1000ms (with uplink loaded).

Retrying that experiment with traceroute shows that the vast majority of that increase occured at the first hop out of my home LAN. First while the link was idle:

traceroute -n 192.147.236.5
traceroute to 192.147.236.5 (192.147.236.5), 30 hops max, 40 byte packets
1 192.168.0.1 1.111 ms 1.014 ms 1.017 ms
2 10.4.178.1 13.796 ms 10.999 ms 11.945 ms
3 24.5.207.193 10.822 ms 12.253 ms 11.131 ms
4 216.197.144.129 13.091 ms 13.744 ms 11.977 ms
5 24.7.74.61 13.064 ms 11.233 ms 12.152 ms
6 24.7.74.170 13.737 ms 12.012 ms 11.456 ms
7 198.32.176.3 16.727 ms 15.236 ms 17.910 ms
8 204.152.184.197 24.631 ms 13.209 ms 65.892 ms
9 192.147.236.254 17.536 ms 17.703 ms 18.794 ms
10 192.147.236.5 16.274 ms 15.714 ms 15.109 ms

Then with the scp file transfer running:

traceroute -n 192.147.236.5
traceroute to 192.147.236.5 (192.147.236.5), 30 hops max, 40 byte packets
1 192.168.0.1 1.077 ms 1.017 ms 1.020 ms
2 10.4.178.1 1186.773 ms 981.150 ms 1110.974 ms
3 24.5.207.193 1195.886 ms 1148.106 ms 1046.820 ms
4 216.197.144.129 945.438 ms 1060.211 ms 1184.985 ms
5 24.7.74.61 1192.730 ms 971.462 ms 1052.090 ms
6 24.7.74.170 1087.762 ms^C

The second trace was truncated because it is clear the largest jump in RTT can be attributed to the cable modem link between hops 1 and 2.

Pinging from WindowsME host while file transfer from xpst450 to game server

To be sure that the RTT increase isn't an artifact of the FreeBSD host's kernel when pinging and scp'ing, I begin pinging from the WindowsME box towards the game server while initiating the same scp file transfer from xpst450:

Pings from the WindowsME box:

Pinging 192.147.236.5 with 32 bytes of data:

Reply from 192.147.236.5: bytes=32 time=14ms TTL=246
Reply from 192.147.236.5: bytes=32 time=13ms TTL=246
Reply from 192.147.236.5: bytes=32 time=14ms TTL=246
Reply from 192.147.236.5: bytes=32 time=14ms TTL=246

Then with the scp started on the FreeBSD box, the pings from the WinME box become:

Reply from 192.147.236.5: bytes=32 time=1236ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1043ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1140ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1057ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1154ms TTL=246
Reply from 192.147.236.5: bytes=32 time=480ms TTL=246
Reply from 192.147.236.5: bytes=32 time=700ms TTL=246
Reply from 192.147.236.5: bytes=32 time=824ms TTL=246
Reply from 192.147.236.5: bytes=32 time=769ms TTL=246
Reply from 192.147.236.5: bytes=32 time=727ms TTL=246
Reply from 192.147.236.5: bytes=32 time=865ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1099ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1098ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1140ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1043ms TTL=246
Reply from 192.147.236.5: bytes=32 time=962ms TTL=246
Reply from 192.147.236.5: bytes=32 time=1002ms TTL=246

(The brief drop in RTT during the scp transfer is suspected due to transient fluctuations in the scp transfer, resulting in fluctuations in the traffic load on the cable modem link.)

Pinging from xpst450 (FreeBSD) host while file transfer from WindowsME host to game server

Trying the reverse - scp from the WindowsME box and ping/traceroute from the FreeBSD box:

First without scp running:

ping 192.147.236.5
PING 192.147.236.5 (192.147.236.5): 56 data bytes
64 bytes from 192.147.236.5: icmp_seq=0 ttl=246 time=17.665 ms
64 bytes from 192.147.236.5: icmp_seq=1 ttl=246 time=18.150 ms
64 bytes from 192.147.236.5: icmp_seq=2 ttl=246 time=17.144 ms
64 bytes from 192.147.236.5: icmp_seq=3 ttl=246 time=16.292 ms
64 bytes from 192.147.236.5: icmp_seq=4 ttl=246 time=20.329 ms
^C
--- 192.147.236.5 ping statistics ---
5 packets transmitted, 5 packets received, 0% packet loss
round-trip min/avg/max/stddev = 16.292/17.916/20.329/1.355 ms

And traceroute along the path while idle

gja@xpst450 :!trac
traceroute -n 192.147.236.5
traceroute to 192.147.236.5 (192.147.236.5), 30 hops max, 40 byte packets
1 192.168.0.1 1.088 ms 1.038 ms 1.031 ms
2 10.4.178.1 13.020 ms 11.665 ms 13.735 ms
3 24.5.207.193 11.292 ms 11.154 ms 17.981 ms
4 216.197.144.129 11.428 ms 11.933 ms 11.760 ms
5 24.7.74.61 12.513 ms 13.783 ms 11.384 ms
6 24.7.74.170 12.112 ms 14.423 ms 12.582 ms
7 198.32.176.3 12.379 ms 14.631 ms 14.619 ms
8 204.152.184.197 12.162 ms 16.686 ms 12.872 ms
9 192.147.236.254 30.773 ms 16.816 ms 18.062 ms
10 192.147.236.5 17.490 ms 48.024 ms 25.636 ms

Now while scp'ing 160MByte file from WindowsME box to 192.147.236.5:

ping 192.147.236.5
PING 192.147.236.5 (192.147.236.5): 56 data bytes
64 bytes from 192.147.236.5: icmp_seq=0 ttl=246 time=306.109 ms
64 bytes from 192.147.236.5: icmp_seq=1 ttl=246 time=514.657 ms
64 bytes from 192.147.236.5: icmp_seq=2 ttl=246 time=506.443 ms
64 bytes from 192.147.236.5: icmp_seq=3 ttl=246 time=498.696 ms
64 bytes from 192.147.236.5: icmp_seq=4 ttl=246 time=412.699 ms
64 bytes from 192.147.236.5: icmp_seq=5 ttl=246 time=625.210 ms
64 bytes from 192.147.236.5: icmp_seq=6 ttl=246 time=715.933 ms
64 bytes from 192.147.236.5: icmp_seq=7 ttl=246 time=406.426 ms
64 bytes from 192.147.236.5: icmp_seq=8 ttl=246 time=580.836 ms
^C

And traceroute along the path while loaded:

--- 192.147.236.5 ping statistics ---
9 packets transmitted, 9 packets received, 0% packet loss
round-trip min/avg/max/stddev = 306.109/507.445/715.933/116.700 ms
gja@xpst450 :traceroute -n 192.147.236.5
traceroute to 192.147.236.5 (192.147.236.5), 30 hops max, 40 byte packets
1 192.168.0.1 1.107 ms 1.027 ms 1.028 ms
2 10.4.178.1 341.363 ms 256.686 ms 215.487 ms
3 24.5.207.193 535.338 ms 505.115 ms 473.631 ms
4 216.197.144.129 460.856 ms 295.035 ms 536.256 ms
5 24.7.74.61 287.425 ms 453.728 ms 297.107 ms
6 24.7.74.170 463.837 ms 288.420 ms 463.148 ms
7 198.32.176.3 288.302 ms 461.846 ms 293.978 ms
8 204.152.184.197 356.235 ms 421.086 ms 297.119 ms
9 192.147.236.254 434.010 ms 544.721 ms 553.579 ms
^C

In other words, the RTT jumps from roughly 18ms (with uplink idle) to roughly 500ms (with uplink loaded).

Comments

The FreeBSD and WindowsME hosts put different loads on the cable modem link during scp - the FreeBSD host averaged 18Kbyte/sec transfer rate whilst the WindowsME host averaged only 13Kbyte/sec. The lower IP traffic load probably accounts for the lower RTT (500ms compared to 1000ms) while the WindowsME host is doing bulk data transfer. (The MSS appeared to be the same for both cases.) The scp process under WindowsME appeared to be somewhat fragile - its throughput fluctuated far more during each test, and at times it would appear to pause - letting the path RTT briefly drop back to 10-20ms before immediately rising back to 500+ms again.

Conclusion

Any online game player sharing their cable modem with other outbound traffic from their home LAN should expect to see wildly intolerable swings in RTT. This will be particularly problematic for first person shooter games, where 'lag' is generally only tolerable if under a couple of 100ms.