Successful High-precision QoS Measurement in Widely Deployed 10-Gbit/s Networks Based on General-purpose Personal Computers

1. Network measurement in high-speed application era

The spread of high-speed networks has made it easy to use broadband applications such as high-quality video streaming applications via the Internet. If we consider video streaming services, the bit-rate depends on the video format and quality, varying from some tens of kilobits per second to several megabits per second. Therefore, considering multiple client accesses in Internet service providers’ networks, the total amount of traffic could reach over 1 Gbit/s. In such an environment, network operators must know the network status in detail in order to provide stable service for users. That is why we have developed a precise, high-speed network measurement technique that can be applied to a wide range of networks.

2. 10-Gbit/s PCI network interface card with network measurement extensions

NTT Laboratories has developed a 10-Gbit/s PCI (peripheral component interconnect) network interface card (NIC) with network measurement extensions (measurement NIC). It works both as a normal NIC for a 10-Gbit/s network and as a measurement NIC. Its extensions enable precise measurements of the network status and quality of service (QoS). It will let us develop a reliable network measurement system based on inexpensive general-purpose personal computers (PCs).

For example, the headers of all the packets flowing in a 10-Gbit/s network can be captured without any sampling and used for analysis. Furthermore, precise timestamps based on timing signals using GPS (global positioning system) can be appended to packets at both the sender and receiver sides. These timestamps are useful for calculating network delays and jitter. Furthermore, while acting as a streaming server’s NIC, our card can append precise timestamps to the streaming packets entering and leaving the server. It makes possible novel application serviceability evaluation systems that perform network measurement while simultaneously providing streaming services. A photograph of the card is shown in Fig. 1, which also lists its specifications.

Fig. 1. 10-Gbit/s PCI network interface card with network measurement extensions.

3. Classification of network measurement techniques

Network measurement can be classified into passive and active measurement. Although each technique is usually chosen for a different purpose, our measurement NIC has hardware-assisted extensions to support both of them.

- Active measurement: a system sends measurement-specific packets (probe packets) into the network under test. Then, by analyzing their status such as inter-packet gap (IPG) and arrival time at a receiver, it obtains the network characteristics, including network delay and maximum available bandwidth.

- Passive measurement: a system monitors the traffic, which is split by utilizing a router’s mirroring port or optical splitters. It obtains network usage in detail, such as the usage rate of network bandwidth by each flow and protocol distribution.

4. Challenges for a high-speed network

The following methods are usually used to measure high-speed network characteristics such as a bandwidth of 10 Gbit/s.
(1) Sampling/non-sampling

Sampling is a nondestructive technique used in commercial routers for collecting information about the amount of traffic travelling through the router. If only one packet out of 1000 packets, for example, is examined, then the amount of collected data can be reduced, so the measurement system can easily analyze the data at the lower load. However, detailed characteristics cannot be obtained if the sampling rate is very low. On the other hand, in non-sampling measurement, where all information is acquired, the system load for collecting all of the packets is very high. Thus, it has been difficult for network routers, switches, and inexpensive general-purpose PCs to perform such measurements.
(2) Header and payload analysis

In header analysis, a system examines only a small amount of data from the head of each packet while ignoring the rest. As a result, the system’s load is low because there is less data to analyze. On the other hand, payload analysis examines the entire contents of each packet: this is useful for detecting computer viruses and for intrusion detection systems.

5. Network measurement extensions of the measurement NIC

We have implemented the following hardware-assisted extensions in our measurement NIC to make the precise measurement of high-speed networks easy (Fig. 2).

Fig. 2. Hardware-assisted features.

(1) Wire-rate packet handling function

This function achieves 10-Gbit/s non-sampling analysis even with a system based on a general-purpose PC. It includes hardware-assisted header examination (called header extraction) and efficient DMA (direct memory access) data transfer mechanisms to fully utilize the limited bandwidth of the PCI bus.
(2) Header extraction function

This function can extract only the header, which reduces the data analysis amount. Since the extraction size can be set to an arbitrary value, our measurement NIC can also be useful for payload analysis.
(3) Precise timestamp function

Our measurement NIC can generate precise timestamps based on external timing signals from GPS receivers, for example. Timestamps are appended to packets at both the sender and receiver sides. They enable precise measurement of network delays and jitter. Even when a GPS signal is not used, the traffic burstiness can be precisely measured by using the internal clock on our measurement NIC.
(4) Packet capturing software library

The packet capturing software library has application programming interfaces compliant with the de-facto standard library “libpcap”. By using this library, existing network measurement software based on libpcap can easily take advantage of our measurement NIC’s hardware-assisted functions.
(5) Traffic playback function

The hardware-assisted traffic playback function can utilize stored characteristics (from previously captured traffic) to generate test traffic and inject it into a network under test while emulating the stored characteristics precisely [1]. In detail, protocol fields, IPGs, and packet lengths can be exactly matched thanks to the hardware-based IPG control with 5-ns resolution. This function is useful because it lets network operators measure the network quality when an application’s traffic flows through a network without deploying actual servers in each site.
(6) Network measurement and traffic controls in servers

Our measurement NIC can work as a normal NIC in end-node servers such as streaming servers while its measurement features are simultaneously being utilized. For example, it can append timestamps to packets carrying audio and video signals for application-specific traffic-QoS measurement.

6. Development of precise measurement systems for 10-Gbit/s networks

We have developed a precise measurement system for high-speed networks by using these network measurement extensions.

7. Applicability evaluation in 10-Gbit/s high-speed networks

We evaluated the applicability of deployed systems at IMC Tokyo 2008 Interop Media Convergence held in Japan in June 2008.

8. Evaluation items

From the streaming server in Osaka, an uncompressed 4K digital-cinema-quality video stream was delivered to the IMC venue at Makuhari at a bandwidth of 6.5 Gbit/s. At the same time, uncompressed HDTV video content was remotely retrieved from Musashino and delivered to the IMC venue by the video editor there at 1.5 Gbit/s. The total amount of traffic was around 8 Gbit/s, which is extremely high compared with existing streaming applications. In the experiment, commonly used existing network measurement software was also used by NTT Communications to verify its applicability to high-speed networks. We, NTT Labs., performed the following measurements.

9. Results

The IPG distributions at the three sites are shown in Fig. 5. In Osaka, some peaks were observed at around 6-, 12-, 19-, 28-, and 32-µs IPGs, but these peaks disappeared after the packets traversed the network to Musashino and Otemachi. From the distribution pattern around the peak values there, the IPG values were dispersed into various values which, we think, may have been caused by queueing in Ethernet switches and routers in the network. We have previously reported that the IPG distribution pattern in a congested switch became broader when the switch congestion became worse [2].

Fig. 5. Observed IPG characteristics (left: in a wide IPG range, right: magnified view).

Figure 5 also shows a magnified view of the IPG distribution at around 6 µs. Since 9-kbyte jumbo packets were transmitted in the experiment, the observation of 6-µs IPGs revealed that the traffic flowed at 10-Gbit/s wire-rate speed with very fine time resolution. Such bursty traffic can sometimes cause network trouble such as packet dropping when intermediate network routers overflow. Furthermore, the number of 6-µs IPGs observed at Musashino and Otemachi increased to 30% compared with 17% in Osaka, which reveals that the traffic burstiness severely worsened. These results support the applicability of our systems to QoS measurement in 10-Gbit/s networks. We successfully clarified the fluctuating behavior of traffic characteristics influenced by the network configuration in real time.

In the high-performance NetFlow analysis, we did not detect any failures to capture packets of 8-Gbit/s traffic even in non-sampling measurements. Although the number of flows was less than 10, the experiment confirmed the improvement in NetFlow probe performance.

10. Conclusion

Through experiments, we have achieved successful realtime measurement of detailed traffic characteristics by using our measurement NIC based on general-purpose PCs, which have previously been difficult to utilize in 10-Gbit/s networks. By widely deploying the measurement systems, network operators can obtain useful information about network status to maintain their service quality.

In this article, we mentioned only some of the many measurement features implemented in our measurement NIC. We are planning to further develop advanced network measurement systems including packet capturing systems incorporating a large amount of storage and sophisticated active measurement systems such as ones providing available bandwidth estimations based on a hardware-assisted traffic generator.

Acknowledgment

This research is partially funded by the National Institute of Information and Communication Technology (NICT).

References

[1]	K. Shimizu, K. Sebayashi, T. Kawano, and M. Maruyama, “The network measurement system for 10-Gbit/s stream traffic with precise traffic playback functions based on a general-purpose PC,” BS-8-1, IEICE Society Conference, 2008 (in Japanese).
[2]	K. Shimizu, T. Ogura, T. Kawano, H. Kimiyama, M. Maruyama, and K. Koyanagi, “Application-coexistent Wire-Rate Network Monitor for 10 Gigabit-per-Second Network,” IEICE Trans. Inf. & Syst., Vol. E89-D, No. 12, pp. 2875–2885, 2006.

↑ TOP