16. Understanding TCP Incast Throughput Collapse in Datacenter Networks

The unique workloads and fast connection low latency characteristics of data centers somewhat differs from what TCP was designed for in the first place. The infamous TCP incast collapse is a major problem in modern data centers. This paper analyzes the dynamics of incast and provides experiment results among different solutions.

The authors approach the incast from three perspectives,

1. it is not limited to particular network environments
2. causes of the problem should be identified and symptoms should be predictable
3. modifications to TCP stack to solve it

Method: synchronized read requests 100 blocks of data from a set of 1-48 storage servers

Things that do NOT help: TCP Reno, New Reno, SACK

Things do:

• reducing min value of RTO (but randomizing the minimum and initial RTO doesn't work, which is against intuition),
• fine-grained OS timers for sub-milisecond RTO,
• randomizing RTO,
• disabling TCP delayed ACKs when possible.

Comment: I like the way they search out all current solutions, but where are the analysis results from all these different working methods to TCP incast problem?

15. Safe and Effective Fine-grained TCP Retransmissions for Datacenter Communication

This paper deals with the TCP incast problem when inbound data overflows the switch buffers by using high resolution timers for TCP time-outs. When there are multiple end-node sending out queries, the client side cannot make forward progress until they receive responses from every server, this is referred to barrier-synchronized requests in the article.

TCP incast collapse:

• high-bandwidth, low-latency networks with small buffers in the switches
• clients issue barrier-sync requests in parallel
• servers respond with small amount of data per request

By enabling microsecond-granularity retransmission time-outs (RTO), the authors intended to solve the incast problem commonly seen in data centers,

1. modify TCP implementation to use high-resolution kernel timers, timeout = 2^backoff (RTO+ rand(0.5) * RTO )
2. prevent TCP incast collapse for up to 47 concurrent senders
3. recover in data centers do not affect performance

Comment: This is a different approach to solve the problems in data centers. The other papers we read strived for backward-compatibility, while this one modifies the TCP stack. Since data centers are usually confined to a fixed location and managed by normally one company, I guess it is a feasible way to solve TCP incast collapse.

14. VL2: A Scalable and Flexible Data Center Network

Today's data centers prevent high utilization, according to the authors, by several ways:

• Not providing enough capacity between the servers.
• Congestion and high resource utilization occurs in some of the servers while others are rather lightly-loaded.
• When one service has a traffic load, all other hosts in the same tree/branch suffer collateral damage.
• Lack of easy migration of VLANs.

By measuring network traffic, flow distribution, analyzing traffic matrix and failure characteristics, Virtual Layer 2 (VL2) is introduced to address the above issues. Design:

• Valiant Load Balancing to do random traffic spreading over multiple paths
• Based on IP
• Ability to host any service on any server, separating server names from locators.
• Replaces ARP queries to VL2 directory

Comments:

1. The last 2 papers are both IP-based to achieve backward compatibility, and replaces ARP broadcasts with queries to a specific server. Seems very promising in today's economy: optimal utilization with existing hardware.
2. It's good to see a paper addressing the 144-ports switches and 24-ports from the cost-effective perspective. It's very practical and yet hardly mentioned in "research", which kind of reminds me when I had my master thesis proposal, my advisors laughed at the idea of my mentioning how much can be saved if the technique could be implemented. I guess after working for Microsoft Research, the authors live in a more practical world than most of us bookworms, haha.

13. PortLand: A Scalable Fault-Tolerant Layer 2 Data Center Network Fabric

With the rapid growth of the Internet, modern computer trends are data centers with multi-core processors and end-host virtualization according to the paper. In view of the shortcomings of SEATTLE and TRILL, the authors proposed PortLand in order to make a plug-and-play-friendly environment for data center system administrators.

PortLand deals with the following issues successfully:

• Virtual Machines (VM) may migrate from one physical machine to another, and they don't need to change their IP addresses
• No switch configuration needed before deployment.
• Any end host can communicate with other hosts in the data center along any physical path.
• No forwarding loops (as SEATTLE would have when the number of hosts grow).
• Rapid failure detection.

Design of PortLand: layer 2 routing, forwarding and addressing for data center

• Fabric manager: centralized network topology, assisting ARP
• Positional Pseudo MAC Address (PMAC): provides location of the host in the topology
• Proxy-based ARP: intercepts IP2MAC and forwards to fabric manager, avoiding broadcast storm
• Distributed location discovery: applying location discovery protocol (LDP), so positioned can be changed without manual overriding.
• Provably loop free forwarding: after LDP, switches update their forwarding tables.
• Fault tolerant routing: switches informs fabric manager; fabric manager updates with the new info; fabric manager informs all affected switches

Comment: The idea of a plug-and-play data center network sounds great, but their simulation seemed to be performed locally. Is there any way it can also be used on 'cloud computing'?

12. Detailed Diagnosis in Enterprise Networks

From experience in small enterprise networks, the authors developed a diagnosic system scalable to large networks by analysing joint behavior of two components in the past and estimate the impact of current events.

• existing diagnostic systems: lack detail, require extensive knowledge, sacrifice details
• NetMedic:
  • framing detailed dignosis as an inference problem
  • estimate when two nodes in the network are impacting each other without knowing how they interact
  • captures state of network using many variables
  • application unrelated

By anaylizing logs of small enterprises for a month with 450,000 entries, a classification of the problems is established. Component states were captured and dependancy was generated to determine which impacts which. Later it is implemented to large enterprise networks and it works successfully and identifies 80% on their top 10 list.

Comment: I remember a couple years ago, NetMedic was THE hip network diagnosis software like Norton SystemWorks. Are they still popular these days? It's interesting to see they "trained" the software with small ethernet network and implements it on large networks.

11. Floodless in SEATTLE: A Scalable Ethernet Architecture for Large Enterprises

The authors try to build a protocol that integrates the scalability of IP and simplicity of Ethernet.

• Advantages of ethernet: persistent MAC addresses, bridged automatically to build routing table (ARP, DHCP); plug-and-play
• Disadvantages of ethernet: initial setup includes broadcasting, consumes resources, security and privacy issues
• Pros of IP: shortest-path routing
• Cons of IP: subnetting wastes address space; Virtual LANs (VLANs) inefficient for large spanning trees

SEATTLE (Scalable Ethernet Architecture for Large Enterprises):

• one-hope, network layer DHT: stores the location of each end-host, distributed directory
• traffic-drive location resolution and caching: routers cache responses to queries; includes location info on ARP replies
• scalable, prompt cache-update protocol: based on unicast, instead of broadcast/timeout to update cache

Simulation: measures on cache eviction timeout, forwarding table size, path stability, switch failure rate, and host mobility rate.

Comment: In view of the high human-error rate of the previous reading "BPG Misconfiguration," SEATTLE's idea about making the network "plug-and-play-able" seems to be very convenient and fool-proof. While at the same time, it provides network administrators to customize network transport. Since this is a 2008 paper, I wonder how it works in the real world?

10. Congestion Control for High Bandwidth-Delay Product Networks

This paper proposed a new protocol in replace of TCP. The new protocol, eXplicit Control Protocol (XCP), addressed the issues of TCP by putting a congestion level field called Explicit Congestion Notification Proposal (ECN) in the packet headers. This XCP protocol does not maintain a per-flow state in routers and thus, requires less CPU cycles in busy routers.

TCP doesn't react well with high bandwidth, high latency links such as satellite and wireless. Under such circumstances, TCP tends to be unstable and oscillate. For example, TCP sends one packet per RTT to get more bandwidth, and it can takes thousands of RTTs to make good use of the high speed network.

Robustness to congestion should be independent of number of flows, according to the authors. And packet loss shouldn't be a signal of network congestion. Thus, XCP introduces decoupling utilization control from fairness control, which adjusts its aggressiveness according to the spare bandwidth in the network and the feedback delay. It also has a per-flow control in the protocol, rather than in routers, to achieve fairness. Furthermore, XCP distinguishes error loss pack-dropping from congestion, which is quite useful on wireless connections.

Comments:

1. I like the fact that in XCP, packet drops caused by congestion are highly uncommon because it separates drops caused by link errors from network congestion. These two factors should be viewed independently in the protocol and acted upon accordingly.
2. Like every new stuff in our lives, backward compatibility is a big issue. XCP is just like IPv6 where everyone knows about it, but no one seems to be using it. Would it be possible to implement XCP concepts one level up in OSI model in order to be compatible with TCP?