Networking for Nutanix Flow: The Prerequisites

Fundamentals · Prerequisites · Nutanix Flow

The general networking you need in order to read a Flow Virtual Networking and Flow Network Security study guide without stopping every third paragraph. Vendor neutral fundamentals, with a note at the end of each section on where Flow uses them.

Most people who end up owning Nutanix Flow did not come from a networking background. They came from virtualization, or storage, or the platform team, and Flow arrived as a checkbox in Prism Central that suddenly required them to have opinions about BGP session states and MTU arithmetic. The Nutanix documentation is good at telling you which field to fill in. It is not written to teach you why the field exists.

This is the missing layer underneath. It is not a networking course and it is not exam material. It is the set of concepts that a Flow study guide assumes you already have, arranged in the order you will hit them, with the Flow specific payoff called out so you can see why each one matters.

1. Layer 2: frames, MAC addresses, and VLANs

A switch moves frames. A frame carries a source and destination MAC address, which is a 48 bit hardware identifier burned into a NIC, or assigned to a virtual NIC by the hypervisor.

Switches are not configured with a map of where things are. They learn. When a frame arrives on a port, the switch records “source MAC X lives on port 3” in its MAC address table. When a frame needs to go to a destination it has already learned, it sends it out that one port. This is why switching scales.

The three things a switch does with a frame

Unknown unicast flooding is the one worth remembering, because it is a behaviour some applications quietly depend on, and modern virtual networking often disables it. Clustered applications that expect to receive traffic for a MAC they have not yet announced will break when flooding is turned off, and the failure looks like an application problem rather than a network one.

Broadcast domains and why VLANs exist

Every port that can receive a given broadcast is in the same broadcast domain. One flat switched network is one broadcast domain, and every device in it hears every broadcast. That does not scale, and it provides no separation at all.

A VLAN cuts one physical switch into several logical switches. Ports in VLAN 10 cannot see traffic in VLAN 20 without something routing between them. VLAN IDs run from 1 to 4094, which is both the strength and the ceiling of the technology.

• Access port. Belongs to exactly one VLAN. The endpoint sends and receives ordinary untagged frames and has no idea VLANs exist. This is how most VMs are connected.
• Trunk port. Carries several VLANs, tagging each frame with a 4 byte 802.1Q header so the far end can separate them. This is how a hypervisor host uplink normally connects to the physical switch.
• Native or untagged VLAN. The one VLAN on a trunk whose frames travel with no tag. Mismatched native VLANs on the two ends of a trunk is a classic and very confusing outage.

ARP, the glue between Layer 2 and Layer 3

A host that wants to send to an IP address on its own subnet does not know the destination MAC. It broadcasts an ARP request: “who has 10.0.0.5?” The owner replies with its MAC, the sender caches it, and the conversation proceeds.

Two consequences worth carrying:

• ARP is a broadcast, so it is confined to the broadcast domain. This is why a VLAN boundary is a real boundary.
• ARP caches go stale. When something moves, or something else starts answering for an address, traffic follows the cache and not reality until the entry ages out. A surprising number of “the network is broken” incidents are stale or contested ARP.

2. Layer 3: addresses, prefixes, and routes

Routing moves packets between subnets. Where switching asks “which port has this MAC”, routing asks “which next hop gets me closer to this network”.

Reading CIDR without thinking about it

An address like 10.20.30.0/24 is a network address and a prefix length. The prefix length says how many leading bits are the network portion. Everything after is host space.

Smaller number means bigger network. A /16 contains 256 /24s. That inversion is the single most common stumbling block for people new to this.

Private address space

Three ranges are reserved for internal use and are never routed on the public internet, defined in RFC 1918:

• 10.0.0.0/8, a very large flat block
• 172.16.0.0/12, which is 172.16 through 172.31 and catches people out
• 192.168.0.0/16

Because everyone uses the same private ranges, overlapping address space between two organizations, or between two tenants, is normal rather than exceptional. Every technology in this document that involves joining two networks has to have an answer for it.

The routing table and longest prefix match

A routing table is a list of destinations and next hops. When a packet arrives, the router checks every entry that could contain the destination and picks the one with the longest prefix, meaning the most specific match. It does not pick the first match, or the cheapest, until specificity has already tied.

The default route, 0.0.0.0/0, matches everything and is therefore always the least specific entry in the table. It is the catch all for “I do not have a better idea, send it to the edge”. Any network that needs to reach things outside itself needs one, or an explicit route for every destination it cares about.

The next hop is the address of the neighbouring router that will take the packet onward. It is always an address the router can already reach directly, which is why a next hop outside the local subnet is a configuration error rather than a clever shortcut.

Static versus dynamic routing

• Static routes are typed in by a human. They are predictable, they never converge on a new path, and they are silently wrong the moment the topology changes.
• Dynamic routing protocols let routers tell each other what they can reach, and recompute when something fails. In this world that means BGP, with OSPF appearing occasionally inside a single site.

3. BGP, only the parts Flow uses

BGP is the routing protocol of the internet and, increasingly, of the data centre. It is large. Flow uses a small and well defined corner of it, and you can read the entire Flow BGP surface knowing six things.

One: autonomous systems and the two flavours of BGP

An autonomous system is a network under one administrative control, identified by an ASN. Private ASNs exist for internal use, the same way private IP ranges do.

• eBGP runs between different autonomous systems. Different ASN on each end.
• iBGP runs inside one autonomous system. Same ASN on both ends.

They behave differently in ways that matter to a network engineer and barely at all here. What matters is that when a product says “eBGP”, it is telling you the two ends are meant to be separately administered networks that agree to exchange routes.

Two: a session is a TCP connection, and its state has a trap in it

BGP peers establish a TCP connection, on port 179, and exchange routes over it. The session walks through a state machine. Only one state means the session is working.

Three: advertisement is a choice, not a consequence

A BGP speaker does not automatically tell its neighbour about everything it knows. It advertises what its policy says to advertise. If a prefix is not reaching a peer, the first question is not “is the session up” but “is this prefix in what we advertise to that peer”.

Four: path selection, and the two ways to influence it

When the same destination is learned from more than one peer, BGP picks one path using a long ordered list of tie breakers. Two of them are the ones you will actually see exposed in products.

The asymmetry is the point. You control your own preferences directly. You can only influence someone else’s, and the polite way to do it is to make your own route look worse, not to try to make it look better.

Five: communities are tags, and that is all

A BGP community is a label attached to a route, conventionally written ASN:value. BGP itself does nothing with them. They exist so that routers downstream can match on the tag and apply policy. Think of them as sticky notes that survive the journey.

Six: what a healthy BGP relationship needs

• IP reachability between the two peer addresses before BGP can do anything. BGP rides on TCP and TCP needs a working path.
• Matching expectations about ASN and peer address on both ends.
• Optionally an MD5 password on the session. Worth knowing because address translation between the peers breaks password verification, since the authentication covers header fields that NAT rewrites.

4. NAT, SNAT, and floating IPs

Network Address Translation rewrites addresses in a packet as it crosses a boundary. It exists because private address space is not routable outside, and because organizations run out of public addresses.

The asymmetry is the thing to internalise. SNAT alone gives you outbound connectivity and no inbound reachability, because there is nothing to tell the translator which internal host an unsolicited inbound packet was meant for. That is not a bug, it is often the point.

Floating IPs

A floating IP is an externally routable address held by the platform rather than by a machine, and bound to a workload on request. It is 1:1 NAT with a management story: the address survives the workload, so you can move it to a replacement instance without anything downstream needing to change.

What NAT quietly breaks

• Anything that authenticates the header. BGP MD5 and IPsec AH both cover fields NAT rewrites.
• Anything that carries an address inside the payload. The classic examples are FTP and SIP, which announce addresses in the data stream that the translator does not know to rewrite.
• End to end identity in logs. Behind SNAT, everything appears to come from one address, which matters more than people expect when investigating an incident.

5. Overlay and underlay

This is the concept that makes modern virtual networking make sense, and it is one idea: put a whole network inside packets on another network.

• The underlay is the real physical network. Switches, cables, IP addresses, ordinary routing. It has one job: carry packets between hosts.
• The overlay is a virtual network built on top, whose packets are wrapped inside underlay packets and carried across as ordinary payload.

The underlay does not know the overlay exists. It sees normal UDP traffic between two host addresses. All the tenant structure, the VLANs, the addressing, lives in the wrapper.

The encapsulation formats you will meet

VTEP: where the wrapping happens

A VTEP, tunnel endpoint, is the thing that adds the wrapper on the way in and strips it on the way out. It can be a physical switch, or software inside a hypervisor. Two VTEPs with underlay reachability between them can carry an overlay segment between them, and the underlay never needs to be told.

This is the whole trick, and it explains why overlays are attractive: the tenant topology becomes software, and the physical network stops needing to change every time the tenant topology does.

6. MTU, which is where most of this actually goes wrong

If you learn one thing here, learn this one. Encapsulation problems almost always present as MTU problems, and MTU problems present as some of the most misleading symptoms in networking.

The two numbers

• MTU, maximum transmission unit, is the largest payload a link will carry in one frame. The classic Ethernet value is 1500 bytes.
• MSS, maximum segment size, is the largest chunk of TCP data in one segment. For IPv4 it is conventionally MTU minus 40, being 20 bytes of IP header and 20 of TCP header.

Jumbo frames

A jumbo frame is anything above the standard 1500, and 9000 is the near universal convention. It exists to reduce per packet overhead on high throughput links, and in the overlay world it exists to give you headroom to absorb encapsulation without shrinking the guest.

The arithmetic of stacked overhead

If the underlay stays at 1500 and the guest keeps its own MTU at 1500, the encapsulated frame does not fit. Something must give. Either the underlay grows, or the guest shrinks by exactly the overhead:

Overheads stack. Each tunnel you nest costs its own headers. This is why the guidance for a simple overlay and the guidance for an extended, encrypted overlay are not the same number, and why a fallback value that works for the simple case is not enough for the complex one.

Path MTU Discovery, and how it gets broken

A sender does not know the smallest MTU along the path. Path MTU Discovery finds it: the sender marks packets Do Not Fragment, and any router that cannot forward one replies with an ICMP Destination Unreachable, Fragmentation Needed message, type 3 code 4, carrying the MTU it can accept. The sender shrinks and retries.

7. The hypervisor is a switch

Inside every virtualization host there is a software switch. VM virtual NICs plug into it, and it plugs into the physical NICs. Understanding its parts removes most of the mystery from host networking.

Bond modes, and why the choice matters

LACP, standardised as 802.3ad, is the protocol two ends use to agree that a set of links is one logical link. It has a negotiation rate, fast or slow, and both ends should agree. It also has a failure behaviour worth knowing: if negotiation does not complete, you want the switch to fall back to treating the ports as individual links rather than leaving them all down, otherwise a partial misconfiguration takes the host completely offline.

8. Stateful firewalling and microsegmentation

The five tuple

A network conversation is identified by five values. Nearly every firewall, load balancer and flow record in existence is built on them:

Stateless versus stateful

• A stateless filter evaluates each packet on its own. It has no memory. To allow a conversation you must write a rule for the request and a matching rule for the reply, which is how classic router access lists work and why they are so error prone.
• A stateful firewall records each permitted conversation in a connection tracking table, and automatically permits packets belonging to a conversation it already approved.

The tracking table is finite. Under a connection flood, or a workload that opens huge numbers of short lived sessions, it fills, and new connections fail while existing ones continue. That failure mode looks like partial, intermittent outage rather than a clean break, which is why platforms expose conntrack utilisation as a health metric.

East west and north south

• North south traffic crosses the boundary of the environment. It goes to or from the outside. This is what a traditional perimeter firewall inspects.
• East west traffic moves between workloads inside the environment. Historically nothing inspected it at all.

In a virtualized data centre east west is the majority of traffic by volume, and it is the direction an attacker moves after gaining an initial foothold. A perimeter firewall sees none of it.

Microsegmentation

Microsegmentation enforces policy at each workload’s own virtual NIC rather than at a choke point. Every VM effectively gets its own firewall, enforced by the hypervisor, and traffic between two VMs on the same host is filtered without ever touching the physical network.

Three ideas travel with it:

• Default deny. Nothing is allowed unless a policy allows it. The opposite of the traditional default allow inside a trusted zone.
• Allowlist. You enumerate what is permitted rather than what is blocked, because the set of permitted things is knowable and the set of bad things is not.
• Identity based grouping. Policy targets labels or tags such as “production” or “web tier” rather than IP addresses, so it survives the workload moving or being rebuilt.

9. Load balancing at Layer 4

A load balancer presents one address and spreads connections across several backends.

Layer 4 versus Layer 7

• Layer 4 balances TCP and UDP connections. It sees addresses and ports. It is fast, protocol agnostic, and cannot make decisions based on content.
• Layer 7 understands the application protocol, typically HTTP, and can route on URL path, header or cookie, terminate TLS, and rewrite requests. It is far more capable and considerably more expensive.

If a product says it provides native Layer 4 load balancing, it is telling you it will not route on URL path, and that anything requiring content awareness needs a separate appliance.

How a connection is placed: hashing

The common approach is to hash the five tuple and use the result to pick a backend. Because every packet of one connection has the same five tuple, every packet of a connection lands on the same backend, which is what makes stateful protocols work. Because different connections hash differently, load spreads.

Note what this is not. It is not round robin, and it is not least connections. A five tuple hash gives you consistency, not fairness. Two heavy clients can hash to the same backend, and the balancer will not correct for it.

Health checks, in detail, because they are where the surprises live

• A TCP probe opens a connection to the configured port and immediately closes it. If the port answers, the backend is healthy. It says nothing about whether the application behind the port is actually working, only that something is listening.
• A UDP probe is harder, because UDP has no handshake. The usual approach is to send a datagram and treat silence as success, since an ICMP unreachable is the only clear negative signal available.
• Thresholds. Health checks are almost always configured with a check interval, a timeout, and a count of consecutive results needed to change state. Requiring several consecutive results is what keeps a single lost packet from ejecting a healthy backend.

10. VPN and IPsec

A site to site VPN builds an encrypted tunnel across an untrusted network so two private networks can talk as if directly connected.

Initiator and responder

Someone has to start. One end is configured to initiate and the other to accept. This matters when one side sits behind NAT or a firewall that only permits outbound connections: that side must be the initiator, because the other end cannot reach it unsolicited.

The two routing questions every VPN has to answer

1. Which traffic goes into the tunnel? Either a static list of prefixes on each side, or a routing protocol running inside the tunnel so each side learns the other’s networks dynamically.
2. What happens when the address space overlaps? Two sites both using 10.0.0.0/8 cannot simply be joined. Something has to translate, or the design has to change.

11. Stretching Layer 2 between sites

Sometimes a workload has to keep its IP address while moving somewhere the address does not belong. The usual drivers are a migration you cannot re address, a disaster recovery plan that needs identical addressing, or an application that hard codes addresses somewhere nobody can find.

Layer 2 extension, also called stretched Layer 2 or Layer 2 stretch, makes one broadcast domain span two locations by tunnelling frames between them, usually over VXLAN.

Why network engineers wince at it

• You are extending a failure domain. A broadcast storm or a loop at one site is now a problem at both.
• Broadcast and ARP traffic crosses the link, consuming inter site bandwidth doing nothing useful.
• The default gateway problem. The subnet exists in two places, but the gateway usually does not, so traffic leaving the subnet at the remote site may have to travel back to the original site first.

Tromboning, also called hairpinning

Tromboning is traffic travelling somewhere it does not need to go and coming back, because the only device that can make a decision about it lives there. On a stretched subnet it is the default outcome unless you deliberately design against it, by giving each site a locally valid gateway for the shared subnet.

Two operational hazards worth knowing in advance

• Overlapping addressing on the two sides. Once one subnet exists in two places, two things being given the same address is no longer a theoretical problem. Whichever side manages addresses has to know about the other.
• Stale ARP responders. If something keeps answering ARP for a workload that has moved or been powered off, traffic follows the answer and disappears. This is a common and genuinely difficult failure during migrations, because nothing at the destination indicates what is wrong.

12. Flow telemetry, and what IPFIX actually is

Packet capture tells you everything and does not scale. Counters scale and tell you almost nothing. Flow telemetry is the middle ground: a summary record per conversation.

NetFlow is the original, from Cisco. IPFIX is the IETF standardisation of the same idea, sometimes described as NetFlow version 10. Its distinguishing feature is that it is template based: the exporter first sends a template describing the fields in a record, then sends records matching it. That is what lets vendors add their own fields without breaking collectors, and it is also why a collector that has not yet received the template cannot interpret the records that follow.

What flow data is good and bad at

That last row on the left is the reason flow telemetry and microsegmentation are always found together. You cannot write a default deny policy for an application you do not fully understand, and flow data is how you come to understand it.

13. Identity plumbing

Two separate things get confused constantly, and separating them explains most access control documentation.

Directory services

• Active Directory is Microsoft’s directory. It holds users, groups and computers.
• LDAP is the protocol for querying a directory, on TCP 389, or 636 for LDAPS. Port 3268 is Active Directory’s global catalog, which searches the whole forest rather than one domain, and is often the right port when a single domain query returns incomplete results.
• Kerberos is the authentication protocol Active Directory actually uses for logons.
• WMI is Microsoft’s management interface, reached over RPC. It starts on TCP 135 and then moves to a dynamically assigned high port, which is why WMI through a firewall requires opening a wide ephemeral range and is a recurring source of pain.

Identity based firewalling

The idea: instead of writing policy about addresses, write policy about users. “Members of the Finance group may reach the finance application” rather than “10.20.30.0/24 may reach 10.50.60.0/24”.

The mechanism is always the same. Something watches the directory for logon events, learns that user X just logged on to machine Y, and programs a rule for Y’s address for as long as that session lasts. It follows that this style of policy:

• Depends entirely on reading logon events from domain controllers, which is why the WMI and LDAP port requirements above suddenly matter.
• Works best for desktop sessions, where one user maps cleanly to one machine, and much less well for multi user servers.
• Breaks quietly when the directory connection breaks, because there are no new logon events to learn from, and the existing mappings age out.

Role based access control

RBAC grants permissions to roles, and roles to users, rather than permissions directly to users. Two details are worth carrying into any platform:

• A role is usually not enough on its own. Many platforms separate the role, which is a set of permissions, from the assignment that binds a role to a user over a defined scope of objects. Granting the role without the assignment produces a user who appears configured and can do nothing.
• Check the unconfigured default. Some systems fail open, granting broad access when no mapping exists. Others fail closed, granting nothing. Assuming the wrong one is a security incident in the first case and a support ticket in the second.

14. The supporting cast

DNS and NTP, which are not optional

Modern infrastructure treats both as dependencies rather than conveniences. Appliances that cannot resolve names or agree on the time frequently refuse to start, and report themselves as unhealthy without saying why. Certificate validation, Kerberos authentication and log correlation all fail when clocks drift. When a newly deployed appliance comes up unhealthy for no visible reason, check DNS and NTP before anything else.

ICMP is not optional either

Blanket ICMP blocking breaks path MTU discovery, breaks health checks, and blinds you during troubleshooting. Blocking echo request selectively is a defensible choice; blocking type 3 is not.

Syslog severities

Eight levels, numbered 0 as the most severe, which is the opposite of most people’s intuition:

Configuring a collector to receive “severity 4 and below” means warning and everything more serious, not warning and everything less. Getting this backwards produces either a silent log server or a flood.

Ports you will see repeatedly

Decoder ring: generic term to Nutanix term

Much of the difficulty in reading any vendor’s networking documentation is that familiar concepts arrive under unfamiliar names. This is the mapping.

If you read nothing else