Enable FNS · Confirm supported versions · Identify required resources

Enable: Prism Central Settings > Microsegmentation > tick Enable Microsegmentation > Save. Disabled by default. Requires a Flow license, or a 60-day trial.

Two enablement side effects and two connectivity requirements

• A Kafka container called flow_data is created on the cluster hosting Prism Central. It stores data essential for Flow visualization. Do not delete it.
• Enabling microsegmentation for the first time also requires enabling Advanced Networking.
• On a Prism Central scale out instance, all PC VMs must be powered on.
• Allow AHV hosts to reach the Prism Central VMs over TCP port 9446 for connection tracking data.

Alert 200614 catches a version mismatch after the fact and tells you to run an LCM inventory of FNS PE on each attached AHV cluster and upgrade the laggards. KB14262.

Create categories and associate to VMs

Policies apply to categories, not to VMs, so any number of VMs starting in a category are secured without administrative intervention. In the Next-Gen policy model any category can be a secured entity, not just AppType. Built-in categories are AppTier, AppType, Environment, Quarantine (immutable), ADGroup, and ADGroup:Default. Bring your own, but never reuse system defined names.

VPC as a category, new in 5.2.0: assign a set of VPCs to a category so a policy scope spans multiple VPCs, usable in inbound and outbound rules, as a secured entity, and inside an entity group.

Association is sourced. Creation is not.

A category is a key value pair that groups similar entities. Associating a policy with a category ensures the policy applies to all entities in the group regardless of how the group scales with time. Currently you can associate only VMs with a category.

• Per VM: Infrastructure > Compute > VMs > click the VM > Categories tab > Manage Categories. Each VM has a one to many relationship with categories, and categories have a many to many relationship with policies.
• In bulk: Infrastructure > Compute > VMs > tick the target VMs > Actions > Manage Categories > type the name in Set Categories and pick from the matching list.
• Hosts, clusters, and images have their own association topics, as does the AHV Administration Guide.
• Timing: a host category attach or detach takes around five minutes to reflect in the applicable VM-Host affinity policies. The Entities tab count updates immediately; the two views use different APIs.

Creating a category

Sourced from the Prism Central Admin Center Guide 7.6, topics 059 to 071. This closed the last documentation gap in the guide.

Procedure: Application Switcher > Admin Center > Categories > New Category.

Assigning: Application Switcher > Infrastructure > the entity’s dashboard > select entities on the List tab > Other Actions > Manage Categories > Set Categories > Save. Policies associated with the category you pick appear in the Possible Associated Policies field, the closest thing the UI gives to an impact preview.

Filtering the Categories List: Key or Value conditions are Contains, Equal to, Not equal to, Doesn’t contain, Starts with, Ends with. You can also filter by Owner Projects, Shared Projects, Type (System or User), Entity Types (Cluster, Host, Image, Subnet, VM, Volume Group) and Policy Types (Affinity, Alert, Image Placement, NGT, Protection, QoS, Security, Storage, VM Startup). The list shows Associated Entities and Associated Policies counts; a dash means not available or not applicable.

Project scoped categories: a 7.6 change, NOT the tested behaviour

Prism Element CVM memory for Flow Network Security

Alert 200613 Flow Network Security CVM Memory Recommendation carries a figure that appears in no other Nutanix document.

The complete Flow memory picture

Checking and increasing CVM memory

Check: Prism UI Menu > VM > Table, tick Include Controller VM, search CVM, review the Memory Capacity column. Or from a CVM, allssh free.

Increase: Prism Element web console > Settings > General > Configure CVM > select Target CVM Memory Allocation > Apply. Run NCC first, per the guide’s own step 2.

Foundation context: at cluster creation Foundation sets CVM memory to available RAM minus 16 GiB, capped at 256 GiB from Foundation 5.3 onward, and CVM vCPU to host cores minus 2, capped at 22 vCPUs through Foundation 5.3.x (the cap no longer applies from Foundation 5.4).

Confirm the Network Controller is enabled and the right version · Cluster compatibility

Enable: Prism Central Settings > Network Controller > Enable, then check the Recommendations section. Auto enabled on X-Large at pc.2023.3 or later.

Four ways incompatibility surfaces

1. At enablement: NC is created but Prism Central raises alert 130201, “Failed to configure host for Atlas networking”.
2. At deployment with a compatible PC package but incompatible AHV package: NC is deployed but not enabled.
3. At registration: NC is not enabled by default on a newly registered PE cluster with incompatible AHV.
4. At upgrade: the NC upgrade fails to start after the pre check if any FVN enabled cluster runs an incompatible AHV version.

Other prerequisites

• Prism Central hosted on an AOS cluster running AHV. Not ESXi, not Hyper-V.
• Prism Admin role, or the task fails with User Denied Access.
• Microservices Infrastructure enabled, on by default from pc.2022.9.
• A Prism Central virtual IP must exist and must not be changed afterwards.
• Connectivity PC to PE, plus internet to ECR (Docker images) and S3 (LCM portal) unless dark site.
• All AHV clusters at the same site as their registered Prism Central. Each site needs a local PC.
• Compute only nodes need AOS 7.0+ and Files 5.1+; CO clusters cannot create VLAN Subnets.

Exclude a cluster with <atlas> config.add_to_excluded_clusters <cluster uuid>, verify with acli atlas_config.get.

Layer 2 subnet extension: prerequisites and best practices

These sit in the back half of the Layer 2 Network Extension chapter and are easy to skim past. Several of them only surface as a broken migration at 2 AM.

Two prerequisites that are easy to miss

• Pair the local and remote Prism Centrals (availability zones) to use the Create Subnet Extension wizard and get bidirectional communication. Paired AZs support both VXLAN over VPN and VTEP based extension. You can also use the manual gateway and connection workflows instead of pairing.
• Set up a default static route, 0.0.0.0/0, with the external network next hop for the VPC used for any subnet extension. That static route is what gives the Network Gateway appliance its NTP and DNS access.

Retaining VM IP addresses across an extended subnet

• With Nutanix IPAM, address ranges in the paired subnets must be unique. From Network Controller 6.0.0 you can no longer update Overlay subnets in a Layer 2 extension to have overlapping IP pools; the product enforces it.
• With third party IPAM on both sides you check for conflicts yourself. With Nutanix IPAM on both sides, Prism Central displays a message when a conflict exists.

Choosing the extension type

• If site to site connectivity is already encrypted, use VTEP only extension and avoid paying for encryption twice.
• Use the Subnet Extension to a Third Party Data-Center workflow when extending to more than one other AZ (point to multipoint), or when extending between clusters managed by the same Prism Central.
• Avoiding tromboning: provide a valid gateway IP on both the local and remote sides. Supplying one on only one side deliberately hair pins all traffic through that side. Do it on purpose or not at all.

Where the work happens: Network & Security → Connectivity → Subnet Extensions tab. Three paths: Create Subnet Extension Across Availability Zones (point to point over VPN or VTEP), Create Subnet Extension To A Third Party Data-Center (point to point or point to multipoint over VTEP), and Update Subnet Extension Across Availability Zones, which carries the same fields as create and is reached by selecting the extension and clicking Update, or from its Summary tab.

Layer 2 extension over VPN specifically

Use it when the two AZs have no underlying secure connectivity, for example across the Internet, where IPSec supplies both connectivity and encryption. Also for lift and shift from a VLAN subnet to a VPC subnet retaining the same VM IP addresses, where VPN provides Layer 3 connectivity and encryption from the VPC segment back to the other VLAN subnets. Consider VTEP only without VPN when encryption is not required.

One more, from the Connections Management chapter: you can enable network segmentation on a Layer 2 Network Extension that has no gateway. That points at the Security Guide topic “Segmenting a Stretched L2 Network for Disaster Recovery”, in the same network segmentation chapter covered under objective 5.4. Layer 2 Network Extension is also called Layer 2 Stretch, so both names appear.

Incompatible clusters, Network Controller updates, FNS updates

Two actions on an incompatible cluster: upgrade AHV with LCM to the version compatible with the Network Controller upgrade version, or exclude the cluster with config.add_to_excluded_clusters. Excluding is also the path to being able to unregister that Prism Element cluster. The one cluster you can never unregister is the one hosting the FVN enabled Prism Central.

Prism Central upgrade in a Nutanix environment

• Only through LCM. The upgrade involves Microservices Infrastructure and the services on it.
• Dark site needs four bundles on the LCM web server: Security Dashboard CVE Data, LCM Framework, Prism Central LCM, and Prism Central MSP Apps. The Service Manager feature installs PC services and applications at a dark site.
• Port 9440 must be open in both directions between the PC VM and any registered clusters.
• Microservices Infrastructure enablement takes up to 30 minutes on a supported single VM PC. Confirm with ecli task.list. Failure is reported after about 2 hours 5 minutes.

Network gateway upgrades

• Version: Connectivity > Gateways > click the name > Gateway Version in Properties.
• Detect: Admin Center > LCM > Inventory tab.
• Upgrading the VPN appliance disrupts traffic for the duration of the operation.
• Update Gateway fields: External Routing Protocol (Static or eBGP), eBGP ASN (1 to 65000 if you have no BGP environment, must not conflict), VTEP IP Address list, VxLAN UDP port default 4789, do not change, BGP Service IP Address and remote eBGP ASN. Some parameters are greyed out; to change those, create a new gateway and delete the old one.

General cluster expansion prerequisites,

These come from the general expansion topic rather than the Flow Virtual Networking variant, and are easy to skip past.

• Check the Health dashboard first and resolve any failing health checks before adding nodes, then run NCC as a final confirmation.
• Wait for any ongoing expand cluster operations to complete. They do not queue.
• Check the Hardware dashboard for metadata state. Any node showing “Metadata store disabled on the node” or “Node is removed from metadata store” must be fixed first with Enable Metadata Store.
• New nodes must be physically connected on the same subnet as the cluster.
• The expansion process compares AOS versions and upgrades whatever is needed so every node lands on the same version. Budget the time.
• The process varies with AOS, hypervisor, encryption and hardware. Rack fault tolerance, data at rest encryption and Hyper-V are handled as special case steps out of sequence.

Segregate East West and North South traffic

• East/West (intra VPC): sent and received on the AHV host internal port br0 by default, Geneve encapsulated, stays within the VPC.
• North/South: enters or exits the VPC. The external subnet determines the virtual switch and VLAN.
• With network segmentation on vs1, the AHV internal interface terminates Geneve East/West traffic on internal port br1.

Procedure

# 1. Create the virtual switch (for example vs1) on every host in the cluster.

# 2. (Optional) tag a VLAN on the non-default bridge, on the AHV host as root:
root@ahv# nmcli -f ovs-port con show ovs-port-<bridge-name>
root@ahv# nmcli con modify ovs-port-<bridge-name> ovs-port.tag <vlan-tag> ovs-port.vlan-mode access
root@ahv# nmcli con up ovs-port-<bridge-name>

# 3. From the CVM, set host IPs and gateway for the virtual switch:
nutanix@cvm$ acli net.update_virtual_switch vs1 \
  host_ip_addr_config='{host-uuid1:10.10.10.15/24;host-uuid2:10.10.10.16/24}' \
  gateway_ip_address=10.10.10.1

# 4. Point VPC east-west traffic at it:
nutanix@cvm$ acli net.set_vpc_east_west_traffic_config virtual_switch=vs1
# add permit_all_traffic=true to allow SSH or SNMP to the secondary host IP
# update later with acli net.update_vpc_east_west_traffic_config virtual_switch=vs1

Virtual switch IP constraints, each with its own failure message

Virtual switch requirements and migration gotchas

• AOS 5.19 or later with AHV 20201105.12 or later.
• Virtual bridges must have the same name, MTU, and uplink bond type on all nodes.
• Do not create, update, or delete any virtual switch while an AOS or AHV upgrade is running.
• Migration needs the same bond type on all hosts, LACP speed fast or 1 second with no lacp suspend-individual, and upstream spanning-tree portfast or edge trunk. Otherwise a 30-second timeout beats the migration’s 20-second non modifiable timer.
• For vs0: all configured uplink ports available, and all host IPs resolvable to the gateway via ARP.

VPN prerequisites, since VPN sits on this plumbing

• Guest VM NIC MTU 1356 for VMs sending traffic over Nutanix VPN connections.
• Peer IP for iBGP, or Area ID in IP address format for OSPF.
• Gateway ASN must not match any on premises BGP ASN; pick from 0 to 65000 if you have none.
• Encryption AES128, AES256, 3DES, AES256GCM128. Authentication MD5, SHA1, SHA256, SHA384, SHA512. DH groups 14 (2048-bit MODP), 19 (256-bit random ECP), 20 (384-bit random ECP).

Segregate UVM, management, and replication traffic

Note that management is defined negatively, as whatever is not backplane. That is Nutanix’s own wording. In an unsegmented cluster the CVM has two vNICs: eth0 on the default external virtual switch carrying all external traffic, backplane and management alike, and eth1 on the internal network to the hypervisor. Segmentation is what moves backplane traffic onto eth2, on its own VLAN or its own physical network.

Five services can be isolated to their own virtual network

• Management, the default network, which cannot be moved off CVM eth0
• Backplane
• RDMA, for Stargate to Stargate over the rdma0 vNIC
• Service specific disaster recovery
• Service specific Volumes

Three segmentation types and their AOS floors: logical network segmentation from AOS 5.5, physical network segmentation from AOS 5.11, service specific traffic isolation from AOS 5.11.

Disaster recovery prerequisites, which are the replication answer

• The VLAN and subnet for the segment must be routable.
• n+1 IP addresses per cluster, where n is the node count. The extra one is the virtual IP.
• Segmentation for DR must be enabled at both sites, local and remote, before you configure remote sites at those sites.

Configurations that do not support network segmentation at all

• CVMs with a manually created eth2 interface.
• CVMs whose eth2 has a manually assigned IP address.
• CVM interfaces connected to port groups backed by NSX NVDS switches.

AOS creates eth2 on every CVM during upgrade whether or not you use it, and you must never configure it by hand. Manual multi homed CVM interfaces are deprecated from AOS 5.15, per KB 9479 and Field Advisory 78.

Troubleshooting hook

“Failed to restart one or more services after Backplane was enabled” means the segmentation task completed but one or more services did not restart in time. SSH to a CVM, run cluster start, and confirm every service reads UP.

Multicluster virtual switches

Everything above describes the single cluster virtual switch created in Prism Element. FVN 6.0 adds a second, different construct created in Prism Central. Objective 5.4 does not qualify which kind, so both belong here. Minimum versions are exactly the tested stack: Prism Central pc.7.3, Network Controller 6.0.0, AOS 7.3, AHV 10.3.

What can attach, and a UI gotcha

Only a Network Controller based VLAN Subnet can be mapped to a multicluster virtual switch. Not a VLAN Basic Subnet, not an individual Overlay subnet. Associating a VPC is v4 API only, with no UI path.

Limitations, and two are design constraints rather than footnotes

• Cannot attach VLAN Basic Subnets or individual Overlay subnets.
• Cannot migrate an existing single cluster virtual switch to a multicluster one. There is no conversion path.
• Cannot migrate VMs using cross cluster live migration (CCLM).
• Cannot protect VMs using Nutanix Disaster Recovery recovery plans.

Creating one

Prism Central > Infrastructure > Network & Security > Virtual Switches > Create Virtual Switch. General tab takes Name, optional Description, and Physical NIC MTU (bytes), which shows a default of 1500 that you must delete before typing. Range 1280 to 9000, and Nutanix recommends a value higher than 1500. Uplink Ports tab takes a Bond Type, then +Add Uplink Ports: expand each cluster, tick the clusters to uplink, click Select Uplink Ports, expand each selected cluster, tick the ports on each host, Save.

Throughput figures assume 2 x 10 Gb adapters and the guide says they are not hard limits. Default LACP settings for Active-Active: Speed Fast (1s), Mode Active fallback-active-backup, Priority Default and not configurable.

Mapping the traffic, which is API only

After creating the switch you must map the secondary IP addresses of the hosts of every cluster in it, and optionally map its UUID to a VPC. Both are v4 API only. You need the secondary IPs with prefix, their gateway IPs, a valid VLAN ID configured in the underlay and connected to the relevant upstream router, and the switch UUID (from its details page, or extId under data from the Virtual Switch API GET).

• "ownerType": "PC" is what marks a virtual switch as multicluster. Any virtual switch created in Prism Central specifies ownerType as PC.
• Per host: extId, internalBridgeName (for example br1), hostNics (for example eth1, eth3), ipAddress.
• Per cluster: extId, hosts, gatewayIpAddress, vlanIdentifier.
• vlanIdentifier is the VLAN ID of the cluster on which you deploy the switch. It is NOT the VLAN ID that the Controller VMs or hosts use.

Viewing, updating, deleting

List attributes: Name, Scope, Cluster, Bond Type, MTU. Scope is only visible if you build a custom view and add it, so the name suffix is the quicker tell. Filters: Name, Scope, MTU range, Bond Type. Details page has only two tabs, Summary and Alerts. The Summary tab carries a Properties widget, an Associated Subnets widget (default subnet type VLAN), and an Uplink Ports widget counting 25G and 10G NIC ports. A virtual switch alert typically indicates a problem with the switch or its associated Open vSwitch (OVS) bridges on AHV hosts.

Prism Element health checks that back this objective

from Prism Element Alerts Reference 7.6. These are the checks that fire when the virtual switch and bond plumbing is wrong.

Which roles can and cannot create a VPC

Prism Admin and Super Admin are two different roles

Distinct built-in Prism Central roles that coexist in every release in the folder. The blueprint’s enablement question turns on this.

FNS RBAC roles and their pre configured permissions

The Built-in Roles List describes them as: Flow Admin, full access including categories provisioning; Flow Policy Author, full access except categories provisioning; Flow Viewer, view access.

Three lines to memorize

1. Category create, delete, update: Flow Admin only.
2. Network Entity Group and Network Function: Flow Admin only, and Policy Author cannot even view a Network Entity Group.
3. Sync Entity Sync Policy: Admin and Viewer, not Policy Author.

Custom roles, authorization policies, and limiting an admin to specific VPCs

Custom role: Admin Center > IAM > Roles > Create Role > New Role (unique name, built-in names not allowed; filter by Entity Type or Operation; add operations and their recommended related operations, which Nutanix recommends selecting in full) or From Existing Role (note: the new role does not include system operations from the existing role, and some operations are pre selected). Save, or Save & Create Authorization Policy. Duplicate, Update, and Delete are under Actions. Built-in roles cannot be updated or deleted, only duplicated, viewed, and given an authorization policy.

Authorization policy for FNS: log in as administrator or super admin, Admin Center > IAM > Roles > select a role > Actions > Add Authorization Policy > Choose Role > Define Scope (Full access or Configure access) > Assign Users. FNS roles do not support cluster and category based scoping for All Entities. Granular RBAC works on Address Group and Service Group by Individual Entity or by Owner for self owned.

Limiting to specific VPCs: edit the authorization policy, and on Define Scope set Entity Type to Subnet or VPC, set Filter to Advanced to add Subnet Type or VPC Type with the AND operator, then tick at least one search value.

Documented RBAC limitations

• No bulk operations on policies for a custom role.
• A user with only import permission can import address groups, service groups, and flow policies without access to them.
• No selective policy import or export. It is all policies or none.
• Importing revokes RBAC permissions on the newly imported policies, since existing policies are overridden. Users with Full Access or All Entities are unaffected.
• A user can see policies in the list even without VPC access.
• An AD server config entity cannot be added to a role scoped to self owned entities only.

A VM inside a VPC cannot reach the Internet, or cannot reach the external network

Transit VPC ERP alerts

Two VMs within the same VPC cannot communicate

Traffic type behavior inside a VPC

• Broadcast forwarded to all guest VMs in the same subnet, regardless of host.
• Unicast follows the configured networking policies.
• Unknown unicast is dropped. Not transmitted to any guest VM inside or outside the source host.
• Multicast forwarded only within a subnet, to all VMs in that subnet. No IGMP snooping in VPCs.

Also check: both VMs are actually in the same VPC (a VM sits in exactly one); the Network Controller is healthy, since an interruption can break connectivity on live migration of VMs in overlay or NC backed VLAN subnets; and alert 130388, vNIC learned IP limit reached, which makes FNS policies involving that VM behave unexpectedly.

The BGP neighbor is not receiving expected routes from the VPC

Transit VPC case: the hub’s BGP gateway services only the hub, so every spoke ERP must be in the hub’s ERP list or alert 802007 fires. VPN based BGP has its own set: 801001 route limit, 801003 eBGP session down, 801004 invalid routes rejected.

Network gateway status issues, and whether a gateway VM is unhealthy

Four documented causes of a Down gateway

1. NTP and DNS unreachable, as above.
2. Missing static routes to the NAT network for Prism Central, NTP, DNS and peer gateway IPs, when the VPC has both NAT and No-NAT and No-NAT is the default next hop.
3. Alert 802005: Prism Central cannot ping the gateway, or its REST server is down.
4. Alert 130403: the Flow gateway VM or the flow-gateway-agent service is down. Check whether the flow-gateway HA event has triggered on MCM.

Where to read status

Log collection

From the Prism Central VM console:

nutanix@cvm$ logbay collect -t msp,anc

nutanix@cvm$ logbay collect -t msp,anc -O msp_pod=true,msp_systemd=true,\
  kubectl_cmds=true,persistent=true --duration=-48h0m0s

• msp collects MSP pod and persistent log volume logs. anc collects the support bundle including database dumps and OVN state.
• PC container logs land under /var/log/containers; ANC persistent logs under /var/log/ctrlog.
• Bundle at /home/nutanix/data/logbay/bundles/<filename>.zip, task detail at /home/nutanix/data/logbay/taskdata/<taskID>/collection_result.txt.

Deletion order: remove all VPN or VTEP connections, BGP sessions, and subnet extensions on a gateway before deleting the gateway itself. Update rules: the Update Gateway window has the same fields as Create, but some parameters are greyed out. To change one, build a new gateway and delete the old one. Same pattern as BGP sessions.

VPN architecture,

A VPN endpoint is three things: a local VPN gateway, a remote VPN gateway, and a VPN connection. You configure a full endpoint at each site.

• The local gateway is a VM running IKEv2 and IPSec, plus BGP and OSPF for routing.
• The remote gateway is not a VM. It is a pointer, a database entry describing the peer. Its key content is the source IP of the remote endpoint, and the local gateway accepts IKEv2 packets only from that source IP. That is a security control, and the first thing to check when a tunnel will not come up after an IP change.
• The VPN connection is the IPSec tunnel. One end is initiator, the other acceptor.

Gateway types: On premises Nutanix VPN Gateway, or On premises Third Party Gateway configured per that vendor’s documentation. Routing: eBGP between remote sites, optionally static, and iBGP or OSPF within a site between the Nutanix VPN appliance and the edge router.

Verify that the subnet extension is active and healthy

Connectivity > Subnet Extensions. Filters give Connection Status and Interface Status. The details page has Summary, Address Table, and Throughput tabs, with an IP Address Pools widget showing the pool range as a pie chart.

PBR based tromboning in an L2 extended subnet

Traffic Mirroring: differentiate from PBR and service chaining, and configure a session

Reference #16. Traffic mirroring replicates traffic from the interfaces of the AHV hosts to the vNIC of guest VMs. Stated uses: security analysis, visibility into traffic flowing through the source ports, packet troubleshooting, and compliance.

Procedure

Prism Central > Application Switcher > Infrastructure > Network & Security > Network Services > Traffic Mirroring tab > Create Mirror Session. General tab takes Name, optional Description, Cluster, and the virtual switch for mirrored traffic. Source & Destination tab takes Host Ports (expand the host, tick an Ethernet port under NICs or a bonded port under Bonds) or Virtual Machines (choose the source VM and the MAC address of its Traffic Mirroring NIC), then the direction. Summary tab offers Create Session, which leaves the session disabled, or Create and Enable Session.

Prerequisite. Configure the traffic mirror type on the destination VM’s vNIC before creating the session. One Traffic Mirroring NIC on the source VM per session using Virtual Machines as source type, and one on the destination VM per session.

Non default virtual switch, only when all three hold: the session captures VM traffic, the source VM’s traffic is mirrored to a destination VM on a different host, and the destination host is reachable only through that switch. Host IPs on it must be reachable in cluster, unique in the cluster, in the same subnet, and in a subnet other than the CVM br0 management subnet.

RBAC. Two built-in roles can configure traffic mirroring: Network Infra Admin and Prism Admin. Not VPC Admin. Custom role permissions: Create, Delete, Update, View, and View Stats for Traffic Mirror, plus View Cluster, View Cluster Networking Capabilities, View Host, View Uplink Bond, and View VM.

Without Prism Central registration, configure the session on the AHV host with aCLI.

Diagnose BGP state using session logs · Determine who made a change and when

BGP reading order: Summary Properties widget for Session Status, eBGP Status, Route Priority, and both gateways’ eBGP ASNs (1 to 65534). Then the Routes tab, which opens on Advertised and also carries Received, each with next hop. Empty Advertised points at the VPC’s ERPs or the session’s Custom advertise filter; empty Received points at a missing No-NAT external subnet. Then BGP Logs. Then the 806xxx alerts.

Read the two status fields as two different things. Session Status on that widget is the overall session status, Up or Down. eBGP Status is the eBGP protocol state, Established or Active. Established corresponds to Up; Active means the network controller is still attempting to establish the session. So Active on a session that was previously up points at the peer or the underlay, not at the session configuration. See objective 1.3 for the full field breakdown and the filter pane inconsistency.

Audit trail

• Prism Central generates audit logs for all Flow networking activity, viewable in Prism Central and forwarded to syslog.
• On the PC VM: /home/nutanix/data/logs/consolidated_audit.log for VM operation audits, athena.* for authentication and IAM activity.
• Timestamps are UTC (ISO 8601) from Prism 5.18. OS logs are not converted, so set server local time to UTC.
• Networking relevant audit events include VsCreateAudit, VsUpdateAudit, VsDeleteAudit, MigrateBridgeToVirtualSwitchAudit, IPAssignmentToVMAudit, VmNicAddAudit, VmNicUpdateAudit.
• Audit module carries UI driven changes. API Audit carries REST API changes. If a policy changed and nobody in the UI did it, look at API Audit.

Analyze IPFIX exports · Interpret alerts and take corrective actions

IPFIX. VLAN Subnets and Overlay subnets both support the IPFIX Exporter, and AHV hosts export IPFIX data for all traffic including overlay subnets. Alert 150011 reports an exporter host update failure with the hypervisor address and error. Alert 150017 reports VM traffic impacted by a down link on a host NIC in a NIC profile, resolved by validating uplink to host NIC connectivity and running a complete NCC health check.

Sitting immediately after 150011: 150012, OVN Connection Unhealthy, “Hypervisor node is disconnected from the Network Controller”, naming the node UUID and host IP. If IPFIX export fails on a host, check whether that host is also raising 150012, because a host cut off from the Network Controller has a larger problem than its exporter.

Troubleshooting the IPFIX exporter

This gap is easy to frame wrongly as “record structure and exported fields”. That was the wrong frame. The knowledge bullet asks you to analyze IPFIX exports to identify network connectivity issues, which is a troubleshooting question, not one about packet anatomy.

The 150011 row is the only part verifiable against the product documentation, and it checks out. The rest is not.

Stated remediations, unverified:

1. systemctl status conntrack_stats_collector, then service conntrack_stats_collector restart if inactive or erroring. The service name is verified from the ports CSV, where it publishes to Prism Central over TCP 9446. The restart procedure is not.
2. manage_ovs enable_bridge_chain plus an Acropolis restart, if bridge chain was disabled while IPFIX was in use. manage_ovs is real and appears in the AHV Administration Guide, but “bridge chain” and enable_bridge_chain appear nowhere in the product documentation. The attached warning is worth repeating anyway: check cluster health first, because restarting Acropolis can cause nontrivial downtime if VM operations are in flight.
3. logbay -o json collect -t dpm_collector for support. That tag is not in the NCC Guide’s documented tag table, consistent with the table being partial. Confirm with logbay list_tags before relying on it.

Two claims in that material that are wrong for the tested version

Worked alert: 806201, load balancer session targets unhealthy

The third row is the cross domain one worth remembering: an FNS policy can break FVN load balancer health checks. The companion alert 806202 covers unhealthy network function vNIC pairs, from service VMs being down, the datapath engine down or blocking health checks, or the service VMs or network function vNICs having been deleted.

Check the Network Controller’s health

From the CLI on a Prism Element cluster, acli atlas_config.get reports enable_atlas_networking: True or False, alongside anc_domain_name_server_list, minimum_ahv_version, and the OVN certificate paths.

What can and cannot be done when the Network Controller is unhealthy

Blocked by design, unrelated to health

• Cannot disable the Network Controller while external subnets or VPCs are in use.
• Cannot unregister the Prism Element cluster hosting the FVN enabled Prism Central.
• Without the Prism Admin role, enabling or using FVN fails with User Denied Access.
• NC is deployed but not enabled with a compatible PC package and an incompatible AHV package, and is not enabled by default on a newly registered PE cluster with incompatible AHV.
• NC upgrade fails after the pre check if any FVN enabled cluster runs an incompatible AHV version.

To exclude a cluster: <atlas> config.add_to_excluded_clusters <cluster uuid>. The prompt states how many external subnets will lose connectivity. Reverse with config.remove_from_excluded_clusters.

Interpret Flow Network Security control plane alerts

Also: alert 130389, an Advanced Networking subnet not recovered from a Prism Central recovery point, where the resolution is to delete the VM vNICs associated with that subnet.

Prism Element alerts and health checks

Reference #62 resolves to the Prism Element Alerts Reference 7.6, the guide Web Console 7.3 topic Prism Central Infrastructure Guide 7.3, Alerts and Events pointed at in a single sentence.

Health dashboard and Manage Checks

Select Health from the pull down list on the left of the main menu. Three columns: the left lists tabs per entity type (VMs, hosts, disks, storage pools, storage containers, cluster services, and when configured protection domains and remote sites), each showing the entity total and the count in each health state; the middle shows detail for the selection; the right shows the rest.

Actions > Manage Checks. Left column lists the health checks. Middle column describes what the check does plus its run schedule and history. Right column describes cause, resolution, and impact. Run Check runs one on demand. Turn Check Off and Turn Check On toggle one.

Network alerts in the Prism Element reference that are not in the Prism Central reference

Flow alerts that appear in both references, so the ID is safe either way: 130201, 130202, 130388, 150011, 200601, 200602, 200606, 801106, 803003, 803005.

The Prism Element reference uses the same six fields as the Prism Central reference: Name, Description, Alert message, Cause, Impact, Resolution. The same alert can appear under more than one entity section. 200613 appears under both Controller VM and Network, 130201 under Node and Network, 130202 under Cluster and Network.

Determine if desired traffic is being prevented by a security policy

1. Mode. Enforce blocks what is not allowed. Monitor blocks nothing.
2. Priority. The traffic may be matching a different policy. A monitor mode match allows and stops all further processing.
3. Discovered traffic. In enforce mode the engine shows what it denied, on the policy details page under Inbounds and Outbounds.
4. IPv6. Rules are IPv4 only and all IPv6 is blocked by default. Left blocked, it stays blocked even in monitoring mode. This is the usual “monitor mode is still dropping traffic” answer.
5. Layer 2. Application policies do not block ARP or layer 2 broadcast. Isolation policies do, dropping ingress and egress broadcast, unknown unicast, and multicast at the destination group.
6. Intra tier. Default is allow all inside a secured entity. A configured rule limits VM to VM traffic to a specific service group, port, and protocol.
7. Scope. A VLAN only policy does not touch VPC entities and vice versa.

Verify VM membership in a policy component

• Common VM count on isolation entity circles, and in the entity side window, shows exactly how many VMs the policy protects.
• Entity groups protect the intersection of their categories.
• vNIC scope: subnet category alone hits every vNIC in that subnet category; an entity group of VM category plus subnet category narrows to one vNIC; adding a VPC category narrows further.
• AppType exclusion: VMs with an AppType category cannot be categorised by ID Based Security.
• Alert 130388: the vNIC hit its learned IP ceiling and will learn no more.

Assess policy hit logs · Identify priority conflicts

Five conditions that suppress or alter hit logs

1. Not generated when both source and destination are in an inbound or outbound category.
2. For isolation policies, generated only in monitor mode.
3. For multi isolation policies, the direction shows as outbound instead of source and destination.
4. Not synchronized in multi Prism Central DR, along with visualization.
5. If Flow is in default mode after a failed mode change (alert 200606), AHV logs nothing at all.

The three priority conflicts

Packet loss with service insertion · Routes present but north south broken (MTU)

Most service insertion packet loss is a scope violation. Check the envelope first: application policies only, Network Controller managed VLAN policies only, VLAN environments only, AHV clusters only, IPv4 unicast only, and no monitor mode. Version floors: AOS/PE 7.3, PC 7.3, AHV 10.3, ANC 6.0.0, FNS 5.2.0, NCC 5.2.0. Then alert 806202 for unhealthy network function vNIC pairs. Note also that FNS Next-Gen does not support Network Function Chain in a VPC, and does not support asymmetric routing in either environment.

Pipe FNS hit logs to an external syslog server

Two halves, both required. Per policy: Define Policy > Advanced Configuration > Policy Hit Logs > Enabled. In Prism Central: configure the remote syslog server with port and protocol, then select modules and severity on the Data Sources tab. Hit logs are redirected only once the syslog server is configured with the hit log module.

Conntrack table status through NCC health checks

The five NCC status types

Each plugin completes independently with one of these, and the result may carry a link to a support portal KB article.

ERR is the one people misread: the check broke, not the cluster. And this explains why alert A200613, the 40 GB PE CVM memory recommendation, is severity INFO: an expected value that cannot be graded, carrying a recommendation. Textbook INFO.

Modules, plugins, and the intrusive check default

A plugin is a component specific code block inside a module, commonly called a check. A module is a logical group of plugins or of modules. NCC is cluster resident, runs hundreds of checks, and depending on the issue raises an alert or automatically creates a Nutanix Support case. It runs as long as the individual nodes are up, regardless of cluster state.

nutanix@cvm$ ncc ncc-flags module sub-module [...] plugin plugin-flags
# default output: /home/nutanix/data/logs/ncc-output-latest.log

Typing ncc with no arguments lists the modules. The Type column shows M (module) or P (plugin), and the Impact tag marks a plugin intrusive or non intrusive.

Modules in the guide’s example output: cassandra_tools, fix_failures, hardware_info, health_checks, help_opts, log_collector, performance_checks, pulsehd_collectors.

How to run NCC and how to find a check name

It is easy to conclude that the Flow documentation carries no NCC content. Four Prism Web Console Guide 7.3 topics cover it.

Web console: Health dashboard > Actions > Run NCC Checks, scoped to All checks, Only Failed and Warning Checks, or Specific Checks (type the name, the field auto populates, the Added Checks box lists the selection). There is a Send the cluster check report in the email checkbox, which needs alert email notification already configured. The Tasks dashboard shows succeeded or aborted. Event triggered checks are marked passed by default.

Command line:

nutanix@cvm$ ncc health_checks run_all      # run everything
nutanix@cvm$ ncc health_checks              # list the available check categories
nutanix@cvm$ ncc health_checks hypervisor_checks   # drill into one category

hypervisor_checks is the guide’s own example and the only category name stated anywhere in the documentation. Anything other than INFO or PASS must be resolved before proceeding with an upgrade.

From the UI: Actions > Manage Checks, select a check, then click the link to the Knowledge Base article for that check. That plus ncc health_checks are the two documented ways to find a check name on a live cluster.

Logbay, the log collection half of this objective

Discovery, the same pattern as NCC: logbay list_tags lists every available tag on a live cluster. The documented tag table is partial (it omits msp and anc, which the FVN guide uses), which is exactly why the command exists.

Documented tags: cvm_config, cvm_logs, cvm_kernel, alerts, ahv_config / esx_config / hyperv_config, ahv_logs / esx_logs / hyperv_logs, and vpn_logs, which is directly relevant to VPN gateway troubleshooting in objective 3.1.

Retrieving the bundle: Tasks dashboard, find the log bundle task, click the Succeeded link in the Status column. The last two runs are available, and a browser pop up blocker will stop the download.

Scheduling NCC

Disabled by default. Health dashboard > Actions > Set NCC Frequency: Every 4 hours, Every Day with a Start Time, Every Week with On days and a Start Time, or Remove Schedule (shown only once a schedule exists). Results email to whoever is configured for alert emails, and the schedule survives AOS and NCC upgrades. Emailed NCC results do not automatically create support cases, unlike proactive NCC, which can.

What the documentation does have, adjacent to this

• TCP port 9446 from AHV hosts to the Prism Central VMs carries connection tracking data, which Prism Central uses to show network flows. This is the one FNS port number the documentation actually gives.
• The flow_data Kafka container is created automatically on the cluster hosting Prism Central when microsegmentation is enabled, and stores data essential for Flow visualization. Do not delete it.
• Alert 200615, high Cadmus service flows, where flows means traffic reaching a secured entity. Additional flows past the limit are not shown in the UI. Fix by reducing policies or flows per policy, or upgrading the Microsegmentation Memory Allocation (S/M/L/XL).

Interpret FNS audit logs to diagnose an issue

Audit logs cover changes to security policy configuration and VM to category mappings: when a policy was changed or applied, and who changed it. Enabled by default, viewable in Prism Central and sent to syslog. The Audit module carries Prism Central changes; the API Audit module carries changes made directly through the REST API endpoint.

The diagnostic pairing: audit logs tell you what changed and who changed it; policy hit logs tell you what the traffic did as a result. Note that hit logs and visualization are not synchronized in multi Prism Central DR, so on a remote Prism Central a synced policy has an audit trail but no hit logs.

Complementary PC VM logs under /home/nutanix/data/logs: alert_manager.*, aplos.out and aplos_engine.out for the v3 API gateway and engine, catalina.out, genesis.out, and cron_time_check.log which checks time difference across PC VMs on a multi VM instance every minute.

Verify AD configuration · Enable and manage referenced AD groups

Configuration to check, at Prism Central Settings > ID Based Security

• Directory URL is the LDAP address including the port number.
• Service Account Username in user_name@domain.com format. Never the Domain Admin account.
• Every domain controller added manually, by IP or host name. DNS on Prism Central is required for host names to work.
• Prerequisites: microsegmentation enabled; WMI access from PC to all DCs through both the network and AD firewalls; AD functional level Windows Server 2008 R2 minimum; Security Groups only; NTP on both AD and Prism Central; AOS 5.17 and PC 5.17 minimum.
• Service account permissions on every DC: Distributed COM Users and Event Log Readers; Local Access and Remote Access on WMI in DCOM Config; Enable Account and Remote Enable on Root\CIMV2, applied to this namespace and subnamespaces; then net stop winmgmt and net start winmgmt.

Validate that group memberships have been applied to a policy

1. Policy type. Must be Application Secure Entities with Secure VDI Groups only selected. Without it, ADGroup categories are not driving it.
2. Scope. FNS does not support VDI policy for a VPC scope. You can attach a VDI VM to a VPC or overlay network, but a VDI policy with VPC scope does not protect the VDI VMs. A VDI policy that silently does nothing is very often this.
3. Default policy. ADGroup:Default applies rules before anyone logs on. If VMs are protected before logon but not after, or the reverse, check this.
4. No visualization. VDI policies do not support it, so verification comes from the configuration, the category on the VM, and the policy hit logs.
5. The union rule. A policy that looks too permissive may be correct: another ADGroup the user belongs to is contributing rules.

Determine whether tenant or a transit VPC is required

A VPC (user or guest VPC, the blueprint’s “tenant”) is the default type: an isolated IP address space of one or more subnets joined by a single virtual router. A VM sits in exactly one VPC and cannot be in a VPC and a VLAN at once, or in two VPCs at once.

A transit VPC is a hub in a hub and spoke topology. Minimum Prism Central pc.2024.1. Choose it when you need to:

• Scale north south routing for many VPCs without advertising each one to the infrastructure routers.
• Route between user VPCs on private IPv4 using ERP routes, keeping that traffic off the physical fabric.
• Host shared services for several VPCs on overlay subnets under the hub.
• Separate a provider layer from a tenant layer, each controlling its own routing and security policy.
• Control cross tenant routing and policy without touching physical infrastructure.

Recognize the purpose or usage of ERP in the VPC

An Externally Routable Prefix is a range on the VPC that the underlay can reach without SNAT. A VPC with external connectivity can carry several. ERPs do three jobs:

• Declare which VPC internal prefixes the underlay may reach directly.
• Feed BGP. The gateway advertises the ERPs with next hop set to the VPC router IP in the No-NAT external subnet, so peers learn the path through the VPC router.
• Supply floating IP addresses to virtual routers, gateways, and the external network.

ERPs must be unique and non overlapping unless overlapping ERPs is explicitly enabled. Entry point on the Create VPC page is Externally Routable IP Addresses, and it is optional. Without an ERP, BGP session creation fails, and a BGP session ignores received routes when the VPC has no routable No-NAT external subnet.

Identify the VPC Gateway nodes

The gateway node is an AHV host the Network Controller picks to run the NAT or No-NAT gateway service. The documentation calls it the redirect chassis.

• Without scale out the Network Controller randomly selects one AHV host from the external VLAN subnet. Load balancing and routing services deploy there.
• Each VPC gets its own redirect chassis, chosen independently, so VPCs do not share the role.
• The Network Controller never uses the host holding the Acropolis Leader.
• Every redirect chassis host gets an IP attached by the Network Controller, taken from the external subnet pool. For No-NAT it may be an RFC1918 address, selected manually or dynamically.

Associate routed and private CIDRs

Private CIDRs are the overlay subnet prefixes inside the VPC, set per subnet with Network IP Prefix and Gateway IP under IP Address Management (mandatory for overlay subnets). Addresses must be unique inside one VPC but may overlap across VPCs and with the physical network. They reach outside through SNAT, and outside endpoints cannot initiate inbound to them.

Routed CIDRs are the ERPs, reachable without translation, used with a No-NAT external subnet. The underlay needs routes pointing at the VPC router IP, or a BGP session that advertises them.

Association work: attach the external subnet under External Connectivity via Associate External Subnet; add ERPs and keep them non overlapping; set Destination Prefixes on the association (selection is by longest prefix match); configure return routes on the physical router.

Objective 1.2: Create and Manage VPC External Networks

Objective 1.3: Configure Connectivity Options

Section 2: Configure Flow Network Security

Objective 2.1: Analyze and Document Application Flows

Objective 2.2: Create and Configure Security Policies

Objective 2.3: Manage Policy Lifecycle and Modes