Nutanix NCP-NS 7.5 Study Guide, Part 2: Troubleshooting Flow Virtual Networking and Flow Network Security

A VM inside a VPC cannot reach the Internet, or cannot reach the external network

Transit VPC ERP alerts

Two VMs within the same VPC cannot communicate

Traffic type behavior inside a VPC

• Broadcast forwarded to all guest VMs in the same subnet, regardless of host.
• Unicast follows the configured networking policies.
• Unknown unicast is dropped. Not transmitted to any guest VM inside or outside the source host.
• Multicast forwarded only within a subnet, to all VMs in that subnet. No IGMP snooping in VPCs.

Also check: both VMs are actually in the same VPC (a VM sits in exactly one); the Network Controller is healthy, since an interruption can break connectivity on live migration of VMs in overlay or NC backed VLAN subnets; and alert 130388, vNIC learned IP limit reached, which makes FNS policies involving that VM behave unexpectedly.

The BGP neighbor is not receiving expected routes from the VPC

Transit VPC case: the hub’s BGP gateway services only the hub, so every spoke ERP must be in the hub’s ERP list or alert 802007 fires. VPN based BGP has its own set: 801001 route limit, 801003 eBGP session down, 801004 invalid routes rejected.

Network gateway status issues, and whether a gateway VM is unhealthy

Four documented causes of a Down gateway

1. NTP and DNS unreachable, as above.
2. Missing static routes to the NAT network for Prism Central, NTP, DNS and peer gateway IPs, when the VPC has both NAT and No-NAT and No-NAT is the default next hop.
3. Alert 802005: Prism Central cannot ping the gateway, or its REST server is down.
4. Alert 130403: the Flow gateway VM or the flow-gateway-agent service is down. Check whether the flow-gateway HA event has triggered on MCM.

Where to read status

Log collection

From the Prism Central VM console:

nutanix@cvm$ logbay collect -t msp,anc

nutanix@cvm$ logbay collect -t msp,anc -O msp_pod=true,msp_systemd=true,\
  kubectl_cmds=true,persistent=true --duration=-48h0m0s

• msp collects MSP pod and persistent log volume logs. anc collects the support bundle including database dumps and OVN state.
• PC container logs land under /var/log/containers; ANC persistent logs under /var/log/ctrlog.
• Bundle at /home/nutanix/data/logbay/bundles/<filename>.zip, task detail at /home/nutanix/data/logbay/taskdata/<taskID>/collection_result.txt.

Deletion order: remove all VPN or VTEP connections, BGP sessions, and subnet extensions on a gateway before deleting the gateway itself. Update rules: the Update Gateway window has the same fields as Create, but some parameters are greyed out. To change one, build a new gateway and delete the old one. Same pattern as BGP sessions.

VPN architecture,

A VPN endpoint is three things: a local VPN gateway, a remote VPN gateway, and a VPN connection. You configure a full endpoint at each site.

• The local gateway is a VM running IKEv2 and IPSec, plus BGP and OSPF for routing.
• The remote gateway is not a VM. It is a pointer, a database entry describing the peer. Its key content is the source IP of the remote endpoint, and the local gateway accepts IKEv2 packets only from that source IP. That is a security control, and the first thing to check when a tunnel will not come up after an IP change.
• The VPN connection is the IPSec tunnel. One end is initiator, the other acceptor.

Gateway types: On premises Nutanix VPN Gateway, or On premises Third Party Gateway configured per that vendor’s documentation. Routing: eBGP between remote sites, optionally static, and iBGP or OSPF within a site between the Nutanix VPN appliance and the edge router.

Verify that the subnet extension is active and healthy

Connectivity > Subnet Extensions. Filters give Connection Status and Interface Status. The details page has Summary, Address Table, and Throughput tabs, with an IP Address Pools widget showing the pool range as a pie chart.

PBR based tromboning in an L2 extended subnet

Traffic Mirroring: differentiate from PBR and service chaining, and configure a session

Reference #16. Traffic mirroring replicates traffic from the interfaces of the AHV hosts to the vNIC of guest VMs. Stated uses: security analysis, visibility into traffic flowing through the source ports, packet troubleshooting, and compliance.

Procedure

Prism Central > Application Switcher > Infrastructure > Network & Security > Network Services > Traffic Mirroring tab > Create Mirror Session. General tab takes Name, optional Description, Cluster, and the virtual switch for mirrored traffic. Source & Destination tab takes Host Ports (expand the host, tick an Ethernet port under NICs or a bonded port under Bonds) or Virtual Machines (choose the source VM and the MAC address of its Traffic Mirroring NIC), then the direction. Summary tab offers Create Session, which leaves the session disabled, or Create and Enable Session.

Prerequisite. Configure the traffic mirror type on the destination VM’s vNIC before creating the session. One Traffic Mirroring NIC on the source VM per session using Virtual Machines as source type, and one on the destination VM per session.

Non default virtual switch, only when all three hold: the session captures VM traffic, the source VM’s traffic is mirrored to a destination VM on a different host, and the destination host is reachable only through that switch. Host IPs on it must be reachable in cluster, unique in the cluster, in the same subnet, and in a subnet other than the CVM br0 management subnet.

RBAC. Two built-in roles can configure traffic mirroring: Network Infra Admin and Prism Admin. Not VPC Admin. Custom role permissions: Create, Delete, Update, View, and View Stats for Traffic Mirror, plus View Cluster, View Cluster Networking Capabilities, View Host, View Uplink Bond, and View VM.

Without Prism Central registration, configure the session on the AHV host with aCLI.

Diagnose BGP state using session logs · Determine who made a change and when

BGP reading order: Summary Properties widget for Session Status, eBGP Status, Route Priority, and both gateways’ eBGP ASNs (1 to 65534). Then the Routes tab, which opens on Advertised and also carries Received, each with next hop. Empty Advertised points at the VPC’s ERPs or the session’s Custom advertise filter; empty Received points at a missing No-NAT external subnet. Then BGP Logs. Then the 806xxx alerts.

Read the two status fields as two different things. Session Status on that widget is the overall session status, Up or Down. eBGP Status is the eBGP protocol state, Established or Active. Established corresponds to Up; Active means the network controller is still attempting to establish the session. So Active on a session that was previously up points at the peer or the underlay, not at the session configuration. See objective 1.3 for the full field breakdown and the filter pane inconsistency.

Audit trail

• Prism Central generates audit logs for all Flow networking activity, viewable in Prism Central and forwarded to syslog.
• On the PC VM: /home/nutanix/data/logs/consolidated_audit.log for VM operation audits, athena.* for authentication and IAM activity.
• Timestamps are UTC (ISO 8601) from Prism 5.18. OS logs are not converted, so set server local time to UTC.
• Networking relevant audit events include VsCreateAudit, VsUpdateAudit, VsDeleteAudit, MigrateBridgeToVirtualSwitchAudit, IPAssignmentToVMAudit, VmNicAddAudit, VmNicUpdateAudit.
• Audit module carries UI driven changes. API Audit carries REST API changes. If a policy changed and nobody in the UI did it, look at API Audit.

Analyze IPFIX exports · Interpret alerts and take corrective actions

IPFIX. VLAN Subnets and Overlay subnets both support the IPFIX Exporter, and AHV hosts export IPFIX data for all traffic including overlay subnets. Alert 150011 reports an exporter host update failure with the hypervisor address and error. Alert 150017 reports VM traffic impacted by a down link on a host NIC in a NIC profile, resolved by validating uplink to host NIC connectivity and running a complete NCC health check.

Sitting immediately after 150011: 150012, OVN Connection Unhealthy, “Hypervisor node is disconnected from the Network Controller”, naming the node UUID and host IP. If IPFIX export fails on a host, check whether that host is also raising 150012, because a host cut off from the Network Controller has a larger problem than its exporter.

Troubleshooting the IPFIX exporter

This gap is easy to frame wrongly as “record structure and exported fields”. That was the wrong frame. The knowledge bullet asks you to analyze IPFIX exports to identify network connectivity issues, which is a troubleshooting question, not one about packet anatomy.

The 150011 row is the only part verifiable against the product documentation, and it checks out. The rest is not.

Stated remediations, unverified:

1. systemctl status conntrack_stats_collector, then service conntrack_stats_collector restart if inactive or erroring. The service name is verified from the ports CSV, where it publishes to Prism Central over TCP 9446. The restart procedure is not.
2. manage_ovs enable_bridge_chain plus an Acropolis restart, if bridge chain was disabled while IPFIX was in use. manage_ovs is real and appears in the AHV Administration Guide, but “bridge chain” and enable_bridge_chain appear nowhere in the product documentation. The attached warning is worth repeating anyway: check cluster health first, because restarting Acropolis can cause nontrivial downtime if VM operations are in flight.
3. logbay -o json collect -t dpm_collector for support. That tag is not in the NCC Guide’s documented tag table, consistent with the table being partial. Confirm with logbay list_tags before relying on it.

Two claims in that material that are wrong for the tested version

Worked alert: 806201, load balancer session targets unhealthy

The third row is the cross domain one worth remembering: an FNS policy can break FVN load balancer health checks. The companion alert 806202 covers unhealthy network function vNIC pairs, from service VMs being down, the datapath engine down or blocking health checks, or the service VMs or network function vNICs having been deleted.

Check the Network Controller’s health

From the CLI on a Prism Element cluster, acli atlas_config.get reports enable_atlas_networking: True or False, alongside anc_domain_name_server_list, minimum_ahv_version, and the OVN certificate paths.

What can and cannot be done when the Network Controller is unhealthy

Blocked by design, unrelated to health

• Cannot disable the Network Controller while external subnets or VPCs are in use.
• Cannot unregister the Prism Element cluster hosting the FVN enabled Prism Central.
• Without the Prism Admin role, enabling or using FVN fails with User Denied Access.
• NC is deployed but not enabled with a compatible PC package and an incompatible AHV package, and is not enabled by default on a newly registered PE cluster with incompatible AHV.
• NC upgrade fails after the pre check if any FVN enabled cluster runs an incompatible AHV version.

To exclude a cluster: <atlas> config.add_to_excluded_clusters <cluster uuid>. The prompt states how many external subnets will lose connectivity. Reverse with config.remove_from_excluded_clusters.

Interpret Flow Network Security control plane alerts

Also: alert 130389, an Advanced Networking subnet not recovered from a Prism Central recovery point, where the resolution is to delete the VM vNICs associated with that subnet.

Prism Element alerts and health checks

Reference #62 resolves to the Prism Element Alerts Reference 7.6, the guide Web Console 7.3 topic Prism Central Infrastructure Guide 7.3, Alerts and Events pointed at in a single sentence.

Health dashboard and Manage Checks

Select Health from the pull down list on the left of the main menu. Three columns: the left lists tabs per entity type (VMs, hosts, disks, storage pools, storage containers, cluster services, and when configured protection domains and remote sites), each showing the entity total and the count in each health state; the middle shows detail for the selection; the right shows the rest.

Actions > Manage Checks. Left column lists the health checks. Middle column describes what the check does plus its run schedule and history. Right column describes cause, resolution, and impact. Run Check runs one on demand. Turn Check Off and Turn Check On toggle one.

Network alerts in the Prism Element reference that are not in the Prism Central reference

Flow alerts that appear in both references, so the ID is safe either way: 130201, 130202, 130388, 150011, 200601, 200602, 200606, 801106, 803003, 803005.

The Prism Element reference uses the same six fields as the Prism Central reference: Name, Description, Alert message, Cause, Impact, Resolution. The same alert can appear under more than one entity section. 200613 appears under both Controller VM and Network, 130201 under Node and Network, 130202 under Cluster and Network.

Determine if desired traffic is being prevented by a security policy

1. Mode. Enforce blocks what is not allowed. Monitor blocks nothing.
2. Priority. The traffic may be matching a different policy. A monitor mode match allows and stops all further processing.
3. Discovered traffic. In enforce mode the engine shows what it denied, on the policy details page under Inbounds and Outbounds.
4. IPv6. Rules are IPv4 only and all IPv6 is blocked by default. Left blocked, it stays blocked even in monitoring mode. This is the usual “monitor mode is still dropping traffic” answer.
5. Layer 2. Application policies do not block ARP or layer 2 broadcast. Isolation policies do, dropping ingress and egress broadcast, unknown unicast, and multicast at the destination group.
6. Intra tier. Default is allow all inside a secured entity. A configured rule limits VM to VM traffic to a specific service group, port, and protocol.
7. Scope. A VLAN only policy does not touch VPC entities and vice versa.

Verify VM membership in a policy component

• Common VM count on isolation entity circles, and in the entity side window, shows exactly how many VMs the policy protects.
• Entity groups protect the intersection of their categories.
• vNIC scope: subnet category alone hits every vNIC in that subnet category; an entity group of VM category plus subnet category narrows to one vNIC; adding a VPC category narrows further.
• AppType exclusion: VMs with an AppType category cannot be categorised by ID Based Security.
• Alert 130388: the vNIC hit its learned IP ceiling and will learn no more.

Assess policy hit logs · Identify priority conflicts

Five conditions that suppress or alter hit logs

1. Not generated when both source and destination are in an inbound or outbound category.
2. For isolation policies, generated only in monitor mode.
3. For multi isolation policies, the direction shows as outbound instead of source and destination.
4. Not synchronized in multi Prism Central DR, along with visualization.
5. If Flow is in default mode after a failed mode change (alert 200606), AHV logs nothing at all.

The three priority conflicts

Packet loss with service insertion · Routes present but north south broken (MTU)

Most service insertion packet loss is a scope violation. Check the envelope first: application policies only, Network Controller managed VLAN policies only, VLAN environments only, AHV clusters only, IPv4 unicast only, and no monitor mode. Version floors: AOS/PE 7.3, PC 7.3, AHV 10.3, ANC 6.0.0, FNS 5.2.0, NCC 5.2.0. Then alert 806202 for unhealthy network function vNIC pairs. Note also that FNS Next-Gen does not support Network Function Chain in a VPC, and does not support asymmetric routing in either environment.

Pipe FNS hit logs to an external syslog server

Two halves, both required. Per policy: Define Policy > Advanced Configuration > Policy Hit Logs > Enabled. In Prism Central: configure the remote syslog server with port and protocol, then select modules and severity on the Data Sources tab. Hit logs are redirected only once the syslog server is configured with the hit log module.

Conntrack table status through NCC health checks

The five NCC status types

Each plugin completes independently with one of these, and the result may carry a link to a support portal KB article.

ERR is the one people misread: the check broke, not the cluster. And this explains why alert A200613, the 40 GB PE CVM memory recommendation, is severity INFO: an expected value that cannot be graded, carrying a recommendation. Textbook INFO.

Modules, plugins, and the intrusive check default

A plugin is a component specific code block inside a module, commonly called a check. A module is a logical group of plugins or of modules. NCC is cluster resident, runs hundreds of checks, and depending on the issue raises an alert or automatically creates a Nutanix Support case. It runs as long as the individual nodes are up, regardless of cluster state.

nutanix@cvm$ ncc ncc-flags module sub-module [...] plugin plugin-flags
# default output: /home/nutanix/data/logs/ncc-output-latest.log

Typing ncc with no arguments lists the modules. The Type column shows M (module) or P (plugin), and the Impact tag marks a plugin intrusive or non intrusive.

Modules in the guide’s example output: cassandra_tools, fix_failures, hardware_info, health_checks, help_opts, log_collector, performance_checks, pulsehd_collectors.

How to run NCC and how to find a check name

It is easy to conclude that the Flow documentation carries no NCC content. Four Prism Web Console Guide 7.3 topics cover it.

Web console: Health dashboard > Actions > Run NCC Checks, scoped to All checks, Only Failed and Warning Checks, or Specific Checks (type the name, the field auto populates, the Added Checks box lists the selection). There is a Send the cluster check report in the email checkbox, which needs alert email notification already configured. The Tasks dashboard shows succeeded or aborted. Event triggered checks are marked passed by default.

Command line:

nutanix@cvm$ ncc health_checks run_all      # run everything
nutanix@cvm$ ncc health_checks              # list the available check categories
nutanix@cvm$ ncc health_checks hypervisor_checks   # drill into one category

hypervisor_checks is the guide’s own example and the only category name stated anywhere in the documentation. Anything other than INFO or PASS must be resolved before proceeding with an upgrade.

From the UI: Actions > Manage Checks, select a check, then click the link to the Knowledge Base article for that check. That plus ncc health_checks are the two documented ways to find a check name on a live cluster.

Logbay, the log collection half of this objective

Discovery, the same pattern as NCC: logbay list_tags lists every available tag on a live cluster. The documented tag table is partial (it omits msp and anc, which the FVN guide uses), which is exactly why the command exists.

Documented tags: cvm_config, cvm_logs, cvm_kernel, alerts, ahv_config / esx_config / hyperv_config, ahv_logs / esx_logs / hyperv_logs, and vpn_logs, which is directly relevant to VPN gateway troubleshooting in objective 3.1.

Retrieving the bundle: Tasks dashboard, find the log bundle task, click the Succeeded link in the Status column. The last two runs are available, and a browser pop up blocker will stop the download.

Scheduling NCC

Disabled by default. Health dashboard > Actions > Set NCC Frequency: Every 4 hours, Every Day with a Start Time, Every Week with On days and a Start Time, or Remove Schedule (shown only once a schedule exists). Results email to whoever is configured for alert emails, and the schedule survives AOS and NCC upgrades. Emailed NCC results do not automatically create support cases, unlike proactive NCC, which can.

What the documentation does have, adjacent to this

• TCP port 9446 from AHV hosts to the Prism Central VMs carries connection tracking data, which Prism Central uses to show network flows. This is the one FNS port number the documentation actually gives.
• The flow_data Kafka container is created automatically on the cluster hosting Prism Central when microsegmentation is enabled, and stores data essential for Flow visualization. Do not delete it.
• Alert 200615, high Cadmus service flows, where flows means traffic reaching a secured entity. Additional flows past the limit are not shown in the UI. Fix by reducing policies or flows per policy, or upgrading the Microsegmentation Memory Allocation (S/M/L/XL).

Interpret FNS audit logs to diagnose an issue

Audit logs cover changes to security policy configuration and VM to category mappings: when a policy was changed or applied, and who changed it. Enabled by default, viewable in Prism Central and sent to syslog. The Audit module carries Prism Central changes; the API Audit module carries changes made directly through the REST API endpoint.

The diagnostic pairing: audit logs tell you what changed and who changed it; policy hit logs tell you what the traffic did as a result. Note that hit logs and visualization are not synchronized in multi Prism Central DR, so on a remote Prism Central a synced policy has an audit trail but no hit logs.

Complementary PC VM logs under /home/nutanix/data/logs: alert_manager.*, aplos.out and aplos_engine.out for the v3 API gateway and engine, catalina.out, genesis.out, and cron_time_check.log which checks time difference across PC VMs on a multi VM instance every minute.

Verify AD configuration · Enable and manage referenced AD groups

Configuration to check, at Prism Central Settings > ID Based Security

• Directory URL is the LDAP address including the port number.
• Service Account Username in user_name@domain.com format. Never the Domain Admin account.
• Every domain controller added manually, by IP or host name. DNS on Prism Central is required for host names to work.
• Prerequisites: microsegmentation enabled; WMI access from PC to all DCs through both the network and AD firewalls; AD functional level Windows Server 2008 R2 minimum; Security Groups only; NTP on both AD and Prism Central; AOS 5.17 and PC 5.17 minimum.
• Service account permissions on every DC: Distributed COM Users and Event Log Readers; Local Access and Remote Access on WMI in DCOM Config; Enable Account and Remote Enable on Root\CIMV2, applied to this namespace and subnamespaces; then net stop winmgmt and net start winmgmt.

Validate that group memberships have been applied to a policy

1. Policy type. Must be Application Secure Entities with Secure VDI Groups only selected. Without it, ADGroup categories are not driving it.
2. Scope. FNS does not support VDI policy for a VPC scope. You can attach a VDI VM to a VPC or overlay network, but a VDI policy with VPC scope does not protect the VDI VMs. A VDI policy that silently does nothing is very often this.
3. Default policy. ADGroup:Default applies rules before anyone logs on. If VMs are protected before logon but not after, or the reverse, check this.
4. No visualization. VDI policies do not support it, so verification comes from the configuration, the category on the VM, and the policy hit logs.
5. The union rule. A policy that looks too permissive may be correct: another ADGroup the user belongs to is contributing rules.