diff --git a/README.md b/README.md index b6bbeda..7d65a39 100644 --- a/README.md +++ b/README.md @@ -3,10 +3,10 @@ This is a tool to sync data from different sources (currently only VMWare vCenter) to a NetBox instance. -**IMPORTANT: READ THIS INSTRUCTIONS CAREFULLY BEFORE RUNNING THIS PROGRAM** +**IMPORTANT: READ INSTRUCTIONS CAREFULLY BEFORE RUNNING THIS PROGRAM** ## Thanks -A BIG thank you goes out to [Raymond Beaudoin](https://github.com/synackray) for creating +A BIG thank-you goes out to [Raymond Beaudoin](https://github.com/synackray) for creating [vcenter-netbox-sync](https://github.com/synackray/vcenter-netbox-sync) which served as source of a lot of ideas for this project. @@ -33,8 +33,11 @@ This ensures stale objects are removed from NetBox keeping an accurate current s * aiodns==2.0.0 ### Environment -* VMWare vCenter >= 6.0 * NetBox >= 2.9 +#### Source: VMWare +* VMWare vCenter >= 6.0 +#### Source: check_redfish +* check_redfish >= 1.2.0 # Installing * here we assume we install in ```/opt``` @@ -142,23 +145,19 @@ optional arguments: It is recommended to set log level to `DEBUG2` this way the program should tell you what is happening and why. Also use the dry run option `-n` at the beginning to avoid changes directly in NetBox. -## Migration -If you migrate from [vcenter-netbox-sync](https://github.com/synackray/vcenter-netbox-sync) do following things: -* rename Tags: - * Synced -> NetBox-synced - * Orphaned -> NetBox-synced: Orphaned - ## Setup Copy the [settings-example.ini](settings-example.ini) sample settings file to `settings.ini`. All options are described in the example file. -You should define the var `cluster_site_relation` which maps a vCenter cluster to an exiting Site in NetBox. -Otherwise a placeholder site will be created. +For VMWare source you should define the var `cluster_site_relation` which maps a vCenter cluster to an +exiting Site in NetBox. If undefined a placeholder site will be created. ## Cron job In Order to sync all items regularly you can add a cron job like this one +``` # NetBox Sync 23 */2 * * * /opt/netbox-sync/.env/bin/python3 /opt/netbox-sync/netbox-sync.py >/dev/null 2>&1 +``` # How it works **READ CAREFULLY** @@ -175,163 +174,20 @@ The program operates mainly like this ## NetBox connection Request all current NetBox objects. Use caching whenever possible. Objects must provide "last_updated" attribute to support caching for this object type. -Otherwise it's not possible to query only changed objects since last run. If attribute is -not present all objects will be requested (looking at you *Interfaces) Actually perform the request and retry x times if request times out. Program will exit if all retries failed! -## Sources -Currently only VMWare vCenter is an available source. But new sources can be added via a new -source class. +## Supported sources +Check out the documentations for the different sources +* [vmware](docs/source_vmware.md) +* [check_redfish](docs/source_check_redfish.md) + +New sources can be added via a new source class. ToDo: * add documentation of source implementation here -## Source: vmware - -To match new/updated objects to existing objects in NetBox a few different steps are taken to find the correct object. - - -### Add/update device/VM object in inventory based on gathered data - -Try to find object first based on the object data, interface MAC addresses and primary IPs. -1. try to find by name and cluster/site -2. try to find by mac addresses interfaces -3. try to find by serial number (1st) or asset tag (2nd) (ESXi host) -4. try to find by primary IP - -#### IP addresses -First they will be checked and added if all checks pass. -* have to pass `permitted_subnets` config setting -* loop back addresses will be ignored -* link local addresses will be ignored - -For each IP address a matching IP prefix will be searched for. First we look for the longest -matching IP Prefix in the same site. If this failed we try to find the longest matching global IP Prefix. - -If a IP Prefix was found then we try to get the VRF and VLAN for this prefix. Now we compare -if interface VLAN and prefix VLAN match up and warn if they don't. We also compare the -length of the found prefix with the prefix length of the configured IP address. -A warning will also be issued if they don't match. - -If the same IP address is found on a different interface (of a different device/VM) within the same realm -(both using same VRF or both are global) then following test are performed: -* If the current interface is enabled and the new one disabled: - * IP stays at the current interface -* If the current interface is disabled and the new one enabled: - * IP will be moved to the enabled interface -* Both interfaces are in the same state (disabled/enable) - * IP will also stay at the current interface as it's unclear which one would be the correct one - -Then we try to add data to the IP address if not already set: - -* add prefix VRF if VRF for this IP is undefined -* add tenant if tenant for this IP is undefined - 1. try prefix tenant - 2. if prefix tenant is undefined try VLAN tenant - -### Finding hosts and VMs from discovered data - -#### 1. try to find exact name match including cluster/site - -If this NetBox object (device, vm) matches, the found object will be used. - -#### 2. Try to find a NetBox object based on list of MAC addresses - -Iterate over all NetBox interfaces of an object (device, vm) and compare MAC address with list -of MAC addresses discovered for this object. If a match was found, count for every object (device, vm) -how many MAC addresses are matching. - -If exactly one NetBox object (device, vm) with matching interface MACs was found then this one will be used. - -If two or more NetBox object (device, vm) with matching MACs were found, compare the two -NetBox object (device, vm) with the highest amount of matching interfaces. If the ratio of matching interface -MAC addresses exceeds 2.0 then the top matching NetBox object (device, vm) is chosen as desired object. - -If the ratio is below 2.0 then None will be chosen. The probability is to low that -this one is the correct one. - -#### 3. Try to find a NetBox object based on the primary IP (v4 or v6) address - -If an exact matching NetBox object (device, vm) was found the object will be used -immediately without checking the other primary IP address (if defined). - -### Try to match current NetBox object (device, vm) interfaces to discovered ones - -This will be done by multiple approaches. -Order as following listing. Whatever matches first will be chosen. - -by simple name: -* both interface names match exactly - -by MAC address separated by physical and virtual NICs: -* MAC address of interfaces match exactly, distinguish between physical and virtual interfaces - -by MAC regardless of interface type -* MAC address of interfaces match exactly, type of interface does not matter - -If there are interfaces which don't match at all then the unmatched interfaces will be -matched 1:1. Sort both lists (unmatched current interfaces, unmatched new interfaces) -by name and assign them each other. -``` - ens1 > vNIC 1 - eth0 > vNIC 2 - eth1 > vNIC 3 - ... > ... -``` - -### Objects read and parsed from vCenter - -#### 1. Add a vCenter datacenter as a Cluster Group to NetBox -Simple, nothing special going on here - -#### 2. Add a vCenter cluster as a Cluster to NetBox. - -Cluster name is checked against `cluster_include_filter` and `cluster_exclude_filter config` setting. - -#### 3. Parse distributed virtual port group -This is done to extract name and VLAN IDs from each port group - -#### 4. Parse a vCenter (ESXi) host - -First host is filtered: -* host has a cluster and this cluster is permitted -* skip host with same name and site, we already parsed it (use unique host names) -* does the host pass the `host_include_filter` and `host_exclude_filter` - -Then all necessary host data will be collected:
-host model, manufacturer, serial, physical interfaces, virtual interfaces, -virtual switches, proxy switches, host port groups, interface VLANs, IP addresses - -Primary IPv4/6 will be determined by -1. if the interface port group name contains "management" or "mngt" -2. interface with this IP is the default route of this host - -#### 5. Parse a vCenter VM - -Iterate over all VMs twice! - -To handle VMs with the same name in a cluster we first iterate over all VMs and look only at the -active (online) ones and parse these first. Then we iterate a second time to catch the rest (also offline). - -This has been implemented to support migration scenarios where you create/copy the same -machine with a different setup like a new version or something. This way NetBox will be -updated primarily with the actual active VM data. - -First VM is filtered: -* VM has a cluster and is it permitted -* skip VMs with same name and cluster if already parsed -* does the VM pass the `vm_include_filter` and `vm_exclude_filter` - -Then all necessary VM data will be collected:
-platform, virtual interfaces, virtual cpu/disk/memory interface VLANs, IP addresses - -Primary IPv4/6 will be determined by interface that provides the default route for this VM. - -**Note:**
-IP address information can only be extracted if guest tools are installed and running. - ### Pruning Prune objects in NetBox if they are no longer present in any source. First they will be marked as Orphaned and after X (config option) days they will be diff --git a/docs/source_check_redfish.md b/docs/source_check_redfish.md new file mode 100644 index 0000000..e69de29 diff --git a/docs/source_vmware.md b/docs/source_vmware.md new file mode 100644 index 0000000..858772e --- /dev/null +++ b/docs/source_vmware.md @@ -0,0 +1,143 @@ +## Source: vmware + +To match new/updated objects to existing objects in NetBox a few different steps are taken to find the correct object. + + +### Add/update device/VM object in inventory based on gathered data + +Try to find object first based on the object data, interface MAC addresses and primary IPs. +1. try to find by name and cluster/site +2. try to find by mac addresses interfaces +3. try to find by serial number (1st) or asset tag (2nd) (ESXi host) +4. try to find by primary IP + +#### IP addresses +First they will be checked and added if all checks pass. +* have to pass `permitted_subnets` config setting +* loop back addresses will be ignored +* link local addresses will be ignored + +For each IP address a matching IP prefix will be searched for. First we look for the longest +matching IP Prefix in the same site. If this failed we try to find the longest matching global IP Prefix. + +If a IP Prefix was found then we try to get the VRF and VLAN for this prefix. Now we compare +if interface VLAN and prefix VLAN match up and warn if they don't. We also compare the +length of the found prefix with the prefix length of the configured IP address. +A warning will also be issued if they don't match. + +If the same IP address is found on a different interface (of a different device/VM) within the same realm +(both using same VRF or both are global) then following test are performed: +* If the current interface is enabled and the new one disabled: + * IP stays at the current interface +* If the current interface is disabled and the new one enabled: + * IP will be moved to the enabled interface +* Both interfaces are in the same state (disabled/enable) + * IP will also stay at the current interface as it's unclear which one would be the correct one + +Then we try to add data to the IP address if not already set: + +* add prefix VRF if VRF for this IP is undefined +* add tenant if tenant for this IP is undefined + 1. try prefix tenant + 2. if prefix tenant is undefined try VLAN tenant + +### Finding hosts and VMs from discovered data + +#### 1. try to find exact name match including cluster/site + +If this NetBox object (device, vm) matches, the found object will be used. + +#### 2. Try to find a NetBox object based on list of MAC addresses + +Iterate over all NetBox interfaces of an object (device, vm) and compare MAC address with list +of MAC addresses discovered for this object. If a match was found, count for every object (device, vm) +how many MAC addresses are matching. + +If exactly one NetBox object (device, vm) with matching interface MACs was found then this one will be used. + +If two or more NetBox object (device, vm) with matching MACs were found, compare the two +NetBox object (device, vm) with the highest amount of matching interfaces. If the ratio of matching interface +MAC addresses exceeds 2.0 then the top matching NetBox object (device, vm) is chosen as desired object. + +If the ratio is below 2.0 then None will be chosen. The probability is to low that +this one is the correct one. + +#### 3. Try to find a NetBox object based on the primary IP (v4 or v6) address + +If an exact matching NetBox object (device, vm) was found the object will be used +immediately without checking the other primary IP address (if defined). + +### Try to match current NetBox object (device, vm) interfaces to discovered ones + +This will be done by multiple approaches. +Order as following listing. Whatever matches first will be chosen. + +by simple name: +* both interface names match exactly + +by MAC address separated by physical and virtual NICs: +* MAC address of interfaces match exactly, distinguish between physical and virtual interfaces + +by MAC regardless of interface type +* MAC address of interfaces match exactly, type of interface does not matter + +If there are interfaces which don't match at all then the unmatched interfaces will be +matched 1:1. Sort both lists (unmatched current interfaces, unmatched new interfaces) +by name and assign them each other. +``` + ens1 > vNIC 1 + eth0 > vNIC 2 + eth1 > vNIC 3 + ... > ... +``` + +### Objects read and parsed from vCenter + +#### 1. Add a vCenter datacenter as a Cluster Group to NetBox +Simple, nothing special going on here + +#### 2. Add a vCenter cluster as a Cluster to NetBox. + +Cluster name is checked against `cluster_include_filter` and `cluster_exclude_filter config` setting. + +#### 3. Parse distributed virtual port group +This is done to extract name and VLAN IDs from each port group + +#### 4. Parse a vCenter (ESXi) host + +First host is filtered: +* host has a cluster and this cluster is permitted +* skip host with same name and site, we already parsed it (use unique host names) +* does the host pass the `host_include_filter` and `host_exclude_filter` + +Then all necessary host data will be collected:
+host model, manufacturer, serial, physical interfaces, virtual interfaces, +virtual switches, proxy switches, host port groups, interface VLANs, IP addresses + +Primary IPv4/6 will be determined by +1. if the interface port group name contains "management" or "mngt" +2. interface with this IP is the default route of this host + +#### 5. Parse a vCenter VM + +Iterate over all VMs twice! + +To handle VMs with the same name in a cluster we first iterate over all VMs and look only at the +active (online) ones and parse these first. Then we iterate a second time to catch the rest (also offline). + +This has been implemented to support migration scenarios where you create/copy the same +machine with a different setup like a new version or something. This way NetBox will be +updated primarily with the actual active VM data. + +First VM is filtered: +* VM has a cluster and is it permitted +* skip VMs with same name and cluster if already parsed +* does the VM pass the `vm_include_filter` and `vm_exclude_filter` + +Then all necessary VM data will be collected:
+platform, virtual interfaces, virtual cpu/disk/memory interface VLANs, IP addresses + +Primary IPv4/6 will be determined by interface that provides the default route for this VM. + +**Note:**
+IP address information can only be extracted if guest tools are installed and running.