Description of the problem:

The following is the connection settings for `ib0-10`:

```network role
network_connections:

• name: ib0
  type: infiniband
  interface_name: ib0

• name: ib0-10
  interface_name: ib0.000a
  type: infiniband
  autoconnect: yes
  infiniband:
  p_key: 10
  transport_mode: datagram
  parent: ib0
  state: up
  ip:
  dhcp4: no
  auto6: no
  address:
• 198.51.100.133/30
• 2001:db8::2/32

```

Before bringing down the connection, the connection is active:
[root@wsfd-netdev23 ~]# nmcli c show --active
NAME UUID TYPE DEVICE
ib0-10 de8ed21e-de6a-4a54-8700-2637cec33ba8 infiniband ib0.000a

[root@wsfd-netdev23 ~]# ip -d a s ib0.000a
28: ib0.000a@ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state LOWERLAYERDOWN group default qlen 256
link/infiniband 80:00:02:16:fe:80:00:00:00:00:00:00:f4:52:14:03:00:33:bd:31 brd 00:ff:ff:ff:ff:12:40:1b:80:0a:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520
ipoib pkey 0x800a mode datagram umcast 0000 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
inet 198.51.100.133/30 brd 198.51.100.135 scope global noprefixroute ib0.000a
valid_lft forever preferred_lft forever
inet6 2001:db8::2/32 scope global tentative noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::b41c:fee:35b1:fc43/64 scope link tentative noprefixroute
valid_lft forever preferred_lft forever

After bringing down the connection, the virtual infiniband device still exists:

[root@wsfd-netdev23 ~]# nmcli c down ib0-10
Connection 'ib0-10' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/17)
[root@wsfd-netdev23 ~]# ip -d a s ib0.000a
28: ib0.000a@ib0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 4092 qdisc fq_codel state LOWERLAYERDOWN group default qlen 256
link/infiniband 80:00:02:16:fe:80:00:00:00:00:00:00:f4:52:14:03:00:33:bd:31 brd 00:ff:ff:ff:ff:12:40:1b:80:0a:00:00:00:00:00:00:ff:ff:ff:ff promiscuity 0 minmtu 0 maxmtu 65520
ipoib pkey 0x800a mode datagram umcast 0000 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535

Version-Release number of selected component (if applicable):
NetworkManager-1.39.0-1.el8.x86_64
NetworkManager-libnm-1.39.0-1.el8.x86_64

How reproducible:
I can reproduce every time

Actual results:
Bring down the IP over infiniabnd connection will not remove the virtual infiniband device automatically

Expected results:
Bring down the IP over infiniabnd connection should remove the virtual infiniband device automatically