running initial pings clientQA ping -------------------- nmap found 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP arm-redhat ping ----------------- nmap found 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP userspace-wireguard ping ----------------- nmap found 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP clientqa02 ping ----------------- nmap found 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP clientqa03 ping ----------------- nmap found 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP clientqa04 ping ----------------- nmap found 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP ipv4-test ping ----------------- nmap found checking routing clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.1 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.6 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.7 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.2 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.3 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.4 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium ipv4-test ip route -------------------- default via 209.38.128.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.5 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.2 209.38.128.0/19 dev eth0 proto kernel scope link src 209.38.128.43 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium getting initial wg configs for clients clientQA wg -------------------- interface: netmaker public key: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= private key: (hidden) listening port: 51821 peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 10.106.0.10:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 second ago transfer: 9.76 MiB received, 9.73 MiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 13 seconds ago transfer: 9.47 MiB received, 9.44 MiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 28 seconds ago transfer: 5.67 MiB received, 6.99 MiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 38 seconds ago transfer: 9.65 MiB received, 9.61 MiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 17 seconds ago transfer: 9.51 MiB received, 9.37 MiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 44 seconds ago transfer: 9.48 MiB received, 9.45 MiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= private key: (hidden) listening port: 51821 peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 14 seconds ago transfer: 9.44 MiB received, 9.47 MiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 30 seconds ago transfer: 9.39 MiB received, 9.33 MiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 40 seconds ago transfer: 9.47 MiB received, 9.29 MiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 48 seconds ago transfer: 9.38 MiB received, 9.38 MiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 59 seconds ago transfer: 9.40 MiB received, 9.33 MiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 50 seconds ago transfer: 5.62 MiB received, 6.93 MiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= private key: (hidden) listening port: 51821 peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 3 seconds ago transfer: 9.42 MiB received, 9.38 MiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 42 seconds ago transfer: 9.34 MiB received, 9.37 MiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 56 seconds ago transfer: 9.48 MiB received, 9.32 MiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 1 second ago transfer: 9.33 MiB received, 9.41 MiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute, 47 seconds ago transfer: 9.45 MiB received, 9.48 MiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 51 seconds ago transfer: 9.36 MiB received, 9.34 MiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= private key: (hidden) listening port: 51821 peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 4 seconds ago transfer: 9.38 MiB received, 9.42 MiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 10.106.0.9:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 5 seconds ago transfer: 9.73 MiB received, 9.76 MiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 51 seconds ago transfer: 9.38 MiB received, 9.38 MiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 22 seconds ago transfer: 9.38 MiB received, 9.42 MiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 46 seconds ago transfer: 9.49 MiB received, 9.33 MiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 57 seconds ago transfer: 9.43 MiB received, 9.39 MiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= private key: (hidden) listening port: 51821 peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 45 seconds ago transfer: 9.29 MiB received, 9.47 MiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 51 seconds ago transfer: 9.36 MiB received, 9.43 MiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 10.116.0.3:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 56 seconds ago transfer: 9.36 MiB received, 9.47 MiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 59 seconds ago transfer: 9.32 MiB received, 9.48 MiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute, 23 seconds ago transfer: 9.37 MiB received, 9.51 MiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 48 seconds ago transfer: 9.33 MiB received, 9.49 MiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= private key: (hidden) listening port: 51821 peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 17 seconds ago transfer: 9.34 MiB received, 9.38 MiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 35 seconds ago transfer: 9.33 MiB received, 9.39 MiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 44 seconds ago transfer: 9.61 MiB received, 9.65 MiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 51 seconds ago transfer: 9.43 MiB received, 9.36 MiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 54 seconds ago transfer: 9.34 MiB received, 9.36 MiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 59 seconds ago transfer: 9.39 MiB received, 9.44 MiB sent persistent keepalive: every 20 seconds ipv4-test wg interface -------------------- interface: netmaker public key: 12T/bwQRxnB6yattNaotXgUk4pCmO6vHJAwF5Tfk1ys= private key: (hidden) listening port: 51821 checking initial logs for errors clientQA log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors -------------------- Apr 23 20:30:28 clientQA-04 netclient[2420253]: {"time":"2024-04-23T20:30:28.734482075Z","level":"ERROR","source":"mqpublish.go 250}","msg":"could not connect to broker at","server":"nm.67-205-143-140.nip.io","error":"not Connected"} Apr 23 20:30:28 clientQA-04 netclient[2420253]: [netclient] 2024-04-23 20:30:28 error occurred during publishing of metrics on node clientQA-04 not Connected ipv4-test log errors -------------------- Apr 23 20:25:43 ipv4-test netclient[124862]: {"time":"2024-04-23T20:25:43.851289601Z","level":"ERROR","source":"daemon.go 212}","msg":"fail to pull config from server","error":"server config not found"} pulling clients and checking config pulling clientQA -------------------- completed pull for server nm.67-205-143-140.nip.io pulling arm-redhat -------------------- completed pull for server nm.67-205-143-140.nip.io pulling userspace-wireguard -------------------- completed pull for server nm.67-205-143-140.nip.io pulling clientqa02 -------------------- completed pull for server nm.67-205-143-140.nip.io pulling clientqa03 -------------------- completed pull for server nm.67-205-143-140.nip.io pulling clientqa04 -------------------- completed pull for server nm.67-205-143-140.nip.io pulling ipv4-test -------------------- [netclient] 2024-04-24 14:39:59 failed to pull server config not found running pings after pull clientQA ping -------------------- nmap found 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP arm-redhat ping ----------------- nmap found 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP userspace-wireguard ping ----------------- nmap found 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP clientqa02 ping ----------------- nmap found 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP clientqa03 ping ----------------- nmap found 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP clientqa04 ping ----------------- nmap found 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP ipv4-test ping ----------------- nmap found checking routing after pull clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.1 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.6 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.7 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.2 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.3 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.4 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium ipv4-test ip route -------------------- default via 209.38.128.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.5 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.2 209.38.128.0/19 dev eth0 proto kernel scope link src 209.38.128.43 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium getting wg configs after pull clientQA wg -------------------- interface: netmaker public key: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= private key: (hidden) listening port: 51821 peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 37 seconds ago transfer: 29.96 KiB received, 30.07 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 45 seconds ago transfer: 27.71 KiB received, 27.37 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 57 seconds ago transfer: 26.70 KiB received, 27.54 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128, 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute ago transfer: 20.84 KiB received, 24.76 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 1 second ago transfer: 28.62 KiB received, 29.51 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= private key: (hidden) listening port: 51821 peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 44 seconds ago transfer: 26.99 KiB received, 28.20 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 52 seconds ago transfer: 27.55 KiB received, 26.96 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 56 seconds ago transfer: 28.55 KiB received, 29.31 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 59 seconds ago transfer: 25.43 KiB received, 26.09 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 2 seconds ago transfer: 18.07 KiB received, 21.64 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute, 2 seconds ago transfer: 29.55 KiB received, 29.06 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= private key: (hidden) listening port: 51821 peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 40 seconds ago transfer: 29.48 KiB received, 28.52 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 57 seconds ago transfer: 26.56 KiB received, 27.29 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 58 seconds ago transfer: 25.59 KiB received, 26.13 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute ago transfer: 27.74 KiB received, 27.23 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 1 second ago transfer: 26.46 KiB received, 26.18 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 2 seconds ago transfer: 26.23 KiB received, 27.64 KiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= private key: (hidden) listening port: 51821 peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 41 seconds ago transfer: 28.48 KiB received, 30.01 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 54 seconds ago transfer: 26.67 KiB received, 26.77 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 56 seconds ago transfer: 26.64 KiB received, 27.67 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128, 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute ago transfer: 31.10 KiB received, 31.61 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute ago transfer: 27.18 KiB received, 26.54 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= private key: (hidden) listening port: 51821 peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 43 seconds ago transfer: 27.16 KiB received, 28.65 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 56 seconds ago transfer: 26.67 KiB received, 26.41 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 10.116.0.3:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 59 seconds ago transfer: 24.13 KiB received, 24.95 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 1 second ago transfer: 28.09 KiB received, 28.65 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 1 second ago transfer: 27.30 KiB received, 27.78 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 2 seconds ago transfer: 26.30 KiB received, 27.18 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= private key: (hidden) listening port: 51821 peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 50 seconds ago transfer: 26.70 KiB received, 26.62 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 52 seconds ago transfer: 30.02 KiB received, 30.76 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 57 seconds ago transfer: 26.37 KiB received, 27.01 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 57 seconds ago transfer: 26.98 KiB received, 26.95 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 1 second ago transfer: 27.57 KiB received, 27.53 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 1 second ago transfer: 25.10 KiB received, 25.81 KiB sent persistent keepalive: every 20 seconds ipv4-test wg interface -------------------- interface: netmaker public key: 12T/bwQRxnB6yattNaotXgUk4pCmO6vHJAwF5Tfk1ys= private key: (hidden) listening port: 51821 checking logs for errors after a pull clientQA log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors -------------------- Apr 23 20:30:28 clientQA-04 netclient[2420253]: {"time":"2024-04-23T20:30:28.734482075Z","level":"ERROR","source":"mqpublish.go 250}","msg":"could not connect to broker at","server":"nm.67-205-143-140.nip.io","error":"not Connected"} Apr 23 20:30:28 clientQA-04 netclient[2420253]: [netclient] 2024-04-23 20:30:28 error occurred during publishing of metrics on node clientQA-04 not Connected ipv4-test log errors -------------------- Apr 23 20:25:43 ipv4-test netclient[124862]: {"time":"2024-04-23T20:25:43.851289601Z","level":"ERROR","source":"daemon.go 212}","msg":"fail to pull config from server","error":"server config not found"} restarting clients and checking config restarting clientQA -------------------- restarting arm-redhat -------------------- restarting userspace-wireguard -------------------- restarting clientqa02 -------------------- restarting clientqa03 -------------------- restarting clientqa04 -------------------- restarting ipv4-test -------------------- running pings after restart clientQA ping -------------------- nmap found 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP arm-redhat ping ----------------- nmap found 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP userspace-wireguard ping ----------------- nmap found 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP clientqa02 ping ----------------- nmap found 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP clientqa03 ping ----------------- nmap found 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP 10.43.138.4 IS UP fd00:eb1b:f9f8:d366::4 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP clientqa04 ping ----------------- nmap found 10.43.138.3 IS UP fd00:eb1b:f9f8:d366::3 IS UP 10.43.138.1 IS UP fd00:eb1b:f9f8:d366::1 IS UP 10.43.138.7 IS UP fd00:eb1b:f9f8:d366::7 IS UP 10.43.138.5 IS UP fd00:eb1b:f9f8:d366::5 IS UP 10.43.138.2 IS UP fd00:eb1b:f9f8:d366::2 IS UP 10.43.138.6 IS UP fd00:eb1b:f9f8:d366::6 IS UP ipv4-test ping ----------------- nmap found checking routing after restart clientQA ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.10 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.1 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.9 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.179 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium arm-redhat ip route -------------------- default via 172.31.16.1 dev eth0 proto dhcp src 172.31.23.197 metric 100 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.6 172.31.16.0/20 dev eth0 proto kernel scope link src 172.31.23.197 metric 100 ::1 dev lo proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium userspace-wireguard ip route -------------------- default via 24.144.80.1 dev eth0 proto static 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.7 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.14 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.13 24.144.80.0/20 dev eth0 proto kernel scope link src 24.144.84.150 ::1 dev lo proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium clientqa02 ip route -------------------- default via 165.22.112.1 dev eth0 proto static 10.16.0.0/16 dev eth0 proto kernel scope link src 10.16.0.11 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.2 10.106.0.0/20 dev eth1 proto kernel scope link src 10.106.0.10 98.216.56.86 via 165.22.112.1 dev eth0 159.223.160.241 via 165.22.112.1 dev eth0 165.22.112.0/20 dev eth0 proto kernel scope link src 165.22.115.252 165.22.115.179 via 165.22.112.1 dev eth0 209.38.230.81 via 165.22.112.1 dev eth0 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:1:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:1:d0::1 dev eth0 proto static metric 1024 pref medium clientqa03 ip route -------------------- default via 159.223.160.1 dev eth0 proto static metric 100 10.10.0.0/16 dev eth0 proto kernel scope link src 10.10.0.5 metric 100 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.3 10.116.0.0/20 dev eth1 proto kernel scope link src 10.116.0.2 metric 101 159.223.160.0/20 dev eth0 proto kernel scope link src 159.223.160.241 metric 100 ::1 dev lo proto kernel metric 256 pref medium 2604:a880:400:d0::/64 dev eth0 proto kernel metric 100 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 1024 pref medium fe80::/64 dev eth1 proto kernel metric 1024 pref medium default via 2604:a880:400:d0::1 dev eth0 proto static metric 100 pref medium clientqa04 ip route -------------------- default via 209.38.224.1 dev eth0 proto static 10.19.0.0/16 dev eth0 proto kernel scope link src 10.19.0.16 10.43.138.0/24 dev netmaker proto kernel scope link src 10.43.138.4 10.114.0.0/20 dev eth1 proto kernel scope link src 10.114.0.13 209.38.224.0/19 dev eth0 proto kernel scope link src 209.38.230.81 ::1 dev lo proto kernel metric 256 pref medium 2a03:b0c0:3:d0::/64 dev eth0 proto kernel metric 256 pref medium fd00:eb1b:f9f8:d366::/64 dev netmaker proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium default via 2a03:b0c0:3:d0::1 dev eth0 proto static metric 1024 pref medium ipv4-test ip route -------------------- default via 209.38.128.1 dev eth0 proto static 10.48.0.0/16 dev eth0 proto kernel scope link src 10.48.0.5 10.124.0.0/20 dev eth1 proto kernel scope link src 10.124.0.2 209.38.128.0/19 dev eth0 proto kernel scope link src 209.38.128.43 ::1 dev lo proto kernel metric 256 pref medium fe80::/64 dev eth0 proto kernel metric 256 pref medium fe80::/64 dev eth1 proto kernel metric 256 pref medium getting wg configs after restart clientQA wg -------------------- interface: netmaker public key: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= private key: (hidden) listening port: 51821 peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 25 seconds ago transfer: 41.12 KiB received, 42.17 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 46 seconds ago transfer: 26.22 KiB received, 32.16 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 11 seconds ago transfer: 41.02 KiB received, 40.71 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 10.106.0.10:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 16 seconds ago transfer: 28.27 KiB received, 29.40 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 27 seconds ago transfer: 35.56 KiB received, 36.67 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 58 seconds ago transfer: 37.66 KiB received, 39.68 KiB sent persistent keepalive: every 20 seconds arm-redhat wg interface -------------------- interface: netmaker public key: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= private key: (hidden) listening port: 51821 peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 8 seconds ago transfer: 38.00 KiB received, 38.97 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 21 seconds ago transfer: 23.54 KiB received, 29.31 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 27 seconds ago transfer: 41.50 KiB received, 41.16 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 13 seconds ago transfer: 35.41 KiB received, 36.21 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 22 seconds ago transfer: 38.07 KiB received, 38.82 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 41 seconds ago transfer: 40.12 KiB received, 39.71 KiB sent persistent keepalive: every 20 seconds userspace-wireguard wg interface -------------------- interface: netmaker public key: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= private key: (hidden) listening port: 51821 peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 2 seconds ago transfer: 36.64 KiB received, 36.20 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 10 seconds ago transfer: 38.30 KiB received, 38.00 KiB sent persistent keepalive: every 20 seconds peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 12 seconds ago transfer: 33.37 KiB received, 33.61 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 27 seconds ago transfer: 35.10 KiB received, 35.80 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 46 seconds ago transfer: 34.31 KiB received, 34.41 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 2 minutes, 8 seconds ago transfer: 35.45 KiB received, 35.51 KiB sent persistent keepalive: every 20 seconds clientqa02 wg interface -------------------- interface: netmaker public key: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= private key: (hidden) listening port: 51821 peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 5 seconds ago transfer: 32.68 KiB received, 33.44 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 10.106.0.9:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute, 20 seconds ago transfer: 29.54 KiB received, 28.71 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 33 seconds ago transfer: 31.34 KiB received, 32.16 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 43 seconds ago transfer: 35.18 KiB received, 35.25 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 46 seconds ago transfer: 34.18 KiB received, 34.59 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 50 seconds ago transfer: 34.63 KiB received, 34.07 KiB sent persistent keepalive: every 20 seconds clientqa03 wg interface -------------------- interface: netmaker public key: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= private key: (hidden) listening port: 51821 peer: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= endpoint: 209.38.230.81:51821 allowed ips: 10.43.138.4/32, fd00:eb1b:f9f8:d366::4/128 latest handshake: 1 minute, 18 seconds ago transfer: 30.50 KiB received, 30.86 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 27 seconds ago transfer: 31.71 KiB received, 32.11 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 30 seconds ago transfer: 32.79 KiB received, 32.61 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute, 34 seconds ago transfer: 30.85 KiB received, 31.29 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 10.116.0.3:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 36 seconds ago transfer: 31.99 KiB received, 32.23 KiB sent persistent keepalive: every 20 seconds peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 36 seconds ago transfer: 32.05 KiB received, 32.21 KiB sent persistent keepalive: every 20 seconds clientqa04 wg interface -------------------- interface: netmaker public key: utGtlnh+za81OIEZxYQym6c3GsTKQhKm/jD6fcF6PUU= private key: (hidden) listening port: 51821 peer: HJf9Da/zWSXOaysKkXKcLMNMu+D/UHFGY+lI+m2nakA= endpoint: 165.22.115.252:51821 allowed ips: 10.43.138.2/32, fd00:eb1b:f9f8:d366::2/128 latest handshake: 1 minute, 8 seconds ago transfer: 31.13 KiB received, 31.46 KiB sent persistent keepalive: every 20 seconds peer: tbkqg/LAKDrpRBQ7xQ8FiJW7GgfZff37YvK12q8U82A= endpoint: 67.205.143.140:51821 allowed ips: 10.43.138.5/32, fd00:eb1b:f9f8:d366::5/128 latest handshake: 1 minute, 11 seconds ago transfer: 30.68 KiB received, 30.70 KiB sent persistent keepalive: every 20 seconds peer: auGWbflM1eaVpWYhJi3qj7HeXOpjte80UtcCt4WIDh0= endpoint: 24.144.84.150:51821 allowed ips: 10.43.138.7/32, fd00:eb1b:f9f8:d366::7/128 latest handshake: 1 minute, 15 seconds ago transfer: 30.74 KiB received, 31.21 KiB sent persistent keepalive: every 20 seconds peer: 4uqSOITLlHgnctfbT0uVKSyGDiNt/X3w4sFCI+a2c0M= endpoint: 165.22.115.179:51821 allowed ips: 10.43.138.1/32, fd00:eb1b:f9f8:d366::1/128 latest handshake: 1 minute, 18 seconds ago transfer: 30.84 KiB received, 31.25 KiB sent persistent keepalive: every 20 seconds peer: X3IJPQgEbE240XBPiL9TTdu+oJT3S8LPkVenzjROmwo= endpoint: 54.91.197.18:51821 allowed ips: 10.43.138.6/32, fd00:eb1b:f9f8:d366::6/128 latest handshake: 1 minute, 18 seconds ago transfer: 29.87 KiB received, 30.54 KiB sent persistent keepalive: every 20 seconds peer: /CIeScKzKqbTz1mRqrAxTX6lO7zPtTQRQE1BfphKARA= endpoint: 159.223.160.241:51821 allowed ips: 10.43.138.3/32, fd00:eb1b:f9f8:d366::3/128 latest handshake: 1 minute, 18 seconds ago transfer: 30.20 KiB received, 30.50 KiB sent persistent keepalive: every 20 seconds ipv4-test wg interface -------------------- interface: netmaker public key: 12T/bwQRxnB6yattNaotXgUk4pCmO6vHJAwF5Tfk1ys= private key: (hidden) listening port: 51821 checking logs for errors after a restart clientQA log errors -------------------- arm-redhat log errors -------------------- userspace-wireguard log errors -------------------- clientqa02 log errors -------------------- clientqa03 log errors -------------------- clientqa04 log errors -------------------- ipv4-test log errors -------------------- Apr 24 14:45:31 ipv4-test netclient[138860]: {"time":"2024-04-24T14:45:31.422704344Z","level":"ERROR","source":"daemon.go 212}","msg":"fail to pull config from server","error":"server config not found"}