WiFi unstable when connecting to public MQTT server on default channel

[stylesuxx]

Category

WiFi

Hardware

T-Beam

Firmware Version

#3026

Description

Follow up to #2796

After flashing artifact from #3026 WiFi is now reconnecting correctly after some time, but it only does work for a couple of seconds. I verified this by pinging the device, also DNS resolution fails when connecting to the MQTT server as can be seen from the serial log:

DEBUG | 12:05:58 50540 [Power] Battery: usbPower=1, isCharging=0, batMv=4182, batPct=100
INFO  | 12:06:11 50553 [mqtt] Using non-TLS-encrypted session
INFO  | 12:06:11 50553 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[50556820][I][WiFiClient.cpp:253] connect(): select returned due to timeout 3000 ms for fd 52
ERROR | 12:06:14 50556 [mqtt] Failed to contact MQTT server directly (1/5)...
DEBUG | 12:06:18 50560 [Power] Battery: usbPower=1, isCharging=0, batMv=4182, batPct=100

---SNIP---

DEBUG | 12:08:35 50697 [Power] Battery: usbPower=1, isCharging=0, batMv=4183, batPct=100
INFO  | 12:08:45 50707 [mqtt] Using non-TLS-encrypted session
INFO  | 12:08:45 50707 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[50721932][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for mqtt.meshtastic.org
ERROR | 12:08:59 50721 [mqtt] Failed to contact MQTT server directly (5/5)...
DEBUG | 12:08:59 50721 [Power] Battery: usbPower=1, isCharging=0, batMv=4182, batPct=100
INFO  | 12:08:59 50721 [WifiConnect] Reconnecting to WiFi access point Takbir
[50721980][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 5 - STA_DISCONNECTED
[50721984][W][WiFiGeneric.cpp:1057] _eventCallback(): Reason: 8 - ASSOC_LEAVE
DEBUG | 12:08:59 50721 [WifiConnect] WiFi-Event 5: Disconnected from WiFi access point
WARN  | 12:08:59 50722 [WifiConnect] WiFi lost connection. Reason: 8
[50727751][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 4 - STA_CONNECTED
DEBUG | 12:09:05 50727 [WebServerThread] WiFi-Event 4: Connected to access point
[50735300][D][WiFiGeneric.cpp:1035] _eventCallback(): Arduino Event: 7 - STA_GOT_IP
[50735301][D][WiFiGeneric.cpp:1098] _eventCallback(): STA IP: 192.168.1.35, MASK: 255.255.255.0, GW: 192.168.1.1
DEBUG | 12:09:12 50735 WiFi-Event 7: Obtained IP address: 192.168.1.35
INFO  | 12:09:12 50735 Using non-TLS-encrypted session
INFO  | 12:09:12 50735 Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: xxxxxxx
INFO  | 12:09:13 50735 MQTT connected

This is the approximate ping section belonging to the above serial log

From 192.168.1.3 icmp_seq=252 Destination Host Unreachable
From 192.168.1.3 icmp_seq=253 Destination Host Unreachable
From 192.168.1.3 icmp_seq=254 Destination Host Unreachable
64 bytes from 192.168.1.35: icmp_seq=255 ttl=255 time=2505 ms
64 bytes from 192.168.1.35: icmp_seq=256 ttl=255 time=1483 ms
64 bytes from 192.168.1.35: icmp_seq=257 ttl=255 time=459 ms
64 bytes from 192.168.1.35: icmp_seq=258 ttl=255 time=6.48 ms
From 192.168.1.3 icmp_seq=266 Destination Host Unreachable
From 192.168.1.3 icmp_seq=267 Destination Host Unreachable
From 192.168.1.3 icmp_seq=268 Destination Host Unreachable

Relevant log output

No response

[GUVWAF]

Can you try with a static IP and set a DNS server (e.g. 8.8.8.8) yourself? Seems there is an issue with the DNS server your DHCP server is giving to the device. It then eventually get disconnected when connecting to the MQTT server fails.

[stylesuxx]

I tried setting up static details, unfortunately I don't fully understand how to do so. I can change address_mode but how am I supposed to set network.ipv4_config? If I try to do it via web UI, it does not seem to take the settings and returns 0 for all fields. I could only find this thread of another person not being able to set it either...

[GUVWAF]

Do you have an Android or Apple device? You should be able to set it with either of those apps.

[stylesuxx]

Thank you, setting static details via Android worked fine. Unfortunately the error still persists, connection does only work for a very limited time after re-connection:

INFO  | 15:10:38 630 [mqtt] Using non-TLS-encrypted session
INFO  | 15:10:38 630 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[633020][I][WiFiClient.cpp:253] connect(): select returned due to timeout 3000 ms for fd 52
ERROR | 15:10:41 633 [mqtt] Failed to contact MQTT server directly (2/5)...
DEBUG | 15:10:51 642 [Power] Battery: usbPower=1, isCharging=0, batMv=4184, batPct=100
INFO  | 15:11:08 660 [DeviceTelemetryModule] (Sending): air_util_tx=2.457250, channel_utilization=0.000000, battery_level=100, voltage=4.184000
DEBUG | 15:11:08 660 [DeviceTelemetryModule] updateTelemetry LOCAL
DEBUG | 15:11:08 660 [DeviceTelemetryModule] Node status update: 70 online, 100 total
INFO  | 15:11:08 660 [DeviceTelemetryModule] Sending packet to phone
INFO  | 15:11:08 660 Telling client we have new packets 152
INFO  | 15:11:08 660 [SerialConsole] getFromRadio=STATE_SEND_PACKETS
DEBUG | 15:11:08 660 [SerialConsole] phone downloaded packet (id=0x45c5cd7b fr=0x94 to=0xff, WantAck=0, HopLim=3 Ch=0x0 Portnum=67 rxtime=1703257868 priority=1)
DEBUG | 15:11:08 660 [SerialConsole] encoding toPhone packet to phone variant=2, 51 bytes
DEBUG | 15:11:11 662 [Power] Battery: usbPower=1, isCharging=0, batMv=4183, batPct=100
INFO  | 15:11:11 663 [mqtt] Using non-TLS-encrypted session
INFO  | 15:11:11 663 [mqtt] Attempting to connect directly to MQTT server mqtt.meshtastic.org, port: 1883, username: meshdev, password: large4cats
[670048][E][WiFiGeneric.cpp:1578] hostByName(): DNS Failed for mqtt.meshtastic.org
ERROR | 15:11:18 670 [mqtt] Failed to contact MQTT server directly (3/5)...
DEBUG | 15:11:31 682 [Power] Battery: usbPower=1, isCharging=0, batMv=4185, batPct=100

Sometimes it's the timeout, sometimes it's the DNS error that hits first. The logic seems to be to restart the WiFi device after 5 failed connection attempts to the MQTT server. The first connection to the MQTT server after restarting the WiFi device works for a couple seconds

[GUVWAF]

Did you try different DNS servers? Maybe it's a firewall issue?

The logic seems to be to restart the WiFi device after 5 failed connection attempts to the MQTT server.

Indeed:

firmware/src/mqtt/MQTT.cpp

Lines 326 to 328 in db8f8db

	if (reconnectCount >= reconnectMax) {
	needReconnect = true;
	wifiReconnect->setIntervalFromNow(0);

So, yeah, if the MQTT connection fails due to DNS failures, it is expected that it tries to reconnect WiFi afterwards.

Edit: Also, you mention the first connection to the MQTT server works for a couple seconds. Are you using the default channel (LongFast)? If so, there's a lot of traffic there and it might be that that is causing problems. You could try changing the channel and see if it stays connected for longer.

[stylesuxx]

I tried 4 different routers, 2 mobile APs, different DNS servers and couple different WAN - I'd say I can exclude any "external" networking issues at this point.

Maybe I was not completely clear - the absolutely first connection after a reset works fine for minutes, up to hours (max was probably around 2 hours). Every subsequent re-connect after the first disconnect will only work for seconds. To me it feels like a buffer filling up and causing issues from there.

Yes, I am using the default channel. I will try on a different channel and see if things change.

Update:
Disabled the primary channel as a base line. WiFi connection was not lost once within 4 hours.

[major-tweaks]

Side comment: As can be seen on the official Discord Channel Announcements,
from User MC Hamster
—
21.12.2023 02:53
MQTT server maintenance tomorrow, Thursday, starting at 9 PM Pacific.

https://discord.com/channels/867578229534359593/871558934916390953

It may be on and offline sporadically.

[stylesuxx]

@major-tweaks thanks for the heads up. I'll probably stop being a pain in the ass over the Christmas holidays anyway.

[major-tweaks]

What I can say, I've got two HelTec V3s ESP32, one via WiFi connected (WebApp), one via BT (iOS App).
The one with WiFi connection to the official MQTT is dead actually at this very moment.
Just turned MQTT on the BT one on, received new nodes and messages (via iOS WiFi Connection), hoping that there are no issues relating to that WiFi connection.. running the latest stable version 2.2.15 beta

[stylesuxx]

Should you run into an infinite WiFi reconnection loop, flash the artifacts of #3026 ;-)

[major-tweaks]

Disabled MQTT on the WiFi connected one, disabled it also on Channel 0, switched off BT also, results in a much much more responsive client.. so there IS actually some issue with WiFi and MQTT.. no problem with just BT and MQTT.

[stylesuxx]

I don't think that the issue is with WiFi and MQTT per say, I think the issue is "just" when you are subscribed to a busy channel. I now disabled the default channel and things are working fine - admittedly with only very little load on that channel. Would be interesting what the "breaking" point is exactly.

[GUVWAF]

Just turned MQTT on the BT one on, received new nodes and messages (via iOS WiFi Connection), hoping that there are no issues relating to that WiFi connection..

Not sure what you mean here. You cannot connect to the iOS app with WiFi, but there is an MQTT proxy via Bluetooth. You also cannot use WiFi and BT simultaneously. If you turn on WiFi, BT is turned off and vice versa.
Edit: Sorry, I now realize you mean the WiFi connection of your iPhone, not the Meshtastic device.

I don't think that the issue is with WiFi and MQTT per say, I think the issue is "just" when you are subscribed to a busy channel. I now disabled the default channel and things are working fine - admittedly with only very little load on that channel.

Yes, when connected to the public MQTT server on the default channel, the device is basically receiving messages non-stop and may not function properly anymore. I have already opened a PR to the documentation to warn users about this (meshtastic/meshtastic#901).

Since there is not really much we can do about this, I'm closing this and converting it to a discussion if you want to discuss further.

[mg212399]

The problem isn't really that wifi disconnects--many things can cause that--but that it never manages to reconnect. This makes unattended MQTT nodes impossible, at least with T-Beams (I don't know whether any other device can manage to reconnect).

If you can't fix the reconnect, can you at least give us an option to reboot the device if it doesn't reconnect within X seconds? That's what I'm going to have to do anyway, so it would save time if it just rebooted itself and I didn't have to go over and press the button.

Also I'm not convinced that the disconnect is load-related. Looking at the device metrics last night the load wasn't particularly high when it disconnected and it had handled much higher loads earlier in the night without problems. But again it wouldn't matter if the device could reconnect after disconnecting.

[GUVWAF]

Which firmware version are you using?
OP mentioned reconnects works again:

After flashing artifact from #3026 WiFi is now reconnecting correctly after some time

This was added in firmware 2.2.17. If you're already using that firmware, it would be good to check what goes wrong still instead of just rebooting.

[mg212399]

I'm on the latest Beta as of a few days ago. 2.2.21?

I'm not sure how to get the serial output, but if you can point me toward instructions I can connect the device to a PC and capture it for you.

Edit: This is a T-Beam Supreme rather than a regular T-Beam, so I don't know whether that would make a difference. It's also possible it's showing the same behaviour of reconnecting for a few seconds and disconnecting behind the scenes, but the screen only shows 'Reconnecting'.

[GUVWAF]

You can use the CLI command meshtastic --noproto or any serial monitor like PuTTY or tio on baud 115200.

Would also be good to see your configuration (with meshtastic --info) after removing private details.

[mg212399]

OK, I'll try Putty. I don't have the meshtastic software installed, I've just been using the web interface. But I can investigate installing that.

[mg212399]

I'll have to trim the logs and upload them for you but it looks like it reconnects a few times but those connections only last a few minutes, then it finally reconnects but doesn't get an IP address and just sits there forever saying Reconnecting.

[mg212399]

Here's a trimmed file with the beginning and end of the log. This appears to be the last connection:

DEBUG | 23:39:33 18478 WiFi-Event 4: Connected to access point

Then nothing. Usually the next message says it got an IP address, but this time there were no WiFi-Event messages at all after that.
3031-trimmed.txt

I have the full log if you need it, but it's about 50MB.

[mg212399]

Here's just the wifi messages:

wifi.txt

[GUVWAF]

It seems your connection is very unstable. ESP32 only supports 2.4GHz Wi-Fi and it generally needs to be quite close to the AP. If it’s in hybrid 2.4/5GHz mode it may work intermittently. Try placing it close to the AP and make sure it's a 2.4GHz network only.
You can also try to set a static IP.

[mg212399]

The Guest network is 2.4GHz.

I moved the T-Beam so the wifi antenna on the T-Beam was about three inches from the router's wifi antenna and while it survived a few hours longer it failed in exactly the same way. Lots of fairly short-lived reconnects followed by a reconnect where it doesn't receive an IP address and never tries to resolve that by requesting it again or disconnecting.

Also that's the thrid wifi router I've tried so it doesn't seem to be a problem at the router end.

[GUVWAF]

Can you try setting a static IP?
If that doesn't work, you could open a new issue with as many details as you can (serial logs, meshtastic --info as well as your WiFi settings) and hope someone can help. I don’t have a T-Beam Supreme and don’t have issues with reconnecting on a normal T-Beam so it’s hard to help further.