8.7 KiB
Runtime Diagnostics and Network Recovery Configuration
This document describes the build-time configuration switches used for runtime diagnostics and automatic network recovery. All options are available under Example Configuration in idf.py menuconfig.
These are compile-time options. Rebuild and flash the firmware after changing them.
Quick reference
| Kconfig symbol | Default in sdkconfig.defaults |
Category | Effect |
|---|---|---|---|
CONFIG_TQ_TASK_MONITOR_ENABLE |
Disabled | Diagnostics | Periodically prints a compact FreeRTOS task table. |
CONFIG_TQ_MEMORY_MONITOR_ENABLE |
Enabled | Diagnostics | Samples internal RAM, DMA-capable RAM, PSRAM, fragmentation, and allocation failures. |
CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE |
Enabled | Diagnostics | Collects detailed DNS, Wi-Fi, heap, ping, and cached-endpoint TCP probe information after DNS failures. |
CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE |
Enabled | Diagnostics | Logs multimodal session generations, transport cleanup stages, and the automatic-reassociation failure streak. |
CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS |
Disabled | Diagnostics | Logs microphone input levels and uplink queue/send statistics. |
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE |
Enabled | Functional recovery | Disconnects and reconnects the Wi-Fi station after the first multimodal HTTPS connection failure. |
Task monitor
CONFIG_TQ_TASK_MONITOR_ENABLE
Enables periodic FreeRTOS task snapshots. It is useful for finding task leaks, stack pressure, and workers that fail to exit.
Related options:
CONFIG_TQ_TASK_MONITOR_INTERVAL_MS: reporting interval.CONFIG_TQ_TASK_MONITOR_MAX_TASKS: maximum number of task entries captured per report.
Runtime impact: periodic task enumeration, formatting, and log output. Keep disabled in normal production builds unless task lifecycle data is required.
Memory watermark monitor
CONFIG_TQ_MEMORY_MONITOR_ENABLE
Enables the runtime heap watermark monitor and the failed-allocation callback. Reports include:
- Internal 8-bit RAM free/minimum/largest block.
- DMA-capable RAM free/minimum/largest block.
- PSRAM free/minimum/largest block.
- Allocated and free block counts.
- Contiguous-memory percentage.
- Current PTT, WebSocket, uplink, and local wake state.
- Allocation failure count and most recent failed allocation.
Related options:
CONFIG_TQ_MEMORY_MONITOR_SAMPLE_INTERVAL_MS: heap sampling interval.CONFIG_TQ_MEMORY_MONITOR_REPORT_INTERVAL_MS: normal periodic report interval.CONFIG_TQ_MEMORY_MONITOR_INTERNAL_WARN_BYTES: internal-memory pressure threshold.CONFIG_TQ_MEMORY_MONITOR_DMA_LARGEST_WARN_BYTES: largest DMA block warning threshold.
Runtime impact: periodic heap queries, state snapshots, and log output. Disable after long-duration memory stability has been established.
Network diagnostics
CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE
Enables detailed diagnostics when HTTPS DNS resolution fails. The diagnostic bundle records:
- STA address, netmask, gateway, DHCP state, RSSI, and DNS servers.
- Internal and DMA heap state.
- One ICMP probe to the gateway.
- One ICMP probe to the fallback DNS server when available.
- One TCP connection probe to the last successfully resolved HTTPS endpoint.
- DNS lookup identifiers, concurrent lookup count, failure count, and lookup duration.
When disabled, the diagnostic functions, probe state, and direct lwip diagnostic dependency are compiled out. Normal DNS retry and HTTPS error handling remain active.
Runtime impact when enabled: a failed DNS attempt can spend additional time running bounded ping and TCP probes, create a temporary semaphore and socket, and generate multiple warning logs. Use during network investigation and disable in production when the extra failure-path latency is undesirable.
Multimodal session diagnostics
CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE
Enables additional logs for:
- Multimodal session generation numbers.
- Waiting for stale transport cleanup before authentication.
- Completion or failure of the cleanup stage.
- Consecutive HTTPS failure count used by automatic Wi-Fi reassociation, when that recovery option is enabled.
Disabling this option does not disable transport cleanup, downstream stall recovery, WebSocket retirement, or error warnings required to understand a functional failure.
Runtime impact: small counter updates and extra log formatting/output. It does not initiate network recovery by itself.
Microphone uplink level diagnostics
CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS
Enables periodic microphone and uplink statistics, including raw channel level, selected uplink channel level, queued packets, sent bytes, drops, and send failures.
Runtime impact: periodic audio-level calculations and verbose logging on the real-time audio path. Keep disabled unless investigating capture mapping, gain, clipping, silence, or uplink congestion.
Automatic Wi-Fi reassociation
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE
This is a functional recovery option, not a diagnostic-only option.
When enabled, the first multimodal HTTPS connection failure requests controlled STA reassociation. The Wi-Fi state worker then:
- Keeps the device awake.
- Disconnects the Wi-Fi station.
- Requests reconnection using saved credentials.
- Waits for the bounded Wi-Fi reconnect timeout.
- Restarts multimodal connection scheduling after successful reassociation.
When disabled, HTTPS and WebSocket retry behavior remains active, but the firmware does not intentionally disconnect a Wi-Fi connection that is still reported as associated.
Runtime impact when enabled: all device network traffic is interrupted during reassociation, including WebSocket, HTTPS, LAN HTTP, and mDNS traffic. It is enabled by default because field testing demonstrated a stale STA data path that repeated transport-level reconnects could not recover. Disable it only when deliberate reassociation is unacceptable for a deployment.
The HTTPS preflight performs one application-level DNS lookup. lwIP already
retransmits DNS queries internally, and the captured failure took about 14
seconds before getaddrinfo() returned. A second application-level lookup only
duplicated the wait and did not recover the stale data path.
Field validation: stale associated STA recovery
The 2026-08-12 device log captured the complete failure and recovery chain:
- Downstream TTS audio stopped arriving and the three-second playback watchdog retired the unresponsive WebSocket after its bounded Stop acknowledgement also timed out.
- The token HTTPS preflight failed to resolve
gz.api.talkingq.com. The STA still reported an IP address, DHCP state, strong RSSI (-38to-41 dBm), and configured DNS servers. - The gateway and fallback-DNS probes received no reply. More importantly, a TCP probe to the last cached server address also timed out, proving that the failure was not limited to name resolution.
- With the old policy, two DNS attempts were made during each of two complete multimodal start attempts. About 69.2 seconds elapsed from retiring the old transport to connecting the replacement WebSocket; nearly 60 seconds were spent waiting for repeated DNS failures.
- After the second failed start, controlled STA reassociation completed in about 1.25 seconds. The next DNS lookup succeeded in 296 ms, subsequent lookups completed in 7 ms or less, authentication succeeded, and the new WebSocket connected about 5.7 seconds later.
- A subsequent local wake completed normally through listening, uplink ASR, and downstream TTS, validating recovery without rebooting the device.
This evidence supports both default reassociation and earlier recovery. With one application-level DNS attempt and reassociation after the first failed multimodal HTTPS start, the expected recovery time for the same fault is about 20 to 25 seconds rather than about 69 seconds. This estimate still requires a device run to confirm because the lwIP DNS timeout dominates the remaining delay.
Recommended profiles
Stability investigation
CONFIG_TQ_MEMORY_MONITOR_ENABLE=y
CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE=y
CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE=y
# CONFIG_TQ_TASK_MONITOR_ENABLE is not set
# CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS is not set
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE=y
Production candidate
# CONFIG_TQ_MEMORY_MONITOR_ENABLE is not set
# CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE is not set
# CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE is not set
# CONFIG_TQ_TASK_MONITOR_ENABLE is not set
# CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS is not set
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE=y
Functional warnings and normal connection recovery remain available in the production candidate profile.