173 lines
8.7 KiB
Markdown
173 lines
8.7 KiB
Markdown
# Runtime Diagnostics and Network Recovery Configuration
|
|
|
|
This document describes the build-time configuration switches used for runtime diagnostics and automatic network recovery. All options are available under `Example Configuration` in `idf.py menuconfig`.
|
|
|
|
These are compile-time options. Rebuild and flash the firmware after changing them.
|
|
|
|
## Quick reference
|
|
|
|
| Kconfig symbol | Default in `sdkconfig.defaults` | Category | Effect |
|
|
| --- | --- | --- | --- |
|
|
| `CONFIG_TQ_TASK_MONITOR_ENABLE` | Disabled | Diagnostics | Periodically prints a compact FreeRTOS task table. |
|
|
| `CONFIG_TQ_MEMORY_MONITOR_ENABLE` | Enabled | Diagnostics | Samples internal RAM, DMA-capable RAM, PSRAM, fragmentation, and allocation failures. |
|
|
| `CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE` | Enabled | Diagnostics | Collects detailed DNS, Wi-Fi, heap, ping, and cached-endpoint TCP probe information after DNS failures. |
|
|
| `CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE` | Enabled | Diagnostics | Logs multimodal session generations, transport cleanup stages, and the automatic-reassociation failure streak. |
|
|
| `CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS` | Disabled | Diagnostics | Logs microphone input levels and uplink queue/send statistics. |
|
|
| `CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE` | Enabled | Functional recovery | Disconnects and reconnects the Wi-Fi station after the first multimodal HTTPS connection failure. |
|
|
|
|
## Task monitor
|
|
|
|
### `CONFIG_TQ_TASK_MONITOR_ENABLE`
|
|
|
|
Enables periodic FreeRTOS task snapshots. It is useful for finding task leaks, stack pressure, and workers that fail to exit.
|
|
|
|
Related options:
|
|
|
|
- `CONFIG_TQ_TASK_MONITOR_INTERVAL_MS`: reporting interval.
|
|
- `CONFIG_TQ_TASK_MONITOR_MAX_TASKS`: maximum number of task entries captured per report.
|
|
|
|
Runtime impact: periodic task enumeration, formatting, and log output. Keep disabled in normal production builds unless task lifecycle data is required.
|
|
|
|
## Memory watermark monitor
|
|
|
|
### `CONFIG_TQ_MEMORY_MONITOR_ENABLE`
|
|
|
|
Enables the runtime heap watermark monitor and the failed-allocation callback. Reports include:
|
|
|
|
- Internal 8-bit RAM free/minimum/largest block.
|
|
- DMA-capable RAM free/minimum/largest block.
|
|
- PSRAM free/minimum/largest block.
|
|
- Allocated and free block counts.
|
|
- Contiguous-memory percentage.
|
|
- Current PTT, WebSocket, uplink, and local wake state.
|
|
- Allocation failure count and most recent failed allocation.
|
|
|
|
Related options:
|
|
|
|
- `CONFIG_TQ_MEMORY_MONITOR_SAMPLE_INTERVAL_MS`: heap sampling interval.
|
|
- `CONFIG_TQ_MEMORY_MONITOR_REPORT_INTERVAL_MS`: normal periodic report interval.
|
|
- `CONFIG_TQ_MEMORY_MONITOR_INTERNAL_WARN_BYTES`: internal-memory pressure threshold.
|
|
- `CONFIG_TQ_MEMORY_MONITOR_DMA_LARGEST_WARN_BYTES`: largest DMA block warning threshold.
|
|
|
|
Runtime impact: periodic heap queries, state snapshots, and log output. Disable after long-duration memory stability has been established.
|
|
|
|
## Network diagnostics
|
|
|
|
### `CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE`
|
|
|
|
Enables detailed diagnostics when HTTPS DNS resolution fails. The diagnostic bundle records:
|
|
|
|
- STA address, netmask, gateway, DHCP state, RSSI, and DNS servers.
|
|
- Internal and DMA heap state.
|
|
- One ICMP probe to the gateway.
|
|
- One ICMP probe to the fallback DNS server when available.
|
|
- One TCP connection probe to the last successfully resolved HTTPS endpoint.
|
|
- DNS lookup identifiers, concurrent lookup count, failure count, and lookup duration.
|
|
|
|
When disabled, the diagnostic functions, probe state, and direct `lwip` diagnostic dependency are compiled out. Normal DNS retry and HTTPS error handling remain active.
|
|
|
|
Runtime impact when enabled: a failed DNS attempt can spend additional time running bounded ping and TCP probes, create a temporary semaphore and socket, and generate multiple warning logs. Use during network investigation and disable in production when the extra failure-path latency is undesirable.
|
|
|
|
## Multimodal session diagnostics
|
|
|
|
### `CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE`
|
|
|
|
Enables additional logs for:
|
|
|
|
- Multimodal session generation numbers.
|
|
- Waiting for stale transport cleanup before authentication.
|
|
- Completion or failure of the cleanup stage.
|
|
- Consecutive HTTPS failure count used by automatic Wi-Fi reassociation, when that recovery option is enabled.
|
|
|
|
Disabling this option does not disable transport cleanup, downstream stall recovery, WebSocket retirement, or error warnings required to understand a functional failure.
|
|
|
|
Runtime impact: small counter updates and extra log formatting/output. It does not initiate network recovery by itself.
|
|
|
|
## Microphone uplink level diagnostics
|
|
|
|
### `CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS`
|
|
|
|
Enables periodic microphone and uplink statistics, including raw channel level, selected uplink channel level, queued packets, sent bytes, drops, and send failures.
|
|
|
|
Runtime impact: periodic audio-level calculations and verbose logging on the real-time audio path. Keep disabled unless investigating capture mapping, gain, clipping, silence, or uplink congestion.
|
|
|
|
## Automatic Wi-Fi reassociation
|
|
|
|
### `CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE`
|
|
|
|
This is a functional recovery option, not a diagnostic-only option.
|
|
|
|
When enabled, the first multimodal HTTPS connection failure requests controlled STA reassociation. The Wi-Fi state worker then:
|
|
|
|
1. Keeps the device awake.
|
|
2. Disconnects the Wi-Fi station.
|
|
3. Requests reconnection using saved credentials.
|
|
4. Waits for the bounded Wi-Fi reconnect timeout.
|
|
5. Restarts multimodal connection scheduling after successful reassociation.
|
|
|
|
When disabled, HTTPS and WebSocket retry behavior remains active, but the firmware does not intentionally disconnect a Wi-Fi connection that is still reported as associated.
|
|
|
|
Runtime impact when enabled: all device network traffic is interrupted during reassociation, including WebSocket, HTTPS, LAN HTTP, and mDNS traffic. It is enabled by default because field testing demonstrated a stale STA data path that repeated transport-level reconnects could not recover. Disable it only when deliberate reassociation is unacceptable for a deployment.
|
|
|
|
The HTTPS preflight performs one application-level DNS lookup. lwIP already
|
|
retransmits DNS queries internally, and the captured failure took about 14
|
|
seconds before `getaddrinfo()` returned. A second application-level lookup only
|
|
duplicated the wait and did not recover the stale data path.
|
|
|
|
### Field validation: stale associated STA recovery
|
|
|
|
The 2026-08-12 device log captured the complete failure and recovery chain:
|
|
|
|
1. Downstream TTS audio stopped arriving and the three-second playback watchdog
|
|
retired the unresponsive WebSocket after its bounded Stop acknowledgement
|
|
also timed out.
|
|
2. The token HTTPS preflight failed to resolve `gz.api.talkingq.com`. The STA
|
|
still reported an IP address, DHCP state, strong RSSI (`-38` to `-41 dBm`),
|
|
and configured DNS servers.
|
|
3. The gateway and fallback-DNS probes received no reply. More importantly, a
|
|
TCP probe to the last cached server address also timed out, proving that the
|
|
failure was not limited to name resolution.
|
|
4. With the old policy, two DNS attempts were made during each of two complete
|
|
multimodal start attempts. About 69.2 seconds elapsed from retiring the old
|
|
transport to connecting the replacement WebSocket; nearly 60 seconds were
|
|
spent waiting for repeated DNS failures.
|
|
5. After the second failed start, controlled STA reassociation completed in
|
|
about 1.25 seconds. The next DNS lookup succeeded in 296 ms, subsequent
|
|
lookups completed in 7 ms or less, authentication succeeded, and the new
|
|
WebSocket connected about 5.7 seconds later.
|
|
6. A subsequent local wake completed normally through listening, uplink ASR,
|
|
and downstream TTS, validating recovery without rebooting the device.
|
|
|
|
This evidence supports both default reassociation and earlier recovery. With
|
|
one application-level DNS attempt and reassociation after the first failed
|
|
multimodal HTTPS start, the expected recovery time for the same fault is about
|
|
20 to 25 seconds rather than about 69 seconds. This estimate still requires a
|
|
device run to confirm because the lwIP DNS timeout dominates the remaining
|
|
delay.
|
|
|
|
## Recommended profiles
|
|
|
|
### Stability investigation
|
|
|
|
```text
|
|
CONFIG_TQ_MEMORY_MONITOR_ENABLE=y
|
|
CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE=y
|
|
CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE=y
|
|
# CONFIG_TQ_TASK_MONITOR_ENABLE is not set
|
|
# CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS is not set
|
|
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE=y
|
|
```
|
|
|
|
### Production candidate
|
|
|
|
```text
|
|
# CONFIG_TQ_MEMORY_MONITOR_ENABLE is not set
|
|
# CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE is not set
|
|
# CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE is not set
|
|
# CONFIG_TQ_TASK_MONITOR_ENABLE is not set
|
|
# CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS is not set
|
|
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE=y
|
|
```
|
|
|
|
Functional warnings and normal connection recovery remain available in the production candidate profile.
|