chore: archive diagnostics and project documentation
This commit is contained in:
chenjintao
@@ -0,0 +1,172 @@
|
||||
# Runtime Diagnostics and Network Recovery Configuration
|
||||
|
||||
This document describes the build-time configuration switches used for runtime diagnostics and automatic network recovery. All options are available under `Example Configuration` in `idf.py menuconfig`.
|
||||
|
||||
These are compile-time options. Rebuild and flash the firmware after changing them.
|
||||
|
||||
## Quick reference
|
||||
|
||||
| Kconfig symbol | Default in `sdkconfig.defaults` | Category | Effect |
|
||||
| --- | --- | --- | --- |
|
||||
| `CONFIG_TQ_TASK_MONITOR_ENABLE` | Disabled | Diagnostics | Periodically prints a compact FreeRTOS task table. |
|
||||
| `CONFIG_TQ_MEMORY_MONITOR_ENABLE` | Enabled | Diagnostics | Samples internal RAM, DMA-capable RAM, PSRAM, fragmentation, and allocation failures. |
|
||||
| `CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE` | Enabled | Diagnostics | Collects detailed DNS, Wi-Fi, heap, ping, and cached-endpoint TCP probe information after DNS failures. |
|
||||
| `CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE` | Enabled | Diagnostics | Logs multimodal session generations, transport cleanup stages, and the automatic-reassociation failure streak. |
|
||||
| `CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS` | Disabled | Diagnostics | Logs microphone input levels and uplink queue/send statistics. |
|
||||
| `CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE` | Enabled | Functional recovery | Disconnects and reconnects the Wi-Fi station after the first multimodal HTTPS connection failure. |
|
||||
|
||||
## Task monitor
|
||||
|
||||
### `CONFIG_TQ_TASK_MONITOR_ENABLE`
|
||||
|
||||
Enables periodic FreeRTOS task snapshots. It is useful for finding task leaks, stack pressure, and workers that fail to exit.
|
||||
|
||||
Related options:
|
||||
|
||||
- `CONFIG_TQ_TASK_MONITOR_INTERVAL_MS`: reporting interval.
|
||||
- `CONFIG_TQ_TASK_MONITOR_MAX_TASKS`: maximum number of task entries captured per report.
|
||||
|
||||
Runtime impact: periodic task enumeration, formatting, and log output. Keep disabled in normal production builds unless task lifecycle data is required.
|
||||
|
||||
## Memory watermark monitor
|
||||
|
||||
### `CONFIG_TQ_MEMORY_MONITOR_ENABLE`
|
||||
|
||||
Enables the runtime heap watermark monitor and the failed-allocation callback. Reports include:
|
||||
|
||||
- Internal 8-bit RAM free/minimum/largest block.
|
||||
- DMA-capable RAM free/minimum/largest block.
|
||||
- PSRAM free/minimum/largest block.
|
||||
- Allocated and free block counts.
|
||||
- Contiguous-memory percentage.
|
||||
- Current PTT, WebSocket, uplink, and local wake state.
|
||||
- Allocation failure count and most recent failed allocation.
|
||||
|
||||
Related options:
|
||||
|
||||
- `CONFIG_TQ_MEMORY_MONITOR_SAMPLE_INTERVAL_MS`: heap sampling interval.
|
||||
- `CONFIG_TQ_MEMORY_MONITOR_REPORT_INTERVAL_MS`: normal periodic report interval.
|
||||
- `CONFIG_TQ_MEMORY_MONITOR_INTERNAL_WARN_BYTES`: internal-memory pressure threshold.
|
||||
- `CONFIG_TQ_MEMORY_MONITOR_DMA_LARGEST_WARN_BYTES`: largest DMA block warning threshold.
|
||||
|
||||
Runtime impact: periodic heap queries, state snapshots, and log output. Disable after long-duration memory stability has been established.
|
||||
|
||||
## Network diagnostics
|
||||
|
||||
### `CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE`
|
||||
|
||||
Enables detailed diagnostics when HTTPS DNS resolution fails. The diagnostic bundle records:
|
||||
|
||||
- STA address, netmask, gateway, DHCP state, RSSI, and DNS servers.
|
||||
- Internal and DMA heap state.
|
||||
- One ICMP probe to the gateway.
|
||||
- One ICMP probe to the fallback DNS server when available.
|
||||
- One TCP connection probe to the last successfully resolved HTTPS endpoint.
|
||||
- DNS lookup identifiers, concurrent lookup count, failure count, and lookup duration.
|
||||
|
||||
When disabled, the diagnostic functions, probe state, and direct `lwip` diagnostic dependency are compiled out. Normal DNS retry and HTTPS error handling remain active.
|
||||
|
||||
Runtime impact when enabled: a failed DNS attempt can spend additional time running bounded ping and TCP probes, create a temporary semaphore and socket, and generate multiple warning logs. Use during network investigation and disable in production when the extra failure-path latency is undesirable.
|
||||
|
||||
## Multimodal session diagnostics
|
||||
|
||||
### `CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE`
|
||||
|
||||
Enables additional logs for:
|
||||
|
||||
- Multimodal session generation numbers.
|
||||
- Waiting for stale transport cleanup before authentication.
|
||||
- Completion or failure of the cleanup stage.
|
||||
- Consecutive HTTPS failure count used by automatic Wi-Fi reassociation, when that recovery option is enabled.
|
||||
|
||||
Disabling this option does not disable transport cleanup, downstream stall recovery, WebSocket retirement, or error warnings required to understand a functional failure.
|
||||
|
||||
Runtime impact: small counter updates and extra log formatting/output. It does not initiate network recovery by itself.
|
||||
|
||||
## Microphone uplink level diagnostics
|
||||
|
||||
### `CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS`
|
||||
|
||||
Enables periodic microphone and uplink statistics, including raw channel level, selected uplink channel level, queued packets, sent bytes, drops, and send failures.
|
||||
|
||||
Runtime impact: periodic audio-level calculations and verbose logging on the real-time audio path. Keep disabled unless investigating capture mapping, gain, clipping, silence, or uplink congestion.
|
||||
|
||||
## Automatic Wi-Fi reassociation
|
||||
|
||||
### `CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE`
|
||||
|
||||
This is a functional recovery option, not a diagnostic-only option.
|
||||
|
||||
When enabled, the first multimodal HTTPS connection failure requests controlled STA reassociation. The Wi-Fi state worker then:
|
||||
|
||||
1. Keeps the device awake.
|
||||
2. Disconnects the Wi-Fi station.
|
||||
3. Requests reconnection using saved credentials.
|
||||
4. Waits for the bounded Wi-Fi reconnect timeout.
|
||||
5. Restarts multimodal connection scheduling after successful reassociation.
|
||||
|
||||
When disabled, HTTPS and WebSocket retry behavior remains active, but the firmware does not intentionally disconnect a Wi-Fi connection that is still reported as associated.
|
||||
|
||||
Runtime impact when enabled: all device network traffic is interrupted during reassociation, including WebSocket, HTTPS, remote logging, LAN HTTP, and mDNS traffic. It is enabled by default because field testing demonstrated a stale STA data path that repeated transport-level reconnects could not recover. Disable it only when deliberate reassociation is unacceptable for a deployment.
|
||||
|
||||
The HTTPS preflight performs one application-level DNS lookup. lwIP already
|
||||
retransmits DNS queries internally, and the captured failure took about 14
|
||||
seconds before `getaddrinfo()` returned. A second application-level lookup only
|
||||
duplicated the wait and did not recover the stale data path.
|
||||
|
||||
### Field validation: stale associated STA recovery
|
||||
|
||||
The 2026-08-12 device log captured the complete failure and recovery chain:
|
||||
|
||||
1. Downstream TTS audio stopped arriving and the three-second playback watchdog
|
||||
retired the unresponsive WebSocket after its bounded Stop acknowledgement
|
||||
also timed out.
|
||||
2. The token HTTPS preflight failed to resolve `gz.api.talkingq.com`. The STA
|
||||
still reported an IP address, DHCP state, strong RSSI (`-38` to `-41 dBm`),
|
||||
and configured DNS servers.
|
||||
3. The gateway and fallback-DNS probes received no reply. More importantly, a
|
||||
TCP probe to the last cached server address also timed out, proving that the
|
||||
failure was not limited to name resolution.
|
||||
4. With the old policy, two DNS attempts were made during each of two complete
|
||||
multimodal start attempts. About 69.2 seconds elapsed from retiring the old
|
||||
transport to connecting the replacement WebSocket; nearly 60 seconds were
|
||||
spent waiting for repeated DNS failures.
|
||||
5. After the second failed start, controlled STA reassociation completed in
|
||||
about 1.25 seconds. The next DNS lookup succeeded in 296 ms, subsequent
|
||||
lookups completed in 7 ms or less, authentication succeeded, and the new
|
||||
WebSocket connected about 5.7 seconds later.
|
||||
6. A subsequent local wake completed normally through listening, uplink ASR,
|
||||
and downstream TTS, validating recovery without rebooting the device.
|
||||
|
||||
This evidence supports both default reassociation and earlier recovery. With
|
||||
one application-level DNS attempt and reassociation after the first failed
|
||||
multimodal HTTPS start, the expected recovery time for the same fault is about
|
||||
20 to 25 seconds rather than about 69 seconds. This estimate still requires a
|
||||
device run to confirm because the lwIP DNS timeout dominates the remaining
|
||||
delay.
|
||||
|
||||
## Recommended profiles
|
||||
|
||||
### Stability investigation
|
||||
|
||||
```text
|
||||
CONFIG_TQ_MEMORY_MONITOR_ENABLE=y
|
||||
CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE=y
|
||||
CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE=y
|
||||
# CONFIG_TQ_TASK_MONITOR_ENABLE is not set
|
||||
# CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS is not set
|
||||
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE=y
|
||||
```
|
||||
|
||||
### Production candidate
|
||||
|
||||
```text
|
||||
# CONFIG_TQ_MEMORY_MONITOR_ENABLE is not set
|
||||
# CONFIG_TQ_NETWORK_DIAGNOSTICS_ENABLE is not set
|
||||
# CONFIG_TQ_MULTIMODAL_DIAGNOSTICS_ENABLE is not set
|
||||
# CONFIG_TQ_TASK_MONITOR_ENABLE is not set
|
||||
# CONFIG_MULTIMODAL_AUDIO_LOG_UPLINK_LEVELS is not set
|
||||
CONFIG_TQ_NETWORK_AUTO_REASSOCIATION_ENABLE=y
|
||||
```
|
||||
|
||||
Functional warnings and normal connection recovery remain available in the production candidate profile.
|
||||
Reference in New Issue
Block a user