feat(runtime): add integration hardening and diagnostics
This commit is contained in:
@@ -0,0 +1,76 @@
|
||||
# On-device integration testing
|
||||
|
||||
This procedure verifies Milestone 010 on the ESP32-C6FH4 without PSRAM or any
|
||||
assumed GPIO. Run it against production firmware and the production LittleFS
|
||||
image. Start from a freshly booted device with no occupied lobby sessions.
|
||||
|
||||
## Build and delivery
|
||||
|
||||
```sh
|
||||
/home/sasa/.platformio/penv/bin/pio run -e esp32-c6-devkitm-1
|
||||
/home/sasa/.platformio/penv/bin/pio run -e esp32-c6-devkitm-1 --target buildfs
|
||||
/home/sasa/.platformio/penv/bin/pio run -e esp32-c6-devkitm-1 --target upload --upload-port /dev/ttyACM0
|
||||
/home/sasa/.platformio/penv/bin/pio run -e esp32-c6-devkitm-1 --target uploadfs --upload-port /dev/ttyACM0
|
||||
```
|
||||
|
||||
Monitor the USB serial log at 115200 baud. Confirm application version
|
||||
`0.10.0`, the 4 MB partition table, successful LittleFS mount, station or
|
||||
fallback network startup, registration of all HTTP/WebSocket routes, and
|
||||
health state `serving`.
|
||||
|
||||
## Resource and load scenario
|
||||
|
||||
Keep the development host connected to `Mazda7`; do not switch it to the
|
||||
device fallback access point. When the board has station address
|
||||
`192.168.1.50`, run:
|
||||
|
||||
```sh
|
||||
python3 scripts/ws_protocol_client.py 192.168.1.50 \
|
||||
--verify-m010 --stress-moves 80 --repeat-matches 2
|
||||
```
|
||||
|
||||
The scenario intentionally takes more than two minutes because both repeated
|
||||
matches use the real 60-second reconnect deadline. It performs these checks:
|
||||
|
||||
1. Connect two players and four observers and assemble all four complete
|
||||
8,192-cell observer projections.
|
||||
2. Verify heartbeat round-trip latency for all six sessions.
|
||||
3. Stop consuming one observer stream while Player 1 performs accepted moves;
|
||||
intermediate observer updates may be dropped, but player acknowledgements
|
||||
must remain within the reported 250 ms budget.
|
||||
4. Flood a second observer with malformed messages. The fixed rate limiter
|
||||
closes sustained abuse, and the players remain responsive.
|
||||
5. Complete and authoritatively reset two matches through the real disconnect
|
||||
timeout. Each reset generates a new server-owned maze while preserving the
|
||||
connected sessions.
|
||||
6. Fetch `/diagnostics` before, during, and after the run. Assert the connection
|
||||
and queue high-water marks remain within six and twelve, internal free heap
|
||||
stays above 32 KiB, the largest internal block stays above 16 KiB, and the
|
||||
post-reset free-heap samples remain within a 32 KiB range.
|
||||
|
||||
Capture the command output, the final diagnostics JSON, and serial lines that
|
||||
begin with `M010 resources:`. The JSON counters are monotonic since boot and
|
||||
include connections, queue use, dropped messages, failures, malformed/rate-
|
||||
limited input, commands, match cycles, maximum handler/send time, and current
|
||||
and minimum internal heap samples.
|
||||
|
||||
## Station, fallback, and browser recovery
|
||||
|
||||
These checks involve network state and must be performed from a separate phone
|
||||
or computer so the development host retains its Mazda7 internet connection.
|
||||
|
||||
1. Reboot with the saved Mazda7 configuration and confirm
|
||||
`/api/wifi/status` reports `station_connected`. Reload both player pages
|
||||
during an active match and verify their tokens restore the same slots and
|
||||
authoritative state.
|
||||
2. From the separate client, exercise the established provisioning procedure
|
||||
with an intentionally unavailable test network. Confirm the bounded station
|
||||
attempts return the device to `maze32-XXXX` at
|
||||
`http://192.168.4.1/provision` and that the Russian page remains usable.
|
||||
3. Restore Mazda7 through the provisioning page without recording its
|
||||
password. Reboot and confirm station connection, browser recovery, `/health`,
|
||||
and `/diagnostics` again.
|
||||
|
||||
Do not record an acceptance result unless the physical logs and samples are
|
||||
available. Do not use an LED, a board pin, or a PSRAM capability as a test
|
||||
oracle.
|
||||
@@ -37,6 +37,9 @@ is only echoed for correlation; it grants no authority.
|
||||
unpaused match may submit movement intent.
|
||||
- `drop_key`: `{}`. Only a player in an active, unpaused match may submit the
|
||||
key-drop intent.
|
||||
- `restart`: `{}`. Only connected Player 1 may reset a finished match, and both
|
||||
players must be connected. Sessions are preserved while a new authoritative
|
||||
maze and game state are created.
|
||||
- `heartbeat`: `{}` after a session has been joined or restored.
|
||||
|
||||
## Server messages
|
||||
@@ -57,6 +60,8 @@ is only echoed for correlation; it grants no authority.
|
||||
- `resume`: reports that both players are attached and authoritative gameplay
|
||||
has resumed.
|
||||
- `match_result`: reports the terminal winner or draw.
|
||||
- `reset`: reports the authoritative return from a finished match to the
|
||||
waiting lobby.
|
||||
- `ack`: successful heartbeat correlation.
|
||||
- `error`: a stable English `code` and Russian player-facing `message`.
|
||||
|
||||
@@ -67,3 +72,13 @@ so a refresh can restore it. A player disconnect pauses the game for at most 60
|
||||
seconds; restoration reattaches the same state, while expiry produces a draw.
|
||||
A disconnected observer token may be restored until its bounded slot is reused
|
||||
by a new observer, and observer disconnects never pause the match.
|
||||
|
||||
## Runtime diagnostics
|
||||
|
||||
`GET /diagnostics` returns bounded operational counters and internal-memory
|
||||
samples without exposing session tokens, Wi-Fi credentials, or hidden maze
|
||||
state. Counters cover connection and queue high-water marks, sent/dropped/
|
||||
failed messages, malformed and rate-limited input, accepted game commands,
|
||||
match cycles, maximum WebSocket handler/send duration, and current/minimum
|
||||
internal free heap and largest block. The response also publishes the fixed
|
||||
limits used by the Milestone 010 physical stress client.
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Runtime resource budget
|
||||
|
||||
Milestone 010 fixes and exposes the following production limits. They are
|
||||
compile-time bounds, not targets that may grow with match duration.
|
||||
|
||||
| Resource | Limit | Defensive behavior |
|
||||
| --- | ---: | --- |
|
||||
| WebSocket connections | 6 | A seventh WebSocket handshake is rejected. |
|
||||
| LwIP sockets | 16 | ESP-IDF remains configured by `sdkconfig.defaults`. |
|
||||
| Client message | 512 bytes | An oversized frame is counted and disconnected. |
|
||||
| Outgoing message | 768 bytes | An over-limit message cannot enter the queue. |
|
||||
| Outgoing queue | 12 entries | Queue overflow is counted; memory cannot grow. |
|
||||
| Game command queue | 16 entries | Excess commands receive `queue_full`. |
|
||||
| Input rate | 20 messages/second | Excess messages are rejected; sustained abuse above 40 in the same window is disconnected. |
|
||||
| Player response budget | 250 ms | The physical stress client fails above this round-trip threshold. |
|
||||
| Observer update policy | Latest bounded projection | A busy observer may lose intermediate dynamic chunks and recover on a newer projection. Failed or stalled required sends close the socket. |
|
||||
| Reconnect deadline | 60,000 ms | Gameplay pauses, then ends in a draw at expiry. |
|
||||
|
||||
## Build-time measurements
|
||||
|
||||
The Milestone 010 production build on 2026-09-05 reported 124,276 of 327,680
|
||||
static RAM bytes and 1,028,600 of 2,097,152 application bytes. The independent
|
||||
binary check measured 1,029,232 bytes in the 2 MiB factory partition, leaving
|
||||
1,067,920 bytes of application margin. The compressed web
|
||||
payload totals 19,253 bytes against its 1.5 MiB budget, leaving 1,553,611 bytes
|
||||
of payload margin. LittleFS remains the fixed 1,984 KiB partition and its image
|
||||
is checked against `partitions.csv` on every build.
|
||||
|
||||
The global authoritative gateway, including its one maze, one projection
|
||||
buffer, fixed queues, sessions, timers, and metrics, occupies 86,840 bytes of
|
||||
static storage in the production ELF. It does not allocate a maze or
|
||||
projection per client. No production source or test requires PSRAM or a GPIO.
|
||||
|
||||
## Runtime measurements
|
||||
|
||||
`GET /diagnostics` returns current and minimum internal 8-bit heap values and
|
||||
the largest internal free block. It deliberately samples
|
||||
`MALLOC_CAP_INTERNAL | MALLOC_CAP_8BIT`, so external RAM cannot make a failing
|
||||
internal-memory result appear healthy. Serial samples use the same capability
|
||||
mask.
|
||||
|
||||
Final six-client, slow-client, repeated-match, and network-recovery samples
|
||||
belong in the active milestone evidence in `PLANS.md`. Build-time numbers in
|
||||
this document must be refreshed if the production image changes before that
|
||||
evidence is recorded.
|
||||
Reference in New Issue
Block a user