132 lines
4.1 KiB
Markdown
132 lines
4.1 KiB
Markdown
|
# HMS MediaEngine – Native Render Bridge (ADR-0004)
|
|||
|
|
|
|||
|
|
Rust/GStreamer-D3D11-Renderkern für den HMS MediaEngine Render-Worker (§6.1C).
|
|||
|
|
Dieser Crate implementiert den nativen Rendergraph als GStreamer-Plugin-Ansatz:
|
|||
|
|
|
|||
|
|
- **GStreamer-Plugin** `hmsrender` mit eigenem Compositor-Element `hmscompositor`
|
|||
|
|
- **D3D11-Layer-Compositor** mit den Blend-Modi V1 (§12.4)
|
|||
|
|
- **HLSL-Shader-Loader** mit dem Standard-cbuffer-Layout `hms_params` (§14.4)
|
|||
|
|
- **FrameReceiver** für binäres MessagePack-`FrameSnapshot` (§11.4)
|
|||
|
|
- **Pipeline-Builder** für die D3D11-Elementkette (§13.1)
|
|||
|
|
|
|||
|
|
## Architektur
|
|||
|
|
|
|||
|
|
```text
|
|||
|
|
Python Render-Worker (§6.1C)
|
|||
|
|
│ FrameSnapshot (MessagePack, IPC)
|
|||
|
|
▼
|
|||
|
|
FrameReceiver ──► LayerCompositor (D3D11)
|
|||
|
|
│
|
|||
|
|
▼
|
|||
|
|
GStreamer-Pipeline: d3d11h264dec → d3d11convert → hmscompositor → d3d11videosink
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Kein CPU-Readback im Normalpfad (§12.6, §33): Alle Operationen laufen auf
|
|||
|
|
GPU-residenten D3D11-Texturen.
|
|||
|
|
|
|||
|
|
## Build-Anleitung (Windows)
|
|||
|
|
|
|||
|
|
### Voraussetzungen
|
|||
|
|
|
|||
|
|
- **Rust Toolchain** (stable, Edition 2021): <https://rustup.rs>
|
|||
|
|
- **GStreamer MSVC Runtime + Development** (1.22+):
|
|||
|
|
<https://gstreamer.freedesktop.org/download/>
|
|||
|
|
- Installiere `gstreamer-1.0-devel-msvc-x86_64` und `gstreamer-1.0-runtime-msvc-x86_64`
|
|||
|
|
- Setze `GSTREAMER_1_0_ROOT_MSVC_X86_64` auf den Installationspfad
|
|||
|
|
- **Windows SDK** (für D3D11, DXGI, HLSL): Teil von Visual Studio Build Tools
|
|||
|
|
- **pkg-config** (für gstreamer-rs): über MSYS2 oder `vcpkg`
|
|||
|
|
|
|||
|
|
### Build
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
cd native/render_bridge
|
|||
|
|
cargo build --release
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
Die kompilierte Bibliothek liegt unter `target/release/hms_render_bridge.dll`
|
|||
|
|
(cdylib).
|
|||
|
|
|
|||
|
|
### Umgebungsvariablen
|
|||
|
|
|
|||
|
|
```bash
|
|||
|
|
export GST_PLUGIN_PATH="$(pwd)/target/release"
|
|||
|
|
export GST_PLUGIN_SYSTEM_PATH_1_0="C:/gstreamer/1.0/msvc_x86_64/lib/gstreamer-1.0"
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## Integration mit dem Python-Orchestrator
|
|||
|
|
|
|||
|
|
Der Python-Render-Worker (§6.1C) lädt die cdylib und ruft die FFI-Funktionen auf:
|
|||
|
|
|
|||
|
|
```python
|
|||
|
|
import ctypes
|
|||
|
|
|
|||
|
|
bridge = ctypes.CDLL("target/release/hms_render_bridge.dll")
|
|||
|
|
|
|||
|
|
# Pipeline bauen (JSON-Konfiguration)
|
|||
|
|
config = {
|
|||
|
|
"canvas_width": 1920,
|
|||
|
|
"canvas_height": 1080,
|
|||
|
|
"fps": 60.0,
|
|||
|
|
"media_uri": "C:/media/clip.mp4",
|
|||
|
|
"layers": {"layer_1": "normal"},
|
|||
|
|
"output_device": None,
|
|||
|
|
}
|
|||
|
|
config_json = json.dumps(config).encode("utf-8")
|
|||
|
|
bridge.hms_build_pipeline(config_json)
|
|||
|
|
|
|||
|
|
# FrameSnapshot als MessagePack übergeben
|
|||
|
|
snapshot = {
|
|||
|
|
"frame_index": 0,
|
|||
|
|
"monotonic_ns": 0,
|
|||
|
|
"state_revision": 0,
|
|||
|
|
"parameters": {"layer_1/opacity": 1.0, "audio/rms": 0.5},
|
|||
|
|
"source_positions": {},
|
|||
|
|
"source_states": {},
|
|||
|
|
"active_asset_ids": {},
|
|||
|
|
}
|
|||
|
|
payload = msgpack.packb(snapshot)
|
|||
|
|
bridge.hms_push_frame(payload, len(payload))
|
|||
|
|
```
|
|||
|
|
|
|||
|
|
## FrameSnapshot-Vertrag (§11.4)
|
|||
|
|
|
|||
|
|
Die Feldnamen im Rust-`FrameSnapshot` sind identisch zum Python-`FrameSnapshot`
|
|||
|
|
in `apps/renderer/hms_renderer/engine.py`:
|
|||
|
|
|
|||
|
|
| Feld | Typ | Bedeutung |
|
|||
|
|
| --- | --- | --- |
|
|||
|
|
| `frame_index` | int | Frame-Nummer |
|
|||
|
|
| `monotonic_ns` | int | Monotone Zeitbasis (ns) |
|
|||
|
|
| `state_revision` | int | Showzustands-Revision |
|
|||
|
|
| `parameters` | dict[str, float] | Flache Parameter-Pfade |
|
|||
|
|
| `source_positions` | dict[str, float] | source_id → Position |
|
|||
|
|
| `source_states` | dict[str, str] | source_id → TransportState |
|
|||
|
|
| `active_asset_ids` | dict[str, str\|None] | layer_key → asset_id |
|
|||
|
|
|
|||
|
|
## Blend-Modi (§12.4)
|
|||
|
|
|
|||
|
|
`normal`, `add`, `multiply`, `screen`, `lighten`, `darken`, `difference`,
|
|||
|
|
`overlay`, `alpha_premultiplied`. Müssen mit Golden-Image-Tests geprüft werden.
|
|||
|
|
|
|||
|
|
## Standard-Shaderinputs (§14.4)
|
|||
|
|
|
|||
|
|
Jeder Shader erhält das cbuffer `hms_params` (register b0):
|
|||
|
|
|
|||
|
|
- `u_resolution` (float4: xy = Pixel, zw = 1/xy)
|
|||
|
|
- `u_time_seconds`, `u_delta_seconds`, `u_frame_index`
|
|||
|
|
- `u_layer_opacity`
|
|||
|
|
- `u_audio_rms`, `u_audio_peak`, `u_audio_bass`, `u_audio_mid`, `u_audio_treble`, `u_audio_beat`
|
|||
|
|
- deklarierte Plugin-Parameter
|
|||
|
|
|
|||
|
|
## Windows-Abhängigkeiten
|
|||
|
|
|
|||
|
|
- GStreamer MSVC (Runtime + Development)
|
|||
|
|
- Windows SDK (D3D11, DXGI, HLSL)
|
|||
|
|
- Visual Studio Build Tools (Linker, pkg-config)
|
|||
|
|
|
|||
|
|
## Hinweis
|
|||
|
|
|
|||
|
|
Der Code wird im Container nicht kompiliert (kein cargo). Die Kompilierung
|
|||
|
|
erfolgt im Windows-Durchlauf gemäß ADR-0004. Gate-0-Messungen bestätigen das
|
|||
|
|
Elementpfad-Budget oder lösen eine Revision aus (eigenständige D3D11-Bridge).
|