\section{Setup and observational semantics}\label{sec:setup}

\subsection{Typed alternating trajectories}

A trajectory of $K$ rounds is
\[
    \xi=(a_1,e_1,\ldots,a_K,e_K)\in\Traj.
\]
At decision state $s_t$, an agent chooses $a_t\in\A(s_t)$ and reaches chance state $s'_t=s_t\oplus a_t$. Nature then draws $e_t\in\E(s'_t)$ from $\mu(\cdot\mid s'_t)$, producing $s_{t+1}=s'_t\oplus e_t$. Action and event tokens form a tagged disjoint union, so their roles cannot be confused by vocabulary overlap. Rewards may occur at both positions:
\[
    R(\xi)=\sum_{t=1}^{K}\{r(s_t,a_t)+r(s'_t,e_t)\}.
\]
The state can be a full history, as in our finite experiments, or a learned representation. A terminal state has value zero.

For a finite reference distribution $b$ and score $f$, define
\[
 \LSE_{b,\tau}(f)=\tau\log\Exp_{Z\sim b}\exp\{f(Z)/\tau\},\qquad \tau>0.
\]
The temperature $\tau$ controls softness relative to a reference; it is not automatically a user's risk preference.

\subsection{Observational model}

Let $\pi_\beh$ be the behavior policy that generated data. The observational trajectory law is
\begin{equation}\label{eq:observational}
    \rho_\beh(\xi\mid x)=\prod_{t=1}^{K}\pi_\beh(a_t\mid s_t)\mu(e_t\mid s'_t).
\end{equation}
Expected teacher-forcing negative log-likelihood is a sum of occupancy-weighted conditional cross-entropies. A function-space optimum therefore matches $\pi_\beh$ at reachable action prefixes and $\mu$ at reachable event prefixes. This identifies a predictive factorization of \eqref{eq:observational}; it neither changes observed behavior into a normative policy nor supplies causal identification by itself.

\begin{assumption}[Policy-invariant event kernel]\label{ass:invariance}
For every evaluation policy under consideration and every supported state--action pair, the conditional event law is the same kernel $\mu(e\mid s\oplus a)$.
\end{assumption}

Assumption~\ref{ass:invariance} requires a sufficient state representation, consistency, and overlap. If it fails, $\mu$ is only a behavior-conditioned predictive law. Planning outputs must then be labeled scenario projections, not intervention effects.

\subsection{Reference policy and objective}

Let $\pi_{\refp}(a\mid s)>0$ be a full-support action reference and define
\[
    B_{\refp}=\sup_s\log\frac{1}{\min_{a\in\A(s)}\pi_{\refp}(a\mid s)}<\infty.
\]
For $M=(\mu,r)$, the finite-horizon KL-regularized control objective is
\begin{align}\label{eq:objective}
J_M^\tau(\pi;x)
&=\Exp_{\pi\times\mu}\!\left[\sum_{t=1}^{K}\{r(S_t,A_t)+r(S'_t,E_t)\}\right] \\
&\quad-\tau\Exp_{\pi\times\mu}\!\left[\sum_{t=1}^{K}\KL\!\left(\pi(\cdot\mid S_t)\,\|\,\pi_{\refp}(\cdot\mid S_t)\right)\right].\nonumber
\end{align}
A uniform $\pi_{\refp}$ recovers entropy regularization up to a fixed additive constant. A non-uniform reference can encode a default or behavior prior, but it must be reported because values and policies are relative to it.

