Methodology: ranking and evidence

How to read portfolio rankings and evidence scores

Current: alpha-p0-v1.4 / scenario-ranking-v2.12. Five evidence measures explain deterministic results. Raw risk-limit usage enters qualified search fitness and final CAGR/risk preference points alongside development growth and the diversity preference cost; the other measures are descriptive.

Measure	Direction	Read with
Risk limit used	Lower = more historical headroom	Limiting metric, period, observed value and cap
Historical growth advantage	Higher = stronger recorded growth relative to seed	Net CAGR and seed/reference differences
Development hurdles passed	More of three blocks passed	Per-block excesses and dates
Downside severity	Lower = smaller full-training drawdown	Training dates
Cost resilience	Higher = less growth lost under stress costs	Both rates/CAGRs, turnover and liquidity

Do not average these measures: risk/downside overlap, persistence reuses returns, and costs already reduce growth. None measures suitability or future success. Model validation requires more than attractive backtest scores.¹

Exact gates, ranking, chronology and research limitations belong to the methodology. The current engine requires activate_cagr_bands_risk_scoring: true; legacy band ranking is unsupported. The combined stability score and ranking index below are unimplemented P1 proposals.

Conventions

Use the run's common training calendar, monthly or no scheduled rebalancing, and effective base/stress costs. Both rates default to zero; identical cases provide no distinct fee stress. Returns/weights in formulas are fractions. G_i is linked-development base-cost net CAGR; seed comparisons use matched paths. Let h=0.005 and:

F(x)=0.5+0.5\frac{x}{|x|+h}

Thus $100F(0)=50$ and $100F(0.005)=75$ . These anchors are product conventions, independent of which alternatives are displayed. Round only for display; unavailable evidence is Not evaluated.

Training stability governs ranked versus exploratory output. Five outer refits rerun the complete enabled policy; outer and later-test outcomes never change frozen historical order. Projections use full analyzed complete-month history, including reserved test rows after selection; their percentiles are conditional resampling summaries, not independent validation.

Risk limit used

For each enforced historical risk scope s and cost c:

\begin{aligned} Dcap_{s,c}&=\min(0.40,D_{seed,s,c}+0.05) \\ Vcap_{s,c}&=\min(0.25,1.10\sigma_{seed,s,c}) \\ U_i&=100\max_{s,c}(D_{i,s,c}/Dcap_{s,c},\sigma_{i,s,c}/Vcap_{s,c}) \end{aligned}

U=100 reaches the tightest limit; above 100 breaches it. Zero risk against a zero cap contributes zero; positive risk against zero cap is a breach. The ranking uses raw U, not its display transform.

Displayed Risk score

\begin{aligned} RiskScore_i&=100\frac{\log(1-\kappa U_i/100)}{\log(1-\kappa)} \\ \kappa&=0.9362362051581328 \end{aligned}

For valid utilization 0–100: 0 maps to 0, 95 to 80, and 100 to 100. Outside the limit, show Limit breached. This logarithmic scale is a presentation convention, not another financial risk measure.

Liquidity remains a feasibility gate. ETF and eligible observed-stock concentration enter the separate diversity preference cost. Report-only provider holdings and partial disclosed-position HHI need coverage/unknown-tail context; an incomplete small value cannot establish better diversification.

Current growth/risk preference

With training risk usage R_train=U, the run's risk coefficient lambda, and diversity cost P as defined in the optimizer paper:

\begin{aligned} f_i &= 100G_i-\lambda R_{train,i}-P_i \\ R_{after,i} &= \max(R_{train,i},U_{later,base,i},U_{later,stress,i}) \\ g_i &= 100G_i-\lambda R_{after,i}-P_i \end{aligned}

Feasible qualified mixes precede feasible unqualified mixes. Higher f ranks first within each group; exact ties prefer lower raw risk usage, then the remaining financial keys. Qualified search uses the same preference and ties; failed-candidate ordering and qualification gates stay unchanged.

lambda defaults to 0.5 annual CAGR percentage points per raw usage point. For example, 12% development CAGR, usage 80 and P=0.8 give f=12−0.5×80−0.8=−28.8. If matched later usage reaches 85, g=12−0.5×85−0.8=−31.3. Negative preference points are valid; the coefficient and concentration strengths are policy trade-offs, not calibrated returns.

Post-run usage includes the reserved later test at both costs, with matching later-period seed caps; paths are not concatenated. Development CAGR, P, the stored coefficient, allocations, qualification and historical ranks stay frozen. Later CAGR is displayed separately. For finite values, R_post≥R_train and g≤f. With zero coefficient, both scores reduce to 100G−P when the later comparison is available. Missing later evidence leaves R_post and g unavailable. A positive risk against a zero cap is explicitly marked as a breach. Using later results for choice or tuning consumes their untouched-validation role.

Historical growth advantage

Upside_i=100F(G_i-G_{seed})

50 means equal CAGR; 75 means +0.5 percentage points/year. Show reported CAGR and matched seed/fixed-reference excesses. This is not factor-adjusted alpha; common equity exposures can affect growth, and no factor attribution is estimated here.²

Development hurdles passed

\begin{aligned} e_{i,k}&=\min_{reference,cost}(G_{i,k,cost}-G_{reference,k,cost}) \\ Persistence_i&=\frac{100}{3}\sum_{k=1}^{3}\mathbf{1}(e_{i,k}\ge0.005) \end{aligned}

Show the count, such as 2/3, with threshold, dates and excesses. The blocks are not independent trials. Qualification requires linked growth, so a qualified mix need not pass all three individual-block hurdles.

Outer positive-excess frequency is separate, once per run: 100 × min_cost(positive continuous-path segments / 5). Its hurdle is strictly positive excess, not +0.5 points. Show median/risk/bootstrap checks too; frequency alone cannot establish an outer pass. All five feasible selections are required for continuous aggregates; standalone resets do not supply gates.

Downside severity

Downside_i=100\max_{cost}(D_{i,full\ training,cost})

Use full-training dates, including context. A value of 28 means 28% observed peak-to-trough loss, not a future-loss cap. Current results do not identify peak/trough dates.

A dated falling-market event table remains proposed: predefine every VT peak-to-recovery episode reaching a 10% loss within training, label unrecovered episodes, and compare matched paths. Do not annualize short event returns or assume the portfolio's worst loss coincides with a broad-market decline.

Cost resilience

\begin{aligned} \delta_i&=G_{i,base}-G_{i,stress} \\ Practicality_i&=\frac{100}{1+\max(0,\delta_i)/0.005} \end{aligned}

100 means no deterioration; 50 means losing 0.5 percentage points/year. Zero/zero rates therefore produce 100 without demonstrating higher-fee resilience. Deterioration below −1e-12 makes the score unavailable (invalid_cost_order); smaller negative roundoff is clamped to zero. Show cost rates, both CAGRs, stress excesses, turnover and liquidity. This score cannot offset a failed feasibility or return gate.

Stability of the search method (proposal)

Unimplemented run-level sensitivity measure. For each predeclared variant v, rerun the same selection rule and compare weights and stressed CAGR on one common evaluation calendar:

\begin{aligned} d_v&=\tfrac12\sum_i|w_{v,i}-w_{0,i}| \\ u_v&=\max(d_v/0.10,|g_v-g_0|/0.005) \\ Stability&=100/(1+\max_v u_v) \end{aligned}

Higher is preferred; 50 reaches a proposed tolerance. Report weight/growth changes separately. Proposed suite: cutoff −21 sessions, quarterly versus monthly execution, alternate seeds and expanded search. Quarterly execution requires new support. Preserve universe/history floors, train-only inputs, matched comparators and the intersection of valid evaluation periods.

A missing required variant gives Not evaluated; alternative-existence disagreement is a failure; no qualifying alternative is N/A. Publish once per run, not per retained mix. Existing repeatability/convergence checks do not implement this combined score.

Ranking index (P1 proposal)

Unimplemented band-based challenger to the current CAGR/risk incumbent. Feasibility and qualification remain fixed. Bootstrap research and selection-bias limits remain those in the methodology's academic notes.

Proposed ranking formulas and ordering

Remove establishment from only the first linked net log return, resample adjusted logs jointly within ordered development blocks, then restore entry once:

\begin{aligned} y_{i,c,t}&=\log(1+r_{i,c,t}) \\ \widetilde y_{i,c,t}&=y_{i,c,t}-\mathbf{1}_{t=1}\log(1-c) \\ z_{i,c,a,p}&=\frac{252}{N}\left[\log(1-c)+\sum_t\widetilde y^*_{i,c,a,p,t}\right] \\ J_i&=\exp\left(\min_{c,a}Q_{10}(z_{i,c,a,*})\right)-1 \end{aligned}

c: base/stress one-way rate; a: expected length 10/21/63; p: replicate; N: linked development observations. Recurring costs stay resampled with returns. Use one primary context shared across candidates/costs; compute J for seed and each feasible group's entire leading 0.5-point band before retention. Q10 uses the one-based ceil(R/10) order statistic, without interpolation. Existing excess-only bootstrap bounds cannot reconstruct J.

Let Q_i=1 for qualified, otherwise 0; L_i=1 inside the group's leading band:

\begin{aligned} x_i&=\begin{cases}J_i-J_{seed}&L_i=1\\G_i-G_{seed}&L_i=0\end{cases} \\ RankingIndex_i&=25(2Q_i+L_i+F(x_i)) \end{aligned}

Open interval	Group/band
75–100	Qualified, leading band
50–75	Qualified, outside
25–50	Feasible unqualified, leading band
0–25	Feasible unqualified, outside

Use canonical integer bands. Infeasible/exploratory rows receive no index; preserve the no-alternative exception. Sort groups/band first, raw J within band, raw G outside. Within-band ties: DD, volatility, ETF HHI, turnover ascending; minimum/median reference excess descending; ID. Outside-band ties: ID.

The complete proposed rule must be compared with the current scoring incumbent. Indices have policy-boundary jumps; differences do not quantify economic gains, future confidence or search-wide correction.

Comparison and validation

Before inspecting outcomes, fix the challenger, primary cost convention, economic materiality threshold and uncertainty treatment. At each earlier-only refit, apply incumbent and challenger to the same full evaluated union before retention, with fixed search budgets, qualification and feasibility. Freeze each selection before its next segment; include transitions and every scheduled period, including fallback/failure outcomes.

Check larger replicate counts/alternate shared streams with candidate sets, qualification, bands, paths and tie-breaks fixed. Compare matched subsequent net compound growth, risk, exposure and turnover. These are offline diagnostics, not additional runtime gates. Choosing a rule from those segments consumes their independence; reserve untouched final evidence. Retain the incumbent absent a material net-growth benefit under the declared criterion.

How AI uses the evidence

Compute supplies calculated scores and raw drivers; AI explains them without recalculating scores or changing rank. Shared stability/fold evidence appears once. Missing optional AI cannot remove deterministic results. Second Opinion uses complete structured records and preserves numeric facts.

References and notes

Display anchors, score transforms and proposed tolerances are product choices. Academic sources support interpretation and validation principles, not their calibration. Resampling and search references are maintained in the methodology white paper.

1: J. Simonian (2024), Investment Model Validation: A Guide for Practitioners, CFA Institute Research Foundation, “Backtesting” and “Cross-Validation.” Validation includes assumptions and implementation; a historical score alone does not establish model adequacy.
2: E. F. Fama and K. R. French (2015), A Five-Factor Asset Pricing Model, Journal of Financial Economics 116(1), 1–22, §§1–2. Market, size, value, profitability and investment exposures describe average stock returns; this tool does not estimate factor-adjusted ETF-mix alpha.

Back to documents