Data mining and prediction

Data mining is a first-class Scopwis workspace. It exposes the registered deterministic analysis catalog instead of making advanced methods discoverable only through hidden Agent tools.

Method families

The current catalog covers workflows such as:

• profiling, robust summaries, missingness, duplicate keys, and tail behavior;
• clustering, density segmentation, PCA, and cluster-quality diagnostics;
• anomaly and local-density outlier screening;
• relationships, nonlinear dependence, partial correlation, lead/lag, and stability;
• time-series trend, seasonality, change points, rolling bands, and forecasting;
• drift and distribution-shift attribution;
• binary ranking, calibration, classification and regression diagnostics;
• experiment/group comparison and robust effect sizes;
• event/state, survival, retention, and gap analysis;
• factor generation, factor mining, signal evaluation, policy simulation, and quantitative performance.

The catalog grows through registered method contracts. UI labels are product-level entry points; the authoritative executable method IDs and parameter schemas come from the Server.

Start a method

1. Open Data mining.
2. Filter by category or search for a method.
3. Select a starter.
4. Review and edit the generated request.
5. Press Start analysis.

The catalog supports category, name, and method-ID filtering. On compact screens, selecting a method opens its detail page; returning to the catalog keeps the edited request for that method.

Selecting a method does not execute anything by itself. The explicit start action sends the request through the same AgentPlan, authorization, query-safety, artifact, and critic gates used by Analyze.

Prediction is leakage-controlled

Predictive and time-series methods must respect chronological boundaries. Forecasting and rolling methods use explicit training/history windows; future observations cannot influence earlier baselines. Methods that cannot satisfy the required ordering, sample, or quality contract fail closed instead of silently changing the analysis.

A predictive output is evidence about the configured method and validation window. It is not automatically a production deployment decision.

Rolling quantile bands

Use rolling_quantile_band to compare each value with the preceding 5–500 observations. The input must contain 6–10,000 rows in strictly increasing time order. Choose lower_percent from 1–49 and upper_percent from 51–99; the defaults are 5 and 95. Set top_n from 1–200 to control how many breaches are returned.

The method checks every window, then returns the largest exceedances first. Equal exceedances retain original row order. Lowering top_n reduces retained ranking candidates and output construction; it does not sample the input or stop validation early. A flat historical band remains valid. Non-finite inputs or an unrepresentable band width/exceedance stop the analysis instead of producing a report with silent null values. A breach is a descriptive comparison, not a diagnosis of its cause.

Comparing distribution shifts

quantile_shift_profile compares the P05/P25/P50/P75/P95 values and interquartile range (IQR) of explicit baseline and current artifacts. The IQR ratio remains defined for every positive baseline IQR, including very small values; changing measurement units must not turn nonzero variation into a zero-baseline diagnosis. Only an exactly zero baseline IQR produces zero_baseline_iqr=true and a null ratio. A zero current IQR with a positive baseline produces a ratio of zero.

Differences or ratios outside the finite numeric range stop the method with a structured execution error. They are not silently replaced with nulls in a successful artifact or report. The result describes the supplied distributions; it does not correct the data or establish the cause of a shift.

Constant factor inputs

Pearson diagnostics in factor mining and combination report zero whenever either finite input series is constant. This includes large repeated values such as [1e100; 10]: rounding during mean calculation must not create a false correlation of 1. Changing series keep the existing correlation calculation and sample-standard-deviation threshold of 1e-12. The result remains a measure of association.

Interpreting mining output

Registered methods return structured artifacts with diagnostics and limitations. Scopwis does not turn correlation into causation, anomaly score into fraud, drift into a root cause, or a backtest into a trading recommendation.

For a decision-ready narrative, continue in Analyze and ask the Agent to synthesize the mining artifact with the rest of the evidence. For a visual deliverable, ask for a report.