Journal of Technology Informatics and Engineering
Journal Title: Journal of Technology Informatics and Engineering Initials: JTIE Frequency: April, August, and December Print ISSN: 2961-9068 Online ISSN: 2961-8215 Editor in Chief: Dr. Ir. Agus Wibowo., M.Kom., M.Si., M.M. DOI: 10.51903 Publisher: University of Science and Computer Technology Journal of Technology Informatics and Engineering is a peer-reviewed scientific journal that publishes original research articles, scholarly studies, and applied research findings related to advances in technology and engineering. The journal is published in both electronic and print formats with e-ISSN: 2961-8215, and p-ISSN: 2961-9068. The scope of the journal encompasses, but is not limited to, Informatics, Mechanical Engineering, Electrical Engineering, Information Systems, and Industrial Engineering. The journal serves as an academic platform for researchers, academics, and practitioners to disseminate research outcomes and technological innovations. Journal of Technology Informatics and Engineering is published three times a year, in April, August, and December. Indexed by:
University of Science and Computer Technology
Narrative-Aware Scientific Claim Verification Agent with Evidence Ranking for ClimateCheck
Climate misinformation often combines a factual proposition with a recognizable narrative, such as denying observed warming, rejecting human causation, minimizing impacts, attacking mitigation, or casting doubt on climate science. This paper presents a lightweight narrative-aware scientific claim verification agent for the official ClimateCheck setting. The revised evaluation uses the official annotated ClimateCheck training data, the official publications corpus of 394,269 abstracts, and a claim-level validation split of the annotated data. The public ClimateCheck test file is treated as a blind claim list because its public fields do not contain verification or narrative labels. The system combines hashed BM25, TF-IDF retrieval, latent semantic analysis, narrative-family probabilities, and a logistic-regression verifier. Full-corpus retrieval shows that BM25 remains the strongest first-stage retriever, with Recall@10 = 0.466, while the narrative-aware hybrid obtains Recall@10 = 0.444. In the judged candidate reranking setting, the narrative-aware ranker obtains the highest Candidate Recall@1 = 0.789 and MAP = 0.848, compared with 0.759 and 0.843 for TF-IDF. End-to-end verification remains difficult: the BM25 top-1 pipeline reaches Macro-F1 = 0.408, while the narrative-aware pipeline reaches Macro-F1 = 0.355. Claim-level narrative evaluation no longer produces a perfect score; single-label top-family Macro-F1 is 0.422, and fine-grained multi-label CARDS-code Macro-F1 is 0.098. These results show that narrative information is useful for reranking already plausible evidence candidates, but it does not replace strong lexical retrieval and does not by itself solve claim verification.
Copyright (c) 2026 Wenhao Su, Siyu Chen, Ethan Qian https://creativecommons.org/licenses/by-sa/4.0
Power-Aware Inventory Planning for AI Infrastructure Using Job-Level Forecasting and LLM Workload Explanations
AI infrastructure planning is commonly expressed as a GPU-count problem, yet operational risk is created by the electric and thermal envelope that accompanies each accelerator. This paper evaluates a power-aware planning method on Dataset A, using the B200 eight-GPU Llama-8B training trace with 45,000 raw 20 ms telemetry rows and 8,940 reproducible supervised decision records after a 100 ms decision stride. The forecasting task predicts total eight-GPU power one second ahead from job-level counters, autoregressive lags, and rolling statistics. The planning task converts forecasts into a peak-aware admission rule and a circuit-inventory simulation for 32 concurrent jobs. XGBoost produced the strongest mean forecast, with MAE 273.26 W, RMSE 636.74 W, and R2 0.923. A calibrated high-quantile forecast produced lower peak-error behavior, reducing the scheduling violation rate from 5.31% under GPU-count-only admission to 0.18% while admitting 61.63% of decision points. In the inventory simulation, XGBoost mean forecasting used 21.00 mean circuits with 1.80% violation risk, whereas the calibrated p95 plan used 22.70 circuits and eliminated observed violations in 1,000 trials. The results show that capacity plans based only on GPU count hide measurable electrical risk. A combined GPU-capacity, power-envelope, and workload-explanation view produces a reproducible basis for AI data center purchasing, placement, and sustainability decisions.
Copyright (c) 2026 Shilu He https://creativecommons.org/licenses/by-sa/4.0
A Therapist-Facing Session Copilot for Live Counseling Support: Reasoning-Guided Retrieval and Ranking from Multi-Turn Counseling Dialogues
This study develops and evaluates a retrieval-based therapist-facing session copilot for live counseling support using the public bilingual Psy-Insight dataset. Rather than providing autonomous psychotherapy or relying on a generative large language model (LLM), the system assists human therapists by ranking historical responses, retrieving interpretable rationales, and providing conservative contextual support. The reproducible pipeline combines TF-IDF representations, class-balanced LinearSVC routers, nearest-neighbor rationale retrieval, and label-aware response ranking without LLM fine-tuning. Experiments use all 520 English and 431 Chinese sessions (6,208 and 5,776 turns, respectively) with session-level train/dev/test splits. Psychotherapy routing achieves strong macro-F1 scores of 0.897 in English and 0.757 in Chinese, whereas strategy routing remains weak (0.253 and 0.268). Label-aware rationale retrieval improves ROUGE-L from 0.145 to 0.152 in English and from 0.142 to 0.151 in Chinese. The best response-ranking approach presents retrieved reasoning in parallel rather than through reasoning-fused reranking, increasing MRR from 0.498 to 0.541 in English and from 0.519 to 0.523 in Chinese while maintaining low latency (6.98–14.64 ms/query). These results demonstrate computational feasibility but do not establish therapeutic safety or clinical effectiveness.
Copyright (c) 2025 Yifan Zhang, Hailey Zhang https://creativecommons.org/licenses/by-sa/4.0
Few-Shot Cold-Start Workload Forecasting for New AI Inference Tenants with Time-Series Foundation Models
This paper presents a reproducible empirical study of few-shot cold-start workload forecasting for new AI inference tenants using the Alibaba GPU-disaggregated DLRM serving trace. Instance lifecycles are transformed into hourly active-demand series, and resource reservations are normalized into capacity units to evaluate 24-hour forecasting under zero-shot, 5-shot, 10-shot, and full-history settings. Seven forecasting methods are compared: archetype mean prior, persistence, moving average, linear trend, seasonal naive, global residual ridge, and CT-TSFM, a compact cross-tenant time-series foundation model. The cold-start evaluation uses 46 held-out tenants, with 110 source tenants for pretraining and calibration. Results show that hourly demand is strongly persistence-dominated. Zero-shot forecasting yields a mean absolute error (MAE) of 326.26 normalized capacity units, whereas only five observations reduce MAE to 4.00 for persistence, global residual ridge, and CT-TSFM. Validation consistently selects a residual gate of 0.0 for CT-TSFM, indicating that retaining the persistence prior and rejecting cross-tenant residual transfer is the most reliable strategy. Calibration intervals achieve approximately 85–87% coverage against a 90% target. The findings demonstrate that a few recent observations substantially improve cold-start forecasting, while source-tenant metadata alone provides limited zero-shot planning capability.
Copyright (c) 2025 Shilu He, Andrew Qian https://creativecommons.org/licenses/by-sa/4.0
Profit-Aware Spot GPU Admission Control with Cost-Sensitive Loss and Evidence-Grounded Policy Memos for AI Workload Supply-Demand Matching
AI clusters increasingly operate with heterogeneous GPU resources where production workloads and opportunistic spot jobs compete for limited accelerator capacity. This study presents a trace-driven admission-control framework using the Alibaba cluster-trace-v2026-spot-gpu dataset, consisting of 466,867 job records and 4,278 GPU-node records. The experiment evaluates GPU demand forecasting, profit-aware spot admission control, and evidence-grounded operational policy generation using chronological training, validation, and test splits. Hourly spot GPU demand forecasting was evaluated across six GPU models, where Ridge regression achieved the best test performance with an RMSE of 38.50 requested GPUs per hour, improving over both last-hour and seasonal naive baselines. The admission-control evaluation compared FIFO, greedy packing, classifier-based acceptance, utility ranking, and the proposed cost-sensitive policy. The proposed approach achieved a test profit of 67,278.96, improving 1.97% over the accuracy-oriented classifier while increasing spot success rate and reducing costly false acceptances by 13.17%. Sensitivity analysis showed that the optimal policy depends on the protection cost assigned to high-priority workloads. A deterministic evidence-grounded explanation layer generated 500 policy memos and passed numeric, policy, and evidence consistency checks. The findings suggest that profit-aware admission control can serve as a practical scheduling guardrail before detailed GPU placement and resource allocation decisions.
Copyright (c) 2026 Siming Zhao, Yuxuan Ren, Xiaohan Chang https://creativecommons.org/licenses/by-sa/4.0
Accounting-Aware Evidence-Constrained Agents for Disclosure, Settlement, and Secondary-Market Risk Monitoring in Tokenized
Tokenized real-world asset (RWA) infrastructure exposes platform operators, investors, and reporting teams to a combined settlement, disclosure, liquidity, and accounting-quality monitoring problem. A tokenized claim can continue to trade while the underlying issuer releases new financial statements, securities fail to deliver in the reference market, or protocol-level liquidity changes in RWA venues. This paper develops an accounting-aware, evidence-constrained agent workflow for risk alerting and source-grounded report generation. The revised experiment replaces the earlier rule-generated monitoring sandbox with external datasets: SEC fails-to-deliver observations, SEC EDGAR XBRL company facts, SEC submissions metadata, Financial PhraseBank sentiment labels, and DefiLlama RWA protocol TVL. The issuer-day panel contains 2,648 surveillance tasks for eight large U.S. issuers from 2024-12-01 through 2026-03-31. Observed settlement stress is defined from external SEC FTD balances rather than from the agent's own rule. Accounting risk is computed from XBRL-derived liquidity, leverage, accrual, and cash-flow indicators. A stronger market-plus-accounting logistic baseline is added alongside single-source baselines and the proposed fusion agent. The machine-learning baseline achieves the strongest F1 score for settlement-stress detection (0.909), while the proposed fusion agent achieves the highest report faithfulness and tool-use correctness (1.000 each) and high recall (0.849). The results support a governance-oriented interpretation: an evidence-constrained agent is most useful not as an opaque high-accuracy classifier, but as an auditable layer that connects settlement evidence, filing metadata, accounting fundamentals, independent sentiment calibration, and RWA protocol liquidity into a reproducible monitoring record.
Copyright (c) 2026 Sihan Zhou, Yuanzheng Chen, Kenny Lee https://creativecommons.org/licenses/by-sa/4.0
Budgeted Multi-Hop Retrieval Agent for Compositional Question Answering: A Retrieval-Policy Evaluation on the Official MultiHop-RAG Benchmark
Multi-hop question answering requires a retrieval system to assemble several complementary evidence documents before an answer module can reason reliably. Single-shot retrieval is efficient, but it often misses later-hop evidence when a question combines source, time, comparison, and entity constraints. This paper evaluates a budgeted multi-hop retrieval agent for compositional question answering on the official MultiHop-RAG benchmark. The benchmark contains 2,556 queries and 609 news-article corpus documents, with answerable evidence distributed across two to four documents. Four retrieval policies are compared under the same sparse lexical scorer: fixed top-k retrieval, iterative retrieval, query decomposition, and the proposed budgeted retrieval agent. The revised evaluation frames the task as retrieval-policy evaluation rather than as a full free-form generative QA system: retrieval-conditioned EM/F1 are reported together with evidence recall, MRR, retrieval rounds, selected documents, and context-token cost. On the official data, the budgeted agent achieves the strongest overall retrieval-conditioned EM/F1 at 62.75% and the highest final evidence recall at 74.67%, using 3.011 average retrieval calls and 509.7 average context tokens. Query decomposition improves over fixed top-k and iterative retrieval but is less stable across question types. Fixed top-k is cheapest but incomplete on longer chains. The four-hop results remain difficult for every policy, showing that a fixed 620-token controller should be extended with hop-aware or dynamic budget allocation. The findings support a moderated contribution claim: explicit budget control is useful for auditable multi-hop retrieval, but it should be evaluated as a cost-accuracy trade-off rather than as a universally dominant RAG architecture.
Copyright (c) 2025 Wenhao Su, Siyu Chen, Chloe Zhao https://creativecommons.org/licenses/by-sa/4.0
Accounting-Aware Evidence Retrieval for Institutional Due Diligence of Tokenized Trade Receivable RWA
Institutional investors evaluating tokenized real-world asset (RWA) transactions need retrieval systems that can answer short, ambiguous, and legally loaded due-diligence questions with traceable evidence. Trade receivable pools are especially difficult because the same question may require accounting policy, financial metrics, footnote disclosure, legal covenants, insurance language, servicer reporting, or waterfall mechanics. This study implements and evaluates an accounting-aware evidence-retrieval pipeline for tokenized trade receivable RWA due diligence. The main experiment uses the official FinDER benchmark with 5,703 query-evidence-answer triples, 6,121 annotated evidence references, and 5,830 deduplicated evidence passages derived from financial disclosures. The pipeline compares vanilla sparse retrieval, accounting-aware query rewriting, feature reranking, section-aware evidence selection, and calibrated abstention. On the official FinDER evaluation, query rewriting increased Recall@10 from 28.25% to 28.62%, reranking increased Recall@10 to 33.86% and answer-support accuracy to 24.57%, and section-aware evidence selection achieved 34.44% Recall@10, 24.04% nDCG@10, 8.32% EvidencePrecision@3, and 25.23% answer-support accuracy. The accounting-relevant subset, defined as Accounting, Financials, and Footnotes, achieved 37.10% Recall@10 and 26.54% answer-support accuracy. A supplementary stress check using a public receivables purchase agreement and SEC 2026-04 financial statement notes showed that the same retrieval logic can surface schedule, lock-box, GAAP, receivable, and note-disclosure evidence, while also highlighting the need for table extraction and field-level numerical validation. The findings support a narrower deployment claim: accounting-aware RAG can improve evidence discovery and analyst review, but it is not yet suitable for autonomous investment or accounting decision-making
Copyright (c) 2025 Yuanzheng Chen, Sihan Zhou, Emma Lin https://creativecommons.org/licenses/by-sa/4.0
Numerical-Reasoning Guardrails for a Quant Research Assistant: A Compact Reproducible Benchmark Using SEC and FRED Data
This paper presents a compact reproducible benchmark for evaluating numerical-reasoning guardrails in a quant research assistant. The revised experiment uses a fixed 2026 source snapshot derived from the SEC 2026 Q1 Financial Statement Data Sets and FRED CSV series for VIXCLS, DGS10, DGS3MO and T10Y3M. The benchmark contains 460 tasks: 300 SEC financial-ratio tasks over 50 issuer-period records, 120 FRED VIX and Treasury-rate change tasks, and 40 macro-regime classification tasks. Each answer is evaluated by five programmatic guardrails: numeric consistency, unit correctness, time-window correctness, formula correctness and citation/source consistency. Four controlled response profiles are tested: Naive-RAG, Calculator-Only, Prompted-Checklist and Guarded-Quant. These profiles are deterministic failure-mode controls rather than performance claims about any particular deployed LLM. The empirical results show that arithmetic alone is not sufficient for financial safety: Calculator-Only reaches 79.78% numeric accuracy but only 0.43% all-guardrails pass rate because source, unit, formula and window fields often fail. Guarded-Quant achieves an 88.48% all-guardrails pass rate, 97.17% numeric accuracy, 100.00% unit pass rate, 96.30% window pass rate, 98.26% formula pass rate and 96.30% citation pass rate. The findings support a modest claim: a compact benchmark can make numerical audit failures visible, but it should not be read as evidence of broad quant-assistant reliability without broader data, live model outputs and operational stress tests.
Copyright (c) 2026 Zeyi Li, Kai Zhang, Annie Wong https://creativecommons.org/licenses/by-sa/4.0
News-Based Uncertainty and Macro-Market Fusion for VIX Direction Forecasting: Evidence from 2015-2024 FRED Panel
This paper evaluates news-based uncertainty and macro-market fusion for one-trading-day VIX direction forecasting using a 2015-2024 daily FRED panel. The dependent variable equals one when the next trading-day VIX close exceeds the current close. The final panel contains 2,512 processed observations from 2015-02-04 to 2024-12-30, with 1,997 training observations from 2015-2022, 257 validation observations in 2023, and 258 holdout observations in 2024. The feature set combines VIX market-state variables, the daily newspaper-based Economic Policy Uncertainty index, the effective federal funds rate, 10-year and 2-year Treasury yields, lagged CPI inflation, lagged unemployment, and interaction terms. Expanding cross-validation on the 2015-2022 training sample gives the highest average ROC AUC to Fusion Random Forest (0.5620). The 2023 validation window selects a weighted fusion ensemble with ROC AUC 0.6005; its weights are fixed before the 2024 test. In the 2024 holdout window, VIX-only Logistic achieves the highest one-day ROC AUC (0.5869) and F1 (0.5448), while the weighted fusion ensemble reaches ROC AUC 0.5638 and F1 0.4786. Event-window diagnostics show that EPU shocks following calm VIX states have a next-day VIX-up rate of 0.5672, compared with 0.4445 on other trading days. The findings support a cautious interpretation: news-based uncertainty contains conditional information, but one-day practical forecasting reliability remains modest and VIX state variables remain the strongest 2024 holdout benchmark.
Copyright (c) 2025 Hailin Zhou, Kai Zhang https://creativecommons.org/licenses/by-sa/4.0
Trajectory Reliability Prediction for Generalist AI Agents: Tool-Use Failure Analysis and Success Forecasting on ZClawBench
Generalist AI agents increasingly perform complex tasks through planning, tool execution, action revision, and artifact generation rather than isolated response generation. This study empirically investigates trajectory reliability prediction on ZClawBench, a public OpenClaw-style agent benchmark containing 696 model-task trajectories across 116 tasks, six model families, and six scenario categories. The study evaluates whether operational trajectory signals—including tool-call volume, replanning behavior, tool errors, invalid-action ratio, recovery patterns, looping indicators, trajectory length, and task category—can predict task success before manual evaluation. Since the dataset provides trajectories and identifiers but lacks explicit per-instance success labels, the binary target is reconstructed from official model-by-category score distributions and treated as a modeling assumption. Five-fold cross-validation with task-level splitting was applied to prevent task leakage. Logistic Regression, Random Forest, XGBoost, sequence TF-IDF classification, and a rubric-based trajectory judge were compared. Logistic Regression achieved the strongest calibrated performance, obtaining ROC-AUC of 0.970, F1-score of 0.913, Brier score of 0.066, and expected calibration error of 0.018. Feature analysis indicated that task difficulty, no-progress behavior, response size, tool errors, and invalid actions contributed most to reliability prediction. The findings suggest that lightweight trajectory diagnostics can support agent monitoring, failure triage, and routing decisions, while further validation with direct case-level evaluation labels is required for deployment claims.
Copyright (c) 2026 Ziliang Samuel Zhong, Chenyu Li, Hengning Rao https://creativecommons.org/licenses/by-sa/4.0
Cost-Aware LLM-Style Routing for AIOps Log Analysis: Log Parsing, Anomaly Detection, Fault Diagnosis, and Incident Summarization on LogEval Task Files
This study investigates a local and cost-aware routing framework for AIOps log analysis using the LogEval benchmark. The evaluation covers four tasks: log parsing, anomaly detection, fault diagnosis, and incident summarization. Instead of relying on external large language model APIs, the experiment implements deterministic local policies that simulate zero-shot and few-shot LLM-style inference under controlled token-cost and latency assumptions. Six approaches were compared: regex normalization, TF-IDF with machine learning, a local character-based classifier, zero-shot policy, few-shot retrieval policy, and a routing cascade. At a risk threshold of 0.20, the router directed only 12.9% of queries to the few-shot retrieval policy while achieving parsing accuracy of 0.991, anomaly F1-score of 1.000, diagnosis accuracy of 1.000, ROUGE-L of 0.743, and BLEU-1 of 0.814. The routing strategy reduced simulated token cost by 80.1% compared with always using few-shot retrieval. Additional unseen-template evaluation revealed limited generalization for closed-label classifiers and retrieval methods when encountering unseen patterns. The findings indicate that routing can effectively reduce AIOps inference costs, while further validation with real LLMs and stronger generalization testing are required before production deployment.
Copyright (c) 2026 Chenyu Li, Ge Liu, Zoe Zhao https://creativecommons.org/licenses/by-sa/4.0
LLM-Inspired Offline Reranking for Financial Search: Query Rewriting, Hybrid Retrieval, and Listwise Relevance Ranking on FiQA
Financial search has high practical value because investors and retail users often ask natural-language questions whose wording differs from relevant financial passages. This paper evaluates a multi-stage retrieval pipeline on FiQA, a financial question-answering retrieval collection in BEIR. The systems include BM25, Dense LSA, BM25-LSA hybrid retrieval, reciprocal-rank fusion, a compact linear reranker, fixed pointwise and listwise relevance rubrics inspired by LLM reranking, query rewriting, and the proposed query rewriting plus hybrid retrieval plus listwise reranking pipeline. The evaluation used the full 57,638-document FiQA corpus, 6,648 available queries, and the 648-query BEIR FiQA test qrels with 1,706 binary relevance judgments. BM25 was the best-performing system, with nDCG@10 = 0.2285, MAP = 0.1863, MRR = 0.2994, and Recall@100 = 0.5207. The proposed full pipeline underperformed BM25. The listwise rubric ranked second on nDCG@10 (0.2228) and improved over the pointwise rubric, suggesting that candidate-list normalization can be useful in this setting. The rubric rerankers are fixed local scoring rules, so these results should be read as an evaluation of LLM-inspired ranking logic rather than as a benchmark of an actual prompt-based LLM reranker. Dense LSA retrieval alone was weak (nDCG@10 = 0.0287), which shows the limitation of a conservative non-neural dense baseline in financial semantic matching. Query rewriting reduced average effectiveness. The findings recommend strong lexical baselines, conservative rewrite gating, and careful evaluation before adopting prompt-based or model-based LLM rerankers in financial search.
Copyright (c) 2026 Siquan Meng, Jing Chen, Isa Zheng https://creativecommons.org/licenses/by-sa/4.0
Evidence-Calibrated RAG for Unanswerable Question Answering: Retrieval Coverage, Abstention Calibration, and Hallucination-Proxy Analysis on SQuAD 2.0
This paper presents a controlled and reproducible empirical study of evidence-calibrated retrieval-augmented question answering (RAG) for answerable and unanswerable reading-comprehension tasks using the SQuAD 2.0 benchmark. The study focuses on whether a system should abstain when retrieved evidence is insufficient rather than always producing an answer. Six lightweight architectures were evaluated on the full validation set of 11,873 questions, including closed-book, BM25, dense, hybrid, reranked, and a proposed evidence-calibrated hybrid RAG model. The proposed approach combines hybrid top-25 retrieval, lexical reranking, deterministic extractive answering, and evidence sufficiency calibration trained on 43,482 examples. On the validation set, it achieved 31.65% exact match, 34.74% F1, 53.01% answerability accuracy, 53.71% refusal F1, and a 37.49% hallucination-proxy rate. Although overall QA performance remains modest, calibrated evidence sufficiency substantially reduced unsupported answers compared with a forced-answer hybrid reranker, lowering the hallucination-proxy rate from 77.80% while improving F1. However, evidence calibration itself remained weak (AUROC 0.5475, ECE 0.1144). The findings demonstrate that retrieval coverage alone is insufficient to prevent hallucinations and highlight the need for stronger evidence calibration in trustworthy RAG systems.
Copyright (c) 2025 Ziliang Samuel Zhong, Jing Chen, Eric Zhong, Xinzhuo Sun https://creativecommons.org/licenses/by-sa/4.0
Evidence-Chain Reliable RAG: Hallucination Detection, Source Attribution, and Deterministic Provenance Explanations
Retrieval-augmented generation (RAG) reduces unsupported generation by grounding answers in source content, but retrieval alone does not guarantee that every output claim is attributable to evidence. This paper presents Evidence-Chain Reliable RAG, an empirical hallucination-detection and provenance framework that scores whether generated response sentences are supported by the corresponding RAG source record. The evaluation uses the complete RAGTruth JSONL data available for this study: 2,965 source records, 17,790 assistant responses, and 14,289 exact-offset hallucination spans across Data2Text, QA, and summarization. The experiment converts word-level spans into response-level, sentence-level, and character-span targets; extracts lexical, BM25, TF-IDF, unsupported-number, unsupported-entity, refusal, and Evidence-Chain Score features; and compares seven methods. On the official held-out test split of 2,700 responses, RandomForest achieved the best case-level F1 of 0.626 and PR-AUC of 0.553. The proposed ECS-Span model achieved case-level F1 of 0.614, ROC-AUC of 0.742, and PR-AUC of 0.536 while also producing deterministic provenance explanations. At sentence level, RandomForest again achieved the highest F1 of 0.321; the proposed method obtained F1 of 0.312, ROC-AUC of 0.777, and PR-AUC of 0.245. Exact character-span localization remained difficult, with character-level F1 of 0.197 because sentence-level predictions often include supported text around shorter hallucinated spans. The findings indicate that evidence-chain features are useful for interpretable RAG auditing, but precise span extraction requires token-level sequence labeling or a comparable fine-grained model.
Copyright (c) 2025 Jiaying Jin https://creativecommons.org/licenses/by-sa/4.0
LLM-Style DevOps Copilot for Cloud-Native Troubleshooting: Retrieval-Augmented Runbook Generation and Command-Safety Evaluation
Cloud-native incident response requires engineers to connect symptoms, observability signals, infrastructure state, and safe remediation commands under time pressure. Large language models can draft runbooks, but an ungrounded assistant can invent commands, recommend the wrong diagnostic path, or reproduce destructive operational shortcuts. This paper evaluates an LLM-style, retrieval-augmented DevOps Copilot simulation for cloud-native troubleshooting on the canonical Szaid3680/Devops Arrow export. The experiment indexes all 42,819 rows with the public Response, Instruction, and Prompt schema and evaluates a deterministic 400-query subset with TF-IDF, BM25, a compact dense-semantic baseline, RAG-style answer construction, reranking, command-safety checking, and the combined reranker-plus-checker pipeline. No live LLM inference is used in the executed experiment; the generation and checking components are deterministic so that the safety effects can be reproduced exactly. Results show that retrieval improves answer grounding but does not by itself guarantee safe automation: RAG-only reaches 0.2966 semantic similarity and emits matched unsafe command text at a rate of 0.0324. The command-safety checker reduces the matched unsafe command rate to 0.0000 for the declared rule set and keeps command validity at 0.9922. The full pipeline obtains 0.3051 semantic similarity, 0.4225 root-cause accuracy, 0.5825 root-category accuracy, and 0.0078 hallucinated-command rate. The findings support treating DevOps copilots as retrieval-grounded and policy-checked workflow systems rather than free-form chat agents.
Copyright (c) 2026 Boning Zhang, Xinzhuo Sun, Ge Liu, Binghua Zhou https://creativecommons.org/licenses/by-sa/4.0
Language-Guided Feature Selection for DDoS and Intrusion Detection on CICIDS2017
This paper reports a complete empirical study of language-guided feature selection for DDoS and intrusion detection on the CICIDS2017 MachineLearningCSV flow data. The central question is whether an LLM-style semantic reading of CICFlowMeter feature names can reduce the feature set while preserving detection performance and lowering false alarms. The experiment used the eight labeled CICIDS2017 CSV sessions, removed only non-finite numeric rows, and retained 2,827,876 flows with 78 original numeric features. A semantic feature screen selected 32 features describing service context, duration, packet and byte volume, flow rates, inter-arrival timing, TCP flags, window sizes, and active/idle behavior. The evaluation compared all features with the language-selected set under full-corpus binary and multiclass stochastic logistic regression, DDoS-specific Random Forest, DDoS-specific stochastic logistic regression, and a compact multilayer perceptron. The best DDoS result was obtained by Random Forest with the selected features: F1 = 0.999896, false-positive rate = 0.000068, and eight errors on 67,714 test flows. The selected features reduced the DDoS Random Forest training time by 23.78% and reduced full-corpus SGD training time by about one half, although the full feature set was stronger for the full binary linear model. Ablation showed that TCP flag/window and destination-port semantics produced the largest DDoS degradation when removed. The findings support language-guided feature selection as a practical compression step for latency-sensitive DDoS mitigation, while retaining all features remains advisable for broad multiclass intrusion detection when a linear learner is used.
Copyright (c) 2025 Yunhe Li, Shenghan Lu https://creativecommons.org/licenses/by-sa/4.0
Uncertainty-Aware Medical Vision–Language Classification on a Lightweight MedMNIST-Compatible Biomedical Patch Benchmark
Medical image classifiers can be accurate while still being unsafe to use when their confidence values are poorly calibrated or when their predictions are communicated in language that overstates diagnostic certainty. This paper presents an uncertainty-aware medical vision-language classification workflow for lightweight 28×28 biomedical images. The target setting is MedMNIST-style classification, where images are standardized to small spatial sizes and where compact CNN, residual, and transformer models can be trained on ordinary hardware. The official MedMNIST v2 collection contains 12 two-dimensional and 6 three-dimensional biomedical image subsets; however, the execution environment used for this manuscript could read the official documentation but could not fetch binary Zenodo files. Three lightweight models were trained and evaluated across three random seeds: a 53,380-parameter CNN, a 392,092-parameter tiny residual network, and a 77,956-parameter tiny Vision Transformer. Each model used the same 2,240/320/640 train/validation/test split, AdamW optimization, and validation-set temperature scaling. The evaluated metrics were top-1 accuracy, macro one-vs-rest ROC-AUC, negative log likelihood, multiclass Brier score, expected calibration error, predictive entropy, and confusion-matrix/class-level metrics. TinyViT achieved the highest mean calibrated top-1 accuracy, 0.9906 ± 0.0016, while SmallCNN achieved the best mean macro ROC-AUC, 0.9993 ± 0.0005, and the best mean post-calibration ECE, 0.0115 ± 0.0028. Temperature scaling reduced ECE for all models, with reductions of 0.1153 for SmallCNN, 0.0853 for TinyResNet, and 0.1189 for TinyViT. A deterministic language-card module converted calibrated predictions into patient-friendly decision-support text that explicitly includes confidence, uncertainty, visual cue wording, and a non-diagnostic safety caveat.
Copyright (c) 2026 Shenghan Lu, Xiaohan Chang, Tracey Zou https://creativecommons.org/licenses/by-sa/4.0
Calibrated Resume-Job Matching for Trustworthy LLM-Assisted Recruiter Screening: Pairwise Matching, Probability Calibration, and Selective Refusal on Two Public Recruitment Datasets
Recruiter screening increasingly relies on large language model (LLM)-assisted workflows, but high-stakes applications require reproducible matching, calibrated probabilities, and reliable handling of uncertain cases. This study evaluates a screening framework combining matching, calibration, and selective refusal using two public datasets: resume-job-description-fit for supervised pairwise learning and Resume-Screening-Dataset for benchmarking and external generalization. After deterministic preprocessing, we compared cosine similarity, alignment features, TF-IDF pairwise models, and hybrid models integrating text, alignment, and title information. The strongest probabilistic models were calibrated with Platt scaling and isotonic regression and evaluated under confidence-based refusal. On the resume-job-description-fit test set, the best three-class model achieved a macro-F1 of 0.450. For binary shortlist-versus-reject screening, the title-augmented hybrid model obtained 0.654 balanced accuracy, 0.647 F1, and 0.699 AUROC. Platt calibration improved probability estimates by reducing the Brier score from 0.232 to 0.226 and negative log-likelihood from 0.772 to 0.675. Selective refusal further improved in-domain accuracy, while cross-dataset transfer remained weak (AUROC 0.47–0.51). These results indicate that matching, calibration, and selective refusal enhance trustworthy within-domain screening, although human review remains essential under distribution shift.
Copyright (c) 2025 Binghua Zhou, Jiaying Jin, David Zhao https://creativecommons.org/licenses/by-sa/4.0
The Impact of Generative AI on University Students’ Academic Performance: A Quantitative and Predictive Analysis
This research is concerned with assessing the effects of generative AI on the academic performance of university students in terms of usage, benefits, and challenges. Quantitative research methodology was used for data collection via structured questionnaires from 256 university students. Data analysis was performed through IBM SPSS (version 27.0) and Stata software utilizing non-parametric statistics considering the type of data. The reliability test showed that the instrument was highly reliable with a Cronbach's Alpha score of 0.842. The results show a statistically significant positive association between the different variables of AI use, with research assistance being the most significant predictor of academic performance. Exploratory factor analysis indicated three important factors affecting academic performance, namely academic integration, perception of challenges, and skill development. Regression analysis showed a positive relationship between academic integration and skill development and academic performance. However, perception of challenges had a negative relationship. In addition to this, another strong relationship between the geographical position of students and awareness about risks related to generative AI was observed. In general, the research offers a balanced look at the role of generative AI in academia as a very helpful technology, as well as a risky one, which has to be approached thoughtfully.
Copyright (c) 2026 Afsana Mimi, Sazeed Hossain, Shahria Zzaman https://creativecommons.org/licenses/by-sa/4.0
Early Warning, Grade Prediction, and Teacher-Facing LLM-Ready Explanations Toward an Open Volleyball Course: Reproducible Evidence from Four Public Education Datasets
Open online courses and public skill-development programs often experience learner dropout not due to content limitations, but because instructors receive delayed and non-actionable feedback. This study proposes and empirically evaluates an integrated framework for an open volleyball course that combines early warning prediction, grade estimation, and teacher-oriented LLM-generated academic status explanations. The predictive models were tested on four lightweight educational datasets: xAPI-Edu-Data, Predict Students’ Dropout and Academic Success, Student Performance, and Higher Education Students Performance Evaluation. A unified preprocessing pipeline was applied using one-hot encoding, an 80/20 train-test split, and 5-fold cross-validation. Decision Tree, Random Forest, and XGBoost models were evaluated for classification, alongside their regression variants for grade prediction. Results show consistent performance across datasets. Random Forest achieved the best macro-F1 on xAPI-Edu-Data (0.799) with a macro-AUC of 0.914, while XGBoost performed best on the dropout dataset (macro-F1 = 0.689, macro-AUC = 0.892). For Student Performance, early-warning models without prior grades reached an RMSE of 3.086, improving to 1.398 when full information was available. On the higher education dataset, performance remained limited due to small sample size and multi-grade targets, with Random Forest achieving a macro-F1 of 0.248. Ablation results confirmed that behavioral and progression features significantly improve predictive accuracy. An explanation layer translated model outputs into structured, teacher-facing natural language linked to key risk indicators and intervention cues. Overall, the framework demonstrates analytic feasibility for structured volleyball course monitoring, though results should be interpreted as pre-deployment evidence rather than validation in real instructional settings. Explanation quality improves when grounded in observed behavioral signals rather than generic generation.
Copyright (c) 2026 Jubin Zhang https://creativecommons.org/licenses/by-sa/4.0
From General Human Activity Recognition to Volleyball-Oriented Wearable Transfer Learning: Cross-Dataset Evidence from UCI HAR and WISDM for Domain Adaptation and Edge Deployment
Wearable human activity recognition has become a practical foundation for coaching analytics, workload monitoring, and interactive sports training, yet volleyball-specific labelled inertial datasets remain much smaller than general-purpose public HAR corpora. This study addressed that gap through a transfer-learning design in which public HAR benchmarks were treated as representation sources and a smartwatch target domain was used as a volleyball-oriented wrist proxy. Full experimental evaluations were conducted on UCI HAR and on a public WISDM-derived smartwatch subset, using three baseline families: logistic regression, a lightweight one-dimensional convolutional network, and a tiny Transformer. The study also measured modality ablation and unsupervised domain adaptation through Deep CORAL in a common four-class transfer space. On UCI HAR, the final measured accuracies were 0.7940 for logistic regression, 0.8039 for CNN-Lite, and 0.3621 for Transformer-Tiny. On the WISDM smartwatch subset, the corresponding accuracies were 0.8207, 0.8682, and 0.7632. Modality ablation on WISDM showed that accelerometer-only input reached 0.8486 accuracy, gyroscope-only input reached 0.6543, and fused accelerometer-plus-gyroscope input reached 0.8505. For cross-dataset transfer from UCI to WISDM, source-only training achieved 0.3376 accuracy, Deep CORAL improved accuracy to 0.4134, and the fully supervised target-only upper bound reached 0.8374. The results establish three concrete conclusions: lightweight convolutional sequence encoders are more reliable than the tested tiny Transformer under these data conditions, accelerometer channels carry most of the discriminative value for wrist-worn deployment, and domain adaptation is necessary when general smartphone HAR is transferred to smartwatch sports analytics. These findings provide a reproducible public-data foundation for volleyball-oriented wearable modelling and for subsequent fine-tuning on sport-specific action labels.
Copyright (c) 2025 Jubin Zhang https://creativecommons.org/licenses/by-sa/4.0
Layout-Aware Progressive PDF Rendering: AI Prioritization of PDF Slices to Reduce Time-to-Functional-First-Frame on FUNSD
Progressive PDF rendering is attractive because users rarely need every visible pixel at once; they need the semantically useful parts of the current viewport early enough for reading and interaction. This paper studies whether layout-aware AI can prioritize PDF slices more effectively than geometric or density-based heuristics. We reconstruct vector PDFs from official FUNSD form annotations and evaluate a tile scheduler that predicts tile utility from inexpensive layout and preview features before high-resolution rendering begins. The empirical study covers 26 reconstructed documents from the FUNSD test split that were fully processed in the present environment, four viewport scenarios, and measured clip-render timings for all visible tiles. The main configuration uses an 8×10 grid and a random-forest regressor trained with page-level 5-fold GroupKFold, then compares the learned scheduler with row-major visible-first, center-first, ink-density, text-density, a hand-tuned layout heuristic, full-page rendering, and an oracle upper bound. The proposed model reaches TTFF-90 in 14.21 ms, compared with 15.18 ms for the best non-AI heuristic, 20.48 ms for full-page rendering, and 24.09 ms for row-major rendering. It also achieves Utility@20ms of 0.941, AUC@25ms of 0.730, NDCG@10 of 0.963, and Recall@10 of 0.969. The results show that slice rendering is not inherently beneficial: the summed visible-tile cost in the main 8×10 setting is 28.80 ms, which is higher than the full-page cost of 20.48 ms, so scheduling quality determines whether slicing improves or harms TTFF. A coarser 6×8 grid reduces AI TTFF-90 to 10.58 ms, while the densest pages favor a full-page fallback. Paired Wilcoxon signed-rank tests over the page-scenario cases yield p < .001 for TTFF-90 improvements of the proposed model over every non-AI baseline. However, those tests should be interpreted as case-level rather than document-level evidence.
Copyright (c) 2025 Heyu Wang, Yuxuan Ren https://creativecommons.org/licenses/by-sa/4.0
Distilling VMAF into an Edge-Deployable Quality Predictor: A Pilot Shot-Level Proxy with LLM-Ready Quality Tokens
This pilot study evaluates whether a compact student model can approximate VMAF well enough to support low-latency release guarding on edge-class CPU environments. The corpus comprises a 62.31-second Big Buck Bunny excerpt at 1280 × 720 and 25 fps, segmented into 13 shots. Twelve distorted variants were generated by crossing H.264/AVC and H.265/HEVC with 180p, 240p, and 360p delivery resolutions and two quality levels per codec-resolution pair, yielding 156 shot-level samples. Frame-level VMAF scores were aggregated into shot-level teacher labels, and a student proxy consumed 14 low-cost no-reference features derived from decoded frames and stream metadata. Shot-grouped five-fold cross-validation was used to prevent content leakage across train-test splits. On this corpus, a 50-tree gradient-boosted decision tree achieved MAE = 6.56 VMAF points, RMSE = 8.32, and Pearson r = 0.913. Relative to simple regressors, the student reduced MAE by approximately 21.5% versus bitrate-only regression and 10.7% versus metadata-only regression. In a single CPU-only benchmark, predictor latency was 0.484 ms per sample and the full decode-feature-predict chain averaged 42.61 ms versus 1117.41 ms for the teacher, corresponding to a 26.22× end-to-end speed-up. As a thresholded guard, the same student reached F1 = 0.826, 0.893, and 0.900 at 60, 70, and 80 VMAF respectively. These findings support the feasibility of a practical edge proxy on this specific pilot corpus, but they should not be interpreted as broad generalization across content classes or production ladders. The paper also introduces an LLM-ready token interface intended for downstream reporting rather than for replacing the underlying quality measurement
Copyright (c) 2025 Xiaohan Chang, Heyu Wang https://creativecommons.org/licenses/by-sa/4.0
Natural-Language Policy Reasoning with Proof Generation: Turning Platform Rules into Verifiable Knowledge
Policy and compliance systems increasingly express rules in natural language, yet enforcement requires deterministic decisions and auditable explanations. This paper studies a practical pipeline that converts natural-language facts and rules into a verifiable knowledge base, answers queries with three-valued semantics (True/False/Unknown), and produces machine-checkable proofs. The contribution is system-level rather than a new reasoning formalism: we integrate controlled-language parsing, symbolic proof extraction, independent proof checking, and proof-based supervision in a single auditable framework. We evaluate the pipeline on two natural-language rule-reasoning benchmarks: (i) a balanced subset of ProofWriter’s open-world-assumption tasks (360 train, 360 test), and (ii) a RuleTaker-style dataset generated from its grammar and label semantics (1800 train, 900 test), both balanced across reasoning depths 0–5. We compare a text-only logistic regression baseline, a retrieval-based “proof” baseline, a symbolic forward-chaining reasoner with proof extraction, and a proof-trained classifier using generated proofs. To ensure fairness, LR-text and LR-proof share the same TF-IDF/logistic-regression setup, and the retrieval baseline uses the same representation with a fixed top-4 configuration. On ProofWriter-Balanced, the symbolic reasoner achieves 0.803 accuracy (0.808 macro-F1), while proof-trained classification reaches 0.825 accuracy (0.825 macro-F1). On RuleTaker-Rep, both methods achieve 1.000 accuracy. Proof verifiability clearly separates faithful from post-hoc explanations: symbolic proofs are verifiable for all predictions, whereas retrieval-based proofs are verifiable for only 31.4%. Sensitivity analyses varying reasoning depth, distractors, and proof corruption show that proof-based methods remain robust to noise but depend on proof integrity. These findings demonstrate the feasibility of auditable natural-language policy reasoning in controlled settings, while highlighting limitations in parser coverage and benchmark regularity.
Copyright (c) 2025 Xiaofei Luo https://creativecommons.org/licenses/by-sa/4.0
Explainable Multi-Hop Question Answering for QA Assistants: Two-Hop Evidence Retrieval, Sentence-Level Supporting Facts, and Explicit Reasoning Paths
Multi-hop question answering (QA) for customer-facing assistants requires not only accurate answers but also an auditable evidence trail that explains how the system arrived at each answer. We present a fully interpretable multi-hop QA pipeline that decomposes inference into three explicit modules—Retriever → Evidence Selector → Reasoner—and produces an explanation consisting of sentence-level supporting facts and an explicit two-hop evidence path. The retriever ranks candidate paragraphs using lexical IDF-weighted token overlap; the evidence selector chooses a small set of high-scoring sentences; and the reasoner extracts a final answer using weighted candidate phrase scoring and deterministic rules for date/number and constrained yes/no comparisons. We conduct full experimental evaluations on the complete development splits of HotpotQA (7,405 questions, distractor setting) and 2WikiMultihopQA (12,576 questions). On HotpotQA, sentence-level evidence selection improves Supporting Fact F1 from 0.334 to 0.419, and adding an explicit two-hop retrieval path further increases Supporting Fact F1 to 0.426 while raising paragraph recall@2 to 0.603. Answer F1 increases from 0.084 to 0.088. On 2WikiMultihopQA, evidence selection improves Supporting Fact F1 from 0.328 to 0.429 and Answer F1 from 0.071 to 0.075. These results quantify the contribution of explicit evidence selection and path-constrained retrieval to explainability and provide a practical, reproducible baseline for knowledge assistants that must justify answers with supporting facts.
Copyright (c) 2026 Xiaofei Luo https://creativecommons.org/licenses/by-sa/4.0
Off-Policy Evaluation and Conservative Policy Selection for Slot-Level Dynamic Bidding and Ranking on the Open Bandit Dataset (Small)
Dynamic bidding and ranking systems must improve revenue or engagement while avoiding harmful regressions during deployment. This paper presents an end-to-end offline OPE and conservative policy-selection workflow for slot-level contextual bandit approximations of ranking decisions. Using the small Open Bandit Dataset (OBD-small) from ZOZOTOWN (ZOZO, Inc.), each logged row is treated as a context-dependent choice among discrete actions (items), with binary click rewards and logged propensity. This formulation is suitable at the slot level but does not capture full listwise ranking or multi-step offline reinforcement learning. Dynamic bidding and ranking systems must improve revenue or engagement while avoiding harmful regressions during deployment. This paper presents an end-to-end offline OPE and conservative policy-selection workflow for slot-level contextual bandit approximations of ranking decisions. Using the small Open Bandit Dataset (OBD-small) from ZOZOTOWN (ZOZO, Inc.), each logged row is treated as a context-dependent choice among discrete actions (items), with binary click rewards and logged propensity. This formulation is suitable at the slot level but does not capture full listwise ranking or multi-step offline reinforcement learning. Empirically, highly deterministic evaluation policies exhibit extreme variance under sparse clicks, while the logistic reward model remains weak (ROC-AUC ≈ 0.5), limiting DM/DR interpretability. Clipped-DR mixing yields only limited certified improvements: in the women’s campaign, gains appear only at moderate confidence (δ=0.10) and for caps up to M=5, whereas stricter or looser settings revert to baseline; in the men’s campaign, certification is largely absent. These findings demonstrate that OPE diagnostics and conservative mixing enable reproducible offline selection under uncertainty, but do not indicate deployment-ready improvements.
Copyright (c) 2026 Tong Ye, Jinyi Mu, James Hunter https://creativecommons.org/licenses/by-sa/4.0
Federated Topic-Preference Learning for Knowledge-Grounded Chat with Differential Privacy
Retrieval-augmented approaches have become central in knowledge-grounded dialogue systems, yet incorporating topical preferences remains difficult due to privacy constraints on user interaction data. This study introduces a lightweight federated topic-preference (FedTP) mechanism that models session-level preferences without centralizing raw data and uses client-level differential privacy (DP). Using the Topical-Chat dataset (8,628 conversations), each conversation is treated as a client, and evidence routing is framed as selecting relevant knowledge snippets based on dialogue context. The proposed method augments a TF-IDF relevance score with a small preference-based component derived from both local session distributions and a DP-aggregated global prior. Experimental results on 9,553 grounded test turns show a consistent but limited improvement in evidence hit rate, from 0.6167 to 0.6194. The small optimal preference weight (λ = 0.005) indicates that the preference signal mainly influences decisions when competing candidates have similar relevance scores, rather than substantially altering routing behavior. A privacy–utility analysis under Gaussian DP (ε ranging from 9.69 to 0.606, δ = 1e−5) shows negligible changes in performance, which is expected given the large number of clients in a one-shot aggregation setting. Additional metrics remain largely stable, suggesting that the method affects selection margins rather than overall alignment. These findings suggest that federated preference aggregation can provide a modest, privacy-preserving bias for evidence routing, but its practical impact remains incremental and context-dependent.
Copyright (c) 2025 Meng-Ju Kuo, Daren Zheng, Julie Hires https://creativecommons.org/licenses/by-sa/4.0
Digital-Twin Dispatching for Urban Mobility via Spatio-Temporal Transformers and Offline Reinforcement Learning
This study addresses the challenge of optimizing ride-hailing dispatch and repositioning under data limitations by proposing an end-to-end digital-twin dispatching framework that integrates spatio-temporal demand forecasting with offline reinforcement learning. Using publicly available NYC FOIL ride-hailing data aggregated at the dispatching-base level, the research aims to evaluate whether coarse-grained data can still support reliable, reproducible decision-making pipelines. The methodology consists of two main components: (i) multivariate time-series forecasting using baseline models, a temporal convolutional network (TCN), and a spatio-temporal transformer to predict next-day demand; and (ii) a digital-twin simulation combined with an action-constrained offline reinforcement learning approach, including behavior cloning (BC) and Conservative Q-Learning (CQL), to optimize fleet repositioning decisions. Experimental results show that the TCN achieves the best forecasting accuracy on the test period, although dominant demand regions largely drive performance gains. In the control phase, conservative policies such as CQL demonstrate stable performance with reduced repositioning costs, but do not significantly outperform behavior cloning due to limited training data. The findings indicate that, in coarse aggregate settings, operational improvements are more influenced by controlling policy sensitivity than by marginal forecasting gains. This study contributes a reproducible benchmark pipeline and highlights the importance of conservative control strategies, transparent assumptions, and sensitivity analysis when deploying AI-driven mobility systems based on limited or aggregated data.
Copyright (c) 2025 Long Zhang, Ruiyan Ma, Peter Greg https://creativecommons.org/licenses/by-sa/4.0
Offline Counterfactual Evaluation for Advertising and Recommendation Slot Policies: A Reproducible Study on the Open Bandit Dataset (Small)
Offline or counterfactual evaluation is a critical capability for iterating advertising and recommender ranking strategies when online A/B testing is slow, expensive, or risky. Off-policy evaluation (OPE) estimates the expected reward of a candidate policy using logged interaction data from a different behavior policy. Still, it can suffer from high variance under poor overlap and can be misleading when the operational objective is choosing among candidate policies rather than minimizing point-estimation bias alone. This paper presents a fully reproducible empirical study of IPS, self-normalized IPS (SNIPS), doubly robust (DR), and Switch-DR estimators on the Open Bandit Dataset (OBD) small release. Using the Men and Women campaigns (10,000 logged item-impressions per campaign and behavior policy) collected by uniform random and Bernoulli Thompson Sampling (BTS), we construct a held-out oracle for stationary slot-wise policies from the random-traffic split and evaluate both value estimation and policy-ranking consistency on random-logged and BTS-logged test sets. Across 1,000 nonparametric bootstrap replications, IPS and SNIPS are accurate on randomly logged data, whereas BTS-logged data exhibit extreme importance weights and very small effective sample sizes (ESS), making IPS-based ranking unreliable under weak support. Switch-DR is most useful in moderate-overlap regimes, where it truncates high-variance corrections. Still, it introduces bias that depends on the switching threshold and must therefore be stress-tested rather than treated as a universally superior estimator. Finally, we provide a structured reporting template—based on oracle decomposition, overlap diagnostics, and estimator components—for explaining why a policy appears better and how reliable that conclusion is.
Copyright (c) 2025 Jinyi Mu, Tong Ye, Priya Patel https://creativecommons.org/licenses/by-sa/4.0