Retaining Negative Returns when Comparing Heterogeneous Games

Authors

  • Gavin Russell Author
  • Sean Spencer Author
  • James Bennett Author

Abstract

The public BBF comparison includes 186 negative normalized scores and 63 exact zeros. A transformation used for aggregation must retain those observations honestly. An equally game-weighted model gives a raw-score contrast of 1.030 and an inverse-hyperbolic-sine contrast of 0.466, both positive but expressed in different units. Their intervals are 0.480 to 1.581 and 0.300 to 0.633 respectively. This secondary analysis explains why a change of scale can be a declared sensitivity analysis while deletion or arbitrary shifting can silently change the represented outcomes.

References

  1. Meng, Zijie, et al. "Make a game: A novel paradigm for interactive game rendering." ICASSP 2026-2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2026.
  2. White, Halbert. “A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity.” Econometrica, vol. 48, no. 4, 1980, pp. 817–838. https://doi.org/10.2307/1912934.
  3. Cameron, A. Colin, and Douglas L. Miller. “A Practitioner’s Guide to Cluster-Robust Inference.” Journal of Human Resources, vol. 50, no. 2, 2015, pp. 317–372. https://doi.org/10.3368/jhr.50.2.317.
  4. Greenland, Sander, Judea Pearl, and James M. Robins. “Causal Diagrams for Epidemiologic Research.” Epidemiology, vol. 10, no. 1, 1999, pp. 37-48. https://doi.org/10.1097/00001648-199901000-00008.
  5. Benjamini, Yoav and Yosef Hochberg. “Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing.” Journal of the Royal Statistical Society Series B: Statistical Methodology, vol. 57, no. 1, 1995, pp. 289-300. https://doi.org/10.1111/j.2517-6161.1995.tb02031.x.
  6. Roberts, David R., et al. “Cross-validation strategies for data with temporal, spatial, hierarchical, or phylogenetic structure.” Ecography, vol. 40, no. 8, 2017, pp. 913-929. https://doi.org/10.1111/ecog.02881.
  7. Varma, Sudhir and Richard Simon. “Bias in error estimation when using cross-validation for model selection.” BMC Bioinformatics, vol. 7, no. 1, 2006. https://doi.org/10.1186/1471-2105-7-91.
  8. Wilkinson, Mark D., et al. "The FAIR Guiding Principles for Scientific Data Management and Stewardship." Scientific Data, vol. 3, 2016, article 160018. https://doi.org/10.1038/sdata.2016.18.
  9. Sculley, D., et al. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems, vol. 28, 2015. https://papers.nips.cc/paper/2015/hash/86df7dcfd896fcaf2674f757a2463eba-Abstract.html
  10. Bland, J. Martin, and Douglas G. Altman. "Statistical Methods for Assessing Agreement between Two Methods of Clinical Measurement." The Lancet, vol. 327, no. 8476, 1986, pp. 307-310. https://doi.org/10.1016/S0140-6736(86)90837-8.
  11. Guo, Chuan, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. “On Calibration of Modern Neural Networks.” Proceedings of the 34th International Conference on Machine Learning, PMLR, vol. 70, 2017, pp. 1321–1330. https://proceedings.mlr.press/v70/guo17a.html.
  12. Geifman, Yonatan, and Ran El-Yaniv. “Selective Classification for Deep Neural Networks.” Advances in Neural Information Processing Systems, vol. 30, 2017. https://arxiv.org/abs/1705.08500.
  13. Ovadia, Yaniv, et al. “Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift.” Advances in Neural Information Processing Systems, vol. 32, 2019. https://arxiv.org/abs/1906.02530.
  14. Bellemare, Marc G., et al.. "The Arcade Learning Environment: An Evaluation Platform for General Agents" arXiv preprint arXiv:1207.4708, 2012. https://arxiv.org/abs/1207.4708.
  15. Agarwal, Rishabh, et al.. "Deep Reinforcement Learning at the Edge of the Statistical Precipice" arXiv preprint arXiv:2108.13264, 2021. https://arxiv.org/abs/2108.13264.
  16. Cobbe, Karl, et al.. "Leveraging Procedural Generation to Benchmark Reinforcement Learning" arXiv preprint arXiv:1912.01588, 2019. https://arxiv.org/abs/1912.01588.
  17. Brockman, Greg, et al.. "OpenAI Gym" arXiv preprint arXiv:1606.01540, 2016. https://arxiv.org/abs/1606.01540.
  18. Ha, David and Jürgen Schmidhuber. "World Models" arXiv preprint arXiv:1803.10122, 2018. https://arxiv.org/abs/1803.10122.
  19. Hafner, Danijar, et al.. "Dream to Control: Learning Behaviors by Latent Imagination" arXiv preprint arXiv:1912.01603, 2019. https://arxiv.org/abs/1912.01603.
  20. Micheli, Vincent, et al.. "Transformers are Sample-Efficient World Models" arXiv preprint arXiv:2209.00588, 2022. https://arxiv.org/abs/2209.00588.
  21. Alonso, Eloi, et al.. "Diffusion for World Modeling: Visual Details Matter in Atari" arXiv preprint arXiv:2405.12399, 2024. https://arxiv.org/abs/2405.12399.
  22. Valevski, Dani, et al.. "Diffusion Models Are Real-Time Game Engines" arXiv preprint arXiv:2408.14837, 2024. https://arxiv.org/abs/2408.14837.
  23. Bruce, Jake, et al.. "Genie: Generative Interactive Environments" arXiv preprint arXiv:2402.15391, 2024. https://arxiv.org/abs/2402.15391.
  24. Zhang, Richard, et al.. "The Unreasonable Effectiveness of Deep Features as a Perceptual Metric" arXiv preprint arXiv:1801.03924, 2018. https://arxiv.org/abs/1801.03924.
  25. Unterthiner, Thomas, et al.. "Towards Accurate Generative Models of Video: A New Metric & Challenges" arXiv preprint arXiv:1812.01717, 2018. https://arxiv.org/abs/1812.01717.
  26. Wang, Zhou, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. "Image Quality Assessment: From Error Visibility to Structural Similarity." IEEE Transactions on Image Processing, vol. 13, no. 4, 2004, pp. 600-612. https://doi.org/10.1109/TIP.2003.819861.
  27. Schwarzer, Max, et al.. "Bigger, Better, Faster: Human-level Atari with human-level efficiency" arXiv preprint arXiv:2305.19452, 2023. https://arxiv.org/abs/2305.19452.
  28. Sutton, Richard S., and Andrew G. Barto. Reinforcement Learning: An Introduction. 2nd ed., MIT Press, 2018. http://incompleteideas.net/book/the-book-2nd.html.
  29. Levine, Sergey, Aviral Kumar, George Tucker, and Justin Fu. “Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems.” arXiv, 2020, arXiv:2005.01643. https://arxiv.org/abs/2005.01643.
  30. Kumar, Aviral, Aurick Zhou, George Tucker, and Sergey Levine. “Conservative Q-Learning for Offline Reinforcement Learning.” Advances in Neural Information Processing Systems, vol. 33, 2020, pp. 1179–1191. https://arxiv.org/abs/2006.04779.

Downloads

Published

2026-10-02

Issue

Vol. 2 (2026): Archive issue of 2026-10-02

Section

Articles

License

Copyright (c) 2026 Gavin Russell, Sean Spencer, James Bennett

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.