Benchmarks - Revision history

BPeat at 03:05, 16 September 2026

2026-09-16T03:05:27Z

BPeat: /* Status: Passed, and Reinterpreted */

2026-09-10T14:23:31Z

‎Status: Passed, and Reinterpreted

BPeat at 14:22, 10 September 2026

2026-09-10T14:22:16Z

BPeat at 14:21, 10 September 2026

2026-09-10T14:21:43Z

BPeat at 14:20, 10 September 2026

2026-09-10T14:20:45Z

BPeat at 14:18, 10 September 2026

2026-09-10T14:18:26Z

BPeat at 14:18, 10 September 2026

2026-09-10T14:18:00Z

Show changes

BPeat at 12:57, 22 November 2024

2024-11-22T12:57:35Z

BPeat: /* Human Evaluation */

2024-11-22T12:52:43Z

‎Human Evaluation

BPeat: /* Human Evaluation */

2024-11-22T12:49:16Z

‎Human Evaluation

@@ Line 10: / Line 10: @@
 * [[Data Science]] ... [[Data Governance|Governance]] ... [[Data Preprocessing|Preprocessing]] ... [[Feature Exploration/Learning|Exploration]] ... [[Data Interoperability|Interoperability]] ... [[Algorithm Administration#Master Data Management (MDM)|Master Data Management (MDM)]] ... [[Bias and Variances]] ... Benchmarks ... [[Datasets]]
-* [[Large Language Model (LLM)]] ... [[Natural Language Processing (NLP)]] ... [[Natural Language Generation (NLG)|Generation]] ... [[Natural Language Classification (NLC)|Classification]] ... [[Natural Language Processing (NLP)#Natural Language Understanding (NLU)|Understanding]] ... [[Language Translation|Translation]] ... [[Natural Language Tools & Services|Tools & Services]]
+* [[Large Language Model (LLM)]] ... [[Large Language Model (LLM)#Multimodal|Multimodal]] ... [[Foundation Models (FM)]] ... [[Generative Pre-trained Transformer (GPT)|Generative Pre-trained]] ... [[Transformer]] ... [[Attention]] ... [[Generative Adversarial Network (GAN)|GAN]] ... [[Bidirectional Encoder Representations from Transformers (BERT)|BERT]]
 * [[Risk, Compliance and Regulation]] ... [[Ethics]] ... [[Privacy]] ... [[Law]] ... [[AI Governance]] ... [[AI Verification and Validation]]
 * [[Case Studies]]

@@ Line 86: / Line 86: @@
 The authors' stated concern is social and economic rather than philosophical: systems that pass as human enable "counterfeit people," with consequences for online trust, employment, genuine social engagement, and the perceived value of human interaction.
 <hr><center>
 <b><i>Today an AI has to dumb down to pass the Turing Test</i></b> - Ray Kurzweil
-</center><hr>
+</center><hr><br>
 <youtube>4VROUIAF2Do</youtube> <youtube>_GCTLciqT0A</youtube>

@@ Line 43: / Line 43: @@
 A recurring caution applies to everything below. '''Benchmarks saturate.''' The history of the field is a sequence of tests that fell and were then dismissed as never having measured intelligence — checkers, chess, Jeopardy!, Go, protein structure prediction, competition mathematics, and most recently the Turing test itself. A high score is evidence about a test, not a verdict on a system. The constraints that make a benchmark meaningful — contamination resistance, construct validity, independent administration — are discussed on [[AI Verification and Validation]].
-<br>
+<br><br>
 <hr><center>

← Older revision		Revision as of 14:21, 10 September 2026
Line 1:		Line 1:
		+	__NOTOC__
	{{#seo:		{{#seo:
	\|title=PRIMO.ai		\|title=PRIMO.ai

@@ Line 23: / Line 23: @@
 * [[Backpropagation]] ... [[Feed Forward Neural Network (FF or FFNN)|FFNN]] ... [[Forward-Forward]] ... [[Activation Functions]] ...[[Softmax]] ... [[Loss]] ... [[Boosting]] ... [[Gradient Descent Optimization & Challenges|Gradient Descent]] ... [[Algorithm Administration#Hyperparameter|Hyperparameter]] ... [[Manifold Hypothesis]] ... [[Principal Component Analysis (PCA)|PCA]]
 * [[Strategy & Tactics]] ... [[Project Management]] ... [[Best Practices]] ... [[Checklists]] ... [[Project Check-in]] ... [[Evaluation]] ... [[Evaluation - Measures|Measures]]
-  ** [[Evaluation - Measures#Accuracy|Accuracy]]
+** [[Evaluation - Measures#Accuracy|Accuracy]]
-  ** [[Evaluation - Measures#Precision & Recall (Sensitivity)|Precision & Recall (Sensitivity)]]
+** [[Evaluation - Measures#Precision & Recall (Sensitivity)|Precision & Recall (Sensitivity)]]
-  ** [[Evaluation - Measures#Specificity|Specificity]]
+** [[Evaluation - Measures#Specificity|Specificity]]
 * [[AI Solver]] ... [[Algorithms]] ... [[Algorithm Administration|Administration]] ... [[Model Search]] ... [[Discriminative vs. Generative]] ... [[Train, Validate, and Test]]
 * [https://www.aitrends.com/ai-insider/machine-learning-benchmarks-and-ai-self-driving-cars/ Machine Learning Benchmarks and AI Self-Driving Cars | Lance Eliot - AItrends]
@@ Line 34: / Line 34: @@
 * [https://spectrum.ieee.org/ai-supercomputer#toggle-gdpr Supercomputers Flex Their AI Muscles: New benchmarks reveal science-task speedups | Samuel K. Moore - IEEE Spectrum]
 * [https://towardsdatascience.com/the-olympics-of-ai-benchmarking-machine-learning-systems-c4b2051fbd2b The Olympics of AI: Benchmarking Machine Learning Systems | Matthew Stewart - Towards Data Science] ... How do benchmarks birth breakthroughs?
 Three pages on this wiki cover overlapping ground. The division of labor:
@@ Line 50: / Line 42: @@
 A recurring caution applies to everything below. '''Benchmarks saturate.''' The history of the field is a sequence of tests that fell and were then dismissed as never having measured intelligence — checkers, chess, Jeopardy!, Go, protein structure prediction, competition mathematics, and most recently the Turing test itself. A high score is evidence about a test, not a verdict on a system. The constraints that make a benchmark meaningful — contamination resistance, construct validity, independent administration — are discussed on [[AI Verification and Validation]].
 = The Turing Test and Related Thought Experiments =

@@ Line 12: / Line 12: @@
 * [[Risk, Compliance and Regulation]] ... [[Ethics]] ... [[Privacy]] ... [[Law]] ... [[AI Governance]] ... [[AI Verification and Validation]]
 * [[Case Studies]]
-  ** [[Gaming]]
+** [[Gaming]]
-  ** [[Algorithm Administration#Model Monitoring|Model Monitoring]]
+** [[Algorithm Administration#Model Monitoring|Model Monitoring]]
 * [[ALFRED]] ... Action Learning From Realistic Environments and Directives
 * [[Algorithm Administration]]

@@ Line 267: / Line 267: @@
 = Human Evaluation =
 <b>CAPTCHA</b> stands for "Completely Automated Public Turing test to tell Computers and Humans Apart". It's a security measure that helps protect users from spam and password decryption by verifying that a user is human and not a computer.
 == I'm not a robot ==

← Older revision		Revision as of 12:52, 22 November 2024
Line 280:		Line 280:

	In a way, humans doing these tests are teaching the bots to get better at beating the tests themselves. It's a fascinating cycle of humans improving AI while trying to outsmart it! Irony at its finest.		In a way, humans doing these tests are teaching the bots to get better at beating the tests themselves. It's a fascinating cycle of humans improving AI while trying to outsmart it! Irony at its finest.
		+
		+	== Future Prospects and Innovations ==
		+	The future of human evaluation CAPTCHA techniques is likely to be shaped by ongoing technological advancements and the need to balance security with user experience. Some promising developments include:
		+	* Advanced AI and Machine Learning Techniques: As AI becomes more sophisticated in solving CAPTCHAs, new techniques are being developed to create CAPTCHA-resistant challenges that can adapt to evolving bot strategies.
		+	* Invisible CAPTCHA: Google's reCAPTCHA v3 represents a significant innovation by eliminating visible challenges for users. Instead, it continuously monitors user behavior to assess the likelihood of a bot interaction, providing a score between 0 and 1.
		+	* Cognitive Deep-Learning CAPTCHA: A 2023 study introduced a new CAPTCHA system that combines text-based, image-based, and cognitive CAPTCHA characteristics. This system employs adversarial examples and neural style transfer to enhance security, making it more resistant to automated attacks.
		+	* Behavioral Analysis and Biometric Verification: Innovations are exploring the use of behavioral analysis to distinguish human actions from bot interactions without explicit challenges. Biometric identification is also being considered for seamless user authentication, leveraging unique user characteristics.
		+	* AI-Powered Solutions: AI algorithms are being developed to create CAPTCHA-resistant challenges that can adapt to evolving bot strategies. This includes employing AI to design intelligent algorithms that better distinguish bot activity from human input.

	= Evaluating Machine Learning (ML) Hardware, Software, and Services =		= Evaluating Machine Learning (ML) Hardware, Software, and Services =

@@ Line 271: / Line 271: @@
 * [https://youtube.com/shorts/rme6PT7-CRI?si=JQnH2Xs6rkIpUPND  I'm not a robot ... explained]
-When you click the checkbox, reCAPTCHA monitors:
+No CAPTCHA reCAPTCHA: Popularized by Google, this involves a simple checkbox labeled "I am not a robot." It works by analyzing user behavior, such as mouse movements to determine if the user is human. If the test is inconclusive, a more traditional image selection CAPTCHA is presented. When you click the checkbox, reCAPTCHA monitors:
 * Mouse movements: Human mouse movements tend to be unpredictable, while bots often exhibit linear or mechanical movements.
@@ Line 277: / Line 277: @@
 == Improving AI while trying to outsmart it ==
-The so-called "bot test"—like CAPTCHAs, where users identify objects in images or complete other seemingly trivial tasks—has a dual purpose. While it's meant to distinguish between humans and bots, the data collected often helps train AI systems to improve at tasks like image recognition, text understanding, or problem-solving.
+The so-called "bot test"—like CAPTCHAs, where users identify objects in images or complete other seemingly trivial tasks—has a dual purpose. While it's meant to distinguish between humans and bots, the data collected often helps train AI systems to improve at tasks like image recognition, text understanding, or problem-solving. the effectiveness of CAPTCHAs is constantly being challenged by advancements in artificial intelligence and machine learning. Recent research has demonstrated that advanced AI can effectively solve image-based CAPTCHAs, such as Google's reCAPTCHAv2, with a 100% success rate using YOLO models for image segmentation and classification. This highlights the need for CAPTCHA systems to evolve in response to AI advancements.
 In a way, humans doing these tests are teaching the bots to get better at beating the tests themselves. It's a fascinating cycle of humans improving AI while trying to outsmart it! Irony at its finest.