Discriminative vs. Generative
YouTube search... ...Google search
Application-specific details ultimately dictate the suitability of selecting a discriminative versus generative model. Model Contrasts...
- Discriminative
- learn the (hard or soft) boundary between classes
- providing classification splits (probabilistic or non-probabilistic manner)
- allow you to classify points, without providing a model of how the points are actually generated
- don't have generative properties
- make few assumptions of the model structure
- less tied to a particular structure
- better performance with lots of example data; higher accuracy, which mostly leads to better learning result
- discriminative models can yield superior performance (in part because they have fewer variables to compute)
- saves calculation resource
- can outperform generative if assumptions are not satisfied (real world is messy and assumptions are rarely perfectly satisfied)
- not designed to use unlabeled data; are inherently supervised and cannot easily support unsupervised learning
- do not generally function for outlier detection
- do not offer such clear representations of relations between features and classes in the dataset
- yields representations of boundaries (more than generative)
- do not allow one to generate samples from the joint distribution of observed and target variables
- generates lower asymptotic errors
- Generative
- requires less training samples
- model the distribution of individual classes
- provides a model of how the data is actually generated
- learn the underlying structure of the data
- have discriminative properties
- make some kind of structure assumptions on your model
- decision boundary: where one model becomes more likely
- often outperform discriminative models on smaller datasets because their generative assumptions place some structure on your model that prevent overfitting
- natural use of unlabeled data
- takes all data into consideration, which could result in slower processing as a disadvantage
- generally function for outlier detection
- typically specified as probabilistic graphical models, which offer rich representations of the independence relations in the dataset
- more straightforward to detect distribution changes and update a generative model
- takes the joint probability and predicts the most possible known label
- typically more flexible in expressing dependencies in complex learning tasks
- a flexible framework that could easily cooperate with other needs of the application
- results in higher asymptotic errors faster
- training method usually requires multiple numerical optimization techniques
- will need the combination of multiple subtasks for a solving complex real-world problem
Generative model takes the joint probability, where x is the input and y is the label, and predicts the most possible known label for the unknown variable using Bayes' Theorem
Discriminative models, as opposed to Generative models, do not allow one to generate samples from the joint distribution of observed and target variables. However, for tasks such as machine learning classification and Regression analysis that do not require the joint distribution, discriminative models can yield superior performance (in part because they have fewer variables to compute). On the other hand, generative models are typically more flexible than discriminative models in expressing dependencies in complex learning tasks. In addition, most discriminative models are inherently supervised and cannot easily support unsupervised learning. Application-specific details ultimately dictate the suitability of selecting a discriminative versus generative model.
Discriminative models and Generative models also differ in introducing the Posterior probability. To maintain the least expected loss, the minimization of result's misclassification should be acquired. In the discriminative model, the posterior probabilities, P(y|x), is inferred from a parametric model, where the parameters come from the training data. Points of estimation of the parameters are obtained from the maximization of likelihood or distribution computation over the parameters. On the other hand, considering that the generative models focus on the joint probability, the class posterior possibility P(k) is considered in Bayes' Theorem
?
