> For the complete documentation index, see [llms.txt](https://sisyphus.gitbook.io/project/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://sisyphus.gitbook.io/project/machine-learning/frequent-questions.md).

# Frequent Questions

### Bias vs Variance :laughing: 

**Simple model**, high bias -> lower the bias -> complex model -> overfitting -> higher variance

**Complex model**, high variance -> lower the variance -> simplified model -> underfitting -> higher bias

High order polynomial function is the more complex model.

### KNN :astonished: 

To determine a new input instance's class/regress value, find the K closest history data points and do a majority vote to predict.

* **Normalize** the features to avoid over/under-estimation for large/small value features. 
* **KD tree**, grids on multi-dimension space,  to accelerate the K neighbors' finding procedure.
* **Weighted neighbors**, the closer the larger weights would be, to make it more sensitive to similarity.

![与当前最近距离半径为圆相交的边界需要左右均检查，不相交则不需要再检查 the other half](https://443921002-files.gitbook.io/~/files/v0/b/gitbook-legacy-files/o/assets%2F-LGHUhl6VYqrZm4Re77O%2F-MXUJkdM98hVmZQg6KGs%2F-MXVUs__yijzNVk4KNu5%2Fkd-tree.png?alt=media\&token=fd4deb79-6d2c-4723-9507-9847f365f2ea)

### K-means  :face\_with\_monocle: 

A way to put unlabeled data into K groups where data points in the same group are similar and data points in different groups are far apart.

* Choose K -> random initialize centers -> iteratively recalculate centers with each cluster's distribution.
*  **Pros**:
  * Easy to understand, guarantee to converge, scalable to large data, relative fast
* **Cons**:
  * manually set K, initial sensitive, outliers sensitive, linear boundaries, O(n) for each step
* **To improve**:
  * verify performance with different K, **select K at the turning point**
  * K-means++, **sequentially select initial centers** in a way that the new center is far away from the previous ones.
  * Pre-processing to **normalize and filter outliers**
  * **Use kernel** to map data points into high dimensions. Then apply linear boundaries there.

### KNN vs K-means :thinking: 

**Both** rely on measuring distances, Euclidean/Manhattan distance, power-> n, cosine similarity, etc. 

* KNN has to have labels for each history data sample ahead. 
* K-means is an unsupervised learning method and doesn't require labels at all.

### Metrics :triumph: 

**Positive/Negative**: the predicted results; **True/False**: whether the prediction is correct.

**TP, TN**: Correctly predicted positive and correctly predicted negative

**FP, FN**: Wrongly predicted positive and wrongly predicted negative

TP / (TP + FP) -> among all positive predictions, how much correct: **precision**

TP / (TP + FN) -> among all positive groud truths, how much correct: **recall, True positive rate**

FP / (FP + TN) -> among all negative ground truths, how much wrong: **False Positive rate**

**ROC curve:** TPR + FPR

### Bayes' Theorem :cowboy: 

P(theta | x) = P(x | theta) \* P(theta) / P(x)

**P(x, theta):** P(x | theta) \* P(theta), joint probability 

**P(x | theta)**: the probability of event B occurring given that A == likelihood of A given B.

**P(theta)**: Priori probability

**P(theta | x)**: Posterior probability 

**Naivety**: the conditional probability is calculated as the pure product of the individual probabilities of components. This **implies the absolute independence of features** — a condition probably never met in real life.

### L1 L2 Regularization 

Only the loss term: **empirical** risk minimization

Loss + regularization: **structural** risk minimization

**L1** means theta prior probability is Laplacian, **L2** means theta prior probability is gaussian. <https://www.bilibili.com/video/BV1aE411L7sj?p=6&spm_id_from=pageDriver>

L2 regularization tends to spread the loss on all terms, L1 is more sparse/binary.

### Type I Error and Type II Error

Type I error: false positive

Type II error: false negative

### Fourier Transform

A Fourier transform converts a signal from time to frequency domain—it’s a very common way to extract features from audio signals or other time series such as sensor data.

### Likelihood

In statistics, the likelihood function (often simply called the likelihood) measures the goodness of fit of a statistical model to a sample of data for given values of the unknown parameters.

### Generative vs Distinctive

A generative model will learn the distribution of data while a discriminative model will learn the distinction between different categories of data. The distinctive model can better at performance.