Ep 43 — Chemometrics: PCA, PLS and Multivariate Analysis

Series: Infrared Spectroscopy Encyclopedia: From Principle to Practice
Chapter: Chapter 4 · Advanced — Cutting-edge Technologies (Part 3, Post-stage)
Target Audience: Graduate students in analytical chemistry/chemometrics/spectroscopy, QC/QA laboratory managers, infrared users transitioning from 'traditional quantification' to 'multivariate quantification', spectral modelers in agriculture/pharmaceutical/food industries
Prerequisites: Ep 18 (Spectra Processing), Ep 20 (Beer-Lambert Law), Ep 22 (Drug Polymorphs), Ep 25 (Food Adulteration), Ep 33 (Biomedical), Ep 35 (Agricultural Soil PLSR), Ep 42 (Hyphenated Data)
Reading Time: ~60 minutes

Introduction: From 'Single-Peak Quantification' to 'Full-Spectrum Quantification'

In 1979, Swedish chemist Svante Wold and American chemist Bruce Kowalski jointly published a landmark paper in Anal Chem [1], formally defining Chemometrics — 'the application of mathematical and statistical methods to extract maximum information from chemical measurement data.' Around the same time, the Wold duo (Svante and Herman) in Sweden and Martens in the United States independently proposed the Partial Least Squares (PLS) regression algorithm [2][3], ushering in a new era for spectral analysis.

"Chemometrics is the science of relating measurements made on a chemical system to the state of the system via application of mathematical or statistical methods."
— Massart D L et al. Chemometrics: A Textbook [4]

In the field of infrared spectroscopy, chemometrics is 'indispensable' due to the following three realities:

1. Severe peak overlap: In a spectrum covering 4000–400 cm⁻¹, almost no completely isolated peaks exist. For example, the amide I band (1600–1700 cm⁻¹) contains subpeaks from various secondary structures such as α-helices, β-sheets, and turns in proteins;
2. Baseline drift and scatter: Particle scattering and KBr moisture absorption can tilt the entire spectrum, and single-peak quantification is strongly affected by the matrix;
3. Component interactions: Hydrogen bonding and dipole interactions among components in a mixture can alter peak positions and intensities.

Traditional single-peak quantification (Beer-Lambert law A = εbc) is inadequate when facing the above complexities. The core idea of chemometrics is: use all data points (thousands of wavenumbers) in the entire spectrum to build a model jointly, averaging out noise, incorporating matrix effects into the model, and extracting hidden information from overlapping peaks [4][5].

In this episode, we will systematically explain the core methods of infrared chemometrics: preprocessing → PCA exploratory analysis → PLS quantitative modeling → model validation → avoiding overfitting → open-source tool practice.

1. Data Preprocessing: The 'Foundation' of Chemometrics

1.1 Why Is Preprocessing So Critical?

There is a saying in chemometrics: 'Garbage in, garbage out.' Preprocessing accounts for 60–80% of the entire modeling workload, and the quality of preprocessing determines the upper limit of the model [4][5][6].

Common 'defects' in infrared spectra:

1.2 Mainstream Preprocessing Methods

1.2.1 Baseline Correction

The purpose of baseline correction is to eliminate 'non-chemical' signals from the spectrum [6][7].

• Polynomial fitting: Use a low-order polynomial (≤ 3rd order) to fit the 'valley' of the spectrum and subtract it to obtain the baseline-corrected spectrum;
• Rubber Band: Treat the spectrum as a 'convex hull', stretch a 'rubber band' along the bottom, and the curve that rebounds is the baseline;
• ALS (Asymmetric Least Squares): Proposed by Eilers in 2005 [7], robust to noise and preserves broad peaks well. It is currently the most recommended algorithm;
• arPLS (Asymmetrically Reweighted Penalized Least Squares): An improved version of ALS that automatically adjusts weights;
• Whittaker filter: Based on difference smoothing and sparse constraints.

📦 Recommended Tool: pybaselines Python library provides 200+ baseline correction algorithms, making it the 'Swiss Army knife' for infrared baseline correction [8].

from pybaselines import Baseline
import numpy as np

# Assume wavenumber (1D), absorbance (1D) are loaded
baseline_fitter = Baseline(x=wavenumber)
bkg, params = baseline_fitter.asls(
    absorbance, lam=1e7, p=0.01
)  # ALS algorithm
corrected = absorbance - bkg

1.2.2 SNV (Standard Normal Variate)

SNV is a classic method to eliminate overall intensity differences and baseline tilt [5][6]:

$$ x_{\text{SNV}}(i) = \frac{x(i) - \bar{x}}{\sigma_x} $$

It performs mean centering and scaling independently for each spectrum. SNV does not require a reference spectrum and is the most widely used normalization method in NIR/MIR.

1.2.3 MSC (Multiplicative Scatter Correction)

MSC specifically targets baseline tilt caused by scattering [5][6]:

1. Compute the mean spectrum $\bar{x}$ of all samples;
2. Perform linear regression for each sample spectrum x: $x = a + b \cdot \bar{x}$;
3. Correct: $x_{\text{MSC}} = (x - a) / b$.

MSC vs SNV: Their effects are similar, but MSC requires a reference mean spectrum, while SNV is more robust. Agricultural soil spectra often use SNV, while polymer spectra often use MSC [5].

1.2.4 Derivative Processing

Derivatives can simultaneously eliminate baseline drift and enhance peak resolution [6]:

• First derivative: Removes constant baseline and highlights changes in peak slope;
• Second derivative: Removes linear baseline, turns overlapping peaks into 'negative peaks', and enhances apparent resolution;
• Savitzky-Golay derivative: Fits a low-order polynomial locally and then differentiates, avoiding noise amplification.

from scipy.signal import savgol_filter
# 9-point window + 2nd order polynomial → first derivative
d1 = savgol_filter(absorbance, window_length=9, polyorder=2, deriv=1)
# 11-point window + 2nd order polynomial → second derivative
d2 = savgol_filter(absorbance, window_length=11, polyorder=2, deriv=2)

⚠️ Derivative Trap: Differentiation amplifies noise. Window length and polynomial order must be chosen carefully — too short a window leads to noise explosion, too long a window distorts peak shapes. Rule of thumb: window length should be 1–2 times the full width at half maximum (FWHM) of the peaks.

1.2.5 Normalization

• Max normalization: Divide each spectrum by its maximum value;

• Area normalization: Divide each spectrum by its total area;

• ATR Correction: The penetration depth of ATR spectra varies with wavelength (longer wavelength penetrates deeper), so ATR correction is needed: $A{\text{corr}} = A \cdot \nu / \nu{\text{ref}}$.

1.2.6 Mean Centering

$$ x_{\text{mc}}(i) = x(i) - \bar{x}(i) $$

Subtract the column mean for each column (each wavenumber). This is a mandatory preprocessing step for PCA and PLS, enabling algorithms to focus on "variation" rather than "absolute value".

1.3 Preprocessing Combination Strategies

In practical modeling, multiple preprocessing steps are usually combined [4][5]:

Table 1: Recommended preprocessing combinations for typical application scenarios

💡 More preprocessing is not always better: Each preprocessing step may lose information or introduce bias. It is recommended to start with the simplest combination (only mean centering), gradually add SNV/derivatives, compare model performance, and use the validation set to decide the final scheme.

II. PCA: Exploratory Analysis and Dimensionality Reduction

2.1 Principle of PCA

Principal Component Analysis (PCA) is a fundamental tool in chemometrics [4][5][9]. Its core idea is to reduce high-dimensional data (e.g., 1000 wavenumbers × 100 samples) to a few principal components (PCs) while retaining as much variance as possible.

Mathematically, for the mean-centered data matrix X (n×p, n = number of samples, p = wavenumbers), singular value decomposition is performed:

$$ X = U \cdot S \cdot V^T $$

where:

• Scores T = U·S: n×k matrix, each sample's "score" on each principal component;
• Loadings P = V: p×k matrix, each wavenumber's "weight" on each principal component;
• Eigenvalues: diagonal elements of S, reflecting the variance explained by each PC.

2.2 Chemometric Applications of PCA

2.2.1 Dimensionality Reduction

Infrared spectra typically have 1000–3000 wavenumber points, but the effective information dimension is usually only 5–20. PCA compresses the data into a few principal components [9]:

• The first 3–5 PCs usually explain > 90% variance;
• Subsequent PCs are mainly noise;
• After dimensionality reduction, the model is more stable, computation is faster, and visualization is easier.

2.2.2 Clustering and Outlier Detection

Scores plot projects samples onto two dimensions (e.g., PC1 × PC2):

• Similar samples should cluster together;
• Different samples should separate;
• Samples far from the main cluster may be outliers.

Outlier detection criteria [5][9]:

• Hotelling T²: Mahalanobis distance in the PC1-PC2 plane; beyond the 95% confidence ellipse indicates an outlier;
• Q residual: Unexplained variance by the model; beyond the 95% confidence line indicates an outlier;
• Leverage: The influence of each sample on the model.

   PC2
    ↑
  3 +     ●●●
    |    ●●●●           ★
  2 +   ●●●●●●         (Outlier)
    |    ●●●●●
  1 +     ●●●
    |   (Class A)       ●●●
  0 + ────────────●●●●●●────────→ PC1
    |               ●●●●●
   -1+                ●●●●
    |                (Class B)
   -2+

Figure 1: Typical PCA scores plot (PC1 vs PC2) showing clustering of two classes and an outlier

2.2.3 Interpretation of Loadings

Loadings plot shows the contribution of each wavenumber to the PC:

• Wavenumbers with large absolute loadings are the "main contributors" to that PC;
• Opposite signs indicate that these wavenumbers vary inversely on that PC;
• Used to interpret the chemical meaning of a PC (e.g., PC1 might represent "moisture content", PC2 might represent "protein content").

2.3 PCA in Practice: Olive Oil Adulteration Detection

A laboratory collected 100 olive oil samples (70 authentic + 30 adulterated), acquired ATR-FTIR spectra, and performed PCA [10]:

1. Preprocessing: arPLS baseline + SNV + Mean centering;
2. PCA: Retained the first 5 PCs, explaining 95% variance;
3. Scores plot: PC1 × PC2 clearly separated authentic and adulterated samples;
4. Loadings plot: PC1 main contributing wavenumbers at 1745 cm⁻¹ (triglyceride C=O) and 3005 cm⁻¹ (=C-H cis double bond);
5. Inference: Adulterated samples deviated from authentic ones in characteristic peaks of triglycerides and unsaturated fatty acids;
6. Application: A discrimination threshold based on PC1 + PC2 scores achieved 96% accuracy.

🔗 Further Reading: For characteristic absorptions of C=O, C-H in olive oil, see ftir.fun carbonyl functional group page and ftir.fun alkyl C-H functional group page.

2.4 Limitations of PCA

PCA is an unsupervised method, not using sample labels (e.g., authentic/adulterated):

• When different classes overlap in PC1-PC2, PCA cannot distinguish them;
• Solution: Use supervised methods (PLS-DA, SVM, see below).

III. PLS: Quantitative Calibration Model

3.1 Principle of PLS

Partial Least Squares (PLS) regression is the "gold standard" for infrared quantification [2][3][5].

PLS decomposes both X (spectra) and y (reference values) simultaneously, building on PCA:

$$ X = T \cdot P^T + E, \quad y = T \cdot q + f $$

By maximizing the covariance between X and y, PLS finds latent variables (LVs) that both explain X variance and correlate with y.

Advantages of PLS:

• Handles highly collinear data in X (adjacent wavenumber points are highly correlated);
• Handles the p >> n situation (more wavenumbers than samples);
• Simultaneously considers the relationship between X and y, more efficient than the two-step PCA-regression approach;
• Strong noise resistance.

3.2 PLS Modeling Procedure

Complete PLS modeling procedure [5][11]:

   ┌──────────────────────────────────────────┐
   │ 1. Sample collection and division         │
   │    - Calibration set 70% (n=70)           │
   │    - Validation set 30% (n=30)            │
   ├──────────────────────────────────────────┤
   │ 2. Preprocessing                          │
   │    - Baseline + SNV + derivative + MC    │
   ├──────────────────────────────────────────┤
   │ 3. Cross-validation to determine optimal   │
   │    number of LVs                          │
   │    - LOO (leave-one-out) or 10-fold CV    │
   │    - Select LV number with minimum RMSECV  │

├──────────────────────────────────────────┤
   │ 4. Build final model with optimal LV number   │
   │    - Train using calibration set              │
   ├──────────────────────────────────────────┤
   │ 5. External validation                      │
   │    - Evaluate using validation set           │
   │    - Calculate RMSEP, R², RPD               │
   ├──────────────────────────────────────────┤
   │ 6. Model maintenance                        │
   │    - Periodically update model with new samples│
   └──────────────────────────────────────────┘

> Figure 2: Complete PLS modeling workflow

### 3.3 Model evaluation metrics

Common evaluation metrics for PLS models [5][11]:

| Metric | Full name | Meaning | Excellent standard |
|--------|-----------|---------|-------------------|
| **R²** | Coefficient of determination | Fit between predicted and true values | > 0.90 |
| **RMSEC** | Root mean square error of calibration | Prediction error in calibration set | Smaller is better |
| **RMSECV** | Root mean square error of cross-validation | Cross-validation prediction error | Smaller is better |
| **RMSEP** | Root mean square error of prediction | External validation prediction error | Smaller is better |
| **RPD** | Residual prediction deviation | SD/RMSEP | > 3 excellent; > 5 can replace laboratory methods |
| **Bias** | Systematic bias | Average offset of predicted values | Should be close to 0 |
| **RER** | Range error ratio | (y_max − y_min)/RMSEP | > 10 excellent |

> Table 2: PLS model evaluation metrics (refer to Williams & Norris [11])

**RPD (Residual Prediction Deviation)** is the most commonly used "rating index" in chemometrics [11]:

- RPD < 2: Model unreliable;
- RPD 2–3: Model usable, but limited accuracy;
- RPD 3–5: Model good;
- RPD > 5: Can replace laboratory reference method.

### 3.4 Cross-validation: Key to preventing overfitting

**Cross-Validation (CV)** is the key to selecting the optimal number of LVs [5][9]:

1. Split the calibration set into K folds (typically K = 10);
2. Leave one fold as "internal validation", train with the remaining K-1 folds;
3. Calculate RMSECV for each LV number;
4. Select the LV number with the smallest RMSECV (or where it starts to plateau).

RMSECV
↑
High│●
│ ●
│ ●
│ ● ← minimum (optimal LV)
│ ●●●●●● ← plateau
│
└──────────────→ LV number
1 2 3 4 5 6 7 8 9 10

> Figure 3: RMSECV curve as a function of LV number. Select the LV number near the minimum to avoid overfitting.

**Leave-One-Out (LOO)** is the extreme case with K = n:

- Advantages: Full use of data;
- Disadvantages: High computational cost, easily underestimates error;
- Recommendation: Use LOO when n < 30, otherwise use 10-fold CV.

### 3.5 PLS in practice: Soil organic carbon prediction

Refer to Ep 35 case [12]:

- **Samples**: 500 Australian soils (CSIRO dataset);
- **Spectra**: MIR-DRIFTS (4000–400 cm⁻¹, one point every 4 cm⁻¹, total 900 points);
- **Reference values**: SOM content determined by Walkley-Black dichromate oxidation method;
- **Preprocessing**: SNV + first derivative (S-G 9 points) + mean centering;
- **Calibration set**: 350 samples; Validation set: 150 samples;
- **LV number**: 8 LVs selected by 10-fold CV;
- **Results**: R² = 0.93, RMSEP = 0.32%, RPD = 3.6 (reaching "excellent" level);
- **Loading analysis**: LV1 main contribution wavenumbers: 2925/2850 cm⁻¹ (CH₂), LV2: 1720 cm⁻¹ (C=O);
- **Application**: Model used for large-scale SOM survey in Australian farmland.

> 🔗 **Further reading**: For infrared characteristics of soil organic matter, see Ep 35, and [ftir.fun alkyl C-H functional group page](https://ftir.fun/ir/group/alkyl-c-h) and [ftir.fun carbonyl functional group page](https://ftir.fun/ir/group/carbonyl).

### 3.6 PLS variants

- **PLS-DA (Discriminant Analysis)**: y is a classification label (0/1), used for discrimination (e.g., genuine/adulterated);
- **PLS2**: y is a multi-column matrix for simultaneous prediction of multiple properties;
- **N-PLS**: Used for three-dimensional data (e.g., Ep 42 hyphenated data);
- **SVM-PLS**: Combines support vector machines with PLS to handle non-linear problems;
- **CARS-PLS**: Uses competitive adaptive reweighted sampling to select key wavenumbers and simplify the model.

---

## 4. Overfitting: The 'ghost' of chemometrics

### 4.1 Symptoms of overfitting

**Overfitting** is the most common pitfall in chemometrics [5][11]:

- Calibration R² extremely high (> 0.99), but validation R² drops significantly;
- RMSEC much smaller than RMSEP;
- Model 'memorizes' training data perfectly, but **predicts poorly on unknown data**.

Error
↑
│ RMSEC ●●●●●●●●●● ← continues to decrease
│
│ RMSECV ●●●●●●↓↑↑ ← starts to increase after a point!
│ ↑
│ optimal LV number
│
└──────────────────→ LV number

> Figure 4: Typical symptom of overfitting – RMSEC continues to decrease but RMSECV increases after the optimal LV number

### 4.2 Causes of overfitting

1. **Too many LVs**: Modeling noise as well;
2. **Too few samples**: n < 10 × LV number;
3. **Excessive preprocessing**: Too many derivatives, too short window length;
4. **Inappropriate wavenumber selection**: Including too many noisy regions;
5. **Inconsistent distribution between calibration and validation sets**: e.g., calibration set at 20°C, validation set at 30°C.

### 4.3 Strategies to avoid overfitting

| Strategy | Description |
|----------|-------------|
| Use RMSECV to select LV number | Select LV number with minimum RMSECV, not highest R² |
| External validation set | Reserve 20–30% of samples for independent validation |
| Increase number of samples | n ≥ 10 × LV number, preferably n ≥ 100 |
| Reduce wavenumbers | Remove noisy regions using variable selection methods like CARS, GA |
| Simplify preprocessing | Use the simplest preprocessing combination |
| Diverse real samples | Calibration set covers future actual application scenarios |

> Table 3: Strategies to avoid overfitting

### 4.4 Model transferability

Model 'transferability' between different instruments and laboratories is another challenge in chemometrics [5]:

- Slight differences in spectra of the same sample on different instruments (instrument response differences);
- Solution: **DS (Direct Standardization), PDS (Piecewise Direct Standardization)** algorithms standardize spectra from the 'slave' instrument to the 'master' instrument;
- **SBC (Slope/Bias Correction)**: Simple slope/bias correction, suitable for cases with small differences.

---

## 5. Other multivariate analysis methods

### 5.1 SIMCA

**SIMCA (Soft Independent Modeling of Class Analogy)** is a supervised discriminant method based on PCA [9]:

- PCA models are built separately for each class;
- New samples calculate the "distance" to each class (residual variance and leverage);
- The class with the smallest distance is the predicted class;
- Advantages: each class is modeled independently, handling class imbalance;
- Applications: drug polymorph identification.

### 5.2 SVM

**Support Vector Machine (SVM)** is a machine learning method (see Ep 44 for details) [13]:

- Handles nonlinearity through kernel functions (RBF, polynomial);
- Performs well in spectral classification;
- Suitable for small sample sizes (n < 100);
- Disadvantages: complex parameter tuning.

### 5.3 MCR-ALS

**MCR-ALS (Multivariate Curve Resolution)** is used for resolving mixture spectra [14]:

- No reference values needed, purely soft-constrained resolution;
- Outputs pure spectra and concentration profiles for each component;
- Applications: Ep 42 hyphenated data analysis.

### 5.4 PARAFAC

**PARAFAC (Parallel Factor Analysis)** is used for three-way data [14]:

- Similar to PCA, but for three-way arrays;
- Outputs three "loading vectors";
- Applications: TGA-FTIR, GC-FTIR, EEM fluorescence data analysis.

### 5.5 ANN (Artificial Neural Network)

ANN was applied to infrared spectroscopy in the 1990s, but has been replaced by deep learning (see Ep 44 for details).

---

## 6. Open Source Tools Recommendations and Practice

### 6.1 pybaselines: Baseline Correction Tool

[**pybaselines**](https://github.com/derb12/pybaselines) [8]:

- 200+ baseline correction algorithms;
- Native Python, compatible with NumPy/SciPy;
- Well-documented with rich examples;
- It is the de facto standard for infrared baseline correction.

```python
from pybaselines import Baseline
import numpy as np

baseline_fitter = Baseline(x_data=wavenumber)
# ALS (most classic)
bkg1 = baseline_fitter.asls(y, lam=1e7, p=0.01)[0]
# arPLS (adaptive weights)
bkg2 = baseline_fitter.arpls(y, lam=1e6, ratio=0.01)[0]
# Polynomial fitting
bkg3 = baseline_fitter.poly(y, poly_order=3)[0]

6.2 SpectroChemPy: Integrated Spectral Analysis Framework

SpectroChemPy [15]:

• 177+ stars, actively maintained;
• NDDataset data structure: spectra + metadata integrated;
• Full workflow: reading (multi-vendor formats) → preprocessing → modeling → visualization;
• API design inspired by scikit-learn, gentle learning curve;
• High educational value.

import spectrochempy as scp

# read Bruker OPUS file
ds = scp.read_opus("sample.0")
# baseline correction
ds = ds.baseline(algorithm='asls', lam=1e7, p=0.01)
# SNV
ds = ds.snv()
# PCA
pca = scp.PCA(n_components=5)
pca.fit(ds)
scores = pca.transform(ds)

6.3 Orange-Spectroscopy: No-Code Spectral Analysis

Orange-Spectroscopy [16]:

• Drag-and-drop visual workflow, zero programming required;
• Seamless integration of preprocessing and machine learning;
• Suitable for classroom teaching and rapid prototyping;
• Built-in PCA, PLS, PLS-DA, Random Forest, etc.;
• Supports various formats such as JCAMP-DX, SPC, SPA.

6.4 scikit-learn: General-Purpose Machine Learning Framework

scikit-learn provides complete implementations of PCA, PLS, SVM, Random Forest:

from sklearn.decomposition import PCA
from sklearn.cross_decomposition import PLSRegression
from sklearn.model_selection import cross_val_predict, KFold
from sklearn.metrics import mean_squared_error, r2_score

# PCA
pca = PCA(n_components=5)
scores = pca.fit_transform(X_centered)

# PLS
pls = PLSRegression(n_components=8)
pls.fit(X_train, y_train)
y_pred = pls.predict(X_test)
rmsep = np.sqrt(mean_squared_error(y_test, y_pred))
r2 = r2_score(y_test, y_pred)

# 10-fold CV to select optimal number of LVs
kf = KFold(n_splits=10)
for n in range(1, 15):
    pls = PLSRegression(n_components=n)
    y_cv = cross_val_predict(pls, X_train, y_train, cv=kf)
    rmsecv = np.sqrt(mean_squared_error(y_train, y_cv))
    print(f"LV={n}, RMSECV={rmsecv:.4f}")

6.5 Tool Selection Recommendations

Table 4: Comparison of Open Source Chemometric Tools

7. Chemometrics Best Practices Checklist

7.1 Data Collection

1. Sample size ≥ 100: models are unreliable when n < 30;
2. Cover future application range: the concentration range of the calibration set should cover the actual application scenario;
3. Replicate measurements: at least 2 measurements per sample to avoid single measurement bias;
4. Randomization: avoid systematic order (e.g., measuring high concentrations before low concentrations);
5. Accurate reference method: the quality of reference values determines the model limit.

7.2 Data Splitting

• Calibration set 60–70%, validation set 30–40%;
• Use Kennard-Stone algorithm for 'uniform splitting', which is more robust than random splitting;
• The validation set must be independent of the calibration set and must not be used in preprocessing parameter selection.

7.3 Model Building

• Use cross-validation to select the number of LVs;

• Use the simplest preprocessing combination.

• Check residual plot: residuals should be random, no trend;

• Check leverage: high leverage samples need attention.

7.4 Model Validation

• External validation set: R², RMSEP, RPD;
• Predicted residual plot: check for systematic bias;
• Stability test: repeat measurements on different days/operators/instruments;
• Long-term monitoring: periodically validate model with fresh samples.

7.5 Model Deployment

• Document preprocessing workflow;
• Provide sample "applicability" criteria (e.g., whether new sample is within model range);
• Regularly update model (drift correction);
• Build a "failed" sample library and retrain model.

8. Typical Cases: From Lab to Industry

8.1 Case 1: NIR Rapid Quantification of Protein Content in Feed

A feed factory needs online monitoring of protein content in feed [11][17]:

• Samples: 1200 feed samples with different formulations;
• Spectra: NIR 1000–2500 nm (2 nm interval, 750 points);
• Reference: Kjeldahl method;
• Preprocessing: SNV + 1st derivative (S-G 11 points);
• Calibration set: 900; Validation set: 300;
• LVs: 12;
• Results: R² = 0.96, RMSEP = 0.38%, RPD = 4.5;
• Application: Deployed on production line for online monitoring, saving $50,000/year in laboratory costs.

8.2 Case 2: ATR-FTIR Quantification of Active Ingredient in Pharmaceutical Tablets

A pharmaceutical company needs rapid determination of acetaminophen content in tablets [18]:

• Samples: 300 tablets from different batches;
• Spectra: ATR-FTIR 4000–400 cm⁻¹;
• Reference: HPLC;
• Preprocessing: arPLS + 2nd derivative;
• Calibration set: 210; Validation set: 90;
• LVs: 6;
• Results: R² = 0.95, RMSEP = 1.2%, RPD = 4.1;
• Application: Replaces part of HPLC testing, saving 4 hours analysis time per batch.

🔗 Further Reading: Characteristic absorptions of amide and hydroxyl groups in drug molecules can be found at ftir.fun amide functional group page and ftir.fun hydroxyl functional group page.

8.3 Case 3: Rapid Screening of Melamine in Food

Extension of Ep 25 case study [19]:

• Samples: 200 milk powder + melamine (0–500 ppm);
• Spectra: ATR-FTIR 4000–400 cm⁻¹;
• Preprocessing: arPLS + SNV + 1st derivative;
• Calibration set: 140; Validation set: 60;
• PLS-DA classification model: detection limit 50 ppm, accuracy 95%;
• Loading analysis: 1550/1500 cm⁻¹ (triazine ring C=N), 3350/3450 cm⁻¹ (NH₂);
• Application: Used for rapid screening at milk collection stations to prevent melamine incident recurrence.

🔗 Further Reading: N-H stretching and triazine ring vibrations of melamine can be found at ftir.fun amine functional group page.

9. Future of Chemometrics: Integration with Machine Learning

The boundary between chemometrics and machine learning is increasingly blurred [13] (see Ep 44):

• Traditional chemometrics: PCA, PLS, SIMCA, strong interpretability, suitable for small samples;
• Machine learning: SVM, Random Forest, XGBoost, strong nonlinear capability, suitable for medium samples;
• Deep learning: CNN, Transformer, automatic feature extraction, requires large samples (> 1000);
• Explainable AI: SHAP, LIME, SSIN, making black boxes transparent.

Future trends:

1. Popularization of explainable AI: models should not only be "correct" but also "know why they are correct";
2. Federated learning: collaborative modeling across labs without sharing raw data;
3. AutoML: automatic selection of preprocessing and models;
4. Quantum chemistry + ML: using DFT-computed spectra to aid small-data learning.

Summary of This Episode

Questions

1. You need to build a PLS model to predict protein content in milk. Describe the complete preprocessing workflow and explain why the NIR region is usually more suitable for such quantification (hint: consider water interference).

2. When building a PLS quantitative model, the calibration set R² = 0.98 but the validation set R² is only 0.65. List at least 4 possible causes and corresponding diagnostic methods.

3. What is the core mathematical difference between PCA and PLS? In what scenarios would you prefer PCA over PLS? Give a practical example.

4. When using the pybaselines library for baseline correction, you find the two parameters lam and p of the ALS algorithm difficult to adjust. Explain the physical meaning of these two parameters and how to choose a reasonable range of values (hint: lam controls smoothness, p controls asymmetry).

5. Given 500 DRIFTS spectra of soil samples and corresponding SOM reference values, you are required to build a PLS model suitable for industrial deployment. Design a complete workflow: data splitting method, preprocessing combination, LV number selection, validation strategy, model performance targets, and deployment considerations.

References

[1] Kowalski B R. "Chemometrics: Views and Propositions." J Chem Inf Comput Sci, 1979, 19(4): 234–238. DOI:10.1021/ci60020a004.
See also Wold S. "Chemometrics: What do we mean with it, and what do we want from it?" Chemom Intell Lab Syst, 1995, 30(1): 109–115.

[2] Wold H. "Estimation of Principal Components and Related Models by Iterative Least Squares." In: Multivariate Analysis, ed. Krishnaiah P R, Academic Press, 1966: 391–420.

[3] Wold S, Sjöström M, Eriksson L. "PLS-Regression: A Basic Tool of Chemometrics." Chemom Intell Lab Syst, 2001, 58(2): 109–130. DOI:10.1016/S0169-7439(01)00155-1.

[4] Massart D L, Vandeginste B G M, Buydens L M C, et al. Chemometrics: A Textbook. Elsevier, 1988. ISBN: 978-0-444-42660-6.

[5] Naes T, Isaksson T, Fearn T, Davies T. A User-Friendly Guide to Multivariate Calibration and Classification. NIR Publications, 2002. ISBN: 978-0-9528666-2-6.

[6] Rinnan Å, van den Berg F, Engelsen S B. "Review of the Most Common Pre-Processing Techniques for Near-Infrared Spectra." Trends in Analytical Chemistry, 2009, 28(10): 1201–1222. DOI:10.1016/j.trac.2009.07.007.

[7] Eilers P H C, Boelens H F M. "Baseline Correction with Asymmetric Least Squares Smoothing." Leiden Univ Medical Centre Report, 2005, 1(1): 5.

[8] pybaselines: A Python package for baseline correction. GitHub: https://github.com/derb12/pyb…

[9] Brereton R G. Chemometrics: Data Analysis for the Laboratory and Chemical Plant. Wiley, 2003. ISBN: 978-0-471-48978-8.

[10] Casale M, Casolino C, Ferrari G, et al. "Olive Oil Adulteration Detection by ATR-FTIR and Chemometrics." Food Anal Methods, 2010, 3: 195–203.

[11] Williams P C, Norris K. Near-Infrared Technology in the Agricultural and Food Industries. 3rd ed. AACC International Press, 2001. ISBN: 978-1-891127-37-6.

[12] Janik L J, Skjemstad J O, Raven M D. "Characterization and Analysis of Soils Using Mid-Infrared Partial Least-Squares." Soil Biology & Biochemistry, 2008, 40(2): 412–423. DOI:10.1016/j.soilbio.2007.09.023.

[13] Buljanić M H, et al. "Machine Learning in Infrared Spectroscopy." Spectrochim Acta A, 2022, 270: 120816.

[14] de Juan A, Tauler R. "Multivariate Curve Resolution (MCR) from 2000: Progress in Concepts and Applications." Crit Rev Anal Chem, 2006, 36(3–4): 163–176.

[15] SpectroChemPy: A Python framework for processing, analyzing and modeling spectroscopic data. GitHub: https://github.com/spectroche…

[16] Orange-Spectroscopy: A package to extend Orange with spectroscopic functionality. GitHub: https://github.com/Quasars/or…

[17] Burns D A, Ciurczak E W. Handbook of Near-Infrared Analysis. 3rd ed. CRC Press, 2007. ISBN: 978-0-8493-7393-0.

[18] Roggo Y, Chalus P, Maurer L, et al. "A Review of Near Infrared Spectroscopy and Chemometrics for Pharmaceutical Analysis." J Pharm Biomed Anal, 2007, 44(3): 683–700.

[19] Lu C, Xiang B, Hao G, et al. "Rapid Detection of Melamine in Milk Powder by Near Infrared Spectroscopy." J Near Infrared Spectrosc, 2009, 17(2): 59–67.

[20] ftir.fun Alkyl C-H Functional Group Page. https://ftir.fun/ir/group/alk…

[21] ftir.fun Carbonyl Functional Group Page. https://ftir.fun/ir/group/car…

[22] ftir.fun Amide Functional Group Page. https://ftir.fun/ir/group/ami…

[23] ftir.fun Hydroxyl Functional Group Page. https://ftir.fun/ir/group/hyd…

[24] ftir.fun Amine Functional Group Page. https://ftir.fun/ir/group/ami…

Next Episode Preview: Ep 44 — Machine Learning in Infrared Spectroscopy
Chemometrics is the "traditional" multivariate approach, while machine learning and deep learning are its "evolutionary version." In the next episode, we will discuss: trade-offs between traditional chemometrics and machine learning; applications of SVM, random forest, and XGBoost in spectral classification; CNN for spectral recognition; and cutting-edge works in 2025 including SSIN explainable AI (detecting functional groups from IR spectra), LLM-driven multitask inference framework for IR spectroscopy, and IR-Bot autonomous robot.

This work is licensed under CC BY-NC-SA 4.0. Images are from public domain or web resources with attributed sources, copyrights belong to their respective owners.