Hyperparameters, Model Optimization & Overfitting Countermeasures

---

Key Takeaways​

Hyperparameters are external configuration settings that control the learning algorithm, model capacity, and training process. Unlike internal model parameters (weights and biases), which are learned automatically from data via backpropagation, hyperparameters are set before training begins.

Finding the optimal combination of hyperparameters is known as Hyperparameter Optimization (HPO) or Automatic Model Tuning (AMT). It directly balances training speed against generalization performance, helping prevent both underfitting and overfitting.

---

Main Discussion​

High-Impact Hyperparameters​

Hyperparameter	Definition & Operational Mechanism	Risk of Too High / Too Many	Risk of Too Low / Too Few
Learning Rate ($\alpha$)	Determines the step size the optimizer takes along the loss gradient when updating internal weights.	• Divergence or oscillating across the loss landscape • Overshooting the global minimum	• Extremely slow convergence • Stalling in poor local minima or saddle points
Batch Size	The number of training samples processed by the network before updating internal weights.	• Requires high GPU memory • May get trapped in sharper local minima	• High training runtime overhead • Noisier gradient updates
Epochs	The total number of complete passes the training algorithm makes through the entire training dataset.	• Overfitting (memorizing sample-specific noise)	• Underfitting (model terminates before learning patterns)
Regularization ($L_1 / L_2$)	Mathematical penalty added to the loss function to penalize overly large weights and constrain complexity.	• Over-simplification / Underfitting (model cannot capture signal)	• High variance / Overfitting (weights grow unrestricted)

$\text{Loss with } L_2 \text{ Regularization (Weight Decay): } \mathcal{L}_{\text{total}} = \mathcal{L}_{\text{data}} + \lambda \sum_{j} w_j^2$

---

Hyperparameter Search Strategies​

To identify the optimal configuration without manual guesswork, data scientists use automated search strategies:

---

Overfitting: Root Causes & Strategic Countermeasures​

Overfitting occurs when a model achieves high accuracy on training data but fails on unseen validation/test data:

Countermeasure	Mechanism & Implementation	Impact on Model Fit
Increase Training Data	Collect more labeled records or pull broader operational data pools.	Most effective resolution; exposes the model to true population variance.
Early Stopping	Monitor validation loss during training and automatically stop when validation error begins to climb, even if training loss continues to fall.	Prevents the model from training for too many epochs and memorizing noise.
Increase Regularization	Increase the $\lambda$ penalty coefficient ($L_1$ Lasso or $L_2$ Ridge) or add Dropout layers in deep neural networks.	Constrains weight magnitudes, forcing simpler decision boundaries.
Data Augmentation	Synthesize new variations of existing training samples (e.g., image rotations, GAN-generated tabular rows).	Expands effective dataset size without collecting new field data.
Ensemble Modeling	Combine predictions from multiple distinct models (e.g., Random Forests, XGBoost).	Reduces individual model variance and smooths idiosyncratic errors.

---

Exam Guide​

Exam Tips​

• Parameter vs. Hyperparameter Disambiguation:
  • Model Parameters: Internal weights ($w$) and biases ($b$) updated automatically during training by the optimization algorithm.
  • Hyperparameters: External settings (learning rate, batch size, epochs, regularization) configured before training begins.
• Overfitting Remediation Hierarchy:
  • If an exam question asks for the best or most fundamental way to fix overfitting, look for increasing the training dataset size or data augmentation.
  • If the question asks for training-time configuration changes, look for early stopping or increasing regularization ($\lambda$).
• Regularization Rule: To reduce overfitting, you increase regularization. If a model is underfitting, you decrease regularization.
• Epochs & Fit:
  • Too few epochs $\rightarrow$ Underfitting (insufficient learning).
  • Too many epochs $\rightarrow$ Overfitting (memorization of training samples).
• AWS Service Mapping: Automated hyperparameter optimization on AWS is handled by Amazon SageMaker Automatic Model Tuning (AMT).

---

Practice Test​

Question 1​

A data scientist is training a deep neural network on Amazon SageMaker. During training, the training loss steadily decreases toward zero, but the validation loss begins increasing significantly after epoch 25. What is happening to the model, and which two actions can mitigate this issue? (Select TWO.)

• A. The model is underfitting due to a high learning rate
• B. The model is overfitting the training dataset
• C. Implement Early Stopping to halt training when validation loss begins to rise
• D. Decrease the amount of training data in the training set
• E. Increase the number of epochs to 200

Correct Answer

• B. The model is overfitting the training dataset
• C. Implement Early Stopping to halt training when validation loss begins to rise
  • Explanation: When training loss decreases while validation loss increases, the model is overfitting (memorizing training data noise). Implementing Early Stopping halts training at the point of optimal validation performance, preventing excessive epochs from degrading generalization.

---

Question 2​

What is the fundamental difference between a model parameter and a hyperparameter in machine learning workflows?

• A. Model parameters are configured manually before training begins, whereas hyperparameters are learned automatically by the optimizer
• B. Model parameters represent internal weights learned from the data during training, whereas hyperparameters are external configuration settings established prior to training
• C. Model parameters are only used for unsupervised clustering, whereas hyperparameters are exclusive to generative transformers
• D. Model parameters are stored in AWS IAM Identity Center, whereas hyperparameters are stored in Amazon S3

Correct Answer

• B. Model parameters represent internal weights learned from the data during training, whereas hyperparameters are external configuration settings established prior to training
  • Explanation: Model parameters (such as neural network weights and biases) are learned automatically from training data through optimization algorithms. Hyperparameters (such as learning rate, batch size, and regularization) are external configuration settings specified before training begins to guide the learning process.

---

Question 3​

A financial services company is developing machine learning models to predict customer credit risk and detect fraudulent transactions. After deploying the models, the company observes that the predictions are not as accurate as desired, leading to potential financial losses or missed fraud detections. To improve the accuracy of these models, the company has hired you as an AI Practitioner.

Which approach would you recommend to enhance the accuracy of the company's machine learning models?

• A. The company should decrease the learning rate, which reduces the size of the steps taken during optimization
• B. The company should reduce the batch size, which means processing fewer samples in each training step
• C. The company should increase regularization to improve the accuracy of the model
• D. The company should increase the number of epochs, which involves training the model for more iterations over the dataset

Correct Answer

• D. The company should increase the number of epochs, which involves training the model for more iterations over the dataset
  • Explanation: Increasing the number of epochs allows the model to learn from the training data for a longer period, potentially capturing more complex patterns and relationships, which can improve accuracy. Multiple epochs are run until the accuracy of the model reaches an acceptable level, or when the error rate drops below an acceptable level.
    

Incorrect Answers:

• A. The company should decrease the learning rate, which reduces the size of the steps taken during optimization
  • Explanation: Decreasing the learning rate can help the model converge more slowly and steadily, potentially avoiding overshooting the optimal point. However, too small of a learning rate can significantly slow down training and cause the model to get stuck in local minima, thereby not necessarily improving accuracy. It's more of a fine-tuning step once other hyperparameters are appropriately set.
• B. The company should reduce the batch size, which means processing fewer samples in each training step
  • Explanation: Reducing the batch size can make the training process noisier, as each update to the model weights is based on fewer examples. While this might help the model escape local minima during optimization, it does not necessarily lead to an improvement in accuracy and could even slow down convergence or lead to less stable training.
• C. The company should increase regularization to improve the accuracy of the model
  • Explanation: Increasing regularization is beneficial when the model is overfitting, as it adds constraints that penalize complexity, encouraging the model to generalize better. However, if the model is already underfitting (not capturing the patterns in the data well), increasing regularization could further decrease its performance, and it might not improve accuracy.

---

Question 4​

A healthcare analytics company is using Amazon SageMaker Automatic Model Tuning (AMT) to optimize its machine learning models for predicting patient outcomes. To ensure the models are performing at their best, the data science team is configuring the autotune settings but needs to understand which parameters are mandatory for successful tuning. Properly setting these configurations will allow the team to enhance model accuracy and performance efficiently.

Which of the following options is mandatory for the given use case?

• A. Number of jobs
• B. Hyperparameter ranges
• C. None
• D. Tuning strategy

Correct Answer

• C. None
  • Explanation: Choosing the correct hyperparameters requires experience with machine learning techniques and can drastically affect your model performance. Even with hyperparameter tuning, you still need to specify multiple tuning configurations, such as hyperparameter ranges, search strategy, and number of training jobs to launch. Correcting such a setting is intricate and typically requires multiple experiments, which may incur additional training costs.
    Amazon SageMaker Automatic Model Tuning can automatically choose hyperparameter ranges, search strategy, maximum runtime of a tuning job, early stopping type for training jobs, number of times to retry a training job, and model convergence flag to stop a tuning job, based on the objective metric you provide. This minimizes the time required for you to kickstart your tuning process and increases the chances of finding more accurate models with a lower budget.

Incorrect Answers:

• A. Number of jobs
• B. Hyperparameter ranges
• D. Tuning strategy
  • Explanation: None of these entities constitute mandatory configuration for Amazon SageMaker Automatic Model Tuning (AMT).

Reference:

• https://aws.amazon.com/about-aws/whats-new/2023/06/sagemaker-automatic-model-tuning-configurations/

---