Building a useful machine learning model begins with a decision the business wants to support, not a list of algorithms. A complete workflow defines the target, collects relevant examples, separates evaluation data, trains a baseline, compares candidates and prepares a controlled handoff to deployment. This tutorial concentrates on model development and evaluation rather than the detailed AWS infrastructure choices covered by separate deployment articles.
Older versions of this guide used Google Cloud AI Platform terminology and broad claims about TensorFlow pipelines. Cloud service names and interfaces change, but the core scientific obligations remain: prevent leakage, reproduce preprocessing, measure appropriate outcomes and record the exact experiment. These principles apply to scikit-learn and current managed ML services alike.
Step 1: formulate a narrow prediction task
Consider routing a support ticket to one of three service teams based on its subject and description. The input is the text available when the ticket arrives; the target is the validated team that should handle it. Define how many classes exist, what a wrong assignment costs, how quickly routing must occur and which tickets require manual review.
A model should not infer sensitive facts outside its permitted purpose. Select an input dataset with authorization, appropriate retention and reviewed fields. A team building an employment recommendation model would need additional scrutiny of fairness and whether predictions are appropriate for the decision. A simple support-routing example avoids turning a training lesson into unreviewed high-impact automation.
Define acceptance before seeing test results. For example, the model may need to improve over the current keyword routing method while keeping false assignment of urgent security tickets below a documented limit. A baseline rule can be valuable even when it is not the eventual production design.
Step 2: collect and label representative examples
Create a dataset with one row per incoming ticket, timestamp, approved text fields and verified team label. Check whether repeated tickets or templated messages could appear in both training and evaluation sets. Such near duplicates can make a model seem much better at generalization than it really is.
Inspect class balance, missing labels and inconsistent categories. A service team may have changed names during the collection period; historic labels must be reconciled according to a documented business rule. If the smallest class has too few examples, overall accuracy may be an unreliable summary of its performance.
Separate personally identifying text or confidential message content unless genuinely needed. Remove secrets and access tokens from samples. Data use should have an owner, and the team must be able to reproduce exactly which records were included in an approved training run.
Step 3: design a realistic train-validation-test split
A basic supervised workflow divides examples into data used to fit parameters, validation data used for choosing models or hyperparameters, and a final holdout set used only after selection. The split should represent actual future conditions. If ticket format changes over time, a chronological holdout may reveal more than a random split.
Group-aware splitting is useful when many tickets come from one customer or are copies of a common incident. Otherwise, near-identical language can leak between partitions. Do not fit a vocabulary, imputer or scaling transform on the entire dataset before the split; such transforms may capture distribution information from the future evaluation set.
Record the split method, random seed where relevant, class distribution and dataset version. If a small result changes dramatically with a different split, that instability needs investigation rather than selective reporting of the best score.
Step 4: establish a baseline and preprocessing pipeline
For a text classifier, a simple baseline could route tickets using a small set of approved keywords or predict the most common class. A stronger candidate might use term-frequency features with logistic regression. Start simple so the team can see whether modeling complexity produces a meaningful improvement.
A reusable preprocessing pipeline should handle text normalization, encoding of categories and any missing-field treatment identically during fitting and inference. Keep the transformation code versioned with the model. If the production service uses a different tokenizer or mapping from training, the reported offline metric may not describe live behavior.
For tabular features, consider whether scaling, imputation or categorical encoding is necessary for the selected algorithm. Fit those transformations only to the training fold inside cross-validation. A preprocessing error can introduce more evaluation distortion than a poorly chosen model.
Step 5: train and compare models
Train a baseline candidate and one alternative under controlled settings. Record data snapshot, feature code, algorithm, hyperparameters, environment and training duration. Use a reproducible experiment tracker or a simple versioned table. The goal is not to run dozens of algorithms but to understand why one performs differently.
Tune hyperparameters with validation or appropriate cross-validation, not by repeatedly checking the final test set. Reusing the holdout to pick the best experiment turns it into another tuning dataset and can make performance estimates overly optimistic.
If deep learning is justified by data volume or task complexity, a supported framework such as PyTorch or TensorFlow can be appropriate. For many ordinary tabular or modest text tasks, a simpler model may be cheaper, easier to interpret and more dependable. TensorFlow eager execution is not inherently unsuitable for all production development; the deployment pattern and tested runtime behavior determine readiness.
Step 6: evaluate the errors that matter
For multiclass classification, inspect precision, recall and F1 for each team as well as aggregate metrics. A high average score can mask consistently misrouting urgent security messages. Use a confusion matrix to see which categories are confused. Review example errors to determine whether the label was wrong, the text was ambiguous or the model missed a useful feature.
For binary classification with rare positive cases, a precision-recall curve may be more informative than overall accuracy. For numeric prediction, compare appropriate error measures and the business tolerance for large mistakes. A metric is useful when it is tied to the decision and the consequences of a false prediction.
If scores will be interpreted as probabilities, assess calibration and threshold effects. A model that ranks examples well can still produce misleading confidence. Document the human-review rule for uncertain cases, especially when the cost of misclassification is substantial.
Step 7: make the result reproducible and deployable
Save the approved preprocessing pipeline, label mapping and trained artifact with a unique version. Record the dataset and experiment configuration so another person can reproduce its creation. Test loading the artifact into a clean environment and confirm it produces compatible outputs for a set of known inputs.
Production deployment requires additional decisions about serving mode, credential scope, concurrency, logging and rollback. Those belong in a separate operational review. A model should not be promoted because one notebook cell returned a plausible prediction; test serialized artifacts and expected error paths.
For those AWS-specific operational choices, machine learning pipelines on AWS provides context, while AWS model deployment and optimization addresses serving and monitoring rather than initial experimental design.
Step 8: monitor outcomes and revisit assumptions
After deployment, compare new inputs with training data and measure actual routing outcomes once the correct team is known. An endpoint can be healthy while misclassifying tickets after product names or support responsibilities change. Define a review interval and what evidence would justify retraining.
Also monitor user corrections and the fairness of error distribution where applicable. If a model relies on an irrelevant protected or proxy attribute, adding more training data without examining the feature may reproduce the problem. Set clear ownership for model changes and for handling a failed release.
A successful model-building project produces a defensible target, documented dataset, realistic evaluation, reproducible artifact and a plan for production measurement. That is more valuable than a high score from a single untracked experiment.