Train/validation/test split
Also: training set, validation set, test set, holdout set, dev set
The training set fits the parameters, the validation set (or cross-validation) chooses hyperparameters and models, and the test set is held back untouched to give one unbiased estimate of performance on new data.
Example
10,000 labeled examples might be split 70/15/15: 7,000 to train, 1,500 to compare learning rates and depths, and 1,500 scored only once at the very end.