Modern AI systems are typically trained by optimizing average losses. Yet, many of the requirements that matter in practice are not, inherently, averages. Safety, robustness, truthfulness, alignment, and invariance often describe conditions that should hold across inputs, outputs, or transformations of both. When such requirements are enforced on average over data, a model may still fail systematically on difficult or under-represented cases. We address this issue through the study of pointwise constrained learning, that is, imposing requirements over each and every data point. We develop generalization theory, principled algorithmic relaxations, and focus on Language Modeling applications where pointwise constraints mitigate failure-modes that are often neglected by average constraints.