1. Behaving in an erratic and unpredictable fashion; unstable. When used to describe the behavior of a machine or program, it suggests that said machine or program is being forced to run far outside of design specifications. This behavior may be induced by unreasonable inputs, or may be triggered when a more mundane bug sends the computation far off from its expected course.
2. When describing the behavior of a person, suggests a tantrum or a flame. “When you talk to Bob, don't mention the drug problem or he'll go nonlinear for hours.” In this context, go nonlinear connotes ‘blow up out of proportion’ (proportion connotes linearity).