1. A piece of hardware that encapsulates some function but can't be used without some kind of front end. Today we have, especially, print engine: the guts of a laser printer.
2. An analogous piece of software; notionally, one that does a lot of noisy crunching, such as a database engine.
The hacker senses of engine are actually close to its original, pre-Industrial-Revolution sense of a skill, clever device, or instrument (the word is cognate to ‘ingenuity’). This sense had not been completely eclipsed by the modern connotation of power-transducing machinery in Charles Babbage's time, which explains why he named the stored-program computer that he designed in 1844 the Analytical Engine.