I'm defining "space monster" as:
"a biological or mechanical being which cannot be communicated with."
The being may be too advanced, too stupid, too uninterested, and/or too different for communication to occur. In some cases it won't be possible to find out which. The Monster may be the last remnant of a powerful ancient civilization, or the result of an experiment gone horribly wrong, or a scout from an unknown dimension, or simply a hungry animal that thinks you taste good.. You may never know.
- may reproduce
- may have goals
- may stay on one planet or travel the starlanes
- may "colonize" planets
- may attack ships and/or colonies
- may leave you alone if you leave them alone
- may do things utterly impossible for player species
- may eat your sun
- may be really weird
- do not communicate
- do not participate in diplomacy
- do not build or research according to the normal tech tree
Of course, this doesn't necessarily mean that the "Monsters" can't in some way be manipulated. A possible interesting twist is the control of space monster "breeding grounds" may allow the player to train and control certain kinds of space monsters.
Left-over Late Game tech
A large, easy to implement category of "monsters" is late-game devices and defenses which remain from some ancient and vanished species (and/or the Orions). This gives the player a taste of the cool things he may eventually be able to build. Of course, not all left-overs will be necessarily harmful.
This should provide plenty of room to add interesting sci-fi beings to the galaxy to provide interesting things to discover but the division between "monsters" and "playable races" should help keep the playable ones manageable and balance-able.