It seems to me that having planets involved in battle would give the game more realism than otherwise. That being said, the features of planetary involvement in combat should be geared towards enriching the player's experience with the secondary goal of enhancing realism.
I propose an system involving both orbitals and planet based weaponry. This planet based weaponry would be housed by a single planetary defensive base, which has customizable weaponry much like ships, as well as special planet-only equipment such as planetary shields or planetary cloaking devices. Construction would involve a one time construction of the base itself as well as several optional add-ons to increase it's capacity for equipment, weaponry, and troops.
The player would have the option of constructing these add-ons at any time after the base has been built. These add-ons would be like increasing the size of a ship. The number of add-ons would be limited based on the technology of the player's empire. The weaponry itself could be updated in much the same manner as refitting a ship. This style gives the player a greater feel of control over planetary defense.
Naturally the player could have only one of these facilities per planet, otherwise the limit of size increases would be ineffective. However, if planets are to have moons, each moon would be able to have their own base. (This, IMO, should be the only way the planets are distinguished from their moons in terms of development and infrastructure.)
Players attacking the planet should perhaps be able to choose what they target on the planet, i.e. specific buildings, housing areas, etc., but only if it can be managed fluidly within battle. Possibly a pop-up if you click the planet giving you a list of the on-planet targets that you have the technology to detect. (Blue with two columns of white text seems cool.) In this case, the planet wide cloaking device will continue to cloak individual buildings and keep them from detection long after the planet itself has been detected. If no buildings can be detected, the attacker would have the option of shooting blindly with much lower chance of damaging buildings.
The option of targeting specific buildings would support more variations of combat goals in the form of surgical strikes on a specific type of building. For example, if you want to damage another empires research capabilities, you make a few small strike forces and send them to destroy the enemy's research labs. Or, if there is a capitol-like building like in MoO2, one could assemble a strike force and attempt to destroy it. Naturally, the player would choose to defend his capitol more thoroughly than other colonies, giving way to interesting strategic scenarios.
The next point is orbitals. This one is tricky. It should (and no doubt will/has) receive great care and thought. My thoughts have led me to these ideas:
Planets can have a number of orbitals in relation to the size of the planet. This makes sense in terms of strategy because a large planet is more valuable and more likely to be besieged than a small planet. The presence of a moon subtracts 1 from the number of orbitals that can be made. A moon is not to be seen as a disadvantage because of it's effect, but an advantage, as the base on the moon is capable of greater weapons storage than an orbital. This makes a colony with a moon preferable to a colony without a moon, thus causing more opportunity for strategic colonization.
Orbitals should be have different options for size and shape, similar to ships, and be customizable. However, a given orbital should have less storage capacity for weapons and equipment than a ship of the corresponding size, even accounting for the exclusion of engines. This is so that orbitals will be seen as lesser defense mechanisms than ships, so players will be encouraged to defend colonies with ships as well as orbitals. Orbitals, however, must be made considerably less expensive than ships so that players won't abuse the system simply by defending with ships, ignoring the idea of orbitals all together.
This type of orbital system will give the player reasonable control over system defense and allow the construction of specialized orbitals for specific systems in which the field of battle may require special strategy to obtain an advantage, while maintaining a fairly simple and understandable system.
The final point is ground troops. This is very important, as this will probably be the main means by which colonies change hands. I suggest that troops be able to be teleported or landed on any planet without a shield. If a shield is present, it must be disabled before ground troops can be sent to the surface. If a troop transport vessel is in combat, the player will have the option of "boarding" the planet in the same way they would board a ship. The difference is that the combat will take place after the battle.
If the battle is won by the attackers, all troops in the attacker's fleet will engage all troops (that weren't in the destroyed weapon base) on the defender's planet. Troop barracks would have heavier armour and be more difficult to detect than regular buildings to avoid abuse of the selective building targeting system. If the battle is won by the defender but the attacker still managed to land troops, the troops that the attacker landed will fight all troops on the planet. If the attacker wins in this situation, space combat reconvenes immediately with the captured planet fighting the enemy forces. In this case, bases on moons are considered ships and will fight against the planetary forces.
This system avoids excessive confusion while providing several opportunities for strategic combat.
This pretty much covers planets involvement in combat. There is, however, the consideration of other non-ship objects such as asteroids and nebulae. As far as asteroids are concerned, I think they should just be immutable obstacles, or even just be part of the background. Nebulae should have the effect of eliminating shielding (possibly counteracted by technology) and massively increasing the stealth statistic of all objects in the area. This makes a planet in a nebula a very interesting strategic position. (Perhaps other empires can't tell that you have colonized inside a nebula without proper sensor technology? Stealth bonuses on the map screen for ships in a nebula?).
These are my ideas for non-ship objects in combat. I hope they are made useful.