I originally had high hopes that the Moo3 combat engine was going to be the space equivalent of the total war engine. On paper it seemed to have everything (ambushes, different task force types having different strengths and weaknesses, tactical manoeuvers, etc) but in practice it just ended up being one bunch of ships shooting at another bunch until one of the bunches was destroyed.
There will always be some serious problems in trying to transfer the intricacies of the TW engine into space combat. For a start many of the things which are key factors in TW combat (morale, terrain, weather, formations, the rock-paper-scissors maneouvring of spears vs swords vs ranged vs cavalry) just dont seem to fit well into a space combat scenario.
The trick will be to try to come up with some analogous attributes to try to make space combat interesting.
A while ago i wrote a massive post over on the Orion Sector about this very thing - just for my ego's sake i'm gonna take the opportunity to paste it here

:
Tactics in Space Combat
Why do we need Tactical Elements?
Tactical elements to gameplay force the players to diversify their approach to the game, otherwise every ship design is the same, and every combat becomes one big blob of ships attacking another big blob, until one side is destroyed. This becomes very, very, boring.
IMO Moo2 has the worst tactical combat of any game ever. It confuses micromanagement with tactics. What wins battles in moo2 is ship design, and nothing else.
Games with Strong Tactical Elements
* Most RTS games are really Real-Time-Tactical with some simple strategic resource management thrown in. However the tactical elements of most of these are pretty limited (i.e. select all of your units, click on enemy units. Wait).
* Turn based wargames, such as the Battle Isle series, the 5-Star General series etc have very strong tactical combat.
* Squad based tactical games (such as the X-Com series) have a very strong tactical element.
* The Total War series have very strong real-time tactical elements, and are the current benchmark to aim for in tactical combat.
All of the above games utilise the some or all of the following gameplay elements (to varying degrees) to achieve tactical gameplay.
(1) Strategic Objectives : These have to be more than just 'wipe out the enemy'. In most RTS games, this takes the form of access to resources, forcing the player to come out and fight, and to try to hold multiple parts of the map, instead of sitting at home waiting for the enemy to come to them. Homeworld is a classic example - if there werent resources to fight over (e.g. if the combat was part of a 4x game) then combat would simply be a case of throwing all of your ships at the enemy and hoping (although its not far off that anyway).
(2) Unit strengths and weaknesses : Example, the Total War series uses a limited number of unit types, each with their own strengths and weaknesses. Knowing which units to use against which is the key to winning. Theres kind of like a rock-scissors-paper thing going on.
(3) Terrain Effects : Some places to attack/defend from are more advantageous than others. e.g. in the Total War series gaining the high ground can be very useful. Similarly, terrain effects can hide units, allowing surprise attacks, and block incoming fire, providing defensive bonuses. This is related to point (1) above - i.e. it matters where the players forces are located.
(4) Detection Range : Not knowing where the enemy is at all times makes a massive difference to the tactics used. For example, in the X-Com games, not knowing where the enemy is hiding means that you must be very cautious in the positioning of your men, using cover at all times, and not leaving one man out on a limb without support to be picked off. Limited detection ranges are necessary if you want players to be able to execute sneak attacks, flanking manoeuvers etc.
(5) Engagement Range : In most games this is fairly limited, which makes the use of offensive and defensive formations an issue. For example, in the Total War series most combat is melee, and even ranged attacks are pretty limited in range, thus positioning is everything, and its pretty much impossible to concentrate all of your attacks on one unit at a time, forcing you to have a cohesive plan.
(6) Unit Behaviour : i.e. how the units AI affects its performance in the field. For example, in the Total War games unit morale is a key factor - winning a combat is not so much a matter of killing the most men (thats more of a long term strategic goal) but of forcing them off of the field. Maneouvers like hitting flanks etc has a significant effect on morale, and if a unit feels it is losing too badly it will break and flee.
Problems with Space Combat
Unfortunately for space based 4x games, most of the above techniques tend to be absent in space combat simulations.
(1) Strategic Objectives : Unfortunately in 4x games these tend to be dealt with outside the scope of the tactical combat - the only objective is usually to destroy the enemy before they destroy you.
(2) Unit Strengths and Weaknesses : If youre going to let players design their own ships from scratch (as in Moo) then it becomes very difficult to achieve this. Moo2 is the worst culprit - players have a tendency to build homogenous fleets with uber-weapons. Zero tactics. Moo3 tried to achieve this, with some limited success, by forcing the player to define ships as a certain type, then limiting the roles those ships could perform.
(3) Terrain Effects : These are usually nonexistent in Moo type games.
(4) Detection Range : This is critical, and if implemented can provide a large tactical element. However, on its own it is not sufficient - there must be incentives for players to split their fleets, otherwise combat will just consist of one huge ball of ships on each side fumbling around looking for each other before combat can begin. Once again, Moo3 attempted to introduce this, but the code appears to be totally broken.
(5) Engagement Range : Again, this is often far too high in this type of game. As a result there is little or no manoevering for position, and combat tends to be a case of focussing all firepower on one enemy unit (ship/fleet/stack) at a time until they are destroyed, then moving onto the next.
(6) Unit Behaviour : Again, this is sorely absent in moo-type games. Morale is nonexistent - ships do exactly what they tell them to. Moo3 had some variation in ship behaviour based on the ship type, but this was limited to target prioritisation, was usually overridden by the player, and was pretty much broken anyway.
Solutions for Space Combat
(1) Strategic Objectives : One simple enhancement can be to implement a time limit on combat - if the attacker has not won within a certain time they will have to retreat. This isnt going to be enough on its own. However if combined with some other elements it may force players to diversify their approaches.
(2) Unit Strengths and Weaknesses : The only way to really achieve this is to strongly limit player choices in ship design, something which doesnt tend to go down to well with fans of the genre.
(3) Terrain Effects : If combat was focussed at a system level rather than on a planetary scale, then planets and stars could be made to provide terrain benefits (e.g. hide behind planets or asteroid fields). There was an ability to hide behind planets/moons in moo3, but this usually had limited effect on the outcome of battles.
(4) Detection Range : Possibly give the defender an advantage by giving them full visibility (possibly they would have to build an array of sensor satellites in the system first) while the attacker's visibility is limited. Possibly this could add some objectives the attacker has an incentive to destroy sensor sattelites, the defender has an interest in defending them. Possibly make ships in orbit around planets harder to detect.
(5) Engagement Range : This has to be carefully balanced with detection range. Obviously missiles are long range (but not necessarily unlimited range) but this advantage has to be offset by limited ammo and the fact that missiles can be shot down. Even the longest range beam weapons shouldnt allow players to snipe at each other from across the map.
(6) Unit Behaviour : Implementing a morale system for ship combat is probably not feasible. Ships go where their captains tell them to.
Summary
Its telling that the moo games which limit the players choices the most are those with the best tactical elements. Moo1 forced all ships of the same design to move and fire together, with a limited number of designs, a simplistic combat arena (10x8 grid IIRC) and limited engagement range. As a result the tactical combat was more like a game of chess. Moo2 gave the player complete control over the design and behaviour of every ship, and as a result every ship was the same, all lined up against each other in a tedious micromanaged battle of attrition. Moo3 looked promising, but the ridiculous level of bugs and poor user interface reduced every battle to a simplistic tarball.
You should probably take the Solutions and Summary sections with a pinch of salt - theyre not very well thought out and most of the ideas probably wouldnt work. However i think that the issues that they try to address are the ones which need to be solved somehow in order to make space combat interesting.
Since i wrote that i've given it a lot more thought, and some of the solutions i've come up with are:
(1) Terrain advantage : Kind of like in total war, have some kind of a 'high ground' which confers an advantage on whoever holds it. One way of implementing this would be to have regions which make ships sitting in them more difficult to target. A technobabble justification for this would have to be knocked up - 'subspace distortion' or something. You would probablyneed to limit the number of ships which can fit into such an area, to try to prevent the defender just parking their entire fleet in there and waiting.
(2) Ship class bonuses : Confer bonuses on ships of a defined class. E.g. PD ships have enhanced accuracy vs missiles. Theres a detailed list of ideas for ship class bonuses
here. More shameless self promotion. Sorry.
(3) Formation bonuses. Allow the player to choose formations for their task forces (if taks forces end up being in the game) which provide bonuses and penalties in certain situations, e.g. make ships take more damage if hit from multiple sides (flanking is a vital tactic to try to encourage), and have some formations provide mitigation for this at the expense of focussed firepower, while others provide maximum offensive/defensive capability in one direction while leaving the task force vulnerable to attack from the sides or rear.
Alternatively (if combat goes the moo2 route of a smaller number of ships which are individually controlled in detail) you could have shield facings which bias shields forwards or provide all round cover.