The answer to this question isn’t simple, or at least, the simple answer is at best in complete. (I am assuming by “nuclear reactor” the o.p. means a nuclear fission reactor, not the more passive and lower performing radioisotope thermoelectric generator or a hypothetical fusion reactor for which we do not have any good models or empirical data to estimate scaling and power output.) The United States has deployed exactly one space-based nuclear fission power supply, as part of the Space Nuclear Auxiliary Power (SNAP) program. The SNAP-10A operated for about a month and a half, exceeding the design output of 500 W(e) by about 20% before a failure in the power regulating system caused it to shut down. The SNAP-10A reactor was metallic salt (NaK) cooled reactor with a semi-passive failsafe design; loss of integrity or deliberate command would cause the beryllium reflectors which maintained the criticality threshold to separate from the craft and the neutron flux to drop sufficiently that the spacecraft would just down. Since it used a metallic salt coolant loop, no pressurization or pressure-volumne heat cycle engine was required, and the system generated electricity via thermoelectric conversion between the heated fluid and the space background. This isn’t terribly efficient but it is simple, reliable, and easily made to be essentially failsafe. Most later concepts for space-based nuclear electric conversion assume using an outer loop with a high temperature gas (typically helium) and using a Sterling or Brayton cycle to attain high thermal conversion efficiencies. However, despite some research programs, no other nuclear reactors have been flown by the US, and most of the research on the use of nuclear fission for space-based applications has focused on nuclear thermal or nuclear electric propulsion, which has a very different set of requirements (addressed below) than just energy production for instruments or habitats.
The former Soviet Union had more expensive experience with nuclear reactors in the Upravlyaemy Sputnik Aktivnyj (NATO designation: RORSAT) and some of the Kosmos-18XX series satellites. These were also metallic cooled thermoelectric units. The reliability of these units varied (likely because of materials quality control issues). Because of the problems experienced by the Soviets and economic troubles with the fall of the Soviet Union, the Russian Federation has largely abandoned the efforts, though they have made the searched done on the TOPAZ-2 system available to the international technical community. Most of their technical expertise has since fled the country to other more lucrative opportunities.
Overall, the specific energy output of existing space-based reactors is between 0.001 and 0.005 kW(e) per kg. Note that this disappointing figure is largely because of the very poor thermal efficiency of around 0.01-0.02. Using more modern thermoelectric conversion and higher operating temperatures, using efficiencies approaching 0.15 or better are possible. The US SAFE-400 design claims a thermal conversion efficiency of around 0.25 (25% efficiency). That sounds absurdly high to me but if true would put it in the ballpark of significantly more complex heat engines. Regardless, it is certainly safe to assert that the output achieved by existing systems is nowhere near the potential maximum output, and with a not-inconsiderable effort into developing the engineering and materials expertise much greater efficiencies could be achieved.
All of this refers to the production of electricity from a nuclear thermal source, and in this regard, the power source and its control system can be considered essentially a black box, especially if the demand cycle is fairly constant, e.g. maintaining a steady power draw for instruments or to support a habitat. However, when it comes to propulsion systems, the hideous inefficiency of thermal electric propulsion for reasonable thrust levels (e.g. those necessary to propel a crewed vehicle or large instrument platform versus a small interplanetary probe) almost certainly dictates the direct use of reactor thermal output to heat the propellant. This, in turn, requires that are are able to finely control the thermal output over a wide range, provide sufficient cooling to keep the system from eating itself, and build the reactor into the load-bearing thrust structure of the vessel. In this case, it is no longer a black box; it is an integral part of your propulsion system and the power output for other functions is almost incidental. At the same time, it has to be highly reliable and essentially maintenance free, which dictates as much robustness and simplicity as possible, while still achieving sufficient mass efficiency and overall performance to be worth the trouble. This is a far more complex set of requirements, but the driving consideration is very likely the thermal management, i.e. getting rid of all of the waste heat generated by such a system.
How do you get rid of waste heat? Well, while space is “cold” in terms of the background temperature, the lack of a working fluid means that there is no way to reject waste heat via convection, as is done with terrestrial reactions, including those in mobile applications like nuclear submarines and naval surface vessels. This means that all of the experience and design that we’ve put into such compact, high performance systems has little applicability in the space environment. (In addition, submarine applications of nuclear fission reactors are often built around using natural convection in order to minimize pump noise; in space, the lack of gravity means that natural convection systems are much more difficult–although not impossible–to implement, so you can’t just take a S9G and plunk it into your spacecraft.)
Radiation from your reactor is really a pretty easy problem to solve; water and waste products can be used to line the view factor between the habitat and the reactor, and as long as the reactor is designed for zero maintenance there is no reason to have any connection between the habitat section and the reactor core, so they can be separated by as much distance as possible. One possible solution is to have the habitat and associated functions on one end of a tether and the reactor/propulsion section on the other, and have them be structurally attached during impulsive phases but then widely separated and used as a counterweight for ballistic flight, thus managing the radiation exposure of the crew. (You still have to mitigate the impingement of high energy cosmic radiation, coronal fluxes, and in the case of a mission to Jupiter, the radiation belts in orbit, but that is a problem regardless of your power source.)
One possible way to deal with thermal issues is to treat the reactor as an expendable component; design it as a module to survive throughout the impulsive maneuver, and then eject it where it can melt down or radiate to its core’s content. Of course, this doesn’t help you with a power supply, but it does address the need to get high specific output for a short period versus a long continuous output through the mission duration. Another, possibly more efficient, is to have a distributed “core”, e.j. a fission fragment reactor, where the coolant is also a carrier for the fissile or fissionable materials and used as part of the propellant, being ejected out into space. With the correct design, a high degree of scalability should be possible without having multiple systems or great complexity.
I think it is not really possible at this point to estimate what the potential output could be of a space-based nuclear fission system based on the state of the art. This work was largely abandoned in the 'Seventies, and only tentatively supported in the 'Eighties and early 'Nineties by the Strategic Defense Initiative, only to again be essentially reduced to toy studies and low priority research by LLNL and LANL since. A lot of the expertise and pretty much all of the practical test systems have been lost or mothballed such that implementing such a system will have a steep learning curve, but if the US had constantly pursued nuclear propulsion and space-based nuclear power such systems could have been viable by the late 'Eighties.
Certainly, space-based nuclear fission power generation is the sin qua non of any crewed missions beyond the orbit of Mars and realistically probably necessary for a sustainable infrastructure to explore at Mars orbit given the low solar flux and issues with solar power generation on the surface of Mars. Developing this capability–along with the ability to extract energy resources in situ–is one of the cornerstones of a future space-based infrastructure for exploitation of space resources and crewed habitation and exploration, and the solutions for the existing state of the art–which demand complex systems, highly processed fuels, and very limited capabilities–do not represent the potential capability and necessary requirements for future use.
The International Atomic Energy Agency’s report on “The Role of Nuclear Power and Nuclear Propulsion in the Peaceful Exploration of Space” has an extensive discussion on the history, state of the art, and future potential for space-based fission power generation.
Stranger