It isn’t an issue of gravity but of inertial mass, which doesn’t change regardless of where you are and will be dominated by the minimum mass per unit area of the solar sail for any useful degree of impulse. At Mars orbit (between 1.38 AU and 1.67 AU) solar irradiance is about 1/2 to 1/3 of what it is at Earth orbit, and a corresponding reduction in impulse per unit of spacecraft mass (including the solar sail). At the orbit of Jupiter (4.95 AU to 5.46 AU) irradiance is about 0.35% to 0.41% of Earth orbit, and at Saturn (9.0 AU to 10.1 AU) you might as well just get out and push for all of the good that the solar wind will do for you. At some point dictated by how thin and lightweight you can make the solar sail (and all of its supporting structure) you just can’t add any useful amount of payload and get a rate of acceleration useful for a anything on the timeline of an interplanetary mission (i.e. a couple of decades or less) or to perform trajectory maneuvers to intercept a celestial body and for current and foreseeable sail materials that is well inside the orbit of Jupiter. All current and seriously proposed solar sail missions that I’m aware of are actually at or inside Earth orbit although there are somewhat fanciful proposals for an Earth to Mars mission within a month using enhanced microwave-powered projection.
Here is a thread from several years ago where I did some general calculations about using solar sails (specifically about going to Mercury but they apply more generally to any use of solar sails).
Stranger