At visible and infrared frequencies, metals show tantalizing promise for strong subwavelength resonances, but material loss typically dampens the response. We derive fundamental limits to the optical response of absorptive systems, bounding the largest enhancements possible given intrinsic material losses. Through basic conservation-of-energy principles, we derive geometry-independent limits to per-volume absorption and scattering rates, and to local-density-of-states enhancements that represent the power radiated or expended by a dipole near a material body. We provide examples of structures that approach our absorption and scattering limits at any frequency; by contrast, we find that common "antenna" structures fall far short of our radiative LDOS bounds, suggesting the possibility for significant further improvement. Underlying the limits is a simple metric, |χ|2/Im χ for a material with susceptibility χ, that enables broad technological evaluation of lossy materials across optical frequencies.