Going to smaller cogs means higher gear which is up shifting. It could be spring tension or maybe clutch is seized? Or the bolt is seized? Those last two issues can make the pulley wheel sit too far from the cassette teeth to shift well. Check the plate that stops the b tension bolt too. Those have been known to fail although that makes the derailleur go too close. Older SRAM derailleurs had the parallelogram run close to the b knuckle and a small tweek could see the parallelogram run against the b knuckle and prevent up shifts but usually only the higher gears depending on the degree of tweek.
Probably start with pulling the derailleur apart as much as you can to clean and re lubricate it. How much play is in that derailleur? Compare the wiggle it has to a new one. If it is really sloppy it might not be worth fixing. Are the pulley wheels in decent condition? Any excessive play there?
Debate? Bikes are made for riding not pushing.