I do plenty of bike packing to explore Vancouver and the gulf islands . I most definitely ride dirt whenever I can. Sometimes semi teck , compared to the Shore, trails. The idea of stuff hanging from my bike plus the extra weight would not inspire me to ride trails.
I use a 60 litre back pack . If I load the back pack properly it feels quite natural except for the extra weight. But that weight is centered on my back . Not an extra five pounds on my handle bars. My bike with extra air in the fork and shock behaves the same as the bike had not changed in weight.
Usually my pack and gear , including camera, weighs about 15 kilos before I factor in water. And I bring a filter. I also frequently bring my DSLR camera gear in a back pack when I ride so I'm used to riding with a heavy pack.
Frame bags to me appear to be fancy bike accessories to give you that bike packing look.
A 60 litre back pack is easy to keep water proof, mud proof. Easy to carry over streams. Much more convenient for pushing bikes up steep terrain.
Bike touring with a credit card ? Drive your bike to your destination , put on those cute looking frame bags and have fun!
Thinking about a multi day grunge fest into the wilderness with your bike and camping gear. Get a good back pack. Packs designed for climbing are best .