This is normal behavior. There is much more to the JVMs memory usage beyond what’s allocated to the heap - there are other memory regions as well. There are additional tuning options for them, but it’s a complicated subject and if you aren’t actually encountering out of memory issues you have to ask if this is worth the effort to tune it.
The ideal case for me is that I don’t need HACS at all. My experience has been the same - I’ve happily been able to switch to core HA components and stop using HACS ones. It’s great to see HA is not idle with success, they are continuing to make new features even when backwards compatibility may break.