French drain and lowering external ground levels

[a twig]

I wouldn’t worry about it - packing a few voids with lime mortar won’t affect the structural integrity one jot.