You would be fine until Southeastern decide to mount a revenue block at Charlton. At this point they could look at your journey history, and seeing the unresolved journeys mount a prosecution for fare evasion.
The possibility of this happening was why some TOCs initially demanded that Oyster Extension Permits were introduced.