Does the digit flash with normal brightness or brighter?
If brighter, it could be a "stuck" digit multiplexer.
If not, it's probably the logic that selects the digits to display depending on mode, a weak or defective transistor selecting the wrong digit at first. Anyway, this would both be chip-internal.