Very interesting behaviour...
It's obvious that there is no button contact or PCB trace problem because the chip recognizes the button in some modes, so IMO all that remains is a chip-internal problem, as you mentioned.
The time button can't set the "display on" flipflop, but other all other circuit parts can. But it DOES make its way to all other parts of the circuitry (setting, display timer retrigger).