Bill (and Amber) have it exactly right! Well-spotted.
Once you know what to look for, you can (barely) see the diagonal dividing line across right-hand-Colman's shoulder. (As Amber points out, it's especially obvious when the stand-in shrugs slightly to switch hands with the hat.) James Wong Howe says this was all done in-camera: most of the frame exposed with a stand-in at right, then the upper-right corner re-exposed with Colman in exactly the same spot. Howe says they did it 13 times before getting one where the match was more or less seamless.
The chutzpah behind this effect is astounding. You have to count on two different men lining up exactly, despite being filmed at two different times, and both having to make small movements as part of their performance. AND they leave the effect on screen for 18 seconds. There's plenty of time before and after the handshake to spot the trickery, but it's still invisible. Over fifty years before the heyday of digital effects!