Right-clicking in the message transcript area will reveal an option to clear the history.
Clear chat transcript option in Messages for OS X
MessagesClearTranscript
MacIssues
How to use, troubleshoot, and repair your Mac
