"Now, let's say that I borrow a CD from a public library. If I copy that CD's songs to my hard drive, is that legal?"
"I believe that copying songs from a library CD is no different from photocopying a book that was borrowed from a library."
If one photocopied an entire book, that likely would also be illegal. People usually photocopy a few pages from a book for reference purposes. That, generally, is legal.
"I should mention that, in my hypothetical scenario, I would be copying the songs to my hard drive and would be using them for myself only."
Doesn't matter. Reproduction is one of the exclusive rights of copyright. If you do it without the consent of the copyright owner (and without some fair use purpose), you are infringing copyright.
"I would NOT be distributing them to other people, for free or for money."
Distribution is a separate right of copyright.