To imply is to hint at something without saying it directly. It goes from you to the person to whom you are talking. Direction-wise it is like throwing a ball.
To infer is to deduce something that was not directly said. Direction-wise it is like catching a ball.
Basically, you imply, I infer.
These two words are incorrectly used so often that in a decade or two they will probably be synonyms. Just like literally has become a synonym for figuratively.
I liked this, it totally reminded me of an episode of Big Bang Theory.