The 44-year-old actor details their relationship in his memoir, "You Can't Be Serious."