Robert with his wife Marlena
Jeffrey Gru (brother)
Marlena Gru (wife)
Despicable Me (photograph)
Robert Gru is Felonius Gru and Dru Gru's Father, Marlena Gru's husband as well, he is also the adopted paternal grandfather of Margo, Edith and Agnes, the father-in law of Lucy Wilde and the son of John Gru.
So far, Robert has only been seen in a picture on the family tree but he never appeared in person during the first film or seen in any of Gru's flashbacks and wasn't even at his son's wedding near the end of the second film, or with him and his wife at the Villain-Con in the film Minions.
Therefore it is unknown what happened to him before the events of Despicable Me, however though, it can be assumed that he may have passed away or he and Marlena are divorced and lives faraway; it's also possibly that he might've commuted a crime and is now in prison for life.
In Despicable Me 3, it is revealed that Gru's family has a long history of villainy. Therefore, it is unknown if he was a villain or not.