Whether we use go or come all has to do with perspective and position.

go

We use go to describe movement away from the place or position where the speaker or hearer is:

• Are you going to the pub tonight?
• Let's go and see Auntie Mary before the holiday is over.
• They've gone to live in Australia and I don't think they'll ever come back.

come

We use come to describe movement to the place where the speaker or hearer is:

• Could you come here for a minute, please, Diane?
  ~ I'm coming.
• We've come to ask you if we can borrow your car for a week.
• I've got some people coming for a meal tonight. Can you and Henry come too?

go back, come back, return

The same rule applies with go back and come back, […], but you can use return for both come back and go back:

• You must have come back / returned very late last night.
  I didn't hear you come in.
• He went back / returned to Mexico when he had finished post-graduate training.

Note, however, that come with and not go with is normally used when we are talking about joining a movement of the speaker or hearer, even though the movement is away from their current place or position:

• I'm going to the hospital this afternoon to get the test results. Could you come with me?
• We're going to Egypt for a week at Christmas . Would you like to come with us?