The speaker wants to abandon his mortal body because bodies as they age become old, raggedy, and useless. He says "An aged man is but a paltry thing, A tattered coat upon a stick". This shows how much he dislikes his old body and wishes to "But such a form as Grecian goldsmiths make", which means he thinks that in Byzantium he will take the form of a golden statue.
