If sweatshops make poor countries better off, then we should allow them because even when benefits are radically disparate, any mutually beneficial exchange, voluntarily arrived at, is by definition non-exploitative and leads to a trend of increasing standards of living.
The power of voluntary exchange is that it naturally leads to two reinforcing effects.
First, it is most likely to capture maximum knowledge of the rational best interest of those parties involved in the exchange. Even if someone outside the exchange feels they have pertinent knowledge unavailable to one of the participants, the trade off would not be to get in between the two parties and take from one or both the final decision, it would be to provide the information to the party lacking it. In this way you are guaranteeing that those most likely to have the most knowledge (those with direct loss or gain at stake) are those also making the final trade decisions.
In the case of a sweatshop that is the employer and employee.