On the open 15th floor of a loft building in Midtown Manhattan, about a dozen skilled workers make their way through piles of pants, stitching each piece together with focus and precision. Some of the items are designed by Outlier, a fashion brand that produces its smaller runs and experimental products with the garment district’s ecosystem of contract manufacturers.

It’s the kind of work that should get a boost from the stiff tariffs newly imposed on products entering the United States from nearly every other country. But the storeroom where Outlier keeps its fabric tells a more complicated story.

The rolls of cloth and boxes of recycled goose down come from Italy and Switzerland, Thailand and New Zealand, countries with specialized industries developed over generations that are unlikely to be recreated in America. Take the linen, made from flax grown in a coastal region stretching from northern France to the Netherlands.

“It would take a decade to get a crop growing,” said Tyler Clemens, Outlier’s co-founder. A linen shipment was headed for the cutting room; Mr. Clemens had just gotten the bill from the Department of Homeland Security with a charge labeled “IEEPA-RECIPROCAL,” after the International Emergency Economic Powers Act, one of the laws used to justify President Trump’s tariff measures.