Caroline Fermin is part of a close circle of friends with children in the Washington Heights neighborhood in Upper Manhattan. They push strollers along tree-shaded blocks and meet up for play dates and birthdays.

And just about every week, they talk about how they can’t afford to stay in the neighborhood. At least half-a-dozen families from her group have left for upstate New York, New Jersey and beyond.

Ms. Fermin, 39, and her husband, Jean-Paul Bjorlin, 42, don’t want to be next. But together, they make under $100,000 a year teaching dance and music at Barnard College, along with side gigs. In October, they moved into a four-bedroom apartment that will cost $50,000 in rent per year, on top of the $36,000 they will spend on day care for their two children.

“We’re kind of treading water,” Ms. Fermin said, adding that they were dipping into their savings, which would only last for a year, and had stopped contributing to retirement funds. “Why put up with this? What are we getting back from the city?”

Ms. Fermin’s family is at the center of an affordability crisis that is reshaping New York City’s population. Their neighborhood of Washington Heights, a hilly enclave of nearly 144,000 residents north of Harlem and sandwiched between the Hudson and Harlem Rivers, has long attracted families with its large apartments, sprawling parks and close community of immigrants from the Dominican Republic. It is often regarded as one of the last affordable places to live in Manhattan.

But increasingly, many families are being pushed out of the neighborhood by a housing shortage and soaring rents and child care costs. The result is a significant drain of a demographic group that is vital to the city’s future.