We research all brands listed and may earn a fee from our partners. Research and financial considerations may influence how brands are displayed. Not all brands are included. Learn more.

couple looking at house

My wife and I recently left Brooklyn for the suburbs, the latest in a long string of emigrants leaving New York City for Long Island. When you have a 20-month-old monster rampaging through your boxed-in two-bedroom apartment, you willingly embrace cliché for space and peace and quiet.

And yet, while my wife and I are adding deck chairs and hanging bookshelves, we're not exactly planting roots. This house is just a rental, not a property we own. That shift may have to wait till we can pull together a down payment -- a priority we have to balance against paying down student loans and funding our child's college account.

There's one lump sum of money that could get us where we need to be, but it's parked in our retirement accounts. Normally we couldn't withdraw that cash without incurring a 10% penalty, but the IRS makes an exception for withdrawals of up to $10,000 for first-time home purchases. So the question for us is: Should we succumb to temptation and raid our nest egg to buy a home? More broadly, should you?

Just Say No

The impulse is a real one, particularly if owning a home is something that's important to you emotionally. (Financially, it can be a tossup, depending on how long you intend to stay put; check out Trulia's rent vs. buy calculator to get a recommendation for your situation.)

Read Next: The Right Way to Take Your IRA Withdrawals

But the experts warn against tapping IRA funds -- even more so if they offer your only avenue toward home ownership. "Generally I would caution against withdrawing simply because if it's difficult for you to come up with a down payment, it will likely be difficult for you to handle the other costs," says Matt Becker, a Pensacola, Fla., financial planner who advises younger families.

Imagine if your roof caved in, or a pipe burst during your first winter. If you're borrowing from your retirement to pay for the house to begin with, you'll likely go into debt to deal with the burdens of owning a home.

There are other issues, too. For one thing, you'd be losing out on a powerful tool that is critical to a successful retirement: compounding interest. "You would be depriving that $10,000 of 60 to 70 years of appreciation," says RBC Wealth Management adviser Darla Kashian. Your $10,000 would turn into about $105,000 in six decades, assuming 6% return and 2% inflation.

And of course, even if your IRA withdrawal doesn't generate a penalty, the money will still be subject to income tax.

Special Exceptions

Are there times when you can, or should, consider a withdrawal? One instance is if you've accumulated a lot of savings in your retirement accounts --perhaps you saved diligently in your 401(k)s, then changed jobs a few times and rolled over but not so much in the bank. If you're actually ahead of where you need to be, then you can consider a withdrawal for a first-time home purchase, says Charlie Farrell, chief executive officer of Northstar Investment Advisors and author of Your Money Ratio$.

Read Next: I'm 22. Should I Invest for Retirement in a Traditional IRA or a Roth?

"If you're ahead of the game when you're younger, you can think about doing that," says Farrell. "It's a good goal to own a home and have a rent-free home to live when you retire. If you don't, your cost of living is going up every year, but you're not building equity."

Farrell offers a few savings guidelines on his website. If you're 30, he suggests, you should have saved 60% of your current annual income (so $30,000 if you earn $50,000) and be putting away 12% of your paycheck in order to retire at 65 with 80% of your income.

Farrell also recommends a mortgage-to-income ratio of 2.0.

My wife and I haven't hit that 60% mark, so we won't be using our retirement accounts for a new home purchase any time soon. It feels slightly atonal to have a child and a spouse, but no mortgage -- as if I'm doing adulthood wrong.

Yet that's actually a smarter reality for young families who live in expensive areas. Borrowing from future wealth to buy a home will only result in a poorer version of ourselves.