The Money Meets Medicine Podcast

Should I Pay Off My Mortgage

June 26th, 2024

With rising interest rates, many people are now considering putting any extra money they have toward their mortgage. While others who refinanced in 2023 to interest rates <3.5% are enjoying the mathematical win of putting their money elsewhere. Should you pay off your mortgage? That’s what we discuss in this episode. 

Notes

In this show we discuss:

  • Paying off your mortgage with a high (or low) interest rate
  • When it makes sense to pay off your mortgage
  • How insurance products impact paying off your mortgage
  • And more…

Show Trancript

 

Jimmy Turner MD

So we had a listener reach out via email, long time listener. And so we’re going to discuss a question that they asked me. And so we’re going to dive into that. And I think it’s a really good question.

 

Jimmy Turner MD

And Justin, based on your response, sounds like it’s a question you commonly get as well.

Disability Insurance for Physicians

 

 

Justin Harvey CFP

Yeah, absolutely. And I had a recent anecdote, which I can describe in the context of our discussion that will further elucidate the point. Yeah.

 

Jimmy Turner MD

I love it. Well, hey, before we dive in, don’t forget, if you help with your student loans, disability insurance, or you want to download a free copy of the Physician Philosopher’s Guide to Personal Finance, you can do that at MoneyMeetsMedicine.com or we help docs with all three of those things and get it from a source you can actually trust.

 

Jimmy Turner MD

So Justin, I want to read this email first because this person’s awesome. I’ve emailed back and forth with them several times of the year. By the way, if you don’t know that about me, jimmy@moneymeetsmedicine.com, you can send me an email.

 

Money Meets Medicine, Jimmy Turner, Justin Harvey

 

Jimmy Turner MD

I will actually respond as best as I can and love having conversations with our community. So I get this email and this fellow car -loving person, I might add, this is one of the reasons why we get along so well, send this email, said, with interest rates on high yield savings accounts being 4% to 5% and a lot of the mortgages being less than 4% because many people, myself included, refinance back when the rates were just crazy historic lows.

 

Jimmy Turner MD

And so I think mine’s 3% asked, does it make sense to pay extra on the mortgage, assuming that there’s no other debt? And they put some comments about there, about initially thinking about putting it towards the mortgage and decreasing the amount that was needed, but then realized that extra payments shorten the duration of the mortgage instead of allowing them to pay their mortgage for a few months in the event of an emergency.

 

Jimmy Turner MD

And so now they’re basically asking the question, hey, do you put it in a high yield account and can you and Justin possibly discuss pros and cons of putting extra money to a high yield savings account versus towards your mortgage?

 

Jimmy Turner MD

Disability Insurance for Physicians

 

And I love this, recognize that there’s not necessarily a right answer here, which I think is going to be one of the take home points, right? Like any time you have this question, should I pay my mortgage off?

 

Jimmy Turner MD

No matter which side you’re on, it seems very black and white. You’re like, you should absolutely pay it off. And here are the 15 reasons why. And you should absolutely not pay it off. And here are the 10 reasons why.

 

Jimmy Turner MD

So hopefully we’re going to find a little bit of nuance on the show. But Justin, now that you got the context, you got the question, which I’d emailed you previously before the show, tell me your story.

 

Jimmy Turner MD

What’s the story here?

 

Justin Harvey CFP

I was recently talking to a gal who had a great, one of those lucky few who locked in that 2 .99 30 year fixed a few years back and was aggressively paying it down because she said her husband was basically worried about what happens if you kick the bucket and these young people in their 30s, but still wanting to cover all their bases.

 

Justin Harvey CFP

And so seeing the mortgage looming as a big potential thing to take care of, especially in the event of stay at home spouse, high earning physician, and not wanting that liability to sort of extend beyond the life of the high earning physician.

 

Justin Harvey CFP

And so the balance was probably a quarter of what it had been even just a few years ago. They had started at a million bucks and it’s at a couple hundred grand and they’re continuing to aggressively pay it off.

 

Justin Harvey CFP

And I got to tell you, Jimmy, I looked at this and it hurt me a little bit on the inside just because if we think about these circumstances, this is one of the things that life insurance is for. And in the event, God forbid, of a post spouse world, this spouse would be covered for a handful of millions and some amount of that, some would have easily been able to be deployed to pay off the rest of the mortgage,

 

Justin Harvey CFP

reduce the monthly net cash outflow needed and provide plenty of support for this family. But what they had been doing was really just socking money into something that’s only, well it’s essentially going to be gathering dust and their rate was so low that once you layer in the tax benefits, that it was like 2% which is less than inflation even.

 

Justin Harvey CFP

And it would behoove them in a number of ways to just string that along as far as possible, not only because the low interest rate, but because every time with a fixed interest mortgage, the payment is the same today as it is next month, next year and 29 years from now.

 

Justin Harvey CFP

And that 29 years from now dollar payment is going to be a tiny fraction in terms of the actual purchasing power of those dollars. And so, stretching it out, as long as you can psychologically handle it, it’s a slam dunk with rates as low as this particular rate was.

 

Jimmy Turner MD

Yeah, no, I really so you land on the don’t pay it off

 

Justin Harvey CFP

Well, I mean, it’s situationally dependent. But I would say with these rates that are like in the twos, yes, yes, I’m saying don’t pay it off if your rate is in the twos, don’t pay it off.

 

Jimmy Turner MD

Yeah, no, I get it. And the reason that I point that out, like the polarization on the topic is because you’ve got the Dave Ramsey’s of the world, right? Who’ve got their baby steps. And part of those baby steps are just like basically paying off all of your debt before you ever invest.

 

Jimmy Turner MD

And maybe at some point on the YouTube channel, I’ll make a video of five reasons that doctors shouldn’t listen to Dave Ramsey. But one of them is certainly this topic. And it’s unfortunate because I have an online business, Justin, you live in the online world too.

 

Jimmy Turner MD

And one of the things that this really impacts is that you have a brand and you have a certain thing that you want to claim as your own and to stand on. And so for example, like my North Star has always been and always will be to do what’s right for people.

 

Jimmy Turner MD

Even if it loses me money, even if financially from a business standpoint, it is not smart, I want to do the right thing for people. And so some people have chosen to not do business with me because I’m not quote unquote financially minded enough.

 

Jimmy Turner MD

And so an example from Dave Ramsey’s side is he’s got these six baby steps or whatever. And he can’t rebrand those. They’re the six steps that were made back when interest rates were different than they are right now.

 

Jimmy Turner MD

And inflation was different than it is right now. But if he rebrands those six steps, it sounds like he’s going back on what he said. It turns out that financial advice can’t be had in a vacuum. Times change, things change.

 

Jimmy Turner MD

And so for those of you that live in the Dave Ramsey camp, you’re like, no, absolutely. You pay off the mortgage, you pay off all your debt before you do anything because you just have to pay off your debt because the psychological snowball burden that’s on you until it’s all gone.

 

Jimmy Turner MD

It’s like, right, I’m not saying there’s not room for that. That can be right in certain situations. But if your interest rate is effectively 2%, you paying off your mortgage provides psychological benefit potentially, but mathematically, it may be the dumbest thing you’ve done financially.

 

Jimmy Turner MD

And so there are lots of dumb things to do, by the way, financially, but that would be up there. The market’s turned 10, 15, 20%. And here you are paying off 2% mortgage debt. Yeah, looking back, you’re going to be like, that’s tough.

 

Jimmy Turner MD

On the other hand, I have also never found, come student loans, mortgages, I’ve never found somebody that after the fact was like, you know what, I can’t stand that I paid off my debt. I don’t know anybody that I know personally, that’s been like, I really regret paying off my debt.

 

Jimmy Turner MD

Usually, there is a burden factor there where they feel like a weight’s off their shoulders. And so I’m not discounting the psychological benefit. I am just saying that there’s nuance to this. And that if you’re in the definitely paid off camp, or the definitely not paid off camp, maybe there’s some room for discussion there.

 

Jimmy Turner MD

Maybe maybe it’s worth reconsidering if you grew up as, for example, a Dave Ramsey fan.

 

Justin Harvey CFP

There was a book that came out a few years back called, How I Invest My Money, I think was the title. We can link in the show notes to whatever the title actually is. And it was essentially interviewing a dozen money managers.

 

Justin Harvey CFP

What do you actually do? I understand you talking head on CNN, talking about this or that stock, but in your 401k, in your Roth IRA, in your taxable account, what do you and your family actually invest in, which was really interesting.

 

Justin Harvey CFP

I love this idea. I haven’t read this book. And I haven’t either, actually. The question they asked was, what do you do with your mortgage? And they all say, well, we tell everyone, don’t pay it off because the interest rate arbitrage, but we all pay it off and we all feel great about it.

 

Justin Harvey CFP

Now, this book was a handful of years ago. And I will tell you, Jimmy, I know people in the last handful of years that regret pay off, that regret an accelerated pay down on their mortgage at 2 .5%, 3%, 3 .5%.

 

Justin Harvey CFP

So this book was written during an interest rate regime of 6%, 7%, 8%. And sure, that makes a lot of sense that you would feel good about that. But those people do exist that regret accelerated pay down on low interest debt, especially if there’s tax benefit.

 

Justin Harvey CFP

And so even in financial advisor land, historically, this has been a, oh, the psychological benefit is so great that it overcomes the mathematical sort of problem. In the last few years, the mathematical argument is so profound.

 

Justin Harvey CFP

It’s better than it’s ever been in the history of these money managers. And so I do think that the conversation has changed a bit in the same way that getting the debt two times your gross income as the ceiling for your mortgage, as the conventional rule of thumb in a much higher interest rate environment, I think it’s worth revisiting some of those assumptions.

 

Jimmy Turner MD

Yeah, we recently discussed buying a home and kind of those rules of thumb on a prior episode a few weeks ago. So if you missed that, go check it out. But I do agree. And I think that this question from our listener is important to put into context, right?

 

Jimmy Turner MD

So the listener has a loan that is less than 3%, right? And so in that context, is how we’re answering this question. Before we do that, I do think it’s important to mention that there’s a different, and maybe we’ll answer this question first, Justin, because I think this is more relevant for you, right?

 

Jimmy Turner MD

And this is actually hilarious to be podcasting about this right now, because you just bought a home with a massive interest rate. I have a home that I refinance when rates were historically low at 3%.

 

Jimmy Turner MD

And so the way that you answer this question and the way that I answer this question are fundamentally different. Because if you said you wanted to put extra money towards your mortgage, because I don’t know, do you mind sharing what your mortgage interest rate was?

 

Jimmy Turner MD

Seven and a quarter.

 

Justin Harvey CFP

Seven a seven one arm, so it’s floating after seven years

 

Jimmy Turner MD

So I have a 15 .1 arm that I got at 30 or 3%. And so for me, it makes a lot of sense to put money towards anything else because I’m probably going to get better than 3% pretty much at anything else. In fact, my money market account gets well more than 3%.

 

Jimmy Turner MD

And so for you, I have people that come up to me at work like, hey, Jimmy, I’ve got this 7% mortgage. I’ve got extra money. What do you think I should do with it? And I’m like, I’m going to be honest with you, like this question five years ago was pretty simple.

 

Jimmy Turner MD

And for those that refinanced, pretty simple. But now you’re sitting at seven and a quarter, right? Extra money, getting a guaranteed 7 .25% return on your money, basically, to pay that thing down faster.

 

Jimmy Turner MD

That’s a fundamentally different question. And so for folks that do have, let’s say, a mortgage north of seven, Justin, like general education, obviously not individualized advice here, but what are your thoughts in that situation?

 

Jimmy Turner MD

Then maybe we’ll answer the listener’s question here in a second.

 

Justin Harvey CFP

The first thing I would always try to do, this is true whether your interest rate is eight or whether it’s two and a half, is like let’s figure out what the math answer is. So we’ve got the math on one hand and then we wanna sort of put the psychological overlay on the other hand.

 

Justin Harvey CFP

So if you’ve got a seven and a quarter percentage rate, we wanna understand if you’re in itemization territory, especially if you do a lot of charity, if you’re maxing out the salt, that’s state and local tax, $10 ,000 cap, and your marriage status, it comes into play based on the standard deduction.

 

Justin Harvey CFP

Bottom line, if you’re well into itemization territory, your effective rate, meaning you gotta net that seven and a quarter percent for the tax benefit that you’re gonna get back in April whenever you file your taxes.

 

Justin Harvey CFP

And it might be 28, 25, 30, somewhere in that percent. So maybe my seven and a quarter should be like five and a half, five, something like that. And so 5% interest guaranteed that I’m paying versus what can I do elsewhere in terms of an after tax return?

 

Justin Harvey CFP

How do I feel about that? Also with the knowledge that I can probably refinance, I’m hoping, refinance this thing down the line eventually. Maybe, maybe not. I have no idea. I’m not an interest rate prognosticator and I tell this to anyone who will listen to me when this comes up.

 

Justin Harvey CFP

But I’m sort of in my mind, hoping that one day I will refinance and I don’t feel the need to aggressively pay down this mortgage even at seven and a quarter right now based on my family circumstances.

 

Justin Harvey CFP

And again, this is why looking at the big picture and if you’re working with an advisor, like hopefully they’ll help you walk through this, but insurance is important. And understanding how are we gonna cover debts if one of the earners, if the primary earner or one of the meaningful earners in a household is no longer around because of death.

 

Justin Harvey CFP

Insurance, life insurance on that person with a term matched to the expected term of your mortgage is one way to guarantee that. And if that is the case, then there’s really no, let me say it this way, there’s much less urgency to accelerate a pay down.

 

Justin Harvey CFP

Now, if I’m uninsurable and I’m the only one in my house that makes money and I’ve got a $1 .2 million mortgage, I probably am gonna be a little, and maybe this doesn’t matter if it’s a 4% or a 2%, like maybe even if it’s 2%, I understand that it’s like one and a quarter is the after tax interest rate, but in order to have the peace of mind to provide for my family because I don’t have the benefit of being able to get the life insurance that I need,

 

Justin Harvey CFP

that’s a totally different.

 

Jimmy Turner MD

conversation. Yeah. I mean, you’re preaching to the choir here, so listeners know, I can’t get disability insurance, which is the entire reason that Money Meets Medicine Disability was created is because I got hosed by an insurance agent, can’t get disability.

 

Jimmy Turner MD

And people ask me like, what do you do? And it’s like, well, I have to keep our guaranteed mandatory monthly expenses at some level that I know that we could pay for and afford should I become disabled.

 

Jimmy Turner MD

And so that means all of the discretionary non -mortgage expenses have to be minimized. I don’t want a bunch of monthly payments. And the reason why is because we have to be able to pay our payments if something goes crazy.

 

Jimmy Turner MD

And so I completely agree that if the breadwinner is not insurable or doesn’t have insurance, it becomes a very different conversation. I think that that’s one of the nuances of personal finance being personal.

 

Jimmy Turner MD

For me, financial decisions that I have to make for me personally, because I can’t get disability insurance, may not and likely are not the right thing for you. And so I agree that asset protection in this context really matters a ton.

 

Jimmy Turner MD

So you mentioned something important previously, but on a prior episode in terms of like, hey, these things should be done first automatically, like you’re 401k and you’re 4 through B and you should have a three to six month emergency fund and all that stuff.

 

Jimmy Turner MD

So this question precludes that you’ve done all of those things, right? You have a student loan plan in place, you’re already paying for disability insurance. You already have that term life insurance that you’re alluding to, Justin.

 

Jimmy Turner MD

So the question is when it gets down to that point where, okay, I’m making all my payments on all the things, my student loan payments there, my disability insurance payments there, my life insurance payments there, I’m maxing out my 401k, 457, back to a Roth, what have you, and then you still have extra money around.

 

Jimmy Turner MD

And we’re talking about having a 3% mortgage paying off that versus investing extra money. The math has to math, right? Like it just does. This is a mathematical conversation. And I do think that your point earlier that when the numbers get low enough, the mathematical argument becomes so strong and so profound, as you put it, that it starts to outweigh the psychological benefit.

 

Jimmy Turner MD

And I think that that’s really, really important because there is potential regret on the other side when you pay down a 2 .5% mortgage, and you look back and you see that 10 years later, your money would have been multiples of what you paid off because you didn’t invest it in the market, right?

 

Jimmy Turner MD

And so that’s a psychological phenomenon too, on the back end. So you can’t just consider like, what is the burden going to feel like when I’m done paying off my mortgage? It’s like, well, what’s the burden going to feel like when you found out that 10 years from now, you should have invested in the market, like every mathematical person told you, because you have a 2 .5% mortgage, and you didn’t,

 

Jimmy Turner MD

right? Regrets are very…

 

Justin Harvey CFP

And as a result, instead of working 0 .4 to spend the last few summers with your 12 -year -old, you’re picking up extra call in order to try to make your retirement plan work. It’s a real -life thing.

 

Justin Harvey CFP

Right.

 

Jimmy Turner MD

Well, and that’s a really important thing to discuss too, right? Is the illiquid nature of the mortgage because that money, right? I mean, we haven’t talked about this on the show before, but like recasting, refinancing, you know, HELOCs, there’s ways to play with the equity that exists in your house.

 

Jimmy Turner MD

I’m not saying it’s not, it’s not impossible to tap into, but it’s definitely a whole lot harder than putting your money elsewhere. And so that exact situation where you’re like, well, I’ve got this money, I paid off my mortgage.

 

Jimmy Turner MD

It’s like my father -in -law would always say, like when Kristen would come home and be like, hey, she bought some clothes and like they were on sale, he’d say, you know, show me the money, like show me the money you saved, right?

 

Jimmy Turner MD

And just the argument that like, you can’t, like you spent money. Like I know psychologically you think you saved money because it was 50% off or BOGO or what have you. Great, you paid off your mortgage, show me the money and you can’t, like you can’t tap into it easily, right?

 

Jimmy Turner MD

And so it’s like, great, like philosophically, you paid off this mortgage, but fundamentally from a liquid asset standpoint, you don’t have anything to show for it.

 

Justin Harvey CFP

Yeah, absolutely. And to put real numbers on it, you could pay off your 2 .5% mortgage and have the ability to get a HELOC, but to take that money back, you’re now paying 9% on that mortgage or on that second, the new debt product that is secured by your house, in this case, a home equity line of credit.

 

Justin Harvey CFP

So you’re totally right.

 

Jimmy Turner MD

And so I’m not downplaying the psychological benefit, but you have to think about the math, and the math’s gotta be math -ing, right? And additionally to that, I would add that you gotta think about other savings goals, right?

 

Jimmy Turner MD

So this high -yield account versus mortgage thing, you could even put more argument towards the high -yield account or a different investment vehicle of some kind. If you have other things that you’re working towards, like a vacation, or maybe you’re thinking about upgrading your home, maybe you are wanting to save for college and your kid’s gonna graduate in the next 10 years, there are other places you might consider putting your money.

 

Jimmy Turner MD

It might not just be a high -yield account versus paying off the mortgage. For just shorter -term goals like the vacation or the reno, that might be a high -yield account sort of thing, but like longer -term stuff, like 10 years from now paying for college, that’s probably needing to be invested and taking some risks so you can build some amount of money that would be reasonable to help pay for some college expenses for your kid.

 

Jimmy Turner MD

So it’s not like this is just a binary, high -yield account versus mortgage. It’s really paying off your mortgage versus all of the other options that you have available to you for all of the other goals that you have.

 

Justin Harvey CFP

Yeah. And what we talked about last episode, the direct indexing, SMA, separately managed account is another good straw man for another thing you could do. A long -term taxable investment account that has great after -tax return compared to even just indexing that is unrestrained in terms of liquidity.

 

Justin Harvey CFP

You can do whatever you want to with it. And the expected return over full economic cycles is going to be a lot better than accelerating pay down your two and a half percent mortgage.

 

Jimmy Turner MD

Absolutely. So hey, if you’re listening to the show right now and you’re like, hey, I want my question answered, you can always email me at jimmy at moneymeetsmedicine .com. I love getting questions, comments from the crowd and from the Money Meets Medicine community.

 

Jimmy Turner MD

So please send me an email. This entire show turned into an episode based on one email that we got from one of our listeners. And so if you have a question you want answered, we’re happy to do that from a general education standpoint and happy to dive into any of the things that interest you on the show.

 

Jimmy Turner MD

If you also want to send topics that you want us to talk about because we get to bend the ear of a certified financial planner in Justin, by all means send them our way and we will be happy to consider them for a future episode.

 

Jimmy Turner MD

Before you head out, don’t forget you can help with student loans, disability insurance, or you want to snag a free copy of the Physician Philosopher’s Guide to Personal Finance. You can do all those things at moneymeetsmedicine .com.

 

Jimmy Turner MD

Thanks for sharing the show with your friends and colleagues in medicine and for rating and reviewing the show on iTunes and Spotify, wherever you might listen. Justin, you and I will see everybody next week.

 

Justin Harvey CFP

Cheers.

 

Jimmy Turner MD

Justin Harvey is a certified financial planner at APM Wealth where he helps anesthesiologists and pain medicine physicians. Dr. Jimmy Turner is a practicing academic anesthesiologist at Wake Forest in North Carolina.

 

Jimmy Turner MD

He’s also a licensed insurance agent. However, either Justin or Jimmy are your financial planner, investment advisor, or insurance agent. This show is expressly for general education entertainment purposes only.

 

Jimmy Turner MD

Nothing should be considered financial advice. All views expressed are solely the views of the guests on the show and do not represent the views or opinions of their employer.

 

Physician Disability Insurance

Are you ready to live a life you love?