Does your group have a policy about being late to a meeting? Most groups have policies about attendance, but how about a policy on how many times a board member can be late before you ask them to leave the board?
People who are late are annoying, they disrupt the meeting and ask for discussions to be repeated so they will be up to date. If you allow people to be late for meetings then you are punishing the people who arrive on time. If a meeting never starts on time, then more and more of the participants will start arriving late. This is a rational response. Why come on time if the meeting does not start on time?
If you don’t establish some rules, then you are not going to have any meetings that start on time and you may never have quorum. Enforcing attendance is important, but so is requiring board members to arrive on time.