This is something that is so often overlooked! I'm based in Brighton where everything is built on hills, so there are so few buildings that don't have steps to get in or within the main body of the building, so finding wheelchair accessible venues is a real challenge, but an important one. I try to keep accessibility as a priority throughout the organising process, and to factor in all forms of ability rather than just focusing on wheelchair users.
As such, checking if spaces have a hearing loop/infrared system is important for attendees who have hearing impairments. If the event is speaker heavy, having a sign interpreter is best though they can be prohibitively expensive sometimes.
We always try to have a break off quiet room/safe space which is useful if we are tackling difficult subjects (I work at a rape crisis centre, so that is common). This is helpful for people with PTSD who can be easily triggered, but is also helpful for people with autism or learning difficulties to have a quieter space to turn to - noise levels can be difficult for some people with autism.
There is training around accessible materials/marketing and I've only recently been made aware of the fact that there is training for accessible materials for people with dementia, which is worth looking in to.
Basically, depending on the nature of the event, I would argue that you should always do everything within your power and your budget to be open to as many people as possible. You benefit from a broader audience and everyone else benefits from being able to attend your awesome events. Win win.