I know that in many Pacifica nations it's a cultural thing. The sea is for fishing not playing. When they do go into water, it's shallow and safe and for cooling off or getting clean. They come to NZ and get caught out on our beaches and, sadly, dominate our drowning statistics.
A bit of a digression, but the college I went to (way back when) had received a large funding grant from a wealthy woman. Apparently, she had lost a loved one to drowning, so her money came with a requirement that all students at the college had to be able to stay afloat for 5 or 10 minutes (I forget the actual requirement) to get their bachelor's degree. I wanted to get that over with, so I took and passed the test in the gymnasium pool as a freshman (I was a strong swimmer, anyway). I was later dismayed to learn that the grant money had run out during my junior year and so the requirement immediately was dropped by the college.
When I was in primary school some of my "class mates" tried to drown me during a swimming lesson. See my profile picture? I was trying to hide how petrified I was, and that was with railings and a wall in the way!
Oh that's awful. It's so unfair that negative childhood experiences stay with us forever. My BIL had a different drowning experience and loathes holidaying anywhere near water.
You've hit on two very significant factors when it comes to why people can't swim! It's rarely one single reason, but often a combination of several. Aquaphobia/Thalassophobia,Geography and Socioeconomic Factors,Lack of Proper Instruction/Technique etc