I started a job 12 days ago and can already tell it’s not a good fit for me, is it too early to quit?

Long story short the work environment is just not a good fit for me. I’m a person who likes to stay busy and the job has long periods of down time which I was not told about. I have asked multiple times if it’s just because I’m new (it’s not, everyone in this division of the company is like this). Sitting at a desk and having 6 hours of downtime just twiddling my thumbs is just not for me. Am I crazy for walking away at this point or is it okay for me to already know that this place isn’t for me?

usa jobs
usa jobs resume
usa hotel jobs
usajobs
usa jobs federal government
usa job in ksa
usa jobs
usa jobs login
usa jobs gov
usajobs.gov
www.usajobs.gov
usajobs.com
usajobs
usajobs.gov official