A smaller bike (like a 50cc) would be worse. The TW is the bike for you.
The TW, even for experienced riders, is known to have a sub-par front tire for gravel riding. And if your tire pressures are too high as well it'll only add to the issue.
The TW was my first bike. I took it to a neighboring, paved parking lot and road it a lot there taking turns and accelerating and getting a good feel for it. I still pick spots on the ground as I'm riding on the road and swerve around them to keep up on reaction and control. The pavement is a confidence builder!! The gravel will take it away.
Practice makes perfect. Grass is probably better than gravel, but pavement is where you need to be. That will allow you to focus on juggling the brake, clutch, gear shifter, throttle, turn signal, horn, high beam, etc. rather than worrying about the next bump in the grass or the next rut in the gravel.
You went and jumped on the bike hoping it'd all go smooth. After a couple falls, some frustrations dealing with the throttle and levers, it didn't go that smooth. You lost confidence in the process. Now it's the uphill battle. Physically you have it, it's a mental game now. Don't worry, you're not alone.
Relax your grip. A death grip is not needed on the handlebars. Your operations will be smoother when relaxed.
But most importantly, get to a paved area where your brain can focus on the hand/foot operations, not the bad manners of the bike on the gravel.
***EDIT: by paved, I don't mean the open road with other vehicles. You need to master these skills before sharing the road with others. What I meant was a large parking lot, dead-end road with no traffic, long paved driveway, etc.