Concerning Japanese numerals, they are really quite tricky. Different uses of numbers require different pronunciations of the kanji. Furthermore, to count things, which is how numbers are usually used, there are a myriad of suffixes you have to know.
As an example, here are some sentences featuring the number 4:
| Japanese | English | 4 is pronounced as: |
|---|---|---|
| ４人でいいですか。 | Is four people okay? | よ yo, ４人 = よにん yonin, four people |
| 人形は四つを買った。 | I bought four dolls. | よっ yo + pause, 四つ = よっつ yottsu, four |
| 本は四冊を読んでる。 | I’m reading four books | よん yon, 四冊 = よんさつ yonsatsu, four + counter word for books |
| 来月は４月です。 | Next month it is April | し shi, ４月 = しがつ shigatsu, four + counter word for month, i.e. the 4th month, or April |
So, in short, I would consider the japanese numeral system to be more a part of grammar than of vocabulary.
Also, there are many irregular uses of numerals (一人 = ひとり hitori instead of いちにん ichinin, which would be what you expect, 一日 = ついたち tsuitachi instead of ひとっか hitokka, 十四日 = じゅうよっか juuyokka, while you would expect じゅうよんにち juuyon’nichi based on other dates in the 10’s or 20’s, and the list goes on)