Unable to use YEAR() and MONTH() inside COUNTIF()

Tim, bit more

What the COUNTIF do is compare one by one A1, A2, etc with criteria you set by second parameter. In your case that's some string, no one day (which is actually the integer number) is equal to any string. Thus result is zero.

And you can't apply any formula to first parameter, something like =COUNTIF(MONTH(A:A),10) returns an error. First parameter shall be a range.

If calculate months for any year usual alternative, as Bryant suggested, is using of SUMPRODUCT. Here you may use calculation on range

0	=COUNTIF(A:A,"MONTH(A:A)=10")
5	=SUMPRODUCT((MONTH(A:A)=10)*1)
5	{=SUM(IF(MONTH(A:A)=10,1))}
5	{=SUM(--(MONTH(A:A)=10))}
3	=COUNTIFS(A:A,"<="&EOMONTH(DATE(2017,10,1),0),A:A,">"&EOMONTH(DATE(2017,10,1),-1))
3	=SUMPRODUCT((MONTH(A:A)=10)*(YEAR(A:A)=2017))

Practically that's the equivalent of third formula, or in more compact form forth one, but they are to be used as array formulas.

However, if your criteria is the month for the concrete year (other words combination of the year and month) you may use COUNTIFS as above (or latest SUMPRODUCT which returns the same)