The pattern name table (the correct name) can be only 32, 64, or 128 wide, and CANNOT be 32 unless in 256 wide mode. It can also only be 32, 64, or 128 tall, and CANNOT be 128 tall if it's also 128 wide.
256x224/256x240(PAL) : 32x32, 32x64, 32x128
256x224/256x240(PAL) or
320x224/320x240(PAL) : 64x32, 64x64, 64x128, 128x32, 128x64
Those are the only sizes the scroll layers can be. They are also the same size - layers A and B cannot be set separately.
I have no idea how BEX sets this... you'd best look at the BEX examples or ask over at Devster's. To set it via assembly, set VDP register 16.
b5-4 = vertical size, 00 = 32 tiles, 01 = 64, 10 = reserved, 11 = 128
b1-0 = horizontal size, 00 = 32 tiles, 01 = 64, 10 = reserved, 11 = 128
The other bits are reserved and should be set to 0.
Remember that the tiles DISPLAYED by the layers depends on the resolution.
256x224 = 32x28 tiles
256x240 = 32x30 tiles
320x224 = 40x28 tiles
320x240 = 40x30 tiles
Where that block of tiles comes from in the pattern name table is set by the vertical and horizontal scroll values. When the scroll values are 0, the displayed tiles start at 0,0. If you have a 40x28 display in a 64x32 pattern name table, you can scroll up to 24 tiles horizontally, and up to 4 tiles vertically. When you reach a border, you might move the pattern names back over top of the ones scrolled off the screen, then write new names into the empty positions while setting the scroll values back to 0,0. There are plenty of examples of scrolling in the BEX forum over at Devster's.
Note that even though the pattern name table is 64x64 (for example), you don't HAVE to use all 64 tiles in each direction. You could use 60 is each direction, but remember that the stride for the table is 64 words. Once you scrolled up to 60, then reset back to 0 and draw in the new data. So you can use any value larger than the display size (say 40x28), and no larger than the pattern name table size (say 64x64).