The Dreamcast itself is just a computer. It can only play the movie types that a program can play. As far as I know, programs exist to play the following media types:
MP2/MP3
Ogg Vorbis
MPEG-1 / Video CD
DivX (the MPEG-4 based video codec, not the crappy crippled DVD format from Circuit City)
A .avi file is not actually any particular type of stream, it's just a standardized format for describing the streams it contains. Thus a .avi could contain MPEG-1, DivX, Indeo, Cinepak, Truemotion, or a bunch of others.