A GIF stores its animation frame by frame using an old, lightweight compression scheme that wasn't designed for photographic motion. It can't use the clever inter-frame compression that modern video codecs (like H.264 in MP4) rely on, where only the changes between frames are stored.
That's why a clip that's a few hundred kilobytes as MP4 can be several megabytes as GIF. To keep GIFs small, reduce the frame rate, dimensions and colour palette — or convert to MP4 when a real video will do.
Try Compress GIF or Resize GIF to bring the size down.