Automatic segmentation of brain tumors is a challenging problem with many inherent difficulties, such as restricted training data, great intra-class variance, and volumetric images with large computational requirements in terms of processing. To overcome these difficulties, we propose Brain Tumor Parser (BTP), a novel convolutional neural network that takes advantage of a refinement module and global 3D information to perform semantic segmentation of brain structures in volumetric images with multiple modalities. We draw inspiration from recent breakthroughs in edge detection and semantic segmentation in natural images, and we build an accurate and effective three-dimensional network that segments small structures while refining large instances in multi-modal Magnetic Resonance Imaging (MRI). We evaluate our approach on the data from the Brain Tumor segmentation (BraTS) 2017 challenge, obtaining comparable results with the best performing algorithms, while using a single yet efficient architecture.