Rail track breakages represent broken structures consisting of rail track on the railroad. The traditional methods for detecting this problem have proven unproductive. The safe operation of rail transportation needs to be frequently monitored because of the level of trust people have in it and to ensure adequate maintenance strategy and protection of human lives and properties. This paper presents an automatic deep learning method using an improved fully Convolutional Neural Network (FCN) model based on U-Net architecture to detect and segment cracks on rail track images. An approach to evaluating the extent of damage on rail tracks is also proposed to aid efficient rail track maintenance. The model performance is evaluated using precision, recall, F1-Score, and Mean Intersection over Union (MIoU). The results obtained from the extensive analysis show U-Net capability to extract meaningful features for accurate crack detection and segmentation.