We present a target tracking system for a robotic air-hockey player, where the state of the puck is optimally estimated by fusing measurements from frame- and event-based vision sensors. In addition to the jumping velocity of the tracked object during the game, the technical challenge of the problem is amplified by the variable delay and irregular sampling intervals of the measurements — a thematic challenge for controls under such visual feedback. An auto-restart Kalman filter is first proposed for compensating sudden jumps in the puck state. Then a memory-enabled auto-restart Kalman filter is derived to additionally accommodate delays and sensing irregularities. Building on physics-based modeling, model-based filtering, and mixed sensor management, the proposed method applies to other vision-based control systems from motion tracking to manufacturing automation. The tracking performance is analyzed in simulation that shows the effectiveness of the proposed tracking algorithm.