Detecting and tracking interesting moving targets are crucial steps in intelligent video surveillance. We proposed a combination algorithm of Frame Difference and Particle Filter accelerated by FPGA parallelism. Parallel pre-processing and post-processing are added to Frame Difference to increase the anti-interference of detection. The improved Particle Filter hardware architecture can judge the tracking and lost status of moving targets by comparing the threshold of particle weight and the number of degenerated particles. Multi-target detection and tracking are implemented by reusing detection and tracking modules. Experimental results show the proposed hardware architecture shows great detection and tracking accuracy and real-time performance.