The Internet of Things (IoT) interconnects Internet and end device networks connected by identification information, which not only extends and explores the range of the Internet, but also constructs bridges of communication between realistic objects and the Internet. With increasing awareness about the energy crisis, the development of intelligent energy-saving systems has become a new trend in all circles. This work proposes an appliance-aware activity recognition mechanism for an IoT energy management system, and especially an intermediate management service layer for the conformability of current household appliances, which not only establishes communication services among various appliances, but also provides services for upper-layer application based on a service-oriented architecture. Further, the presented system has the capability to deduce human activities from appliances being used and the variation of their states, and provide energy management services. In other words, the system can identify home activities at present, and notify users what unused appliance is or turn them off automatically. According to experiment results, improving the usage behaviour of appliances by means of a remainder system improves the efficiency in saving household electricity.