Toward Conversational Agents with Context and Time Sensitive Long-term Memory