Highly Automated Driving technology will be facing major challenges before being pervasively integrated across production vehicles. One of them will be monitoring drivers' state and determining whether they are ready to take over control under certain circumstances. Thus, we have explored their physiological responses and the effects on trust of different scenarios with varying traffic complexity in a driving simulator. Using a mixed repeated measures design, twenty-seven participants were divided in two reliability groups with opposite induced automation reliability expectations -low and high-. We hypothesized that expectations would modulate participants' trust in automation, and consequently, their physiological responses across different scenarios. That is, increasing traffic complexity would also increase participants' arousal, and this would be accentuated or mitigated by automation reliability expectations. Although reliability group differences could not be observed, our results show an increase of physiological activation within high complexity driving conditions (i.e., a mentally demanding non-driving related task and urban scenarios). In addition, we observed a modulation of trust in automation according to the group expectations delivered. These findings provide a background methodology from which further research in driver monitoring systems can benefit and be used to train machine learning methods to classify drivers' state in changing scenarios. This would potentially help mitigate inappropriate take-overs, calibrate trust and increase users' comfort and safety in future Highly Automated Vehicles.